The SEO price of public data (and who really pays it)

Open the source of any page of your product and count the data in it. Not what is displayed — what is there. It is almost always far more, and almost nobody decided it should be. To be found you have to expose to machines exactly the information assets your company is built on — catalogue, prices, availability, reviews. It is a legitimate and largely unavoidable trade-off. The problem is not that it exists. It is that almost nobody treats it as a choice, with a perimeter and limits.

TL;DR

Every search-optimised page exposes structured data readable by anyone, not only by search engines.
A modern page carries two layers of machine-readable data: the one for search engines, and the state the interface builds itself from. The second is almost always richer than what the page shows.
The problem is not any single field: it is the scale. Thousands of records reachable with the same request a user makes when clicking "next page".
In the EU this data stays protected even when public, under the sui generis database right. "Public" does not mean "free for all".
The answer is not to hide everything. It is to decide what to expose, to whom, with what limits — and that is a few lines of code, not only of policy.

The trade-off nobody writes down

To be found — by Google or by a conversational assistant — you have to make machine-readable precisely what sets you apart: how wide the catalogue is, how much ground you cover, how well you are rated. There is no version of "good SEO" that avoids it. If you sell visibility, visibility is paid for in data.

So far, healthy. It breaks somewhere else: between "this data is needed to be indexed" and "this data ended up on the page because it was easier that way" there is almost never anyone drawing the line. It is not a wrong decision — it is the absence of a decision.

What is actually in a page

What the browser shows and what the page contains are two different things. A modern page carries at least two layers meant for machines, both in plain sight in the source:

Structured data for search engines (schema.org). It powers rich results: name, address, rating, images, URL. Anyone who can write a ten-line parser reads it, not just Googlebot.
The application state the interface builds itself from. This is how modern apps "hydrate" a page on first load. And this is where the fat accumulates: precise category, multi-dimensional ratings, opening hours, staff, geographic coordinates — and often fields the interface never displays.

To see the difference (invented data, shape only):

<!-- 1. Meant for search engines — schema.org, standard format -->
<script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "HairSalon",
    "name": "Example Salon",
    "address": { "streetAddress": "Via Roma 1", "addressLocality": "Milan" },
    "aggregateRating": { "ratingValue": "4.7", "reviewCount": 128 }
  }
</script>

// 2. Hydration state — much richer, and it includes things the page never shows
{
  "venue": {
    "id": "8231",
    "name": "Example Salon",
    "category": "Hairdresser",
    "rating": { "overall": 4.7, "cleanliness": 4.8, "value": 4.5 },
    "openingHours": { "mon": "09:00-19:00" },
    "staff": [{ "name": "..." }],
    "address": { "lat": 45.4642, "lon": 9.19, "street": "Via Roma 1" },
    "contact": { "phone": "+39 02 XXXXXXX" }
  }
}

Neither asks for a login, an API key, or anything else. They are the page any visitor receives.

A case I checked

So this does not stay theoretical: I ran this examination on a real European beauty and wellness marketplace, which I do not name. Both layers present, neither protected. On the detail page the application state also exposed a field absent from the search listing: the direct phone number.

But the single field is not the problem. The scale is. A large Italian city returned between one and one and a half thousand businesses across some seventy result pages, and every next page was reachable through an ordinary link in the HTML — the same one a user follows when clicking forward. Rebuilding the full catalogue of a category in a city is a matter of minutes, not hours, with anonymous requests and respecting the delays the site itself declares.

Two details worth more than the rest:

The robots.txt was well written. It precisely blocked dozens of filter combinations, and explicitly opened to AI crawlers with a declared list. This is not a forgotten file: it is a curated one, written by someone who knew what to keep out of the index. The listing and detail pages, though, stayed open — consistent with the visibility choice, and for that reason never compensated elsewhere.
Zero external links in the payload. No own domain, no social profile, no URL outside the marketplace. Consistent with a model that does not want to be disintermediated. The corollary is interesting: a business visible only there is a business with no site of its own — which says more about the market than about the platform.

Honestly: there is nothing exceptional about this case. I picked it because it is ordinary. You can run the same analysis, right now, on your own product.

What the law says

A point that's often underestimated: in the EU, data collections carry specific protection — the sui generis database right (Directive 96/9/EC) — which applies even when the individual data points are public. The protection doesn't cover originality of content, but the investment made in collecting, verifying, and organizing it. Systematically extracting and reusing a substantial part of a protected database without authorization can infringe this right regardless of whether the individual data points were freely visible.

If the data includes elements traceable to natural persons — professionals' names, reviews with identifiable content — GDPR also applies, with specific obligations around legal basis and purpose of processing that go well beyond the simple public availability of the original data.

What a company can actually do

There's no need to choose between "strong SEO" and "protected data": there's a need to stop treating them as unrelated. Some of these countermeasures cost almost nothing, others need real architectural work — worth separating the two, and a bit of code helps show what actually changes.

Low-effort countermeasures (days, not sprints)

Trim state to what's needed, not to what's convenient. Client-side hydration state should carry only what's needed to render that screen at that moment — not the entire object returned by the backend. In practice, an explicit field-selection step before serializing is usually enough:

// Bad: passes the whole domain object straight to the client
return res.json({ props: { venue } })

// Better: explicit whitelist of the fields the interface actually uses
const { id, name, category, rating, openingHours, address } = venue
return res.json({ props: { venue: { id, name, category, rating, openingHours, address } } })
// no "contact.phone", no internal ranking fields, etc. — if that page doesn't need them

Separate the structured data meant for search engines from the application state. The schema.org block should come from a dedicated function, with an explicit list of fields meant for indexing — not a dump of the same object the frontend uses:

function toSearchEngineJsonLd(venue: Venue) {
  // Only the fields we actually want indexed, listed by hand
  return {
    '@type': 'HairSalon',
    name: venue.name,
    address: venue.address,
    aggregateRating: venue.rating,
    url: venue.publicUrl,
  }
}

Put a real rate limit in place, not just a declared one. A Crawl-delay in robots.txt is a polite request, not enforcement: well-behaved bots respect it, anyone wanting to bypass it simply ignores it. Even a simple server-side limit cuts out most unsophisticated attempts:

// Minimal example, in-memory — in production I'd use Redis or an edge rate limiter
const recentRequests = new Map<string, number[]>()

function isOverLimit(ip: string, maxRequests = 60, windowMs = 60_000) {
  const now = Date.now()
  const history = (recentRequests.get(ip) ?? []).filter((t) => now - t < windowMs)
  history.push(now)
  recentRequests.set(ip, history)
  return history.length > maxRequests
}

Alert on anomalous patterns, not just on errors. Thousands of sequential requests to consecutive listing pages, from the same IP or IP block, in a time window no human would use to browse a catalog, is a pattern detectable with a single log query — if someone writes it and someone watches it.

Architectural countermeasures (need a real project)

Move from "state serialized to the client" to a BFF (Backend-For-Frontend) or Server Components pattern. If rendering happens server-side and the client only receives ready HTML plus minimal targeted requests for interactions, there's no longer a single monolithic JSON blob to "vacuum up" in one shot — every piece of data requires its own request, individually traceable and limitable.
Sign or time-bind responses (short-lived tokens, subtle data watermarking to trace the source of an eventual aggregated leak) to make systematic large-scale reuse costly — not impossible, costly.
A dedicated anti-bot layer (behavioral fingerprinting, invisible challenges, not necessarily visible CAPTCHAs that degrade the human experience) in front of the most valuable listing and detail routes, with an explicit, verified exception for the crawlers you actually want to authorize (search engines, deliberately chosen AI crawlers) — the difference from today is that the exception must be declared and verified (checking the User-Agent isn't enough, it's trivially spoofed), not left as the default behavior for anyone.

The part almost nobody does

An official data channel for anyone with a legitimate need for bulk access — partners, aggregators, researchers — with a dedicated API, clear terms of use, and a point of contact. Anyone who genuinely needs a lot of data usually prefers a stable, documented channel over a raw, unguaranteed one; offering it removes the "there was no other way" excuse from anyone who still wants to go through the back door.
Explicitly cite sui generis protection and usage terms right on the page itself, not only in the general Terms of Service nobody reads. It makes the legal position explicitly defensible instead of implicit, and raises — psychologically, before even legally — the perceived cost of ignoring it.

None of these measures, on its own, is a definitive fix — and that's exactly the point: securing a data exposure isn't a switch, it's a sum of friction points. The goal isn't making systematic extraction impossible: it's making it costly, traceable, and deliberate enough to discourage anyone without a legitimate, declared reason to do it.

What I would do tomorrow morning

If I had to look at a product I do not know, I would do three things in this order. The first costs twenty minutes.

Open the source of a listing page and a detail page, and read the application state line by line. Do not hunt for vulnerabilities: take inventory. One question per field — does this page use it? Whatever is left over is already your answer.

Then see how easy it is to reach page one thousand. If the "next" link is an ordinary URL and there is no server-side limit, your catalogue is protected by nothing at all. robots.txt is a polite request, not a gate.

Then decide, and write it down somewhere someone will re-read when the data model changes. Because it will, and the new field will enter the payload for the same reason the phone number did: serialising everything is faster than choosing.

This is not sprint work, it is half a day. And almost nobody spends it — which is exactly why, when you do, you always find something.

FAQ

Is large-scale scraping of public data legal?

It depends on the jurisdiction and the specifics. In the EU, the sui generis database right protects collections even when individual data points are public, and GDPR applies when personal data is involved. "Public" doesn't equal "free of any legal protection."

Why would a company authorize AI crawlers if that also exposes its data?

Because visibility in generative answers (AEO/GEO) is becoming a discovery channel comparable to classic search. The right move isn't blocking everything, but deciding deliberately what to authorize and in what form — separating the minimum data needed from the rest.

How do you balance SEO and protecting your data asset?

By explicitly separating what's needed for indexing from what's only needed for the interface to work, monitoring anomalous access patterns, and offering a dedicated channel — with clear terms — to anyone with a legitimate need for access beyond a single human user.

If you're working out how to balance SEO/GEO visibility against protecting your data asset — or want to know what your product actually exposes today — that's an audit I run regularly: let's talk. Related reading: GEO, what it actually is and SEO for AI and Google AI Overview.

Sources: Directive 96/9/EC on the legal protection of databases, Regulation (EU) 2016/679 — GDPR, schema.org — ItemList.