llms.txt Does (Almost) Nothing: The Data, and Why I Recommended It

Straight answer: no, llms.txt does not get you cited by AI today. The 2026 data is brutal: out of roughly 38,000 domains with a valid file, 97% received zero requests for it (Ahrefs analysis across 137,000 domains, May 2026); an SE Ranking study of 300,000 domains finds no statistically significant correlation with AI citation frequency; and Google has stated plainly that no Search system reads it. There is one case where it genuinely helps — and it isn't the one it's sold for.

And yes: in June's GEO article I recommended it too. Let's look at the numbers that came out after.

TL;DR

• Adoption: 10.13% across 300,000 domains (SE Ranking) — heavily skewed toward technical sites.
• Actual usage: 97% of valid files got zero requests in a month (Ahrefs, May 2026). GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended crawl the HTML directly.
• Effectiveness: no correlation with AI citations; removing the variable made the predictive model more accurate — i.e. it was noise.
• Google: John Mueller compared it to the keywords meta tag; Google's May 2026 AI optimization guide debunks it in a mythbusting section.
• Where it truly works: technical documentation read by agentic IDEs (Cursor, Claude Code, Copilot, Cline, Aider), which do fetch /llms.txt and /llms-full.txt.

What it is, in two lines

llms.txt is a 2024 proposal: a Markdown file at the site root summarizing who you are, what you offer and which pages matter, so a language model can understand you without wading through HTML stuffed with menus, cookie banners and JavaScript. The idea is good. It's exactly the sort of thing an engineer wants to exist.

The problem isn't the idea. It's that none of the intended consumers are using it.

The numbers, no spin

Source	What it measured	Result
SE Ranking	300,000 domains	10.13% have an llms.txt
SE Ranking	Correlation with AI citations	None significant; removing the variable improved the model
Ahrefs	137,000 domains, ~38,000 with a valid file	97% zero requests to the file in May 2026
Google	Search systems	None reads or acts on llms.txt

That 97% should end the discussion. We're not talking about "limited benefit": we're talking about a file that, in the overwhelming majority of cases, nobody ever downloads. AI crawlers hit the HTML, as they always have.

Mueller's comparison with the keywords meta tag is unkind but apt: a signal publishers dutifully fill in, no engine uses, and that survives because it's easy to sell as a service.

Why I recommended it (and what I got wrong)

In June's GEO piece I listed it among the things to do. The logic was: twenty minutes of work, no harm, and if models start reading it tomorrow you're ready. As option value, that still holds.

The mistake was positioning: putting it in a list of GEO actions makes it look effective, when it's merely harmless. That distinction matters, because attention is finite — every item on a checklist crowds out another. And the other items on that list — server-side structured data, entity definition, self-contained content — have evidence behind them that llms.txt doesn't.

So, correction: do it last, if you have time to spare, and expect nothing.

Where it does work: documentation

Here's the part almost nobody covers. There is one class of consumers that genuinely reads llms.txt: agentic coding tools. Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, Aider — point them at a documentation site and they look for /llms.txt and /llms-full.txt.

Which is perfectly logical: those tools need to load a library's docs into a context window, and clean Markdown is worth ten times a docs page full of sidebars and interactive widgets.

Hence a sharp practical rule:

• Brochure site, e-commerce, company blog? llms.txt is optional and essentially useless. Do it for fun.
• API, SDK, developer product with documentation? llms.txt (and especially llms-full.txt) is useful today, measurable today, and improves life for anyone integrating your product with an agentic IDE. Do it well.

We keep ours for the second reason, not the first.

What to do instead (to actually get cited)

1. Make content readable server-side. Many AI crawlers don't execute JavaScript. If your best answer only appears after React hydration, it doesn't exist to them.
2. Real structured data in the HTML. Organization, Product, FAQPage, Article. That's the machine format systems genuinely use, and have for years.
3. Self-contained answer blocks. A paragraph that fully answers one question without depending on surrounding context. That's the unit a model can lift and cite.
4. Define your entity and stay consistent. Same name, same description, same sameAs everywhere. Models build a graph — give it clean nodes.
5. Check robots.txt for AI crawlers. This one has immediate effect: if you block GPTBot or ClaudeBot, no llms.txt on earth saves you. First thing to verify, and it happens more often than you'd think.
6. Measure citations, not files. Track which assistants send visits and mentions. It's the only metric that matters in this channel.

FAQ

Should I delete my llms.txt?

No, keeping it does no harm: it's a static file, it doesn't meaningfully consume crawl budget, and if someone starts reading it tomorrow you're ready. The point is not to count on it, and not to pay for it as if it were a ranking lever.

If every competitor has one, am I at a disadvantage?

No, for the simple reason that almost nobody downloads it. A real competitive disadvantage looks different: not rendering content server-side, or blocking AI crawlers in robots.txt. That's where the gap shows.

Does this apply to llms-full.txt?

Same story on the AI search side, different story for developer tooling: llms-full.txt is designed to hold entire documentation in Markdown, and agentic IDEs like it a lot. If you sell a developer product, that's the file worth curating.

What about agents browsing the web on a user's behalf?

They read HTML too, today. The channel that's actually growing isn't a special file at the root: it's the protocols where agents query your data in a structured way — feeds and APIs in commerce, MCP in enterprise systems.

---

If you're investing in GEO and want to know which moves have evidence behind them and which are folklore, happy to talk. The bigger picture, updated with this data, is still here.