The llms.txt Autopsy: What Happens When 137,000 Sites Actually Get Checked

The promise, and the test that finally checked it

The pitch for llms.txt is intuitive enough to sell itself. A website is a tangle of navigation, scripts, and boilerplate; large language models have finite context windows; therefore, the argument goes, a single clean file at the root of a domain, listing a curated summary and the pages that matter, should help AI systems read a site correctly and represent it accurately in generated answers. Framed that way, publishing one sounds like basic hygiene for the AI era, and a large number of vendors began recommending it as a lever for AI-answer visibility.

Intuition, however, is not evidence, and a proposed convention is not an adopted standard. The relevant empirical question is narrow and answerable: when a site publishes llms.txt, does anything actually read it, and does any major AI platform treat it as an input to what it cites? For most of the file's short life that question went unmeasured, and the marketing filled the vacuum. In mid-2026 it stopped being unmeasured.

Ahrefs ran a server-log analysis across 137,210 domains and reported the result in a study titled, without much ceremony, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. The value of the work is not rhetorical. It replaces an assumption about how AI systems behave with a direct observation of what they did, at a scale large enough that the finding is not an artifact of one unusual site or one unusual week.

Does llms.txt work? What 137,000 sites reveal

The headline number is the one that ended the debate on its own terms. Across the domains that carried a valid llms.txt file, 97 percent of those files received zero requests during the measurement month of May 2026. Not a low read rate. Zero. For the overwhelming majority of sites that had done exactly what the vendors recommended, nothing came to read the file at all.

Is llms.txt a ranking signal? Google's own answer

Behavioral data tells you what happened; it cannot by itself tell you what a platform intends. For that, the most authoritative source is the platform speaking about its own systems, and on llms.txt the platform has spoken. Google's John Mueller has stated publicly that llms.txt is not done for search, characterizing it, at most, as a token-saving convenience for AI coding tools that read developer documentation, not as a signal that shapes how a business surfaces in AI features.

This is consistent with Google's own generative-AI optimization guidance, which states explicitly that no AI-specific text file or special markup is required to appear in AI features. The instruction from the platform is not "publish this file to be included." It is closer to the opposite: there is no dedicated file to publish for this purpose.

The same pattern recurs across this field. Google has been equally explicit that E-E-A-T, the Experience, Expertise, Authoritativeness, and Trust framework, is a set of criteria for the human raters who evaluate search quality, not a machine-computed score a page can optimize into a ranking factor. In both cases the industry took a genuine concept and reframed it as a controllable lever, and in both cases the primary source says the lever is not what it is being sold as. Reading the platform's own documentation, rather than the secondary content built on top of it, is the single most reliable defense against this class of claim.

Why a text file was never going to be the lever

The empirical result is clear, but the mechanism behind it explains why the result was predictable. Generative answer engines are retrieval systems. When a query arrives, the engine assembles candidate passages from an index, ranks them, and generates an answer grounded in what it retrieved. The unit of work is the passage and the entity, drawn from the live index of the open web, not a hand-written manifest a site owner left at its root and hoped would be consulted.

A useful contrast is schema.org. Where llms.txt is a proposal, structured data via schema.org is a vocabulary founded jointly by Google, Bing, Yahoo, and Yandex in 2011, governed by a standards body, and actually consumed by the systems that were built to consume it. The difference is not popularity; it is provenance and adoption. A convention becomes an input to a machine when the machine's owners commit to reading it, and there is no comparable commitment behind llms.txt.

There is also a structural reason a static summary file is a weak instrument for AI visibility even in principle. Research on how AI Overviews assemble answers indicates that the passages a system reads and the pages it credits are not guaranteed to be the same set, and a page well outside the top organic results can still be cited. That reading of the underlying mechanics is emerging rather than settled, and rests on technical analysis of platform disclosures rather than a single peer-reviewed source, so it should be held loosely. But the direction is instructive: in a system where citation is decoupled from a site's own declarations, a self-authored file describing what a site would like to be cited for is exactly the kind of input such a system is least likely to weight.

What actually works instead: the generative engine optimization evidence

The most useful thing the llms.txt data does is clear space to ask what the evidence supports in its place. Here the field is not empty. The founding, peer-reviewed study of the category, published at ACM SIGKDD in 2024, ran a controlled benchmark of roughly 10,000 queries and measured which content interventions changed whether a source was surfaced inside a generated answer. The strongest levers were not files or markup. They were adding citations to credible sources, including direct quotations, and replacing vague claims with specific statistics, with citing authoritative sources the single most consistent driver, producing a meaningful relative lift on the study's visibility metric.

A large-scale 2025 follow-up points in the same direction from a different angle. It found that AI search engines are systematically biased toward earned and third-party media over brand-owned and social content, a sharp contrast with classic Google, which sources more evenly. That result is emerging rather than established, one methodologically large study not yet widely replicated, and it should be cited with that caveat. But taken with the founding benchmark, the shape of the advice is stable: what moves AI visibility is being a credible, well-cited, corroborated entity that other trustworthy sources reference, not a manifest a business writes about itself.

The gap between those two pictures is the whole point. The tactic with the strongest marketing behind it, llms.txt, is the one the read-rate data undercuts. The tactics with the strongest evidence behind them, credible citations and earned third-party corroboration, are slower, harder, and less packageable as a checkbox, which is precisely why they are undersold relative to their support.

The infrastructure honesty gap

llms.txt is not an isolated case. It is one instance of a broader pattern in which the tooling layer promoted as "how you control AI visibility" turns out to be younger, leakier, and less standardized than the marketing implies. The same year the llms.txt data landed, Cloudflare documented a case in which a retrieval crawler continued fetching pages after being blocked, using an undeclared browser-spoofing agent and rotating network identity to evade the block, and verified that answers still reflected content from domains that had explicitly disallowed it. That specific incident is documented and independently verifiable; its characterization as ongoing practice is contested by the vendor named in it, and it is a single company's report rather than a third-party audit, so it belongs in the "established as an incident, contested as a general claim" category.

Stack these findings together and a defensible, evidence-first position emerges. The public no-crawl directive is not always honored. The self-authored summary file is not read. The entity-identity mechanisms underneath "be the same business everywhere" are looser in practice than their specifications suggest. None of this means AI visibility is unmanageable. It means the levers being sold as clean, controllable switches are, on the evidence, neither as clean nor as controllable as advertised, and that any responsible read of the space has to tier its tactics by how much primary evidence actually stands behind each one.

How to read any AI-visibility claim honestly

The practical residue of this autopsy is a small set of questions that separate a substantiated tactic from a sales story, and they generalize well beyond llms.txt. First, is the mechanism a governed standard the platforms have committed to reading, like schema.org, or a proposal one party authored and others are hoping gets adopted? Second, is there primary data on whether the thing is actually consumed at scale, or only an intuitive argument that it should be? Third, does the platform itself describe the input as a signal it uses, or has it said the opposite? And fourth, is the supporting evidence established, emerging, or contested, and is whoever is selling the tactic honest about which?

Applied to llms.txt, all four questions resolve against it: a proposal, not a standard; measured and found unread; disavowed by the platform as not done for search; and defended, when it is defended, by intuition rather than data. Applied to earned citations and credible corroboration, they resolve the other way. That is not a reason for cynicism about the field. It is a method for spending effort where the evidence is, and it is the same method that should govern how a business decides which of its own visibility investments are worth making.