When a Crawler Doesn't Play by the Rules: The Cloudflare-Perplexity Incident, Explained

A text file is a request, not a wall

robots.txt is one of the oldest conventions on the web. A site publishes a small text file at its root that lists which paths crawlers are asked not to fetch. It is an advisory standard: the server does not enforce it, and it carries no technical barrier. A compliant crawler reads the file and stays out of the disallowed paths because its operator has decided to be a good citizen. A crawler that ignores the file faces no wall, only reputational and, potentially, legal consequences.

This design has held up for thirty years because the crawlers that mattered were run by a small number of search companies with a strong incentive to be trusted. The generative-answer era stresses that arrangement. There are now many more crawlers, run by many more operators, fetching for many more purposes, and not all of them behave the way their published names imply. The question owners now ask, does robots.txt stop AI crawlers, has a more complicated answer than the file's history would suggest.

What Cloudflare documented in August 2025

On 4 August 2025, Cloudflare published a technical report describing behavior it attributed to Perplexity. According to the report, when sites blocked Perplexity's declared crawler through robots.txt and by its stated user-agent, Perplexity continued to fetch those pages using an undeclared, generic crawler that impersonated a normal web browser, presenting itself as Chrome running on macOS rather than as any AI bot.

Cloudflare reported that this stealth crawler rotated both its IP address and its ASN (the network identifier that ties an address to an operator) to move around the blocks, and that the activity spanned tens of thousands of domains and millions of requests a day. Crucially, Cloudflare said it verified the effect rather than merely inferring it: it set up test domains that explicitly blocked Perplexity, then queried Perplexity about content that lived only on those blocked domains, and reported that the answers still reflected that content. Cloudflare subsequently removed Perplexity from its list of verified bots.

Perplexity disputes the framing

This is a dispute, not a settled finding, and the analysis here says so plainly. Perplexity publicly rejected Cloudflare's characterization, contesting the methodology and the framing of the behavior. The report is a single vendor's account; it was not independently audited by a neutral third party, and it describes a snapshot of behavior at one point in time.

So two things are true at once. The incident as documented, a controlled test in which blocked content surfaced anyway, is a real, sourced event that a site owner can reason about. The wider claim, that this represents an ongoing, deliberate practice by a particular company, is contested and should be treated as such. We cite the first tier confidently and flag the second as unresolved. The value of the episode does not actually depend on how the corporate dispute resolves, because the structural lesson holds either way.

The structural lesson: blocking a bot is not the same as stopping it

Strip away the specific companies and the episode reduces to a simple, durable point. A robots.txt disallow, and a block keyed to a declared user-agent string, both depend on the crawler telling the truth about who it is. A user-agent is self-reported. A well-behaved crawler declares itself accurately and honors the file. A crawler that presents itself as an ordinary browser, from rotating addresses, is difficult to distinguish from a real human visitor using that same browser, which is exactly why generic-browser traffic is hard to block without also blocking people.

This is why blocking AI bots in robots.txt is a necessary hygiene step but never a guarantee of exclusion. The file governs the crawlers that choose to be governed. It cannot, on its own, stop a fetch that refuses to identify itself as a bot. Enforcement, as opposed to a polite request, requires a different layer: verified-bot allowlists, behavioral detection, and network-level controls of the kind a CDN or edge provider operates. The lesson is not that robots.txt is useless. It is that robots.txt is an instruction sitting one layer above the enforcement that would make it binding, and the two are frequently confused.

Crawlability is not one setting: the kinds of AI bots

The incident also exposes a more basic modeling error, the idea that there is a single lever called "AI crawler access." There is not. The AI-crawler ecosystem is split by function, and a single blanket rule treats categories that behave differently as if they were one thing.

Broadly, current technical accounts converge on three types. Training crawlers (for example GPTBot, CCBot, anthropic-ai) bulk-ingest pages to build model training sets and generally respect robots.txt. Retrieval and search crawlers (for example OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch live at query time to assemble an in-session answer. User-triggered fetch agents (for example ChatGPT-User, Claude-User) act more like a browser a person is driving, and enforce robots.txt less consistently because a human initiated the request. Because these behave differently, a one-line "block AI" or "allow AI" directive is a category error: the right posture differs by crawler purpose, and by whether you want to be ingested for training, fetched for answers, or both.

• Training crawlers ingest for model building and generally honor robots.txt; blocking them affects future training corpora, not live answers.
• Retrieval crawlers fetch at query time to power a synthesized answer; blocking them can affect whether you are drawn on in that moment.
• User-triggered agents behave like a person-driven browser and apply robots.txt least consistently, which is where the declared-versus-actual gap is widest.

The pattern behind it: an infrastructure-honesty gap

The stealth-crawler episode is not an isolated oddity. It is one instance of a broader pattern in the discovery-science stack: the tooling layer that vendors promote as "how you control AI visibility" is younger, leakier, and less standardized than the marketing implies.

Consider the file that was supposed to be the polite counterpart to a block, llms.txt, the proposed root-level file meant to hand AI systems a clean, curated summary of a site. Ahrefs analyzed 137,210 domains and found that 97 percent of valid llms.txt files received zero requests in a single month of 2026, and Google has stated publicly that the file is not used for search. So one file that asks bots to stay out can be circumvented by a crawler that refuses to identify itself, and another file that invites bots in is, at scale, not being read. Both facts point the same way: the control surface owners are told to configure is real but partial, and confident claims about "controlling" AI access should be met with the question of which layer actually enforces the control.

What this means for a site owner deciding whether to block AI crawlers

The practical takeaways are unglamorous, which is the point. First, keep robots.txt correct and specific, treating it as the declared-intent layer it is, not as a security boundary. Second, decide access per crawler purpose rather than with a single switch, because being ingested for training and being fetched for a live answer are different trades with different consequences for visibility. Third, if genuine exclusion matters to you, understand that enforcement lives at the edge and network layer, not in a text file, and plan accordingly.

Above all, verify rather than assume. The one durable instruction from the Cloudflare-Perplexity episode is that what a crawler declares and what it actually fetches can diverge, so the only reliable knowledge of how your pages are being crawled, rendered, and reached comes from inspecting the real behavior against your real logs and index state. That reading, not a default configuration, is the real starting point for any decision about AI-crawler access.