Crawlability Is Not One Setting: A Field Guide to the Three Kinds of AI Bots

One robots.txt rule, three different machines

A single line in a robots.txt file feels like a switch: allow the bots, or block them. That intuition is where most AI-crawler advice goes wrong. The label "AI bot" collapses at least three distinct kinds of automated visitor, each arriving for a different purpose, each with a different relationship to the content it reaches, and each with its own record of whether it honors the rules you write.

The distinction that matters is not which company sent the crawler. It is what the crawler is for. A bot that copies your page into a training set that a model will read months from now is doing something categorically different from a bot that fetches your page this second to compose a live answer for a person who just asked a question. Treating both with one rule means you cannot express the posture most businesses actually want, which is usually some form of "do not ingest me for training, but do read me when a buyer is asking about me right now."

This is the crawlability-is-not-one-setting problem. Before you can decide what to allow, you have to know the three functional categories the ecosystem has settled into, and you have to hold them honestly, because the behavior of any individual bot is a moving target, not a fixed fact.

The three kinds of AI crawlers, by function

The AI-crawler ecosystem is bifurcated by function, not just by company. Multiple independent technical analyses converge on the same three-way split, which is what makes it a usable framework rather than one vendor's taxonomy. The categories are training crawlers, retrieval crawlers, and user-triggered fetch agents.

AI bot user agents you will actually see in your logs

The framework only helps if you can map it to strings in your access logs. The named user agents below are the ones commonly cited in current technical write-ups, grouped by the three functions. Treat the list as a starting inventory rather than a closed set, because vendors add, rename, and split agents regularly.

• Training crawlers (bulk ingestion for pretraining): GPTBot, CCBot, anthropic-ai.
• Retrieval crawlers (live fetch at query time to build an answer): OAI-SearchBot, PerplexityBot, Claude-SearchBot.
• User-triggered fetch agents (a person initiated the request): ChatGPT-User, Claude-User.

Why one rule cannot govern all three

Once the three functions are separated, the category error in "block AI" or "allow AI" becomes obvious. The posture a business usually wants is not uniform across the three. A common, defensible stance is to decline training ingestion while remaining fully reachable by retrieval crawlers and fetch agents, so that you are not feeding future models for free but you are still eligible to be the answer when a live buyer asks. A single directive cannot express that, because it cannot tell the three apart.

There is a second reason a blanket rule misleads. The three classes are enforceable to different degrees. The training class is the one that most reliably honors a named-agent block. The retrieval class is the one you most likely want to keep open. The fetch-agent class is the one whose compliance is least predictable. A rule that treats all three identically will over-block the traffic you value and give you false confidence about the traffic you meant to stop.

This is also why crawlability belongs in the Technical Foundation pillar and not in a content or authority discussion. Whether a retrieval crawler can reach and read your key pages is an eligibility question that sits below everything else. A page that a retrieval crawler cannot fetch cannot be cited in an answer whatever else is true about it, in the same way that a page an engine cannot index cannot appear in a classic result.

Does robots.txt actually stop AI crawlers?

Writing a rule and having it obeyed are different things, and the gap is documented. In August 2025 Cloudflare reported that Perplexity was using an undeclared, generic browser-spoofing crawler, impersonating Chrome on macOS, to continue fetching pages after being blocked through robots.txt and its declared user agent, rotating both IP address and ASN to evade the blocks across tens of thousands of domains. Cloudflare said it verified through live query-testing that Perplexity's answers still reflected content from domains that had explicitly blocked its crawler, and it subsequently de-listed Perplexity as a verified bot. Perplexity publicly disputed Cloudflare's methodology and framing.

Two things are true at once here. The incident itself is independently documented and verifiable. Whether it characterizes Perplexity's ongoing, general practice is contested, because it is a single vendor's report rather than a third-party audit, and the company denies the framing. The durable lesson is not "distrust one named engine." It is structural: a robots.txt directive is a request that a well-behaved crawler honors, not a wall that enforces itself, and the fetch-agent class in particular treats it inconsistently. Assuming a block is airtight is the mistake.

The same caution applies in the other direction, to the tooling sold as a way to control AI visibility. The llms.txt file, promoted as a way to hand AI systems a curated summary of your site, is not read at meaningful scale: an analysis of 137,210 domains found 97 percent of valid llms.txt files received zero requests in the month studied, and Google has stated publicly that llms.txt is not used by its Search systems. Crawl control is younger and leakier than the marketing around it suggests, which is a reason to measure what reaches your pages rather than trust a setting to do the governing for you.

Being reached is not the same as being read faithfully

Suppose you get the posture right and a retrieval crawler fetches your page cleanly. That is necessary, but it is not the finish line, and pretending otherwise is where a lot of AI-visibility claims overreach. Two separate gaps sit between "my page was fetched" and "my business was named accurately."

The first is rendering. Retrieval crawlers vary in how much of a page they execute, and content that only appears after client-side JavaScript runs can be invisible to a crawler that does not render it (Google Search Central, JavaScript SEO documentation, describes the two-wave crawl-render-index behavior for Google's own systems). A page that looks complete in a browser can be partly blank to the machine that fetched it, so reachability and readability are distinct checks.

The second gap is attribution. Even when a system does cite sources, the citation and the reasoning can be decoupled: an attribution survey of tested open-source models reported that over 95 percent of their answers contained at least one unattributed sentence, and the RAG faithfulness literature shows that a cited source is not reliably the source the model actually reasoned from. Being fetched, being rendered, and being faithfully credited are three separate outcomes. A field guide to the crawlers is the first of the three, not a shortcut past the other two.

Should you block AI crawlers? A posture, not a switch

The right question is not "should I block AI crawlers" as a yes or no. It is "which function do I want to allow, for which pages, and how will I know whether the rule is being honored." That reframing turns a switch into a posture, and a posture is something you can hold and revisit as bot behavior changes.

A workable default for a business that wants to be found: keep retrieval crawlers and user-triggered fetch agents fully able to reach the pages that describe what you do, decide training ingestion on your own view of the trade-off, and instrument your logs so you can see which of the three functions is actually arriving rather than assuming your robots.txt describes reality. The table below tiers the claims in this article by how settled they are, so you can weight them accordingly.

Where this fits in a technical read of your site

Crawlability sits at the technical floor of visibility, underneath content and authority. The functional taxonomy tells you what to check for; a technical read tells you what is actually happening on your own domain. Which retrieval crawlers are reaching your key templates, whether your important pages are readable without JavaScript execution, whether your robots.txt is expressing the posture you think it is, and whether anything is fetching you in ways your rules did not anticipate, are all questions answered by inspection, not assumption.

That inspection is a bounded, reproducible engineering exercise against published standards, which is exactly why the Technical Foundation is the most objective surface to measure. The rest of this guide is the framework. The value of the framework is realized only when it is pointed at a specific site and turned into a ranked list of what to fix first.