The Crawl-to-Click Gap: What AI Engines Take From the Web vs. What They Give Back

Why Cloudflare's number is different from a survey

Most claims about how AI engines treat the web come from surveys, sampled studies, or a single site's server logs, useful but limited vantage points. Cloudflare is a different kind of source. It operates content-delivery and security infrastructure in front of a large share of the world's websites, which means it can observe actual crawler and referral traffic across a vast footprint of real sites, not a sample of them. That is the specific reason its crawl-to-referral figures below carry more weight than most competing claims in this space: it is a network-observed measurement, not a survey response.

The core numbers: 1,700 to 1, and 73,000 to 1

Cloudflare's analysis, published as "The crawl-to-click gap," found that by mid-2025 roughly 80 percent of all AI crawling activity was for model training purposes rather than live retrieval to answer a specific user query. Within the crawling it could attribute to specific companies, the ratio of pages crawled to visitors referred back to the site was approximately 1,700 to 1 for OpenAI and approximately 73,000 to 1 for Anthropic, as of June 2025.

To make that concrete: for every single visitor Anthropic's crawlers sent to a site, that site had already been crawled roughly 73,000 times. That is not a rounding difference from a typical referral relationship. It is a fundamentally different kind of traffic, overwhelmingly extractive, with visitor referral as a small and, for some crawlers, a vanishingly small byproduct.

The gap widened for some crawlers into 2026

Cloudflare's follow-up analysis, "The crawl before the fall of referrals," tracked the same ratios into the first quarter of 2026 and found the pattern had not eased for every crawler; for some it worsened. ClaudeBot's ratio reached approximately 23,951 to 1 by Q1 2026, close to Anthropic's already-extreme mid-2025 figure and moving in the wrong direction from a publisher's standpoint.

Not every crawler behaved the same way, and the differences are informative on their own. Perplexity's ratio sat far lower, at roughly 111 to 1, a figure closer in kind, if still substantially higher, to Google's own traditional search referral ratio of about 4.9 to 1. The spread across companies, from single digits at Google, through low hundreds at Perplexity, to tens of thousands at Anthropic and ClaudeBot, shows this is a design and business-model choice each company is making, not an unavoidable feature of AI crawling in general.

Two independent studies converge on roughly the same referral share

A single source, however strong its vantage point, is one lens. Two independently conducted studies, using entirely different methods, arrive at a consistent order of magnitude for how little referral traffic AI chatbots currently send. Ahrefs studied 3,000 websites directly and found AI chatbots deliver approximately 0.17 percent of the average site's total referral traffic today. Separately, an AEO-platform analysis by Conductor, examining ten major industries, put AI referral share at roughly 1 percent of total web traffic.

These figures should be read as emerging rather than settled to the decimal point, since both are single studies measuring a fast-moving landscape and the two numbers themselves differ by roughly six times from each other, 0.17 percent versus 1 percent. But the order of magnitude both land on, a small single-digit-percent share of total traffic at most, is the important, corroborated finding: whichever exact figure is closest to true for a given site, AI referral traffic is currently a minor fraction of what classic search referral traffic still provides.

What the asymmetry means, stated precisely

This data does not show that "AI steals your content" in a moral sense, a framing that adds heat without adding evidence. It shows a structural fact: the current generation of AI-answer engines consumes far more of a site's content, through crawling for training and for live retrieval, than it currently returns in the form of visitors sent back. Cloudflare's own 80-percent figure for training-purpose crawling versus live-retrieval crawling shows most of that consumption is not even tied to a specific user question being answered in the moment; it is model-building activity that happens regardless of whether any user ever asks about that page at all.

For a business used to measuring digital marketing success through referral traffic in an analytics dashboard, this is the single most important number in this piece to internalize: an AI engine can be reading your content constantly, using it to inform the answers it gives to thousands of users, and send your Google Analytics report a number statistically indistinguishable from zero. The absence of referral traffic is not evidence the content is not being used. It is evidence the wrong metric is being watched.

Why this reframes AI visibility as a citation problem, not a traffic problem

If referral traffic from AI engines is currently a fraction of a percent of a typical site's total, and is unlikely to become a large fraction any time soon given the crawl-to-referral ratios above, then building an AI-visibility strategy around driving click-through traffic from these engines is optimizing for the wrong outcome. The actual value at stake is whether a business is named, described accurately, and cited inside the answer itself, since for a large share of AI-assisted queries, that answer is the entire interaction a buyer has with the information before making a decision.

This is the precise argument for treating AI visibility as a share-of-answer measurement, how often a business is named when its buyer population asks a relevant question across these engines, rather than as a referral-traffic-driven channel to be judged by the same click-based standard as classic search. The two are structurally different problems that happen to look similar from a marketing dashboard, and conflating them leads to the wrong investment.

What the data does not yet establish

Two caveats belong alongside these findings. First, Cloudflare's crawl-to-referral ratios, while network-observed and credible, are not audited third-party figures and represent Cloudflare's own calculation methodology; they should be treated as the strongest available evidence in this space, not as an independently reconciled industry standard. Second, both the Ahrefs and Conductor referral-share figures describe a fast-moving landscape as of the studies' publication and are likely to shift, potentially significantly, as AI-answer products change how and whether they link out to sources at all.

What is unlikely to change in the near term is the underlying direction: engines built to synthesize an answer from many sources have a structural reason to consume more than they refer, since referring a click away from the answer works against the product's own design goal of resolving the query on the spot. That structural incentive, not a temporary technical limitation, is why this piece treats the asymmetry as established rather than a transitional phase.