AI crawler statistics

Headline numbers

Terms used on this page

Two terms recur throughout and are worth defining precisely before the data, since imprecise use of either is a common source of confusion in secondary coverage of this topic.

Why this data matters now, not just as trivia

Crawl statistics can read as an operations footnote, the kind of number an infrastructure team monitors quietly without it ever reaching a strategy conversation. That undersells what's actually happening. Every page an AI crawler fetches is server load, bandwidth, and in some hosting setups direct cost, incurred whether or not that crawl ever produces a citation, a mention, or a single referred visitor.

A site owner deciding whether to allow or restrict a given bot is making an economic decision with a real, measurable cost side and an uncertain, hard-to-measure benefit side. That's exactly the asymmetry this page's numbers are meant to make visible, rather than leave to guesswork.

Crawl-to-referral ratios

The crawl-to-referral ratio is the single most useful number in this page: how many pages a bot crawls from your site for every visitor it sends you. A 100:1 ratio means it crawled 100 pages before sending one person.

Read plainly: Mistral's crawler took roughly 3,389 pages for every visitor it referred back, the most extractive ratio recorded among major operators. Anthropic's ClaudeBot and OpenAI's GPTBot follow, at 2,237:1 and 217:1 respectively. Google's traditional crawler, included for contrast, sits at 4.6:1. That's a difference of nearly three orders of magnitude between the most and least extractive crawlers tracked.

How the ratio has moved over the past year

That jagged note is deliberate, not an oversight. It's an honest illustration of exactly the problem this whole page exists to flag. Secondary aggregator coverage of Cloudflare data reports figures at different points using seemingly different methodologies. Reconciling them into one clean trend line would misrepresent how confident the underlying picture actually is. A first-party, consistently-measured version of this exact chart is one of the concrete deliverables planned once the Citation Index's crawler-monitoring component is running.

Market share by crawler

The year-over-year swing is the more interesting finding than the snapshot itself: a crawler's relative share moved by several points within twelve months, which means any specific ranking quoted today should be assumed to be temporary rather than structural.

AI's share of all bot traffic

Who blocks whom

A rising share of the top web domains now block at least one AI crawler in robots.txt. The exact percentage circulating in secondary coverage varies by which crawlers are counted and which domain set was sampled. This is precisely the kind of figure the field needs a dedicated, transparent census for, rather than a single oft-repeated number.

We aren't publishing a specific blocking-rate figure on this page until we can verify one directly against a stated sample, for the same reason the provenance audit flagged similar figures elsewhere. The dedicated study designed to close this gap is the robots.txt AI-Blocking Census.

Why the ratio is so lopsided

Three structural reasons, none of which require assuming bad faith on anyone's part:

• Different business models. A search engine's product is the click — crawl efficiency against referral volume has been optimized for two decades. An AI model's product is the answer itself; sending a referral is not what the user is paying for.
• Training crawls are bulk, one-time-ish operations. A crawler building a training corpus has no per-query reason to convert a crawl into a visit — the entire notion of "referral" doesn't apply to that use case the way it does to retrieval.
• Retrieval bots still cite rather than redirect. Even AI-search retrieval bots that fetch pages live to answer a specific question typically show a citation link rather than driving the same click-through rate a blue link in classic search does.

Does the ratio vary by industry or site type?

This is a reasonable, untested hypothesis, not a documented finding. A plausible mechanism exists. A reference-heavy site, documentation, an encyclopedia-style resource, a large product catalog, offers an AI crawler more distinct, individually valuable pages per crawl session than a site with a small number of pages updated rarely. That could push the ratio in either direction, depending on how citation likelihood interacts with page count.

No published, disclosed-method breakdown by vertical was located during this page's research. Open question on the site's evidence-tiering scale, and a candidate addition to a future edition of the Citation Index's crawler module.

What to do with these numbers

A lopsided ratio is not automatically a reason to block a crawler — it's an input to a decision, not the decision itself. If an engine sends negligible traffic but citing your site there drives brand awareness, inquiries, or reputation, allowing the crawl may still be worthwhile. If it sends negligible traffic and you can't identify any offsetting benefit, blocking the training variant of that bot costs you very little. See the registry for the exact robots.txt syntax per bot.

How AI crawlers compare to legacy search bots, historically

It's worth a moment of historical framing, because the "crawl a lot, refer relatively little" pattern isn't entirely new. Early-2000s search engines also crawled aggressively before their indexes and ranking systems matured, and crawl efficiency improved over subsequent years as those products optimized for it.

Do AI crawlers follow a similar trajectory, improving their crawl-to-referral efficiency as the underlying products mature and as referral-driving UI patterns, clearer citations, more prominent source links, become more established? Or does the structural difference described in §7 keep the ratio permanently more extractive than classic search? That's itself an open, falsifiable question worth revisiting in a year.

A worked example: reading your own server logs against these ratios

These industry-wide figures become genuinely useful once compared against a specific site's own log data. Here is how that comparison works in practice, using invented numbers for a hypothetical mid-traffic blog:

1. Pull a month of server logs and filter for requests matching a known AI bot user agent — say, ClaudeBot makes 45,000 requests to the site that month.
2. Cross-reference with analytics for referral visits attributable to Claude — say, 12 visits with a Claude-attributable referrer in the same window.
3. Compute the site-specific ratio: 45,000 ÷ 12 ≈ 3,750:1 — notably higher than the 2,237:1 industry figure cited above, suggesting this hypothetical site is, for whatever reason, less efficiently converting ClaudeBot's crawl volume into referral traffic than the broader network average.
4. Decide what that comparison is worth acting on. A ratio meaningfully worse than the published baseline is a prompt to investigate further — is content stale, poorly structured for citation, or is the site simply in a less AI-search-visible category — not an automatic instruction to block the bot.

This kind of comparison is exactly what the crawl-side half of the Citation Index's panel program is designed to make easier at scale, once enough participating sites contribute anonymised logs to build a properly segmented baseline rather than one industry-wide average.

Methodology & verification status

This page is a case study in its own subject matter — see where AI SEO statistics actually come from for the full audit this methodology section is trying to hold itself to.