Multiple "X% of websites block AI crawlers" figures are already in circulation. We checked several and none disclosed a reproducible sampling frame — which domains, how many, from when. Rather than add another unsourced number, this page pre-registers a census we can actually run and verify.
Why a census, not another statistic
"What share of sites block GPTBot?" is one of the most commonly asked questions in AI-search technical SEO, and one of the least reliably answered. Reports exist that quote a specific percentage. But tracing them the way the provenance audit traces other claims, the sampling frame, which domains, how many, what date, is rarely stated clearly enough to reproduce or even sanity-check.
That doesn't mean the existing figures are wrong. It means nobody — including us, without doing the work below — is in a position to say whether they're right.
What is actually at stake in this number
A robots.txt blocking rate sounds like a narrow technical curiosity. But it functions as an early indicator for a much bigger, actively unfolding negotiation between publishers and AI companies over how content gets used. A rising blocking rate signals growing publisher resistance. A falling one could signal either resolved licensing arrangements or simple resignation. Getting this number right, tracked consistently over time, matters to more than one audience.
What is currently claimed
A meaningful and rising share of large sites block at least one major AI crawler. Consistent with the broader publisher-hostility narrative around AI crawl-to-referral economics, but the specific percentage in circulation could not be traced to a disclosed, reproducible sample.
Blocking correlates with publisher category — news and media sites are widely assumed to block more aggressively than e-commerce or SaaS sites. Plausible, but we found no rigorous cross-category breakdown with a stated method.
Existing figures, compared
Applying the same provenance-tracing method used elsewhere on this site to two figures currently in circulation illustrates exactly the gap this census is meant to close:
| Claim | Stated sample? | Stated date? | Grade |
|---|---|---|---|
| Coverage citing a rising share of top sites blocking at least one AI crawler | Not disclosed at the level needed to reproduce | Approximate only | Partial |
| Coverage citing a specific crawl-to-refer-derived blocking estimate | Not disclosed | Approximate only | Partial |
Neither source is being singled out as unusually weak — this is the norm for the category, not the exception, which is precisely the point.
Census design
- 01 Sample domains A published top-N list, stratified by category
- 02 Fetch robots.txt Direct HTTP fetch, no cache, dated
- 03 Parse directives Per-bot Allow/Disallow, structured
- 04 Classify Blocked / allowed / no rule, per crawler
- 05 Publish raw CSV of every domain × bot × rule
What a sample row will look like
To make the eventual dataset concrete rather than abstract, here is the shape of a single row, using invented values to illustrate structure only:
domain: example-news.com category: news robots_txt_exists: true last_modified: 2027-01-09 GPTBot: disallow OAI-SearchBot: allow ClaudeBot: disallow Claude-SearchBot: allow PerplexityBot: no_rule collected_at: 2027-01-14T00:00:00Z
A dataset of thousands of rows in this shape supports exactly the kind of cross-tabulation — by category, by bot type, by presence of any rule at all — that a single aggregate percentage cannot.
What will be recorded
Why per-bot differentiation matters more than a single blocking number
A single "X% block AI crawlers" figure conceals the distinction this whole site's technical-SEO cluster argues is the most important one in the field. Training bots and retrieval bots are different decisions with different consequences, covered in full in the AI Bot Registry. A domain that blocks GPTBot but allows OAI-SearchBot has made a coherent, defensible choice. That's visible only if the census records per-bot data, rather than a single yes/no "blocks AI" flag.
Hypothesis C4 below is a direct test of whether that kind of differentiated policy is actually common in practice, or whether most site owners are applying an undifferentiated rule regardless of what the registry recommends.
Pre-registered hypotheses
| # | Hypothesis | Prediction |
|---|---|---|
| C1 | Training crawlers (GPTBot, ClaudeBot) are blocked more often than retrieval crawlers (OAI-SearchBot, PerplexityBot) on the same domains | Supported |
| C2 | News and media domains block at a higher rate than e-commerce domains | Supported |
| C3 | Blocking rate has increased since the prior year across the same domain set, re-checked | Supported |
| C4 | A large majority of domains apply a uniform "block all AI" policy rather than differentiating by bot category | Not supported |
C4 predicts against the intuitive assumption. Our expectation, unverified until the census runs, is that most site owners either take no action or apply a blanket rule copied from a template — not the differentiated per-bot policy that the registry recommends.
Multiple reports quote a specific percentage of sites blocking AI crawlers. None we checked disclosed a reproducible sample. So instead of repeating one, we're pre-registering a census that will.
Share on XSchedule
First collection targeted alongside Index v1 in Q1 2027, re-run semi-annually against the same domain list so year-over-year change is comparable rather than an artifact of a different sample.
What this means for publishers, whichever way the numbers land
Suppose C1 and C4 both hold: training bots blocked more than retrieval bots, but most sites still applying an undifferentiated policy. Then a large share of publishers are leaving citation-driving retrieval bots blocked alongside training crawlers they meant to restrict. That's an easily fixable mistake, once the census makes the pattern visible.
Suppose C4 fails instead, and differentiation turns out to be common. That would be a genuinely positive finding about how quickly nuanced technical-SEO guidance has actually been adopted, worth reporting with equal enthusiasm to a null result, consistent with this site's registry of pre-registered null predictions.
Will this cover sites outside English-language, US/EU markets?
Not comprehensively at first. A top-domain list skews toward large, globally-recognized properties, which themselves skew toward English-language and Western markets in most published rankings. Do sites in markets with different data-protection and AI-training regulation apply different blocking policies? That's a genuinely interesting question this initial design can't answer well. It's registered as a candidate expansion for a later edition, not solved here.
Limitations
- A top-domain list is not a random sample of the web — findings describe large, established sites, not the long tail.
- robots.txt states policy, not enforcement. Whether a "disallowed" bot actually complies is a separate, harder question — see the registry's behavior section.
- A domain's robots.txt can change between the census date and when you read this — every figure will carry its collection date.
- Deliberate vs. default-template rules cannot be distinguished from the file alone, as noted in the FAQ — a genuine measurement gap this design does not close.
Namdev, R. (2026). The robots.txt AI-Blocking Census (v0). Retrieved from https://ritiknamdev.com/blog/robots-txt-ai-blocking-census Published under CC BY 4.0 — reuse freely with attribution.
This follows the same discipline as the AI Citation Index — publish the method before the number. See the AI Bot Registry for the crawlers this census will check for.