No one has taken a current top 50 from Ahrefs, Profound, or Promptwatch and checked each domain's robots.txt. The only opened join of "was cited" to "blocks a crawler" is a news probe. HasData, in July 2026, ran 10 Google AI Mode queries, found 82 cited domains, and could score 52 of them. Twenty-seven of those 52, 51.9%, disallow at least one AI crawler, against 15% of the crawl sample. The names are newsrooms, plus LinkedIn. The probe was not repeated in September. YouTube, Reddit, and Wikipedia were not in it.
- 51.9% is not the block rate of the famous domains. It is 27 of 52 cited domains from ten news queries, restricted to domains already inside a crawl sample.
- The printed blockers are nbcnews.com, nytimes.com, cnn.com, cnbc.com, theguardian.com, linkedin.com, finance.yahoo.com, thehill.com, bbc.co.uk, and apnews.com. Citation counts in the probe run from 1 to 3.
- News publishers block far more often than the open web. In the September re-run, any AI crawler is disallowed by 53.0% of news publishers and 8.3% of the top web. The drop from July is mostly a Cloudflare robots.txt change, not a wave of sites opening up.
- A disallow line is a weak predictor of absence from Google AI answers. Those answers can ground from Google's index after the training crawlers have been turned away.
- LinkGuard, Attensira, and Stackscan are popularity or convenience counts. None of them starts from a most-cited list. Ahrefs' May 2025 bot-block study was not opened and is not used.
The only join that exists
HasData's AI Crawler Block Index scores robots.txt. The citation join is a sidebar of that project, measured in July and explicitly not repeated when the index was re-run on 16 September 2026. Ten Google AI Mode queries on a general news mix produced 82 unique cited domains. Fifty-two of them sat inside the 10,894-domain crawl sample. Twenty-seven of those 52 disallow at least one AI crawler. HasData reads that as more than three times the sample rate of 15%. Partial Ten queries. News-weighted. One month.
"Blocked," in this index, means the bot's own user-agent group has Disallow: / with no overriding Allow. A global User-agent: * disallow does not count. That definition matters. A site can look closed in a browser test and open in this table, or the reverse.
The domains that lead the US citation lists were not in the scored set. YouTube, Reddit, Wikipedia, and Google were not checked here. LinkedIn is the exception, because it showed up in the news answers. Treating "news domains block crawlers" as "the most-cited domains block crawlers" swaps a query mix for a market. The news-domain list, which is a citation question and not a robots question, is on most-cited news domains.
The domains in that probe
| Domain | AI crawlers disallowed | Citations in the 10 queries |
|---|---|---|
| nbcnews.com | 8 | 3 |
| nytimes.com | 8 | 2 |
| cnn.com | 7 | 2 |
| cnbc.com | 8 | 2 |
| theguardian.com | 6 | 2 |
| linkedin.com | 7 | 2 |
| finance.yahoo.com | 6 | 2 |
| thehill.com | 8 | 1 |
| bbc.co.uk | 8 | 1 |
| apnews.com | 5 | 1 |
These are the names HasData printed, not a complete ranking of the 27. A site can block GPTBot, ClaudeBot, CCBot, and PerplexityBot and still be the footnote on an AI Mode answer. HasData's mechanism note is the useful one. Google-Extended is separate from GPTBot. A site can refuse the training crawlers and still be grounded from Google's index. Sites that also disallow Google-Extended can still appear via Google's existing search cache. The bot that fetches for an answer is not the bot that trains a model. That split is the subject of GPTBot vs OAI-SearchBot.
The census registered on this site has not produced this table. The robots.txt AI-blocking census is a protocol. Quoting HasData as if it were that census would merge two projects.
The web-wide rates are a different question
The index itself is large, and it answers "who blocks," not "who is cited." Sample: Tranco top 10,000, list 46XLX from 9 July 2026, plus 1,148 news homepages, deduplicated to 10,894 registrable domains. Re-measured 16 September 2026, last updated 29 September.
| Rule | Top web, July | News, July | Top web, September | News, September |
|---|---|---|---|---|
| Any AI crawler | 10.3% | 56.4% | 8.3% | 53.0% |
| GPTBot | 7.9% | 50.5% | 5.7% | 46.9% |
The drop is largely Cloudflare removing a managed robots.txt notice on 15 September 2026, not a decision by thousands of publishers to open the door. Live fetch told a stricter story than the file. Of sites that disallow GPTBot, 39.5% still returned HTTP 200 to a request naming GPTBot, in the July enforcement subset. On publishers, the live GPTBot pass rate fell from 54.2% in July to 47.2% in September, while a browser control stayed near 83%. A robots.txt line and a refused connection are different measurements. This page uses the line, and says when it is using the line.
Three other counts are popularity censuses. They are not joins, and they disagree with each other because the samples differ. LinkGuard, 31 May 2026, is a convenience sample: news and media 90% (27 of 30), reference and community 60% (9 of 15), blogs and publishers 23% (3 of 13), SaaS and marketing 15% (3 of 20). Attensira, 3 September 2026, read 8,036 of the Tranco top 10,000: GPTBot blocked on 14.3%, OAI-SearchBot on 5.8%. The author treats those as a floor, because a CDN rule does not appear in robots.txt. Stackscan, 10 September 2026: 6.88% of 90.8 million sites that have a robots.txt block GPTBot. Of the top 1,000 sites that serve a robots.txt, 13.0% block it. None of these starts from a citation rank. Using Attensira's 14.3% as "14% of cited domains block GPTBot" would be a substitution this page exists to refuse.
The wider crawler statistics, which are about traffic rather than about who gets cited, are on AI crawler statistics. The engine whose news answers were probed is AI Mode.
What this does not prove
Open question It does not say Reddit, YouTube, or Wikipedia block or allow any crawler. Those files were not scored against the citation lists. It does not say a newsroom that blocks GPTBot will vanish from ChatGPT. Search crawlers and training crawlers are different tokens, and the opened censuses were not cross-tabulated with citation rank. It does not say 51.9% of the most-cited web blocks AI. Ten news queries selected the kind of site that blocks, and then found that those sites block.
How to use it
If the goal is to be cited on a news query, the domains that already win those queries often disallow training crawlers and still appear in Google AI Mode. Copying their robots.txt is not the strategy. If the goal is to know whether the global head blocks crawlers, that table has not been published. It would be a short piece of work, and it is the work the protocol on this site describes. It is not done.
The argument for keeping domain lists and crawler lists apart is the same argument as the rest of the series, on most-cited domains in AI search.
No one has checked the robots.txt of the global most-cited domains. On ten AI Mode news queries, 27 of 52 cited domains in the sample block at least one AI crawler. YouTube, Reddit, and Wikipedia were not in the test.
Share on XVerification status
HasData's index page is the join and the census. LinkGuard, Attensira, and Stackscan are labelled as samples that do not start from citations. The Ahrefs bot-block study of 21 May 2025 was not opened. No robots.txt was fetched for this page.
Namdev, R. (2026). Which most-cited domains block AI crawlers (v1). Retrieved from https://ritiknamdev.com/blog/most-cited-domains-blocking-ai-crawlers Published under CC BY 4.0 — reuse freely with attribution.
A full census of the open web is a protocol on the robots.txt AI-blocking census. It has not scored these domains. The training-crawler versus search-crawler split is on GPTBot vs OAI-SearchBot.