GPTBot vs OAI-SearchBot: the AI crawler guide

1. The decision, in two lines

Every major AI company now splits crawling into separate, independently controllable bots. The expensive mistake is a copy-pasted "block all AI bots" rule that catches the citation crawlers along with the training ones. Here is the whole decision on one row per bot:

OpenAI's documentation is explicit: sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." The common instinct to block GPTBot is perfectly coherent — it just has nothing to do with search visibility, which a different bot governs. The citation numbers that make this worth caring about are on the ChatGPT citation statistics page.

Blocking GPTBot protects your training data. Blocking OAI-SearchBot deletes you from ChatGPT search. They are not the same decision.

2. Pick a policy, copy the robots.txt

Three policies cover almost every site. Find the one that matches your business model and paste the block.

Policy A — maximum visibility. You want to be discovered and cited, and you have no objection to being trained on. Most brands, businesses and creators using content to attract customers are here.

# Policy A: allow everything. Every crawl is a chance to be cited.
User-agent: *
Allow: /

Policy B — cited, but not trained on. The default recommendation for most sites with a content-rights position: keep full AI-search visibility, opt out of model training.

# Policy B: stay visible in AI search, opt out of training.

# --- Allow AI SEARCH & CITATION bots ---
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

# --- Block AI TRAINING bots ---
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /

Policy C — keep content out of AI entirely. If your paid product is the writing, this is a legitimate stance. But robots.txt alone will not achieve it: the non-compliant crawlers ignore it, so this file is the polite half of an enforcement layer, not the whole of it.

# Policy C: robots.txt half only. Pair with a WAF or Cloudflare AI Crawl Control.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

Recognise the trade-off in Policy C: fully walling off content also removes it from the answers where discovery increasingly happens. If you are unsure which policy you are in, default to A. The visibility upside of being crawled generally outweighs the theoretical downside of being trained on, and you can tighten later. While you are in the file, if you are also considering an llms.txt file, read whether llms.txt actually works first — the published evidence is a null.

3. Why every vendor has three bots

OpenAI, Anthropic, Perplexity, Apple and Meta all separate crawling into three distinct jobs. Once you see the pattern, every vendor's bot list reads itself:

Anthropic has the cleanest separation after OpenAI: ClaudeBot trains, Claude-SearchBot indexes for Claude's search answers, Claude-User is the live fetch. Anthropic says all three respect robots.txt; the citation behaviour is on our Claude citation statistics page. A deprecated anthropic-ai/Claude-Web token is worth keeping in old block lists.

Perplexity declares no training crawler at all — its crawler documentation says it does not train foundation models — so both PerplexityBot and Perplexity-User exist for visibility and you would generally allow both. The asterisk is enforcement, covered in section 6. Apple splits by suffix: keep Applebot allowed for Siri and Spotlight, block Applebot-Extended to exit Apple Intelligence training. Meta uses Meta-ExternalAgent for Llama training and Meta-ExternalFetcher for live fetches. Common Crawl's CCBot has no citation product; its dataset feeds many models' training.

4. Every AI crawler, and what to do

Every named agent across every vendor is catalogued in our bot user-agent registry, cross-checked against third-party references such as Momentic's crawler list. Here is where crawl volume actually concentrates. Googlebot dwarfs everything at roughly 50% of all crawler requests — as the same report notes — so this chart shows the AI-bot subset for readability:

5. Google: the one case with no clean lever

Google-Extended lets you opt out of Gemini and Vertex training, which is genuinely useful. But AI Overviews are not a separate crawler — they are generated from the standard Googlebot search index. You cannot appear in Google Search while excluding yourself from AI Overviews. Anyone promising an "AI Overviews opt-out" that keeps your rankings is selling something that does not exist; Google's own AI features documentation says as much. What that surface actually cites is covered in AI Overview statistics and the source delta between AI Mode and AI Overviews.

6. How to verify the rule actually worked

Nothing in the robots.txt chain reports back to you. There is no confirmation, no error, no dashboard. Your logs are the only feedback loop that exists — so here is the verification sequence.

1. Fetch the file for every host you own. Root domain, every subdomain, any separate docs or shop host. Each is independent — a rule on example.com does nothing for docs.example.com.
2. Check the citation bots specifically. Confirm that no rule, including a broad * group, blocks OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot.
3. Check for a stray global disallow. A User-agent: * plus Disallow: / left over from staging is the single most damaging line a site can ship.
4. Filter your logs by user-agent for the tokens that matter: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Bytespider.
5. Verify by IP before you trust a count. User-agent strings are trivially spoofed. OpenAI publishes machine-readable ranges at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json; Google, Anthropic and others publish equivalents. Reverse-DNS plus an IP match is how you separate real crawlers from impersonators.
6. Read the result over the following week. A working block shows the bot fetching /robots.txt and then requesting nothing else. If it keeps requesting pages, either your rule did not match its token or the bot is not honouring it.

Two things about that last step. Bots cache robots.txt, so a rule added this morning may not be observed for hours or longer — no operator publishes how long they cache. And a rising crawl cadence is the earliest available signal of AI visibility: a 30-day log study of agentic crawling reported GPTBot re-fetching high-value pages roughly every 2.4 days once it had discovered them. Whether a recrawl rate is associated with a citation that lasts is what our citation half-life study asks. Finally, write your intended policy into a comment at the top of the file — otherwise nobody can tell a deliberate block from an accident.

Enforcement, where it matters. robots.txt is a request, not a wall. GPTBot, OAI-SearchBot, ClaudeBot, Googlebot and Applebot honour it reliably by their operators' accounts; CCBot and Bytespider have documented compliance gaps. In August 2025 Cloudflare de-listed Perplexity as a verified bot after catching it rotating IPs and spoofing a Chrome user-agent to reach sites that had blocked it. If you have content you genuinely must keep out, layer a WAF, Cloudflare's AI Crawl Control, or IP rules on top.

7. Common failure modes

Most robots.txt mistakes are mechanical, not strategic — the rule is written with the right intent and fails somewhere in the chain above.

• The stray blank line. A blank line ends a group. Rules placed after one belong to a different group than you intended.
• The token mismatch. Matching is on the token, not the full user-agent string. GPTBot does not match OAI-SearchBot, even though both are OpenAI. This is exactly where the "block all AI" instinct goes wrong.
• Assuming a wildcard covers AI bots. There is no agreed wildcard token for "AI crawler". A * group with Disallow: / blocks Googlebot too, which is almost never what anyone means.
• Reading documented compliance as verified compliance. Open question Every "honours robots.txt" statement here reflects what the operator published about itself.
• Reading "allowed" as "will be crawled". Allowing removes a barrier; it does not create demand. A small site can allow every bot in the table and see almost no AI crawler traffic. Which domains do get surfaced is its own finding — see the most-cited domains page.
• Reading crawl volume as visibility. A high crawl count from a training bot tells you nothing about whether any engine will cite you.

The common thread: robots.txt is a permission layer, not a growth lever. It can only take visibility away; it cannot add any. The levers that might add some — and how much evidence each actually has — are graded in the tactic evidence scoreboard. Revisit the file quarterly: new crawlers appear, and a robots.txt configured perfectly a year ago may be silently blocking a citation bot that did not exist when you wrote it.

8. Why this suddenly matters: the economics

Cloudflare reported that training now drives roughly 80–82% of all AI crawling, up from 72% a year earlier, while search crawling fell to about 18%. Most AI crawling, in other words, is not the crawling that gets you cited.

The return side is more lopsided still. Per Cloudflare's network data, for every visitor referred:

All four ratios are Cloudflare's, restated in independent write-ups. Whether the referrals that do arrive are worth anything is handled in AI referral traffic statistics and conversion benchmarks.

That imbalance is driving two reactions. The first is mass blocking. GPTBot is now the single most-blocked AI crawler, disallowed by roughly half of top news sites. AI-blocking among reputable sites climbed from around 23% in late 2023 to around 60% by mid-2025 — a trend visible in top-1000 robots.txt analysis, and the design for measuring it independently is registered as our robots.txt blocking census, which has not yet run. The second is monetisation: Cloudflare launched pay-per-crawl, returning HTTP 402 responses at scale. The free-for-all era of AI crawling is ending.

9. What nobody has actually verified

Open question Policy here is documented; behaviour is not. Three gaps stand out, and each is cheaply testable with a controlled site and some patience.

• Compliance at scale. No public, independent study has systematically tested whether each of these bots respects a fresh robots.txt block across many sites. The disputed cases we know about surfaced through individual investigations, not systematic testing.
• Whether training and search indexes are genuinely separate. The three-bot model implies they are. That has not been tested, and a shared internal pipeline could couple them in ways outside observers cannot see.
• Whether retrieval bots render JavaScript. If they do not, a client-rendered page is invisible to them whatever robots.txt says. Our rendering study is the attempt to settle it. Even for Google, JavaScript costs roughly nine times the crawl time of HTML.

Until someone runs those tests, treat every compliance claim on this page — and on every other page like it — as the operator's word rather than an observation. When we run them and find nothing, the result goes in the null results registry rather than quietly into a drawer.