The AI Bot User-Agent Registry

Why a registry, not a listicle

Every SEO publication has an "AI bots to know" article. Almost none of them are maintained after publication. Six months later, half the user-agent strings are outdated, and a new bot from a launch that quarter is simply missing. A registry that isn't kept current is worse than no registry. It gives a false sense of completeness.

This page is designed to be corrected. If an operator changes a user-agent string, publishes a new bot, or updates its crawl behavior, the row updates. The change gets logged at the bottom with a date. Same discipline argued for in the provenance audit.

How we ended up with sixteen different bots to track

It's worth a moment on how this got so complicated, since the complexity itself is informative. Five years ago, a website owner's crawler vocabulary began and ended with Googlebot and Bingbot: two agents, one purpose each, well-documented for decades. The generative-AI wave changed this almost overnight, and in a specific way. Rather than one company shipping one crawler, each AI lab shipped multiple distinct bots for distinct purposes, often within months of each other. "Collect training data," "answer a live search query," and "fetch a page a user explicitly asked about" are genuinely different engineering problems, with different rate limits, different freshness requirements, and, crucially for a site owner, different implications for whether blocking one affects the others.

The result is that "block AI bots" stopped being a single decision. It became, for any site owner who wants to reason about it carefully, a per-bot policy matrix. Most robots.txt files on the web haven't caught up to this reality yet. See the robots.txt AI-blocking census for the planned measurement of exactly how many sites are still applying a single blanket rule where a differentiated one would serve them better.

The full registry

"Documented yes" reflects the operator's own published policy. It is not independently verified compliance — see §9 for what has actually been tested.

Training bots vs. retrieval bots — the distinction that matters

Nearly every mistake in this area comes from treating "AI bot" as one category. It is at least three:

A worked scenario: a mid-size publisher deciding its policy

Abstract categories are easier to apply with a concrete walk-through. Consider a hypothetical mid-size publisher — call it a review site with strong organic traffic but growing concern about AI answer engines paraphrasing its content without sending readers back.

Working through the registry above, a defensible policy for this hypothetical publisher looks like this. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, the three retrieval bots most likely to drive citation and, occasionally, referral traffic. Block GPTBot and ClaudeBot in their training role, since the publisher has decided its archive is worth more as a citation source than as free training data. Leave the agentic on-demand fetchers, ChatGPT-User, Claude-User, Perplexity-User, alone, since they only fire when a specific reader already has the URL.

This is one reasonable configuration among several. A publisher prioritizing licensing revenue over citation reach might instead block everything and pursue direct commercial deals. A publisher indifferent to training-data use might allow all of it. The registry doesn't prescribe one right answer. It makes the trade-off visible enough to choose deliberately.

A second scenario: an e-commerce product catalog

The publisher scenario above optimizes for citation reach. A different business shape reasons through the same registry differently. Consider a hypothetical mid-size e-commerce store weighing AI-driven product discovery against a real concern: its product descriptions and pricing are commercially sensitive, and its margins are thinner than a media publisher's, so training-data use carries a more direct cost if a competitor's model ends up able to reproduce its catalog copy.

A defensible policy here looks different in emphasis, even using the same three-category framework: allow the retrieval bots that can drive AI-assisted shopping discovery (increasingly relevant as agentic commerce grows — see AI shopping and agentic commerce statistics), but block training crawlers more aggressively than the publisher scenario did, and consider blocking PerplexityBot specifically in its training role given its disputed robots.txt compliance noted in §9 — since for this business, the downside of unclear compliance (catalog data used for training despite an intended block) outweighs the upside of Perplexity's citation reach. Product pages behind a login or a "for humans, not bots" pricing negotiation flow are a separate, harder case this registry does not attempt to resolve — that scenario depends on business specifics no general registry can generalize across.

How to verify a crawler is real

A user-agent string is just a header any script can set. Spoofed "GPTBot" traffic scraping content or probing for vulnerabilities is common enough that verification matters before you trust a log line.

robots.txt recipes

Three common intents, and the rule that matches each:

There is no universal "block all AI" directive respected by every operator — each bot must be named individually. A wildcard Disallow: / under User-agent: * blocks everything, including Googlebot and your organic traffic, which is rarely the actual intent.

A decision framework, at a glance

What we actually know about behavior

The registry above states documented policy. Independent, testable evidence of actual behavior is much thinner — this is a genuine research gap, not a settled question.

Both of these are exactly the kind of question a controlled test site can answer cheaply, and both are on the roadmap for the AI Citation Index's crawler-behavior lab — and the JavaScript-rendering question specifically has its own dedicated pre-registered experiment at do AI crawlers render JavaScript?. Until either runs, treat the "honors robots.txt" column as policy, not proof.

What getting this wrong actually costs

It's worth being concrete about the two failure directions, because they have different, asymmetric costs. Blocking a retrieval bot by mistake, say, applying a blanket "AI" rule that catches OAI-SearchBot along with GPTBot, silently removes a site from that engine's citations. There's no error message, no notification, just an absence that could persist for months before anyone notices traffic from that engine never grew.

Allowing a training bot you meant to block costs nothing immediately visible, but forecloses a future option. Once content has been crawled and potentially incorporated into a training run, blocking afterward doesn't retroactively remove it. The first mistake is invisible and reversible. The second is invisible and, for that specific crawl, irreversible. Both are strong arguments for getting the per-bot decision right the first time, rather than treating it as a setting to revisit casually later.

How this compares to other published AI bot lists

Several other publications maintain their own version of this table. Worth being direct about what distinguishes this one, rather than implying uniqueness that doesn't exist. Cloudflare Radar publishes live, aggregate bot-classification data at genuinely large scale, better than anything a single registry like this can offer for traffic-share questions. This page leans on that data directly in §9, rather than duplicating it.

Several SEO tool vendors publish their own bot lists as part of a broader technical-SEO product, updated on whatever cadence their content team maintains. What this specific registry adds is the three-way training/retrieval/agentic taxonomy applied consistently across every row, the explicit distinction between documented policy and independently verified behavior, and a changelog that makes every addition and correction checkable, rather than silently folded into a "last updated" date with no detail on what changed.

None of that makes this the only registry worth reading — cross-referencing against Cloudflare Radar's live data and an operator's own documentation directly is always the more authoritative path when the two disagree. This page's job is to be the fastest correct starting point, not the final word.

What changes as the agentic web matures

This taxonomy will likely need a fourth category within the next year or two. Emerging agent-to-website protocols, covered in more depth in WebMCP and the agent-readable web and MCP servers as a visibility channel, describe a mode of interaction that's neither a bulk training crawl, nor a live search retrieval, nor a simple on-demand page fetch. It's a structured, standardized request for specific data or actions, initiated by an autonomous agent rather than a human explicitly following a link.

As those protocols mature past early Chrome Canary previews into broadly adopted standards, this registry's table will need a fifth column, and possibly a fourth top-level category, to describe them accurately. That update will be logged here when it happens, not retrofitted quietly.

Changelog

Limitations

• Operator documentation can lag actual behavior. A published policy is not a guarantee of current practice.
• New bots ship faster than any registry can track them. This list covers the operators large enough to matter for most sites, not every AI company crawling the web.
• Regional and enterprise-only bots are excluded — this registry is scoped to consumer-facing AI search and assistant products.
• The worked scenario in §5 is illustrative, not a universal recommendation — the right policy genuinely differs by business model, as the scenario itself notes.