Robots.txt tester for AI crawlers

How do I check if my robots.txt blocks AI crawlers?

Paste your robots.txt above and enter a path. The tester runs the RFC 9309 matching rules for 20 crawlers, including GPTBot, ClaudeBot, PerplexityBot and Googlebot, and shows each one as Allowed or Blocked with the line that decided it. A bot with its own User-agent group ignores the * group, so check each row, not just the summary.

What this robots.txt tester does

Your robots.txt file now decides two different things: whether search engines crawl your site, and whether AI companies can use it, for training future models or for citing you in answers. The robots.txt tester for AI crawlers checks your file against 20 bots at once and shows the verdict for each, the exact line that decided it, and what that bot is for.

Google retired its own robots.txt Tester from Search Console in 2023, and it never covered AI crawlers anyway. This tester applies the matching rules from the robots.txt standard, RFC 9309, which Google follows, so the results match how compliant crawlers read your file.

The bots it checks

The list builds on this site's AI bot user-agent registry, which records each bot's operator, purpose and documentation, and adds three crawlers the registry does not yet cover: CCBot, Bytespider and DuckAssistBot. Categories are simplified to one per bot; the registry notes where a bot has more than one role, for example PerplexityBot and Amazonbot.

How robots.txt matching works

1. Pick the group. A crawler looks for a User-agent line matching its name (case-insensitive). If several match, the most specific wins. If none match, it uses User-agent: *. If there's no * group either, everything is allowed.
2. Find matching rules. In that group, every Allow and Disallow whose path matches the start of the URL is a candidate. * matches any characters; $ anchors the end.
3. Longest wins. The rule with the longest path wins. On a tie, Allow wins.
4. No match means allowed. An empty Disallow: allows everything.

A common trap: a bot only reads its own group. If you write User-agent: GPTBot with some rules, GPTBot ignores everything under User-agent: *, including your blocks on admin or private folders. The tester shows the group each bot actually used.

Training vs. retrieval: what to block

Many sites block AI bots wholesale and lose citations they wanted. Blocking a training crawler stops future crawls for model training; it doesn't remove what was already collected, and it doesn't take you out of AI search. Blocking a retrieval bot does: OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to cite them in answers. The "Block AI training" template reflects that split. This site's robots.txt AI-blocking census looks at how sites actually configure this.

Worked example: the default template

The box loads with the "Block AI training" template. With the path set to /, the seven training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider) show as Blocked by the Disallow: / line of their shared group. The other 13, including OAI-SearchBot and Googlebot, fall back to User-agent: * and show as Allowed, because /admin/ doesn't match /.

Change the path to /admin/settings and every bot is blocked, but for two different reasons: the training bots by their own group, the rest by the * group's Disallow: /admin/ (marked "via *"). The warning under the table points out the trap: if the training group had only Disallow: /private/, those bots would be free to crawl /admin/, because they never read the * rules.

What the tester doesn't do

• It doesn't fetch your live file. Browsers can't read another site's robots.txt from this page, so paste it in.
• It checks rules, not behaviour. Whether a bot actually respects the file only shows up in your server logs; the AI crawler log analyzer reads those.
• It matches exact product tokens. A custom user agent you type is checked as a whole token, the way RFC 9309 compliant crawlers match it.
• It ignores Crawl-delay and Sitemap lines for the verdict, and only warns about unsupported directives such as Noindex:.

Common robots.txt mistakes

• Blocking Googlebot to "stop AI Overviews". That removes you from Google Search entirely. Google-Extended is the AI-training control, and it doesn't affect AI Overviews.
• A bot group without the shared rules. A named group replaces the * group for that bot.
• Using robots.txt to hide pages. Blocked URLs can still be indexed from links. Use noindex.
• Blocking CSS and JavaScript. Google needs them to render the page.
• Typos in bot names. "GPT-Bot" or "Claude-Bot" match nothing; the tester flags user-agent lines that don't match a known bot.

Related tools

Robots.txt controls crawling. To give AI systems a summary of what to read, see the llms.txt generator and llms.txt validator. For what crawlers do once they arrive, read do AI crawlers render JavaScript?

Privacy

The file is parsed entirely in your browser. Nothing you paste is uploaded.

Last checked and updated: 1 November 2026. Everything on this page runs in your browser; nothing you enter is sent anywhere.