AI crawler log analyzer

Short answer: to see whether AI bots crawl your site, load a raw server access log into the analyzer above. It matches each request's user agent against known crawlers such as GPTBot, ClaudeBot and PerplexityBot and shows what they fetched. Google Analytics can't show this, because these bots don't run your analytics script.

Troubleshooting

• "This browser cannot decompress .gz files." Your browser lacks the decompression API. Unzip the log first, or use a current Chrome, Edge, Firefox or Safari.
• "This looks like Common Log Format…" That format has no user-agent or referrer field, so no bot can be identified. Switch the server to the combined format and export again.
• "Could not read this file" The file is damaged or isn't a text log. Open it in a text editor to check.

Why AI crawlers only show up in server logs

Analytics tools count visitors that run JavaScript. AI crawlers mostly do not: they request the HTML and leave. So GPTBot, ClaudeBot and PerplexityBot are invisible in Google Analytics, and the only complete record of what they fetched is your server access log. This AI crawler log analyzer reads that log in your browser and answers three questions: which AI bots came, what they took, and how many human visits the same platforms sent back.

Most tools that do this are paid SaaS dashboards, Python packages or CLI scripts. This one runs in the page with no install, and because your log contains visitor IP addresses, it never uploads the file.

How bots are classified

Each request's user-agent string is matched against the crawlers in this site's AI bot user-agent registry, plus a few the registry does not yet list (CCBot, Bytespider, DuckAssistBot, Meta-ExternalFetcher and the older anthropic-ai token). The robots.txt tester uses nearly the same list. The grouping follows what each operator documents the bot is for:

Two names never appear in logs. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: Google and Apple crawl with their usual bots and read the token to decide whether content may be used for model training. Google states that "Google-Extended doesn't have a separate HTTP request user agent string" in its crawler documentation. If you blocked Google-Extended, you will still see Googlebot.

The spoofing problem, stated plainly

A user agent is a claim. Scrapers routinely pretend to be Googlebot or GPTBot because some sites wave those through. Real verification needs the request IP:

• Google documents a reverse DNS lookup (the name must end in googlebot.com, google.com or googleusercontent.com) followed by a forward lookup that returns the same IP, plus published IP range files. See Verifying Googlebot and other Google crawlers.
• OpenAI publishes IP lists per bot, for example gptbot.json and searchbot.json, listed on its crawler overview.

A web page cannot run DNS lookups or keep those lists current, so every count here is of claimed identity. Treat a sudden spike from one "bot" across many unrelated IPs as suspect, and verify a sample on your server before quoting the numbers anywhere.

AI referrals and the crawl-to-referral ratio

Crawling is the cost side. The return is people clicking through from an AI answer. The analyzer counts requests from non-bot user agents whose Referer is chatgpt.com (or chat.openai.com), perplexity.ai, claude.ai, gemini.google.com or copilot.microsoft.com, plus requests whose URL carries utm_source=chatgpt.com and similar tags. It then divides all AI bot hits by those visits.

Read the ratio with care. It describes your log, nothing more. It is not a benchmark, there is no agreed healthy value, and a crawl in March does not "cause" a click in April. Undercounting is built in: apps and some browsers strip the referrer, so a share of AI clicks arrive as direct traffic. Training-bot hits, by design, never send anyone back. The per-type split matters more than the headline: compare retrieval and on-demand hits, which are tied to answers people see, with referrals. This site's AI referral traffic statistics and crawl-to-citation latency study go further into what can and cannot be read from these numbers.

Supported log formats

• Combined Log Format (Nginx and Apache default): IP, time, request, status, bytes, referrer, user agent. Referral counting needs this format or JSON.
• Common Log Format: no referrer or user agent, so bots cannot be identified. The tool tells you if this is all it finds.
• JSON lines: one object per line. Cloudflare Logpush HTTP request fields (ClientRequestUserAgent, ClientRequestURI, EdgeResponseStatus, EdgeStartTimestamp, ClientRequestReferer) are read directly, as are common names like user_agent, path, status, time and referer.
• Gzip: .gz files are decompressed in the browser where it supports the DecompressionStream API.

What to do with the results

• 4xx or 5xx for retrieval bots. If OAI-SearchBot or PerplexityBot are getting 403s, a firewall or CDN bot rule is blocking the crawlers that cite you. Check it against your robots.txt.
• Did anything fetch /llms.txt? Click each bot and look at its top URLs. Then check the file with the llms.txt validator.
• Heavy training crawls, no referrals. That is the case for blocking training bots while keeping retrieval bots, the split explained in GPTBot vs OAI-SearchBot.

Limitations

Counts are of requests that claim a bot identity. Bots that use a browser user agent, as some agentic browsers do, are counted as humans. A log from one server behind a load balancer or CDN may miss requests served from cache. Very large files are limited by your device's memory for the per-URL tables; the tool keeps the top URLs per bot rather than every path.

Last checked and updated: 1 November 2026. Everything on this page runs in your browser; nothing you enter is sent anywhere.