AI Crawler Log Analysis: What to Measure

AI crawler log analysis is the practice of isolating requests from AI training and retrieval bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar) inside raw server or CDN access logs, confirming which of those requests are genuinely from the operator they claim to be, and tracking crawl frequency, page coverage, and error rate over time. It is a server-side discipline, not a dashboard metric — the raw log line, with its user-agent string, source IP, timestamp, requested path, and response code, is the only unit of evidence that exists before an AI system decides whether to cite you.

Why this now matters more than it did a year ago

The volume alone justifies the work. Vercel's network-level analysis of AI crawler traffic across Next.js sites found that OpenAI's GPTBot generated 569 million requests in a single month, with Anthropic's Claude crawlers close behind at 370 million, and Applebot and PerplexityBot adding hundreds of millions more — combined, those crawlers accounted for roughly 28% of Googlebot's request volume over the same period (Vercel, "The rise of the AI crawler"). In a companion post, Vercel's engineering team also found that none of the major AI crawlers currently render JavaScript — they fetch and parse raw HTML only, which changes what "coverage" even means for a client-rendered site (Vercel, "The three types of AI bot traffic and how to handle them").

Cloudflare's Radar team, working from network-wide bot classification data rather than a single customer's logs, corroborates the growth trend but adds nuance: across October-November 2025, Googlebot still reached 11.6% of unique pages on the median site, more than three times GPTBot's 3.6% and ClaudeBot's 2.4% (Cloudflare Blog, "From Googlebot to GPTBot: Who's crawling your site in 2025," Dec 2025). The same analysis found that AI platforms crawl far more than they refer traffic back — Cloudflare measured Anthropic's crawl-to-referral ratio as high as 500,000:1 and OpenAI's at up to 3,700:1, versus Perplexity's comparatively low ratio. That give-back gap is precisely why log-level measurement matters: pageview analytics won't show you the crawl side of that ratio, only the log does.

What to filter for

Direct answer: Every major operator ships a distinct, documented user-agent string, and most now separate their training crawler from their live-retrieval (user-triggered) crawler — a distinction that changes how you should read the traffic:

• OpenAI: GPTBot (training), OAI-SearchBot (search indexing), ChatGPT-User (live user fetch) — documented at developers.openai.com/api/docs/bots
• Anthropic: ClaudeBot (training), Claude-SearchBot (search quality), Claude-User (live user fetch) — documented in Anthropic's support article, "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
• Perplexity: PerplexityBot (declared crawler) and Perplexity-User (user-triggered fetch)
• Google: Google-Extended (an AI-training opt-out signal layered on top of standard Googlebot)
• Plus Applebot-Extended, Amazonbot, Meta-ExternalAgent, and Bytespider, all of which show up in general-purpose crawler traffic once you start grepping

Filtering starts as a simple case-insensitive match against the user-agent field for these tokens, either via a log query (most CDN log explorers support this natively) or a command-line pass with grep or an equivalent over raw access logs. That gets you a candidate set. It does not get you a trustworthy set — and that gap is the part most GEO guidance skips.

The verification problem: the user-agent string is not evidence

A string like "ClaudeBot/1.0" in a log line is just text an attacker's script chose to send; it proves nothing about who actually made the request. This isn't a theoretical concern. Imperva's long-running research on Googlebot impersonation found that among sites targeted by fake-Googlebot traffic, roughly 1 in 5 requests claiming to be Googlebot were impostors, with impersonation rates on individual sites running as high as 25% (Imperva, "Was That Really a Google Bot Crawling My Site?"). HUMAN Security's Satori threat intelligence team ran a two-week study specifically on AI crawler user agents and found an average spoof ratio of 1:17 — meaning roughly 1 in every 18 requests claiming to be an AI crawler (GPTBot, ClaudeBot, PerplexityBot, and others) was not actually from that operator, equivalent to more than 2 million spoofed requests per day across their observed traffic (HUMAN Security, "AI Crawler Spoofing: Attackers Impersonate ChatGPT & Perplexity").

The most consequential documented case of the reverse problem — a real AI company obscuring its own crawler rather than a third party impersonating one — is Cloudflare's August 2025 investigation into Perplexity. Cloudflare set up newly registered, undiscoverable domains with robots.txt files disallowing all crawling, then asked Perplexity's own assistant questions about those domains. It found that when Perplexity's declared crawler hit a robots.txt block, a second, undeclared crawler continued fetching the content while presenting a generic Chrome-on-macOS user agent and rotating across IPs and ASNs not listed in Perplexity's public documentation. Cloudflare ran the same test against OpenAI's crawler and found it respected the block. As a result, Cloudflare de-listed Perplexity as a verified bot (Cloudflare Blog, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives," Aug 2025). Perplexity publicly disputed the findings, calling the report a publicity move. The dispute itself doesn't matter for log analysis purposes — the operational lesson does: a robots.txt disallow and a user-agent match are both signals a sophisticated crawler can choose to ignore, and your logs are the only place that discrepancy would ever show up.

How to actually verify a request

Two verification mechanisms exist, and they differ by operator:

IP allowlist matching. OpenAI publishes machine-readable JSON files of its crawlers' IP ranges — separate feeds for GPTBot, OAI-SearchBot, and ChatGPT-User — documented at developers.openai.com/api/docs/bots. Anthropic now publishes an equivalent feed for ClaudeBot, Claude-User, and Claude-SearchBot; its support article states plainly that "if a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic" (support.claude.com/en/articles/8896518) — a meaningful change from Anthropic's earlier position that it did not publish IP ranges at all. Either way, treat these as live feeds to pull on a schedule, not values to hardcode, since ranges change as providers add infrastructure.

Forward-confirmed reverse DNS (FCrDNS). For crawlers without a published IP feed, or as a second check even when one exists, Google's own documentation lays out the canonical method: run a reverse DNS lookup on the source IP, confirm the resulting hostname belongs to the operator's domain (for Google, that's googlebot.com, google.com, or googleusercontent.com), then run a forward DNS lookup on that hostname and confirm it resolves back to the original IP (Google for Developers, "Verify Requests from Google Crawlers and Fetchers"). The forward step exists specifically because a reverse lookup alone can be spoofed by anyone who controls their own PTR record — Google documented this exact technique as far back as a 2006 Search Central blog post, and it remains the reference implementation other operators' guidance echoes.

Log metrics compared

Step-by-step analysis process

1. Export raw access logs with full fidelity. You need, at minimum: timestamp, source IP, requested path, user-agent string, response code, response size, and latency. Sampled or pre-aggregated logs (common in default CDN dashboards) will hide the spoofing signal you're looking for.
2. Filter to candidate AI crawler traffic. Match user-agent strings against the documented tokens for each operator (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and so on). Keep the training and retrieval crawlers separate — they answer different questions.
3. Verify each candidate request. Pull the operator's current IP allowlist (OpenAI's and Anthropic's JSON feeds) and check membership; for operators without a published feed, or as a cross-check, run forward-confirmed reverse DNS on the source IP.
4. Split the dataset into verified and unverified buckets. Report both, but never blend them into a single "AI crawler traffic" number — a spoofed spike will otherwise look like a real one.
5. Compute crawl frequency trends per crawler, per week. Look for step changes, not just totals; a drop that coincides with a robots.txt or infrastructure change on your side is diagnostic, not noise.
6. Compute coverage against your sitemap. Cross-reference the distinct paths each verified crawler has fetched against your full URL list to find sections it's never reaching.
7. Break down error rates by path pattern. A cluster of 403s or 429s on a specific directory usually points to a WAF rule or rate limit inadvertently catching legitimate crawlers alongside bad actors.
8. Join crawl volume against referral analytics. Even an approximate crawl-to-referral ratio, computed monthly, tells you whether a given AI platform is a source of downstream visibility or purely a cost center on your infrastructure.
9. Check the JS-rendering gap on your most important pages. Diff what a crawler actually receives (raw server HTML) against what a user sees (rendered DOM) for your highest-priority URLs, since this is the single most common reason content that looks fine to a human is invisible to these systems.

Limitations — what this doesn't guarantee

Log analysis tells you who crawled you and how successfully — it does not tell you whether an AI system cited you, summarized you accurately, or attributed the answer to your domain at all. Crawl access is a precondition for inclusion, not a proxy for it. It also can't fully close the verification gap: IP allowlists go stale between publication cycles, some operators still don't publish one, and reverse DNS can be defeated by a sufficiently resourced adversary controlling their own infrastructure — HUMAN Security's own numbers show real, unresolved spoofing even against defended traffic. Finally, none of this measures the retrieval-time behavior of AI answer engines that don't crawl at all for a given query, relying instead on a cached index or a partner data source; a page can be well-crawled and still never surface in an answer for reasons log analysis simply can't see.

Where nqzai fits

Direct answer: nqzai's GEO tooling is built to sit on top of exactly this kind of log-derived signal rather than replace it: it correlates verified crawler activity against a site's actual content structure, flags coverage gaps and JavaScript-rendering blind spots before they cost you inclusion, and rolls the crawl-to-outcome picture into the same reporting where SEO and AI-search performance are already tracked together — so the question stops being "did a bot hit this page" and becomes "is this page actually reaching the systems that generate answers."

FAQ

Do I need to filter AI crawler traffic separately from general bot traffic?

Yes. General bot-management tooling typically classifies traffic as "good bot / bad bot" without distinguishing an AI training crawler from a search crawler from a live-retrieval fetch — and those three behave, and matter, differently. GPTBot visits are about training-data inclusion; ChatGPT-User visits happen because a live user asked about your page right now.

Is user-agent filtering alone ever good enough?

For rough trend-watching, sometimes. For any decision that involves blocking, rate-limiting, or reporting a specific number to stakeholders, no — the spoofing rates documented by Imperva and HUMAN Security (roughly 1-in-5 and 1-in-18 requests respectively, depending on the study and crawler) are too high to treat user-agent strings as trustworthy on their own.

Which crawlers currently publish verifiable IP ranges?

OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) and, as of a recent policy update, Anthropic (ClaudeBot, Claude-User, Claude-SearchBot) both publish machine-readable IP feeds. Google publishes IP ranges for Googlebot and recommends the reverse-DNS method as a documented alternative. Not every AI crawler operator currently does either.

Why don't AI crawlers pick up JavaScript-rendered content?

Vercel's direct testing across multiple production sites found that none of the major AI crawlers execute JavaScript — they fetch raw HTML only. Content that depends on client-side rendering to appear in the DOM is effectively invisible to them regardless of how well the page performs for human visitors or traditional search engines.

How often should I re-run this analysis?

Crawl frequency and error rate are worth checking weekly, since a robots.txt change, a WAF rule adjustment, or an operator-side deprioritization can show up within days. IP allowlists should be re-pulled on the same cadence rather than cached indefinitely, since operators update their published ranges as infrastructure changes.