TL;DR
Cloudflare data shows GPTBot requests jumped 305% year-over-year through mid-2025, while PerplexityBot exploded 157,490% off a tiny base. OpenAI runs three independently controllable crawlers—GPTBot, OAI-SearchBot, and ChatGPT-User—so blocking only one leaves the others untouched. Anthropic warns that IP-based blocking is unreliable because it can prevent the crawler from ever reading robots.txt. Among top news sites, 34.2% now disallow GPTBot, rising to 55% among outlets with the strongest fact-checking records.
The verdict: diagnose which specific crawler and which stack layer—robots.txt, WAF, rate limiting, or CDN—is causing the block, because default CDN-level blocking is becoming common and a blanket approach will miss the real failure point.
An AI crawler block is any server-side condition that prevents an AI company's automated fetcher — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and similar agents — from successfully retrieving a page, even though the page is publicly reachable in a browser. The failure can happen at four distinct layers of the stack (robots.txt, web application firewall rules, rate limiting, or CDN-level bot management), and each layer produces a different signature in server logs. Diagnosing an AI crawler block means identifying which layer is responsible, not just confirming that a block exists.
This distinction matters more than it did two years ago. Cloudflare's mid-2025 analysis of crawler traffic across its network found that combined AI and search crawler traffic grew 18% from May 2024 to May 2025, with GPTBot's share of crawler requests jumping from 2.2% to 7.7% and raw GPTBot requests up 305% year-over-year — while ClaudeBot's share fell from 11.7% to 5.4% over the same window, and PerplexityBot recorded a 157,490% increase in raw requests off a tiny base (Cloudflare, "From Googlebot to GPTBot: Who's crawling your site in 2025," July 1, 2025). Cloudflare's subsequent 2025 Radar Year in Review confirmed the trend held for the full year and added a pointed detail: AI crawlers were the most frequently fully disallowed user agents found across the robots.txt files Cloudflare's network observed, with GPTBot, ClaudeBot, and CCBot leading that list (Cloudflare Radar, "2025 Year in Review," December 2025). In other words, blocking is now the norm for a meaningful slice of the web, and it is getting more common, not less.
Who's actually knocking on the door
Direct answer: Each AI company runs more than one crawler, and they behave differently under a block:
- OpenAI operates three separate, independently controllable user agents: GPTBot (fetches pages as training-data candidates), OAI-SearchBot (builds the index behind ChatGPT's search feature), and ChatGPT-User (fires when a person asks ChatGPT to open a specific URL). OpenAI publishes machine-readable IP ranges for each at dedicated JSON endpoints and says robots.txt changes can take about 24 hours to take effect in its systems (OpenAI, "Overview of OpenAI Crawlers").
- Anthropic runs ClaudeBot (training-data collection), Claude-User (fetches pages in response to a live Claude query), and Claude-SearchBot (search indexing). Anthropic's own guidance is explicit that blocking by IP address alone is unreliable because it can prevent the crawler from ever reading robots.txt in the first place — the supported method is a robots.txt Disallow rule, verified against Anthropic's published IP list (Anthropic / Claude Privacy Center, "Does Anthropic crawl data from the web, and how can site owners block the crawler?").
- Perplexity runs PerplexityBot for general indexing. Perplexity's help center states the crawler will not index the full or partial text of a page disallowed via robots.txt, though the domain and a brief factual summary may still surface (Perplexity Help Center, "How does Perplexity follow robots.txt?"); the technical specification for the crawler, including its user-agent string, lives in Perplexity's developer docs (Perplexity, "Perplexity Crawlers").
The practical implication: a block that catches "GPTBot" in a WAF rule does nothing to OAI-SearchBot or ChatGPT-User, and vice versa. Diagnosis has to be crawler-specific, not company-specific.
Why blocking has become the default posture
Two independent lines of evidence show this isn't anecdotal. A CommonCrawl-based longitudinal study submitted to arXiv in October 2025 reconstructed robots.txt history back to January 2023 across the top one million sites and found that GPTBot disallowance rose sharply after OpenAI's August 2023 announcement, with roughly a quarter of the top 1,000 sites and about 10% of the top million now restricting AI crawlers — and among news outlets specifically, 34.2% disallow GPTBot, rising to 55% among outlets with the strongest fact-checking track records (Bouchaud & Ramaciotti, "Web Crawler Restrictions, AI Training Datasets & Political Biases," arXiv:2510.09031, submitted October 10, 2025). Separately, Cloudflare's own reporting shows Anthropic's crawl-to-refer ratio — pages crawled per visitor actually sent back to the site — running as high as 500,000:1 for stretches of 2025, an imbalance publishers increasingly cite as their reason for blocking training crawlers even while leaving search-oriented bots alone (Cloudflare Radar, "2025 Year in Review"). Cloudflare has responded by shipping default blocking: starting in the second half of 2026, ad-monetized pages on Cloudflare get training and agent bots blocked by default while search bots remain allowed, configurable per category (Cloudflare Blog, "Your site, your rules: new AI traffic options for all customers"). That default is exactly why "it's probably robots.txt" is no longer a safe assumption — a growing share of blocks now originate at the CDN/WAF layer, sometimes without the site owner touching robots.txt at all.
Four causes, compared
| Cause | What it looks like at the origin | How to detect it |
|---|---|---|
| robots.txt disallow | Compliant crawlers (GPTBot, ClaudeBot, PerplexityBot) simply never request the page — no log entry from that user agent at all. Non-compliant or user-triggered agents (e.g., ChatGPT-User) may still fetch it. | Fetch /robots.txt directly and check for a User-agent block matching the crawler with Disallow: / or a path-specific rule. Test against the specific crawler's documented user-agent string, not just "AI bots" generally. |
| WAF / firewall rule | A crawler request reaches the edge and gets a 403 (or similar) response, often instantly, sometimes without appearing in standard security-event logs if it's a platform-managed rule. | Check WAF/managed-rule logs for the exact user-agent or ASN; look for rules named around "AI bots," "AI Scrapers and Crawlers," or "Block AI bots." Cloudflare's own community forum has documented cases where a managed AI-crawler-category rule produces an "invisible" 403 not visible in standard security event logs even when dashboard toggles show allow (Cloudflare Community, bot-management category). |
| Rate limiting | The first N requests in a window succeed; subsequent ones from the same IP/ASN get throttled or blocked (429 or 403), often mid-crawl. Effect is intermittent, not constant. | Look for a sudden run of 429/403 responses clustered by source IP after an initial burst of 200s. Cloudflare notes rate-limiting counters have up to a few seconds of lag, so a handful of requests can slip through before enforcement kicks in — a signature distinct from a hard block (Cloudflare WAF docs, "Rate limiting best practices"). |
| CDN bot management | Requests get challenged (CAPTCHA/JS challenge the bot can't solve), silently dropped, or served a degraded/empty response — distinct from a clean 403, and often driven by a bot-score threshold rather than a named rule. | Check the CDN's bot-management or "verified bots" dashboard for a classification (search / agent / training) and its allow/block state per category; test whether the crawler's declared IP range is being challenged despite passing user-agent matching. |
Because these layers stack — a request can pass robots.txt, get past the WAF, and still be dropped by rate limiting — a single test at one layer is not sufficient to clear a site. Cloudflare's own AI Crawl Control documentation notes that its WAF-based crawler rules are evaluated before pay-per-crawl logic, so a rule blocking traffic from a country or ASN can override an otherwise-correct AI-crawler allow rule (Cloudflare, "AI Crawl Control with WAF").
Step-by-step diagnostic process
- Confirm the crawler's real identity first. Pull the exact user-agent string and current IP range from the source's own documentation (OpenAI's
/gptbot.json,/searchbot.json,/chatgpt-user.json; Anthropic'sclaude.com/crawling/bots.json; Perplexity's developer docs) rather than a third-party list, since strings and ranges change over version bumps. - Fetch robots.txt as that exact user agent (not a browser UA) from an IP outside your own network, and check for a matching
User-agentblock and anyDisallowrules — including on every relevant subdomain, since robots.txt does not inherit across subdomains. - Check the raw server/CDN access logs for the crawler's user-agent string and declared IP ranges. No log entries at all, from a fully compliant crawler, usually confirms a robots.txt-level block (the crawler self-excluded). Log entries followed by non-200 responses point to a layer downstream of robots.txt.
- If entries exist, read the response codes and timing pattern. A consistent 403 on essentially every request suggests a WAF/firewall or CDN bot-management rule. A pattern of initial 200s followed by 429/403 after a burst suggests rate limiting. A CAPTCHA/challenge page in the response body (which a bot cannot solve) suggests bot management with a challenge action rather than a hard block.
- Cross-check against the CDN/WAF dashboard directly. Search security-event logs by the crawler's ASN or IP range, not just user-agent string, since spoofed user-agents are common and some managed rules key on IP reputation or ASN rather than the header. If the platform ships a named "AI bots" or "AI Crawl Control" category, check its allow/block state for each of the three behavior classes (search, agent, training) separately — blocking "training" does not mean the site also blocked "search."
- Test from a second network/IP if possible to rule out a narrower cause (e.g., a geo-block or ASN-specific rule) rather than a blanket AI-crawler rule.
- Check for conflicting layers. If the CDN's dashboard shows AI crawlers "allowed," but logs still show blocks, look for a lower-priority custom WAF rule, an IP-reputation score threshold, or a country/ASN rule that executes before the AI-crawler allow rule — these can override an explicit allow, as Cloudflare's own documentation describes for its rule evaluation order.
- Verify against a spoof-resistant signal where available. Because user-agent strings are trivially spoofable, prefer IP-range matching or, where supported, cryptographic bot-authentication signals (Cloudflare has begun implementing Web Bot Auth, an IETF draft where compliant crawlers sign requests with an Ed25519 key) over user-agent string matching alone when confirming which requests are genuinely from the AI company versus an impersonator.
- Re-test after any config change and allow for propagation delay. OpenAI states robots.txt changes take roughly 24 hours to reflect in its search crawling systems; CDN/WAF rule changes are typically near-instant but should still be re-verified with a live fetch, not just a dashboard toggle check.
What this doesn't guarantee
Clearing every layer above does not guarantee inclusion, citation, or even a subsequent crawl. A crawler that is technically "allowed" may still choose not to visit a given page based on internal prioritization, may crawl but not use the content for the purpose you expect (e.g., OAI-SearchBot indexing without GPTBot training), or may crawl once and not return for months. robots.txt is also a voluntary honor system — nothing in the protocol prevents a non-compliant scraper from ignoring it entirely, and Perplexity's own history includes a documented dispute over whether user-initiated fetches must honor robots.txt at all. IP-range verification is only as current as the crawler's published list; ranges do change. And because CDN-level "AI bot" categories are usually maintained and updated by the CDN vendor rather than the site owner, a rule that worked last month can silently start or stop matching a given crawler after a vendor-side classification update, with no change on the site's end. Diagnosis tells you what is currently blocking a request — it does not certify that the page will be crawled, indexed, or cited going forward.
Where nqzai fits
nqzai's platform includes AI-visibility monitoring built to sit on top of exactly this problem: it tracks whether a site's pages are actually being surfaced and cited across AI answer engines, and it flags when technical access issues — not content quality — are the likely cause of an absence. Rather than asking a site owner to manually reconcile robots.txt, WAF logs, and CDN dashboards across three different vendor consoles, nqzai's diagnostics correlate crawler-access signals with citation and mention tracking, so a drop in AI visibility can be traced back to an access failure at the right layer instead of triggering a content rewrite that wouldn't have fixed the actual problem.
FAQ
Direct answer: Does blocking GPTBot in robots.txt also block ChatGPT from citing my site in a user's conversation?
No. GPTBot governs training-data collection only. ChatGPT-User (user-initiated fetches) and OAI-SearchBot (search indexing) are separate, independently controlled crawlers, and OpenAI's own documentation notes that robots.txt rules may not even apply to ChatGPT-User's on-demand fetches since they're triggered by a specific person's request.
If my CDN shows AI bots as "allowed," why are requests still failing?
Rule evaluation order matters. A broader WAF rule (country block, ASN reputation, generic rate limit) can execute before or independently of an AI-crawler-specific allow rule and override it. Check for other active rules, not just the AI-bot toggle.
Is a 403 from an AI crawler always intentional blocking?
Not necessarily. Bot-management systems sometimes issue a challenge (CAPTCHA/JS) that a legitimate crawler cannot complete, producing a failure that looks like a block but is really a false positive from risk scoring rather than a deliberate rule targeting that crawler.
Can I trust a request's user-agent string to identify the crawler?
Only in combination with IP verification. User-agent strings are client-controlled and can be spoofed by scrapers pretending to be GPTBot or ClaudeBot; cross-check against the company's published IP ranges (or, where supported, cryptographic request signing) before concluding a specific crawler was blocked or allowed.
How long after fixing robots.txt should I expect a crawler to return?
It varies by crawler and purpose. OpenAI states its search-indexing systems typically pick up robots.txt changes within about 24 hours; there's no published SLA for when a training crawler will next visit a previously blocked page, since crawl scheduling depends on the company's own prioritization, not the site's timeline.
