TL;DR
Only 37% of the top 10,000 domains even have a robots.txt file, meaning most sites have no stated preference at all. OpenAI alone operates three independently controllable crawlers—GPTBot, OAI-SearchBot, and ChatGPT-User—so a single "block AI" directive will miss at least one. Cloudflare's bot verification checks IP, ASN, reverse DNS, and behavioral signatures because user-agent strings are trivial to spoof.
The article's verdict: an accurate audit must reconcile all three layers—live robots.txt, active WAF rules, and raw server logs—because a gap between any two is the actual finding, not what you intended.
An AI crawler access audit is the process of reconciling three independent data sources — a site's robots.txt directives, its firewall or WAF bot-management rules, and its raw server log evidence — to determine which AI crawlers can actually reach a site's content, as opposed to which crawlers a site owner merely intended to allow or block. The word "actually" is doing the work in that definition. Most sites only ever check one of the three layers, usually robots.txt, and treat the result as fact. It isn't. It's a stated preference, and preferences and outcomes diverge more often than most teams assume.
Why one layer isn't enough
Direct answer: Robots.txt is a request, not a lock. RFC 9309, the IETF standard that formalized the Robots Exclusion Protocol in 2022, is explicit about this: the rules it defines "are not a form of access authorization." A compliant crawler reads the file and honors it voluntarily. A crawler that ignores it, spoofs its user-agent, or was never told about the standard in the first place will fetch the page anyway, and robots.txt has no way to know.
That gap is not theoretical. Cloudflare, which sits in front of a large share of the web's traffic, reported in mid-2025 that only about 37% of the top 10,000 domains had a robots.txt file at all, and used that finding to justify shipping a managed robots.txt product for customers who had never configured one (Cloudflare Blog, July 1, 2025). If more than 6 in 10 of the largest sites on the internet have no stated preference, "check robots.txt" answers a much smaller question than most audits assume.
The inverse problem is just as common: a site has a robots.txt file that looks airtight, but the firewall in front of it either isn't enforcing anything equivalent, or is enforcing something different — blocking by user-agent string while a crawler rotates through residential IPs, or allowing a "verified" bot that turns out to be forging its own identification. Cloudflare's own bot-verification system checks IP address, ASN, reverse DNS, and behavioral signatures precisely because user-agent strings alone are trivial to spoof — a distinction laid out in its verified-bots documentation (Cloudflare, Verified bots). If your WAF is only pattern-matching on the "ClaudeBot" or "GPTBot" string, it is checking the one signal every impersonator also controls.
And even a robots.txt file plus a correctly configured WAF only tells you about rules. It tells you nothing about what already happened. That's what server logs are for, and they're the layer most audits skip because they're the most tedious to read.
The three layers, compared
| Layer | What it tells you | What it can't tell you | How to check it |
|---|---|---|---|
| robots.txt | Which crawlers you've told to stay out, and from where | Whether any crawler actually obeyed the file, or whether the file is even being served correctly (wrong content-type, redirect loops, 5xx errors all silently break it) | Fetch it directly; Cloudflare's dashboard now has a dedicated compliance tab for this (Cloudflare changelog, Oct 21, 2025) |
| Firewall / WAF / CDN bot management | What you're actively blocking or challenging at the edge, and by what signal (IP range, ASN, behavioral score, user-agent) | Whether the rule you think is active is the rule that's actually deployed, and whether a spoofed request slipped past a weak signal | Review the live rule set and verified-bot allowlist in your WAF or CDN dashboard, not the documentation of what you meant to configure |
| Server log evidence | What actually connected to your origin — user-agent, IP, path, status code, timestamp, request volume | Intent — a hit in the logs doesn't tell you whether the bot respected robots.txt on the way in, only that a request landed | Reverse-DNS or ASN-verify sampled IPs against each crawler's published ranges, the same method Google documents for confirming Googlebot (Google for Developers) |
None of the three rows is sufficient alone. Together, they let you answer three different questions: what did we ask for, what are we enforcing, and what actually happened. A gap between any two of them is the finding.
A numbered audit process
- Pull the live robots.txt, not the one in your repo. Fetch it from the production URL with a plain HTTP client, not a browser (which may apply caching or redirects you won't see live). Confirm it serves as
text/plain, returns a 200, and isn't silently redirecting to a login page or CDN error page for bot-flagged user-agents specifically — a WAF rule that "protects" the site by challenging the crawler fetching robots.txt itself is a real, self-defeating misconfiguration.
- Enumerate every AI-relevant user-agent by name and by purpose. Don't audit "AI crawlers" as one bucket. OpenAI alone operates three separately controllable agents — GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for live, user-triggered fetches — and blocking one has no effect on the others (OpenAI, Overview of OpenAI Crawlers). Anthropic runs the same three-way split: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-initiated retrieval, each independently addressable in robots.txt and each carrying different visibility trade-offs if disallowed (Claude Help Center). An audit that treats "block AI" as one directive will misconfigure at least one of these bots by accident. For the complete set of per-crawler robots.txt rules — including the newer Content Signals extension — and a recurring process for keeping them current as vendors add or rename crawlers, see our robots.txt governance guide for AI crawlers.
- Enumerate every AI-relevant user-agent by name and by purpose. Don't audit "AI crawlers" as one bucket. OpenAI alone operates three separately controllable agents — GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for live, user-triggered fetches — and blocking one has no effect on the others (OpenAI, Overview of OpenAI Crawlers). Anthropic runs the same three-way split: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-initiated retrieval, each independently addressable in robots.txt and each carrying different visibility trade-offs if disallowed (Claude Help Center). An audit that treats "block AI" as one directive will misconfigure at least one of these bots by accident.
- Check for Content Signals, not just Allow/Disallow. As of September 2025, Cloudflare's Content Signals Policy lets a robots.txt file express how content may be used after a crawler accesses it — search, AI input, or AI training — as separate, layered preferences rather than a blanket block (Cloudflare Blog, Sept 24, 2025). A site can now legitimately allow indexing while declining training use, and an audit that only checks for a bare Disallow line will misread that nuance as "wide open."
- Pull the live WAF/CDN rule set and diff it against what you believe is configured. Export the actual active rules — not a screenshot from a planning doc, not last quarter's change ticket — and check specifically for AI-bot-category rules, IP-range allowlists tied to verified bots, and any generic "block automated traffic" rule that might catch or miss AI crawlers as a side effect of unrelated bot-fraud protection.
- Confirm your verified-bot allowlist matches the crawler's currently published ranges. Crawler IP ranges change. A rule built against a stale IP list will either fail to block a rotated range or fail to allow a legitimate crawler that moved.
- Pull 30-90 days of raw origin logs and isolate every request whose user-agent string claims to be an AI crawler. Do this at the origin, behind the CDN cache, where possible — cached responses served entirely by the edge may never generate an origin log line, which will undercount real crawl activity if your audit only looks at CDN-layer logs.
- Sample-verify the claimed identity of those requests. Take a statistically meaningful sample of IPs per claimed crawler and run the same forward-confirmed reverse-DNS check Google documents for Googlebot: reverse-lookup the IP, confirm the hostname resolves to the expected domain, then forward-resolve that hostname back to the same IP (Google for Developers, Verify Requests from Google Crawlers and Fetchers). Anything that fails this check and still claims to be GPTBot, ClaudeBot, or Googlebot is either spoofed traffic or a misconfigured legitimate tool — either way, it's a finding, not noise.
- Reconcile the three data sets and name every disagreement explicitly. A crawler disallowed in robots.txt but present in verified origin logs is a compliance gap. A crawler allowed in robots.txt but absent from logs entirely may mean it isn't crawling the site yet, or that a WAF rule is silently blocking it upstream of where you're logging. A crawler blocked at the WAF but still appearing at the origin means the WAF rule isn't actually wired to that path or environment.
- Re-run the audit on a cadence, not once. Crawler behavior shifts fast: Cloudflare's own traffic data showed AI crawling volume up 32% year-over-year in April 2025, cooling to just 4% growth by July of the same year, with individual crawlers' market share moving by several points in either direction over a matter of months (Cloudflare Blog, "The crawl-to-click gap"). An audit performed once and filed away is describing a snapshot of a system that keeps moving.
What this doesn't guarantee
Direct answer: Even a clean three-layer audit has real limits, and naming them matters more than pretending the process is airtight.
It doesn't guarantee compliance. RFC 9309 is explicit that Robots Exclusion is advisory. A crawler operator can read your Disallow line and ignore it; your audit will surface that only after the fact, in logs, not before it happens.
It doesn't guarantee you've found every crawler. New AI products launch new user-agents regularly, and an audit built against today's known-crawler list will miss a launch from next month until someone updates the list. Independent trackers like Paul Calvano's ongoing analysis of AI-bot robots.txt adoption are useful precisely because they track this churn over time rather than as a single snapshot (Paul Calvano, "AI Bots and Robots.txt," Aug 2025).
It doesn't tell you what happens after a crawler successfully fetches your content. Whether that content ends up in a training set, a live answer, or a citation is outside what robots.txt, a WAF, or your own server logs can observe — the audit stops at the edge of your own infrastructure.
And it doesn't resolve inconsistent behavior across a crawler's own sub-agents. Academic research examining robots.txt adoption across news publishers found blocking rules varying widely even within a single operator's family of bots and found substantial differences in how reputable versus low-quality sites were treating the same crawlers — evidence that "AI crawler policy" is not applied uniformly even by the crawler operators themselves (arXiv, "Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web," Oct 2025).
Where nqzai fits
nqzai's AEO/GEO tooling is built to sit on top of this kind of audit rather than replace the judgment it requires. It surfaces which AI crawlers and search-answer systems are showing up as sources of traffic or citations for a given domain, and pairs that with the site's visible technical posture — the kind of reconciliation described above between what a site permits and what's actually reaching it — so that a content or SEO team can see the gap between "we allowed this crawler" and "this crawler is generating measurable outcomes" in one place, without hand-assembling logs, WAF exports, and robots.txt history themselves.
FAQ
Does blocking GPTBot in robots.txt stop ChatGPT from citing my site?
No. GPTBot governs training data collection only. ChatGPT's live, user-triggered fetches use a separate agent, and OpenAI's own documentation treats the two as independently controllable settings, so blocking one has no effect on the other (OpenAI, Overview of OpenAI Crawlers).
If my WAF shows zero blocked AI crawler requests, does that mean none are trying to access my site?
Not necessarily. It could mean no crawler has attempted access, or it could mean your WAF's detection rule for that crawler isn't matching real traffic — for instance, because it's keyed to a user-agent string a crawler no longer uses, or an IP range that changed. Cross-checking against raw origin logs is the only way to tell the difference.
Is a robots.txt Disallow line legally enforceable?
No. RFC 9309 states plainly that the rules "are not a form of access authorization" — it's a voluntary protocol, not an access control mechanism (RFC 9309). Enforcement, to the extent it exists, happens at the WAF/firewall layer or through legal terms of service, not through the file itself.
How often should I re-run this kind of audit?
Quarterly at minimum, given how fast crawler market share and behavior have shifted recently — Cloudflare recorded a swing from 32% year-over-year crawl growth to 4% in the space of three months during 2025 (Cloudflare Blog). Sites making active robots.txt or WAF changes should check sooner, immediately after the change ships.
Can I trust the user-agent string alone to identify a crawler in my logs?
No. User-agent headers are client-controlled and trivially spoofed. Reliable identification requires IP-range or forward-confirmed reverse-DNS verification against the crawler operator's published ranges, the same method Google documents for verifying Googlebot (Google for Developers).
