TL;DR
GPTBot logged 569 million fetches in a month, Claude 370 million, yet neither executes JavaScript — they download the file and move on. Over a third of AI crawler requests hit 404s, suggesting they follow stale links rather than curated queues. Googlebot is the exception: it runs a Web Rendering Service but may skip resources it judges non-essential.
The core failure is parsing: if content isn't in the raw HTML, a non-rendering crawler cannot see it, regardless of robots.txt. The fix: server-render or statically generate all primary content, avoid hiding text behind CSS or interaction, and write substantive alt text for images.
A page can return a 200 status, load perfectly in a browser, and still be invisible to the systems that power AI answers. That's because most AI crawlers don't render a page the way a browser does — they fetch raw HTML and read what's there. Any resource that determines what ends up in that raw response — a CSS rule that hides text, a script that injects content after load, a lazy-loading pattern that waits for a scroll event that never comes — can silently remove content from what the crawler actually sees, even though nothing about the request ever failed.
This is a narrower problem than "SEO," and it's worth being precise about it. Retrieval failure isn't a ranking penalty or a crawl-budget issue. It's a parsing failure: the crawler got a response, but the response didn't contain the content a human visitor would see.
What we actually know about how AI crawlers fetch pages
Direct answer: The clearest public data on this comes from Vercel's analysis of crawler traffic across its network, published in its report "The rise of the AI crawler". Looking at a month of traffic, Vercel found GPTBot generating 569 million fetches and Claude's crawler 370 million, against Googlebot's 4.5 billion — together, GPTBot, Claude, AppleBot, and PerplexityBot accounted for roughly 28% of Googlebot's volume. Search Engine Journal covered the finding with the same figure.
The more consequential finding in the same report: none of the major AI crawlers execute JavaScript. ChatGPT's crawler fetches JavaScript files on about 11.5% of requests and Claude's on about 23.84%, but the analysis found no evidence either one runs the code — they download the file and move on. ChatGPT's crawler spent 57.7% of its fetches on HTML; Claude spent over a third of its requests on images. Both crawlers also hit 404s on more than a third of their fetches (34.82% for ChatGPT, 34.16% for Claude), a much higher error rate than Googlebot's, which suggests these crawlers are following stale or generated links rather than a curated crawl queue.
Google is the exception, not the rule, and it's worth citing its own guidance directly rather than assuming AI crawlers behave the same way. Google's Fix Search-Related JavaScript Problems documentation confirms that its Web Rendering Service does execute JavaScript, but notes that Googlebot "continuously analyze[s] and identify[s] resources that don't contribute to essential page content and may not fetch such resources" — meaning even a renderer that runs JS can decide not to bother loading something it judges non-essential. That single sentence is a good summary of the whole risk category this article covers: a resource being technically reachable doesn't mean a crawler will use it.
OpenAI's own documentation, Overview of OpenAI Crawlers, confirms there are three separate agents — GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for live user requests — each independently controllable through robots.txt, and each with different fetch behavior. Cloudflare's analysis of AI bot traffic adds another layer: robots.txt compliance itself is inconsistent across vendors, so a rule that blocks a resource for one crawler may be ignored by another. None of this changes the core fact — if the content isn't in the raw HTML response, a non-rendering crawler will not find it, regardless of what your robots.txt says.
How each resource type actually fails
| Resource | What happens when blocked/mismanaged | Who's affected | Typical fix |
|---|---|---|---|
| CSS | Content usually stays in the DOM even if visually hidden, but layout-dependent techniques like display:none, off-screen positioning, or accordion/tab panels that never expand can strip real text from what a non-rendering crawler treats as visible. Google has said blocking CSS entirely in robots.txt prevents it from understanding how a page looks, per reporting on John Mueller's comments and Search Engine Land's coverage of Search Console's blocked-resource severity warnings. | Rendering crawlers (Googlebot) more than raw-HTML crawlers, but hidden-content patterns hurt both | Keep CSS crawlable; don't hide primary content behind interaction-only disclosure |
| JavaScript | Content injected client-side (React/Vue hydration, "load more" buttons, client-rendered product data) never reaches the raw HTML response that non-rendering AI crawlers read. Vercel's data shows ChatGPT and Claude fetch JS files without executing them. | Non-rendering AI crawlers (GPTBot, ClaudeBot, PerplexityBot) severely; Googlebot much less | Server-render or statically generate the primary content; treat JS-only content as invisible to most AI systems |
| Web fonts | Text characters remain in the DOM regardless of whether the font file loads, so plain body copy usually survives. The real failure is icon fonts and ligature fonts that map real words or meaningless code points to glyphs — Font Awesome's own writeup on the problem describes how a single letter like "L" carries no meaning to a machine reading the raw character | Any system extracting semantic meaning from characters, including AI crawlers and screen readers | Use ligature icon fonts that keep real words in markup, or pair icon glyphs with visually-hidden text labels |
| Images | Images provide zero text content on their own; a crawler that doesn't render pages and doesn't run an image model depends entirely on alt text, captions, and surrounding copy. Google's guidance on fixing lazy-loaded content warns that scroll- or interaction-triggered lazy loading can prevent images (and the content around them) from ever loading during a crawl | All crawlers for the visual content itself; non-rendering crawlers additionally lose anything gated behind a scroll trigger | Use loading="lazy" or IntersectionObserver-based loading (which fires without a real scroll event), and write substantive alt text |
| Blocked scripts/robots.txt rules | A resource explicitly disallowed in robots.txt is a resource a compliant crawler will never fetch at all, independent of whether it would have mattered for rendering | Depends entirely on which crawler; compliance varies — Cloudflare's AI bot traffic analysis found compliance is inconsistent across vendors | Audit robots.txt for accidental blanket disallows on /js/, /assets/, or template directories |
A step-by-step audit process
- Pull the raw HTTP response, not the rendered DOM. Fetch the page with a plain HTTP client set to a crawler user-agent, or use "View Source" rather than DevTools' Elements panel. What's in that raw response is what a non-rendering crawler sees.
- Diff raw HTML against the rendered page. Open the same URL in a browser, wait for it to fully load, then compare the visible text to what showed up in step 1. Any content present only after JS execution is a retrieval gap for most AI crawlers.
- Check robots.txt for accidental resource blocks. Look for disallow rules on JS bundle directories, CSS paths, or image CDNs — these were a common misconfiguration Google explicitly warned about, and the pattern still shows up in modern builds that disallow static-asset directories by default.
- Test lazy-loaded sections without simulating a scroll. If a crawler doesn't scroll or hover, content behind scroll-triggered or event-based lazy loading effectively doesn't exist. Google's guidance specifically recommends IntersectionObserver-based or native lazy-loading implementations because they don't require simulated interaction.
- Search the rendered DOM for hidden or off-screen primary content. Accordions, tabs, and "read more" truncation are common places real copy gets hidden by default and never un-hides during a crawl.
- Audit icon/ligature fonts for glyph-mapped words. If your icon system maps meaningful words to custom glyphs, confirm the underlying markup still contains readable text, not arbitrary Unicode code points.
- Check image-dependent content for a text equivalent. Any information conveyed only through an image (pricing tables, infographics, screenshots of specs) needs alt text or adjacent copy that says the same thing in words.
- Run the page through Google's URL Inspection Tool to see Googlebot's rendered output and any blocked-resource warnings — useful as a proxy signal even though most AI crawlers don't render like Googlebot does.
- Check server logs for actual crawler user agents and their response codes. Confirm GPTBot, ClaudeBot, and PerplexityBot requests are landing on 200s and not 404s or redirect chains — Vercel's data found AI crawlers hitting 404s on over a third of requests industry-wide, often from stale or malformed links, which is worth ruling out on your own domain specifically.
What this doesn't guarantee
Direct answer: Fixing every item above gets you to "the content is retrievable." It does not get you to "the content will be cited." Whether an AI system chooses to surface a page in an answer depends on relevance, authority signals, and competing sources — none of which this audit touches.
It also can't guarantee compliance. A crawler ignoring your robots.txt rules, or one spoofing another crawler's user-agent string, isn't something a resource audit can detect from your side; you'd need IP-range verification against each vendor's published list, and even that only covers the crawlers that publish one. And because none of the major AI crawlers currently render JavaScript, "the page looks right when I load it in a browser" and "the page is retrievable" are simply different claims — passing a visual check tells you nothing about what's in the raw response.
Finally, this is a moving target. Google's own rendering behavior has changed materially over the past decade — it moved from a largely text-based indexer to an evergreen, browser-based renderer, and its documentation has been revised accordingly. AI crawler vendors are earlier in that curve and could add rendering capability at any point, which would change several of the recommendations above.
Where nqzai fits
Direct answer: nqzai's technical audit surface is built around exactly this gap between "the page loads" and "the page is retrievable." It fetches pages the way a non-rendering crawler would — reading the raw response rather than a browser-rendered DOM — and flags where primary content only exists after client-side execution, where lazy-loaded sections depend on interaction that a crawler won't perform, and where robots.txt rules are blocking resources that affect what gets parsed. The output is a concrete list of what to fix, tied to the specific pages and resources involved, rather than a generic crawlability score.
FAQ
Do AI crawlers like GPTBot and ClaudeBot execute JavaScript at all?
Based on large-scale traffic analysis, no. Vercel's review of crawler traffic found GPTBot and Claude's crawler fetch JavaScript files but show no evidence of executing them — they read the raw HTML response and stop there, per the Vercel report.
If content is hidden with display:none, will AI crawlers see it?
The HTML element and its text are still present in the raw response, so a crawler reading raw HTML will technically encounter the markup. But content that's never visible to a real user is a cloaking risk with search engines, and several AI-focused audits report that answer engines specifically deprioritize or discard content that never becomes visible — so hiding real content this way is a bad trade even where it's technically retrievable.
Does blocking CSS or JS in robots.txt actually hurt indexing?
For Googlebot, yes — Google's own guidance and public statements attributed to John Mueller confirm that blocking CSS prevents Google from understanding how a page renders, which affects indexing quality (documented in Search Engine Roundtable's coverage). For crawlers that don't render pages at all, blocking JS specifically matters less for content extraction, but blocking image or resource paths that carry captions or metadata can still remove content from what gets crawled.
Is lazy loading safe to use if I care about AI crawler visibility?
Yes, if implemented with native lazy-loading attributes or the IntersectionObserver API, which load content based on viewport visibility rather than requiring a real scroll or interaction event. Google's lazy-loading guidance specifically warns against event-based triggers that depend on simulated user actions a crawler won't perform.
How do I know if my robots.txt is actually blocking something that matters?
Fetch a sample of your JS, CSS, and image paths with a plain HTTP client using an AI crawler's user-agent string and check the response codes, then cross-reference those paths against your robots.txt disallow rules. Google's URL Inspection Tool will also flag blocked-resource warnings specifically for Googlebot, which is a useful proxy even for crawlers that behave differently.
