TL;DR
A single Vercel/MERJ analysis found GPTBot hit 404 pages on 34.82% of its fetches and ClaudeBot on 34.16%, versus 8.22% for Googlebot, and that neither AI crawler executes JavaScript despite sometimes downloading .js files. OpenAI runs three separate bots (GPTBot, OAI-SearchBot, ChatGPT-User) with distinct robots.txt rules, and blocking GPTBot does not stop ChatGPT Search citations. Perplexity-User, a real-time fetcher, generally ignores robots.txt because a human initiated the request.
An audit must check five layers in one pass — permission, delivery, content availability, discovery, and machine-readable meaning — because a failure in any single layer breaks retrieval regardless of the others. Run this full checklist on a schedule, not as isolated projects, since stale sitemaps and redirect chains cost AI crawlers proportionally more crawl budget than they do Googlebot.
An AI search technical audit is a single, end-to-end check of whether AI crawlers and retrieval systems — GPTBot, ClaudeBot, PerplexityBot, Google's Gemini (via Googlebot), and the on-demand fetchers each of those companies also run — can reach a page, read its content, and extract enough structured meaning from it to cite or summarize accurately. It covers five layers that all have to hold at once: permission (does robots.txt actually allow the agent that matters), delivery (does the server return the right status code, without redirect loops or silent failures), content availability (is the content in the raw HTML or only in JavaScript the crawler never executes), discovery (do sitemaps and internal links surface every page worth finding), and machine-readable meaning (does structured data and canonical markup describe the page correctly and consistently). Site owners often audit these areas one at a time — a robots.txt review here, a schema markup pass there — but a crawler failure in any single layer breaks retrieval regardless of how well the others are handled. This is a checklist for running all five as one pass, on a schedule, rather than as isolated projects.
Why the technical surface is different for AI crawlers than for Google
Direct answer: The starting assumption for most technical SEO — "if Googlebot can index it, it's fine" — doesn't transfer cleanly to AI retrieval, for three concrete, documented reasons.
The crawlers are not one thing. OpenAI operates three separate, independently controllable bots: GPTBot, which collects training data; OAI-SearchBot, which builds the index ChatGPT Search cites from; and ChatGPT-User, which fetches pages live when a user's prompt triggers a real-time browse (OpenAI crawler documentation). Blocking GPTBot in robots.txt does not remove a site from ChatGPT Search citations — that requires blocking OAI-SearchBot specifically, and the two are commonly conflated. Anthropic runs the equivalent split: ClaudeBot for training, Claude-User for user-triggered fetches, and Claude-SearchBot for its search index, all of which respect robots.txt including the user-triggered agent (Claude Help Center, "Does Anthropic crawl data from the web"). Perplexity separates PerplexityBot, an indexing crawler that honors robots.txt, from Perplexity-User, a real-time fetcher that Perplexity's own documentation says generally does not treat robots.txt as binding since a human initiated the request (Perplexity crawler documentation). An audit that checks only "is my site blocked from AI" without naming the specific agent is checking the wrong thing.
Most AI crawlers do not render JavaScript. Google's own documentation describes Googlebot processing pages in two distinct passes — an initial HTML fetch, then a separate queued render through its Web Rendering Service, a Chromium-based renderer that executes JavaScript before indexing (Google Search Central, "Understand JavaScript SEO basics"). That second pass is specific to Google. A joint analysis by Vercel and the technical SEO firm MERJ, published December 17, 2024, examined hundreds of millions of AI crawler fetches on Vercel's network and found GPTBot generated 569 million requests and Claude 370 million in the analyzed month — against 4.5 billion from Googlebot — and that neither crawler executes JavaScript despite sometimes fetching the files: GPTBot requested .js files in about 11.5% of its fetches, ClaudeBot in about 23.84%, and in neither case did the content execute (Vercel, "The rise of the AI crawler"). Any content that only renders client-side — a price loaded via API call, an FAQ answer injected after hydration, a comparison table built in React — is invisible to GPTBot, ClaudeBot, and PerplexityBot today, full stop, regardless of how well the page performs in a Lighthouse audit or in Google Search Console.
AI crawlers waste more of their own budget on dead ends than Googlebot does. The same Vercel/MERJ analysis found ChatGPT's crawler hit 404 pages on 34.82% of fetches and Claude's on 34.16%, compared with 8.22% for Googlebot, and that ChatGPT spent an additional 14.36% of its fetches following redirects versus 1.49% for Googlebot. That gap is a direct signal that most sites' internal linking, sitemaps, and redirect hygiene are tuned for Google's more tolerant, more mature crawl patterns and not for newer, less forgiving AI crawlers — a stale sitemap or an old redirect chain costs proportionally more against these bots' effective crawl budget.
Comparison: what to check, and how
| Audit area | What to check | How to check it |
|---|---|---|
| robots.txt permissions | Correct Allow/Disallow for each named agent (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot) — not a blanket AI block | Manual read of robots.txt; cross-reference against each provider's published agent list |
| IP/UA verification | Confirm traffic claiming to be an AI crawler actually originates from that provider's published IP ranges | Compare server logs against OpenAI's, Anthropic's, and Perplexity's published IP JSON endpoints |
| Rendering dependency | Whether page content (main copy, prices, FAQ answers, comparison data) exists in the raw HTTP response or only after JavaScript execution | Diff raw HTML (view page source) against the rendered DOM in browser devtools; or fetch the URL with a bare HTTP client and check what text is present |
| HTTP status codes | Every crawlable URL returns a clean 200, or a real 301/404/410 — no soft 404s (200 status on a "not found" page), no redirect chains over one hop | Crawl the full site with a headless HTTP crawler recording status codes per URL; check Search Console's crawl stats report |
| Sitemaps | Present, under the 50MB / 50,000-URL limit, contains only indexable canonical URLs (no redirects, no noindex, no 4xx), referenced from robots.txt, updated on change | Fetch and parse the sitemap XML; validate each listed URL resolves 200 and is self-canonical |
| Structured data | Valid schema.org markup matching visible on-page content, using the most specific applicable type, present on every duplicate of a page (not just the canonical) | Google's Rich Results Test or a schema validator; manual comparison of markup fields against rendered content |
| Canonical signals | A single, consistent canonical URL indicated the same way across redirects, rel="canonical" tags, and sitemap entries — no conflicting signals | Check rel="canonical" per page, cross-reference against sitemap and redirect targets |
| llms.txt (optional) | Present at domain root if used; understood as a voluntary hint, not an access-control mechanism | Fetch /llms.txt; check server logs for any agent fetches of it |
The consolidated audit process
- Name your crawlers before you check anything. List every AI agent relevant to your visibility goals by exact user-agent string — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot — and pull each provider's current published IP ranges. A blanket "block AI" or "allow AI" robots.txt rule is almost always wrong because training bots, search-index bots, and user-triggered fetchers serve different purposes and often warrant different rules.
- Read robots.txt line by line against that list. Confirm each named agent has an explicit, intentional rule. Missing an agent means the default (usually allow) applies, which may not be the intended outcome; a wildcard
Disallow: /underUser-agent: *combined with no specific allow for a named bot will silently block it too.
- Fetch the raw HTML the way a non-rendering crawler would. Request the URL with a plain HTTP client (no JavaScript execution) and check whether the primary content — body copy, prices, structured data, FAQ text — is present in that response. If it only appears after client-side rendering, GPTBot, ClaudeBot, and PerplexityBot will not see it, per the rendering behavior documented above.
- Crawl the site for status codes end to end. Record the HTTP status for every internal URL and every URL in the sitemap. Flag 4xx and 5xx responses, redirect chains longer than one hop, and soft 404s — pages that display "not found" copy but return a 200. Google's own guidance is explicit that a 200 on a not-found page should instead be a genuine 404 or 410 so crawlers don't keep spending budget re-fetching content that doesn't exist (Google Search Central, "HTTP status codes, network and DNS errors").
- Validate structured data against what's actually rendered. Run each templated page type through a schema validator and confirm every markup field matches visible, crawlable content — Google's structured data guidelines specifically warn against markup describing content that isn't visible to users, and require the same markup be present on every duplicate of a page, not only the canonical version (Google Search Central, general structured data guidelines).
- Check canonical signals for internal consistency. A page's redirect target, its
rel="canonical"tag, and its sitemap entry should all point to the same URL. Google's documentation notes these signals stack in strength — redirects strongest, canonical tags next, sitemap inclusion weakest — and that specifying different canonical URLs through different methods for the same page actively confuses the signal (Google Search Central, "How to specify a canonical").
- Audit sitemap health independently of the crawl. Confirm the sitemap is under the 50MB/50,000-URL cap, contains only 200-status canonical URLs (no redirects, no noindex pages, no known 404s), is declared in robots.txt, and gets regenerated on content change rather than on a stale schedule (Google Search Central, "Build and submit a sitemap").
- Pull real log-file data on the named agents. Filter server logs for the exact user-agent strings from step 1, verify the requesting IPs against each provider's published ranges (user-agent headers are trivially spoofable), and compute each agent's own 404 rate, redirect rate, and crawl frequency on your site specifically — not industry averages.
- Re-run the full pass on a cadence, not a one-time basis. Crawler behavior, agent names, and robots.txt conventions have changed multiple times in the past two years — Anthropic split one bot into three, OpenAI did the same earlier — so an audit that was correct six months ago can silently be wrong today. Treat this as a recurring check tied to major site changes (CMS migration, redesign, new templated content type) and a fixed quarterly minimum otherwise.
What this doesn't guarantee
Passing every item above gets a page in front of AI crawlers cleanly. It does not guarantee the page gets cited. Accessibility and inclusion are different problems: a technically perfect page can still lose out to a competitor's page that answers the query more directly, is judged more authoritative, or simply gets sampled differently by a given model on a given day. This audit also can't control the split between training use and retrieval use — blocking GPTBot stops training collection but has no effect on ChatGPT Search citations, which run through OAI-SearchBot instead, and the reverse is equally possible depending on intent. It doesn't fix underlying content quality, depth, or accuracy, which matter independently of crawlability. And llms.txt, despite growing use, remains a voluntary convention with no confirmed systematic ingestion commitment from any major AI provider as of this writing — treat it as a hint you can offer, not a control you can rely on (Search Engine Land, "Meet llms.txt, a proposed standard for AI website content crawling"). Finally, this is a snapshot: the crawler landscape has changed enough times in two years that an audit result has a shelf life, not a permanent status.
Where nqzai fits
Direct answer: nqzai's audit tooling runs this consolidated check as a single automated pass rather than a set of separate one-off reports: it checks robots.txt against the current named-agent list, diffs raw HTML against rendered content to flag JavaScript-only sections, crawls the full site for status-code and redirect issues, validates structured data against what's actually on the page, and checks sitemap and canonical consistency — then surfaces the findings as a prioritized list rather than a wall of disconnected metrics, so a team can fix the highest-impact gaps first instead of triaging five separate audits by hand.
FAQ
Does blocking GPTBot remove my site from ChatGPT search results?
No. GPTBot collects training data; ChatGPT Search citations are built from a separate crawler, OAI-SearchBot. Blocking GPTBot alone has no effect on search citations — you'd need to block OAI-SearchBot specifically, which also then removes the option to appear in citations at all (OpenAI crawler documentation).
Do I need full server-side rendering to be visible to AI search?
For GPTBot, ClaudeBot, and PerplexityBot, yes for any content you need those crawlers to read — none of them execute JavaScript, per the Vercel/MERJ analysis of over 500 million fetches. Google's Gemini is the exception, since it rides Googlebot's rendering infrastructure. If your primary content loads only via client-side JavaScript, it's currently invisible to every major AI crawler except Google's.
How is this different from a regular technical SEO audit?
The individual checks overlap with technical SEO, but the emphasis differs: a technical SEO audit is tuned around Googlebot's tolerant crawl behavior and ranking signals, while an AI search technical audit specifically accounts for crawler fragmentation (multiple named agents per provider with different purposes), zero JavaScript execution outside Google's ecosystem, and measurably higher error-rate sensitivity — AI crawlers hit 404s and redirects at roughly four times Googlebot's rate on the same infrastructure, per the Vercel/MERJ data.
How often should this audit run?
At minimum quarterly, and immediately after any CMS migration, redesign, or new templated content type, since crawler rules and agent names have changed multiple times industry-wide over the past two years and a passing result has a limited shelf life.
Is llms.txt worth implementing?
It's low-cost to add and can't hurt, but treat it as a hint, not a guarantee — no major AI provider has publicly committed to systematically ingesting it the way Googlebot ingests sitemap.xml, and adoption across the industry has been fragmented.
What's the single highest-impact fix if I can only do one thing?
Check whether your primary content — the text a user or an AI system would actually need to answer a question — exists in the raw, unrendered HTML response. Every other item on this checklist is moot if the content itself isn't reachable without JavaScript execution, since that one gap silently excludes GPTBot, ClaudeBot, and PerplexityBot regardless of how clean the rest of the technical surface is.
