View-Source vs. Rendered DOM: The AI Search QA Check Most Teams Skip

View-source HTML is the document your server sends before any JavaScript runs — the raw response body. The rendered DOM is what a browser produces after executing that JavaScript: the live tree of elements a human actually sees and can inspect. For most of the last decade this distinction was a Googlebot curiosity. For AI search QA in 2026, it is close to the whole ballgame, because most AI crawlers stop at the first document and never build the second one.

That gap is not a hypothesis. It has been measured at scale, on real traffic, by people who could see the requests.

Quick Answer

• If you're serving a single-page app that hydrates content client-side → test with curl using each crawler user-agent string, because GPTBot, ClaudeBot, PerplexityBot, and Bytespider all see only the raw HTML response and never execute JavaScript.
• If you're optimizing for AI search visibility and rely on Google's rendering pipeline → diff view-source against Search Console's rendered HTML, because Googlebot is the only major AI crawler that reliably executes JavaScript (and even then on a delay), while all others skip it.
• If you manage robots.txt for multiple AI crawlers from the same operator (e.g., OpenAI or Anthropic) → audit each agent string (e.g., GPTBot vs. OAI-SearchBot vs. ChatGPT-User) independently, because blocking one does not block the others—Search Engine Roundtable notes ClaudeBot blocking stops training collection but not Claude-User or Claude-SearchBot.

What the research actually shows

In December 2024, Vercel published a joint analysis with the technical SEO firm MERJ that tracked crawler behavior across its network, including nextjs.org and production sites on both Next.js and non-Next.js stacks. The finding, laid out in Vercel's "The rise of the AI crawler": across more than 500 million GPTBot fetches, they found zero evidence of JavaScript execution. The same held for Anthropic's ClaudeBot, Meta's external agent, ByteDance's Bytespider, and PerplexityBot. Some of these bots do request JavaScript files — Vercel's data shows ChatGPT's crawlers pulling .js files on about 11.5% of requests and Claude's on about 23.8% — but requesting a file is not the same as running it, and neither crawler executed what it fetched. They read the initial HTML response and moved on.

The scale here matters for QA priorities. In the same dataset, GPTBot generated 569 million requests in a month and Claude 370 million, against Googlebot's 4.5 billion — roughly 20% of Googlebot's volume and growing. These are not edge-case bots; they're a meaningful, non-rendering slice of your total crawl traffic.

Google is the outlier, and it's an infrastructure accident rather than a design philosophy. Google's own documentation, Understand JavaScript SEO Basics, lays out a two-phase process: Googlebot crawls and queues a URL, then a separate rendering service executes the JavaScript later, sometimes with a real delay between the two passes. Google-Extended, the crawler that feeds Gemini and AI Overviews, inherits this same rendering pipeline because it rides on Googlebot's infrastructure — it's the one major AI-facing crawler that actually sees your post-JavaScript page reliably, and even then, on a delay.

The two crawler operators most relevant to AI search each publish their own crawler documentation, and neither claims JavaScript execution. OpenAI's Overview of OpenAI Crawlers describes GPTBot, OAI-SearchBot, and ChatGPT-User as distinct fetch-and-filter agents, each independently controllable via robots.txt, none described as a rendering browser. Anthropic's support article on how its crawlers work and how to block them similarly documents ClaudeBot, Claude-User, and Claude-SearchBot as separate retrieval agents — and the distinction matters operationally: Search Engine Roundtable's coverage of Anthropic's documentation update notes that blocking ClaudeBot in robots.txt stops training-data collection but does nothing to Claude-User or Claude-SearchBot, which is a QA trap of its own if you only test one agent string and assume the rest behave identically.

Why this is a QA problem, not just a trivia fact

If your framework hydrates the headline, the body copy, product details, FAQ answers, or citations client-side — a common pattern in single-page apps and some server-rendered apps with client-side content swaps — a human opening dev tools sees a complete page. A crawler that never executes JavaScript sees whatever was in the initial response: often a shell, a loading skeleton, or an empty root container with nothing inside it. Your page can rank on Google, look flawless in a design review, and still be functionally invisible to the AI engines summarizing or citing sources — because Google's renderer papered over the gap and nothing else did.

View-source vs. rendered DOM: a reference table

A step-by-step QA process

1. Pull the raw response with curl. Run a plain curl request against the page and read the output as text. This is close to what a non-rendering crawler receives, no browser involved.
2. Identify a load-bearing sentence and grep for it. Take a specific line from your body copy, product spec, or FAQ answer and search the curl output for it. No match means the text is injected after the initial load and is invisible to non-rendering crawlers.
3. Check with real crawler user-agent strings, not just a generic curl. Some origins, CDNs, or WAFs serve different responses (or block outright) based on user-agent. Test with the actual tokens: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, and PerplexityBot. A 403 or timeout under a crawler UA and a clean 200 under a browser UA is a QA failure independent of rendering.
4. Open browser View Source (Cmd/Ctrl+U) as a second check. It's a fast manual sanity pass that catches the same gap curl does, useful when you want to eyeball things quickly rather than script a check.
5. Diff view-source against the rendered DOM. Open DevTools, inspect the live Elements tree, and compare it against what you saw in steps 1–4. Anything present in Elements but absent from source — headline text, structured data, internal links, canonical tags, meta descriptions — is JavaScript-dependent content.
6. Confirm what Google specifically sees using Search Console. Google is the exception that renders, so testing it needs a different tool. Use the URL Inspection tool's Test Live URL, then View Tested Page, to pull the actual rendered HTML Googlebot produced, and compare that against your view-source result from step 1 — a mismatch here confirms your content is JS-dependent, even if Google is currently handling it fine.
7. Audit robots.txt per crawler, not as one block. Since each operator runs multiple independently-controllable agents, verify your intent is actually encoded — check that you haven't accidentally disallowed OAI-SearchBot while trying to keep out GPTBot, or blocked Claude-SearchBot while meaning to block only ClaudeBot.
8. Re-run the same checks after any rendering fix. If you move to server-side rendering, static generation, or pre-rendering for a section, re-diff view-source against the DOM to confirm the content actually landed in the initial response — don't assume the fix worked from the deploy log alone.
9. Monitor server access logs for crawler user-agents over time. A one-time check tells you today's state; logs tell you whether GPTBot, ClaudeBot, and PerplexityBot are actually completing fetches with 200s, or hitting 404s and redirects at the volumes Vercel's data suggests are common industry-wide.

What this doesn't guarantee

Direct answer: Passing every check above proves your content is retrievable — it does not prove it will be cited, summarized, or ranked well by any AI engine. Visibility in raw HTML is a precondition, not an outcome. A page can be perfectly server-rendered and still lose out to a competitor's page on relevance, structure, or authority signals the engine weighs after retrieval.

It also doesn't cover every failure mode. A crawler can see your content and still fail to fetch the page at all — blocked by a WAF, rate-limited, or 404ing on a stale URL, all of which Vercel's data shows happening at real scale even for crawlers with no rendering problem. User-agent strings are trivially spoofed, so a curl test that returns clean HTML doesn't prove the real crawler gets the same response from your CDN or bot-mitigation layer; where that matters, cross-check against published IP ranges or reverse DNS, not the UA string alone.

Crawler behavior also isn't static. Google's own documentation describes rendering as a distinct, sometimes-delayed second pass, not a promise, and the non-rendering behavior of GPTBot, ClaudeBot, and PerplexityBot reflects a resource tradeoff these operators have made at today's traffic volumes — not a permanent architectural commitment. Retest periodically rather than treating one clean audit as settled.

Finally, none of this substitutes for content quality. A perfectly retrievable page with thin, generic, or inaccurate content isn't going to out-cite a better-written competitor just because its HTML is clean.

Where nqzai fits

Direct answer: nqzai's AI-search auditing checks a site's pages the way a non-rendering crawler actually reads them — pulling the raw response, verifying that headlines, body content, structured data, and internal links needed for AI search visibility are present before any JavaScript runs, and flagging pages where the browser-rendered version and the raw-source version diverge. Instead of asking a team to manually curl every template variant and diff the output, it surfaces the gap as a concrete, page-level finding tied to what's actually missing, so JavaScript-dependent content gets fixed at the template level rather than caught page-by-page after the fact.

FAQ

Direct answer: Does view-source show exactly what every AI crawler sees? Close enough for QA purposes on GPTBot, ClaudeBot, PerplexityBot, and similar non-rendering agents, per Vercel's traffic analysis — but always verify with the crawler's actual user-agent string, since some infrastructure serves different responses by UA.

Is Google an exception across the board? Only Google's own crawling infrastructure — Googlebot and, through it, Google-Extended for Gemini and AI Overviews — is documented as rendering JavaScript, per Google's JavaScript SEO basics guide. OpenAI's and Anthropic's own crawler documentation describes fetch-and-filter agents, not rendering browsers.

If my page ranks well on Google, is it safe for AI search too? Not necessarily. Google's rendering pipeline can surface content that a non-rendering AI crawler never sees, since GPTBot and ClaudeBot don't get the benefit of that second rendering pass.

What's the fastest single check I can run today? Fetch the raw HTML for your URL and search the output for a sentence you know is on the page. If it's not in the output, it's invisible to most AI crawlers regardless of how the page looks in a browser.

Does blocking one AI crawler in robots.txt block them all? No. OpenAI and Anthropic each operate multiple independently-controlled agents — disallowing GPTBot doesn't affect OAI-SearchBot or ChatGPT-User, and disallowing ClaudeBot doesn't affect Claude-User or Claude-SearchBot, per each company's own crawler documentation.