---
title: "How Claude Crawls and Cites the Web: Technical Breakdown"
description: "A technical breakdown of ClaudeBot, Claude-User, and Claude-SearchBot, how robots.txt controls each one, and what's actually verified about how Claude finds and cites content."
answer_summary: "A technical breakdown of ClaudeBot, Claude-User, and Claude-SearchBot, how robots.txt controls each one, and what's actually verified about how Claude finds and cites content."
canonical: "https://nqz.ai/blog/geo-claude-integration-1"
published_at: "2026-07-03T16:58:11.820Z"
updated_at: "2026-08-21T07:37:43.000Z"
author: "Lina Voss"
category: "GEO"
tags: ["GEO","Claude","Anthropic","ClaudeBot","AI crawlers","robots.txt"]
image: "https://images.unsplash.com/photo-1515879218367-8466d910aaa4?w=1200&h=630&fit=crop"
---

# How Claude Crawls and Cites the Web: Technical Breakdown

Most "GEO for Claude" advice treats Claude like a black box that occasionally mentions your brand. It isn't one. Anthropic publishes exactly how its bots identify themselves, exactly what its web search and web fetch tools return, and independent researchers have measured exactly what those bots do and don't render. This piece sticks to what's documented or independently verified — no invented token thresholds, no fictional model IDs, no guessed chunking behavior.

## The three bots, and why the distinction matters

**Direct answer:** Anthropic operates three separate crawlers, each with its own `robots.txt` user-agent token and its own job. This wasn't always the case — Anthropic previously operated under the now-deprecated `Claude-Web` and `Anthropic-AI` agent strings before splitting the function into three named bots, a move that mirrors OpenAI's split of GPTBot into GPTBot, OAI-SearchBot, and ChatGPT-User ([Search Engine Journal](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/); [Search Engine Land](https://searchengineland.com/anthropic-claude-bots-470171)).

| Bot | Purpose (Anthropic's own wording) | Blocking it means |
|---|---|---|
| **ClaudeBot** | "Helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." | Future content is excluded from training data. Does not affect whether Claude can still fetch your page live. |
| **Claude-User** | "Supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent." | Claude can't retrieve your content in response to a specific user query — the fetch that happens when someone asks Claude about your page directly. |
| **Claude-SearchBot** | "Navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses." | Your content isn't indexed for Claude's search-quality pipeline, which Anthropic says "may reduce your site's visibility and accuracy in user search results." |

Source: [Anthropic's Claude Help Center — "Does Anthropic crawl data from the web, and how can site owners block the crawler?"](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

The practical takeaway: these are independently controllable. A publisher who wants to keep content out of training but still wants to be cited when users ask Claude about them should block `ClaudeBot` while leaving `Claude-User` and `Claude-SearchBot` alone. Blocking all three collapses that distinction and removes you from all three functions at once.

## robots.txt mechanics that actually apply

Per Anthropic's own documentation, blocking any of the three is a standard directive block:

```
User-agent: ClaudeBot
Disallow: /
```

A few details that are easy to get wrong:

- **This has to be repeated per subdomain.** Anthropic's docs explicitly note the directive must be applied "for every subdomain you wish to opt out from" — a `Disallow` on `example.com` doesn't cover `blog.example.com`.
- **Crawl-delay is supported**, as a non-standard extension: `Crawl-delay: 1` under a given user-agent block throttles that bot's request rate rather than blocking it outright.
- **All three bots respect robots.txt**, including Claude-User — which is a meaningful difference from some competitors. Anthropic's documentation and third-party reporting both note that this contrasts with OpenAI's guidance that robots.txt rules may not apply to `ChatGPT-User`, and with Perplexity, whose user-triggered fetcher generally ignores robots.txt entirely ([ppc.land](https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/); [Search Engine Roundtable](https://www.seroundtable.com/anthropic-updates-its-crawler-docs-40978.html)).
- **IP blocking is explicitly discouraged.** Anthropic's own guidance: blocking IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." Anthropic also states it does not currently publish fixed IP ranges, since it uses cloud-provider public IPs that can change.
- **Anti-circumvention is respected** — Anthropic states its bots won't attempt to bypass CAPTCHAs or similar protections.

## The JavaScript rendering gap — verified, not assumed

**Direct answer:** This is the part most GEO content gets vague about, so it's worth citing the actual measurement rather than repeating the claim. A large-scale crawl-log analysis by Vercel (in a joint study referencing MERJ's crawler research) tracked hundreds of millions of AI crawler fetches across production sites and found that **none of the major AI crawlers execute JavaScript**, ClaudeBot included ([Vercel — "The rise of the AI crawler"](https://vercel.com/blog/the-rise-of-the-ai-crawler)).

The Vercel data on ClaudeBot specifically, over the measurement period:

- **370 million fetches**, versus 569 million for GPTBot in the same window.
- **23.84% of ClaudeBot's requests were for JavaScript files** — meaning it *does* fetch `.js` assets — but there is no evidence it executes them. (For comparison, ChatGPT's crawlers fetched JS files at an 11.50% rate.)
- **35.17% of ClaudeBot's total fetches were images** — a notably higher proportion than other major AI crawlers in the same dataset, suggesting Anthropic is investing more in visual content ingestion than text-only competitors.
- **34.16% of ClaudeBot's fetches returned a 404.**
- Requests were observed originating from a single U.S. data center location (Columbus, Ohio) during the study period.

The practical consequence follows directly from "doesn't execute JavaScript": if your page's actual content — product details, article body, pricing, FAQ answers — only exists after a client-side render (a typical React/Vue single-page app fetching data after load), ClaudeBot, Claude-User, and Claude-SearchBot never see it. They see whatever is present in the raw HTML response. Content embedded directly in that initial HTML payload — including data serialized into `<script>` tags, or content streamed via server components before the client takes over — can still be read, because it's already in the response Claude's crawlers fetch. The fix isn't a GEO trick, it's the same one that's mattered for SEO for a decade: server-side render or statically generate anything you want an AI crawler to be able to read, and verify with a raw fetch (`curl` or "view source"), not by looking at the page in a JS-executing browser.

## How Claude's web search tool actually decides to search

**Direct answer:** For teams building on the Claude API (as opposed to optimizing for the consumer Claude.ai product), Anthropic's own tool documentation spells out the decision logic precisely. Claude searches when a request "depends on information that is current, changing, or outside its training data" — recent events, current prices or statistics, or specifics about people/organizations/products that might have changed. It answers directly, without searching, for "established facts, math, science fundamentals, or coding concepts," creative work, or anything already present in the conversation ([Claude Platform Docs — Web search tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool)).

This has a direct GEO implication that's often missed: a page that only restates evergreen, stable knowledge Claude already "knows" from training is competing to be recalled from parametric memory, not to be *retrieved*. Pages have the best shot at being surfaced through live search specifically around content that's time-sensitive, comparative, numeric, or otherwise the kind of thing Claude is documented to treat as requiring a fresh look.

On the mechanics: each search result Claude receives carries a `url`, `title`, `page_age` (when the page was last updated), and encrypted content that Claude references but that developers never see in plaintext. Citations in Claude's response are built from `web_search_result_location` blocks, each carrying the source `url`, `title`, and — critically — `cited_text` capped at **150 characters**. That's a real, documented constraint: whatever gets quoted back to a user is a short, self-contained snippet, which argues for writing key facts as crisp, standalone sentences rather than burying them in long compound clauses that don't excerpt cleanly. Developers can also scope search with `allowed_domains`/`blocked_domains` and `max_uses`, and — on newer tool versions — Claude can run "dynamic filtering," writing code to filter search results before they ever reach its context window, which Anthropic says is meant to cut token usage on search-heavy requests rather than a documented change to how much of a page gets read.

Anthropic's web fetch tool, which retrieves the full content of a specific URL Claude already has in hand (rather than searching), carries the same rendering limitation as the crawlers: Anthropic's own documentation notes it does not support pages that are dynamically rendered via JavaScript.

## What actually gets cited: the evidence, and its limits

**Direct answer:** Separately from Anthropic's own documentation, independent measurement firms have tracked what Claude actually cites in practice. Two findings are worth taking seriously, with the caveat that these are third-party studies, not Anthropic disclosures, and methodologies vary:

- A domain-overlap study from Otterly.ai found only about **13% of domains cited by Claude and ChatGPT overlapped** for the same query set, dropping to roughly **4.2% overlap at the individual URL level** — and that Claude leans toward citing established brand and product pages, where ChatGPT leans more toward community content like forum threads ([Otterly.ai — Claude AI Citations Study](https://otterly.ai/blog/claude-ai-citation-study/)). If your content strategy is built for "AI search" as one undifferentiated channel, this is evidence that Claude and ChatGPT are functionally separate audiences with different winners.
- Analytics firm Profound has reported Claude showing the **lowest social-content citation rate** among major AI assistants in its tracking, and a roughly **86.7% overlap between what Claude cites and what surfaces in Brave Search's own top organic results** for the same queries — consistent with Claude's search tool drawing on Brave as a retrieval backend (Brave Search sits on Anthropic's published subprocessor list, though Anthropic has not officially named its search provider in its own documentation) ([reporting via Erlin.ai's Claude SEO guide](https://www.erlin.ai/blog/claude-seo)). If that correlation holds, the practical implication is unglamorous but important: ranking well in traditional organic search is not obsolete as a GEO lever for Claude specifically — it's still a strong proxy signal.

Treat both of these as directional, not gospel — they're vendor research from GEO-tooling companies with their own methodology, not Anthropic-published statistics, and citation behavior on any AI product can shift with model or product updates.

## Practical, Claude-specific guidance

Given everything above, here's what's actually actionable and defensible:

1. **Decide independently on ClaudeBot vs. Claude-User vs. Claude-SearchBot.** Don't block all three by habit. If your concern is training-data usage, block `ClaudeBot` specifically and leave the other two able to serve your content into live answers and Claude's search index.
2. **Verify your key content survives a JS-free fetch.** `curl` the page (or use "view source," not the rendered DOM) and confirm your core facts, pricing, and claims are present in the raw HTML — not injected after a client-side data fetch. This is the single highest-leverage fix, because it's a documented hard limitation, not a heuristic.
3. **Don't rely on IP blocking to manage AI bot access.** Anthropic's own guidance says it isn't reliable. Use `robots.txt`, with `Crawl-delay` if load is the concern rather than presence itself.
4. **Invest more than usual in image accessibility.** ClaudeBot's measured fetch pattern shows unusually heavy image traffic relative to other AI crawlers — descriptive alt text, meaningful filenames, and captioned context around images are more likely to matter for Claude visibility specifically than for a text-only crawler.
5. **Write citable facts as short, self-contained sentences.** With citation snippets capped at 150 characters in Claude's API responses, a fact buried in a long dependent clause is less likely to excerpt cleanly than the same fact stated plainly.
6. **Don't neglect ordinary SEO fundamentals.** The reported overlap between Claude's citations and traditional organic search results suggests domain authority, structured content, and technical crawlability aren't superseded by "GEO" — they're still a substantial part of it for this particular engine.
7. **Aim content at genuinely time-sensitive or comparative queries if you want live-search visibility.** Claude's own documentation states it answers stable, well-established facts from training rather than triggering a search — so a page competing purely on evergreen information is competing to be *recalled*, not *retrieved and cited*.

None of this requires guessing at internals Anthropic hasn't published. The crawler behavior, the tool mechanics, and the citation constraints above are all independently verifiable — which is a better foundation for a GEO strategy than working from a chunk-size number nobody can confirm.

Sources:
- [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Claude Platform Docs — Web search tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool)
- [Claude Platform Docs — Web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool)
- [Vercel — The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [Search Engine Journal — Anthropic's Claude Bots Make Robots.txt Decisions More Granular](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/)
- [Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them](https://searchengineland.com/anthropic-claude-bots-470171)
- [Search Engine Roundtable — Anthropic Updates Its Crawler Documentation](https://www.seroundtable.com/anthropic-updates-its-crawler-docs-40978.html)
- [ppc.land — Anthropic clarifies what its three web crawlers do](https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/)
- [Otterly.ai — Claude AI Citations Study](https://otterly.ai/blog/claude-ai-citation-study/)
- [Erlin.ai — Claude SEO: How to Get Cited by Claude AI](https://www.erlin.ai/blog/claude-seo)