Robots.txt Rules for AI Crawlers: A Governance Guide

Robots.txt governance for AI crawlers is the ongoing practice of writing, testing, and revising per-crawler rules in a site's robots.txt file so that each named AI bot — a model-training crawler, a search-indexing crawler, or a live user-query fetcher — gets exactly the access a site owner intends, and re-checking those rules as vendors split, rename, or add crawlers. It is not a file you write once and forget. Every major AI lab now operates multiple distinct crawlers with different purposes, and blocking one does nothing to the others.

That last point trips up more site owners than any other. A single line reading User-agent: GPTBot / Disallow: / feels like an AI opt-out. It isn't. OpenAI runs three separate, independently configurable crawlers — GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for live, user-triggered fetches — and explicitly documents that "each setting is independent of the others," meaning a site can disallow training while still appearing in ChatGPT's search answers, or vice versa (OpenAI, Overview of OpenAI Crawlers). Anthropic runs the equivalent three-way split for Claude, and updated its own documentation in February 2026 specifically to make that split clearer after site owners kept misconfiguring it (Search Engine Journal, "Anthropic's Claude Bots Make Robots.txt Decisions More Granular," Feb. 2026).

Why this is governance, not a one-time edit

Three things make this a maintenance practice rather than a set-and-forget file:

Vendors keep changing the crawler roster. GPTBot launched in August 2023; OAI-SearchBot and ChatGPT-User followed later as OpenAI added product surfaces. Anthropic's crawler lineup similarly grew from a single ClaudeBot entry into the current three-bot structure, replacing older, now-deprecated agents like Claude-Web and anthropic-ai (support.claude.com, "Does Anthropic crawl data from the web, and how can site owners block the crawler?"). A robots.txt file written in 2023 is very likely missing crawlers that exist today.

Blocking rates are moving, and moving fast. Independent tracking of the top 1,000 global websites found GPTBot blocking climbed from roughly 5% shortly after its August 2023 launch to over a third of top sites within about two years, as major publishers — Reuters was reportedly the first top-100 site to block it, within a day of launch — reassessed the trade-off between AI visibility and training-data exposure (ppc.land, "Top websites increasingly block AI web crawlers amid privacy concerns"). Whatever your competitors and peers are doing with their robots.txt files today is not what they were doing a year ago.

The economics behind the decision keep shifting, too. Cloudflare's own analysis of crawl-versus-referral traffic found that as of June 2025, Google crawled sites roughly 14 times for every visitor it referred back — while OpenAI's ratio was about 1,700 crawls per referral, and Anthropic's was about 73,000 crawls per referral (Cloudflare Blog, "Control content use for AI training with Cloudflare's managed robots.txt"). That imbalance is a big part of why blocking rates keep rising, and it's a number worth re-checking periodically rather than assuming it's static.

The crawlers, compared

Direct answer: Here is the reference table for the crawlers site owners most commonly need to make explicit decisions about. "What blocking controls" describes the specific consequence of disallowing that user-agent — not a general AI opt-out.

Sources for the table: OpenAI's own crawler documentation (developers.openai.com); Anthropic's support article on its crawlers (support.claude.com); Perplexity's own crawler documentation (docs.perplexity.ai); Search Engine Journal's reporting on Google's Bard-to-Gemini documentation update for Google-Extended (searchenginejournal.com); and Apple's own support page on Applebot (support.apple.com).

Two patterns are worth naming explicitly. First, every major vendor now separates "train on this" from "answer questions using this" from "index this for search" — three different consent decisions, not one. Second, the crawlers built for live, user-triggered fetches are the ones most likely to sidestep robots.txt entirely, because the vendors treat a user-directed request differently from an automated crawl. That distinction is the single most common source of misconfigured AI governance.

How the underlying protocol actually works

Direct answer: Robots.txt itself is standardized in RFC 9309, published by the IETF in September 2022, which formalized rules that had existed as an informal convention since 1994 (IETF Datatracker, RFC 9309: Robots Exclusion Protocol). Two details from the RFC matter directly for AI-crawler governance:

• Rule groups are matched by user-agent name, and the most specific matching group wins — a Disallow under User-agent: ClaudeBot overrides a broader rule under User-agent: * for that bot, but has no effect on Claude-User or Claude-SearchBot, which need their own groups.
• The RFC is explicit that these rules "are not a form of access authorization." Robots.txt is a request, not a lock.

Crawl-delay is not part of RFC 9309 but is a widely supported non-standard extension; Anthropic's documentation confirms its bots honor it, which is useful for throttling rather than fully blocking a crawler (support.claude.com). More recently, Cloudflare proposed an additional, non-RFC extension called Content Signals — a Content-Signal line expressing separate search, ai-input, and ai-train preferences — which it rolled out by default across millions of domains on its managed robots.txt service starting in September 2025 (Cloudflare Blog, "Your site, your rules: new AI traffic options for all customers"). Content Signals is a preference layer on top of the standard, not a replacement for per-crawler Disallow rules, and — like robots.txt itself — its effectiveness still depends entirely on crawlers choosing to honor it.

A governance process, not a one-time task

1. Inventory actual crawler traffic first. Pull server or CDN logs and list every AI-related user-agent that has actually hit the site in the last 30-90 days, rather than assuming the current well-known list is complete. New crawlers appear without much notice.
2. Separate the three decisions per vendor. For each company, decide independently: should this content train future models, should it be searchable inside that company's assistant, and should the assistant be able to fetch it live when a user asks. Treat these as three answers, not one.
3. Write one explicit User-agent group per crawler you have an opinion about. Don't rely on the wildcard * group to express AI-specific intent — it's a fallback, and specific-agent groups always take precedence over it under RFC 9309.
4. Add a Content-Signal line as a supplementary preference, not a substitute, if the platform supports it — it's a newer, less universally honored layer.
5. Cross-check against each vendor's published IP ranges or verified-bot lists where available (OpenAI and Anthropic both publish JSON IP lists) to distinguish real crawler traffic from spoofed user-agent strings claiming to be GPTBot or ClaudeBot.
6. Validate propagation timing. OpenAI's own documentation notes it can take roughly 24 hours for a robots.txt change to take effect for its search-related crawler — don't assume changes are live immediately, and don't re-test too early and conclude a rule failed.
7. Re-scan logs after each change to confirm the targeted crawler actually stopped (or started) hitting the disallowed paths, rather than trusting the file alone.
8. Set a recurring review cadence — quarterly at minimum — because vendors add crawlers, rename them, and rewrite their own documentation (as Anthropic did in February 2026) without necessarily notifying site owners.
9. Keep a short decision log: which crawler, which decision, which date, and why. When a new stakeholder asks "why do we block ClaudeBot but not Claude-SearchBot," the answer should be retrievable, not re-litigated.

Limitations — what this doesn't guarantee

Direct answer: Robots.txt governance is real, useful, and worth doing well. It is not a technical control, and treating it like one leads to false confidence.

• It's voluntary by design. RFC 9309 states plainly that these directives are not a form of access authorization — any crawler that chooses to ignore the file technically can, and the standard doesn't provide any enforcement mechanism.
• Some of the biggest AI vendors say, in their own documentation, that certain of their crawlers may not fully honor it. OpenAI's docs state that ChatGPT-User's robots.txt compliance "may not apply" since it's user-triggered (developers.openai.com), and Perplexity's own crawler documentation says Perplexity-User "generally ignores robots.txt rules" for the same reason (docs.perplexity.ai).
• Independent testing has found outright non-compliance, not just documented exceptions. A survey of 47 UK business websites over three weeks in late 2025 found AI crawlers violating disallow directives on 72% of sites, averaging over 150 violating requests per site (365i, "AI Crawler Compliance Tracking: Cloudflare Data," Jan. 2026). Cloudflare separately delisted Perplexity from its verified-bot program in 2025 after finding it used stealth, non-declared user-agents to keep crawling sites that had explicitly blocked its declared bot.
• IP blocking is not a reliable backstop. Anthropic's own guidance warns that its crawlers run on shared public cloud IP ranges, so blocking those ranges risks blocking legitimate, unrelated traffic — and the company doesn't publish IP ranges for that reason.
• Blocking a training crawler going forward does not retroactively remove content already ingested into a model trained before the block was added.
• A block only covers vendors you know about and named. Datasets like Common Crawl feed multiple downstream AI labs indirectly; blocking the named crawlers above says nothing about how your content flows through general-purpose crawl datasets you never explicitly addressed.

None of this is an argument against maintaining rules — it's an argument for treating compliance as probabilistic, monitored, and revisited, not a switch you flip once.

Where nqzai fits

Getting this right by hand means tracking a moving list of crawlers across several vendors' documentation, translating each into correctly scoped robots.txt groups, and then actually watching server logs to confirm the rules are being respected — work most teams don't have a standing process for. nqzai's technical SEO and AI-visibility tooling is built to close that loop: it surfaces which named AI crawlers are actually hitting a site, maps that traffic against the site's current robots.txt rules to flag gaps or drift, and rolls the governance checklist above into a recurring check rather than a one-time audit — so the file stays aligned with what site owners actually decided, as the list of crawlers keeps growing.

FAQ

Does blocking GPTBot remove my content from ChatGPT entirely?

No. Blocking GPTBot only opts your site out of future training-data collection. It has no effect on OAI-SearchBot (ChatGPT Search visibility) or ChatGPT-User (live fetches when a user asks ChatGPT to visit your page) — those require their own separate rules.

What's the actual difference between blocking a crawler for training versus for search?

Training crawlers (GPTBot, ClaudeBot) collect content that may shape a future model's weights, with effects that persist indefinitely once ingested. Search-indexing crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build a retrievable index that determines whether your page can be cited or shown in real-time answers — blocking these affects visibility going forward, not model training.

Is one Disallow: / rule enough to opt out of AI use of my site?

Only for the single named user-agent it targets. Every major vendor runs multiple crawlers for different purposes, so a comprehensive opt-out requires a separate, correctly named User-agent group for each crawler you want to restrict.

Should I bother with IP-range blocking in addition to robots.txt?

Generally no, as a primary defense. Anthropic explicitly advises against it because its crawlers share IP ranges with unrelated cloud traffic, and neither OpenAI nor Anthropic guarantees IP lists stay complete or current. Use published IP/JSON lists to verify legitimate traffic, not as the main enforcement mechanism.

How often should I revisit my AI crawler robots.txt rules?

At minimum quarterly, and immediately after any major vendor documentation change — Anthropic restructured its crawler documentation as recently as February 2026. Because compliance itself isn't guaranteed, pair the review with a log check to confirm the rules you wrote are actually being honored.