AI Crawler Response Code QA

TL;DR

Validate AI crawler response codes, redirects, error handling, rate limiting, and server behavior with a repeatable technical QA process.

AI crawlers—GPTBot, ClaudeBot, PerplexityBot, Google-Extended—use HTTP response codes to decide whether to index, cache, or discard your content, directly influencing whether your site appears in ChatGPT, Claude, Perplexity, Gemini, or Google’s AI Overviews. This guide explains how to tune your server responses so AI models cite your content, not your competitors’.

What is AI Crawler Response Codes: A Technical QA Guide?

AI Crawler Response Codes refer to the HTTP status codes (200, 301, 404, 410, 429, 503, etc.) that a web server returns when an AI agent requests a page. Unlike traditional search crawlers, AI crawlers are more sensitive to rate-limiting, stale content, and ambiguous redirects. A 200 OK with fresh, structured content signals “cite me”; a 429 Too Many Requests or 503 Service Unavailable tells the AI to skip you or deprioritize your content. This guide maps each status code to AI crawler behavior and provides actionable optimizations to increase citation probability.

Why AI Crawler Response Codes Matters for AI Search

AI crawlers are less forgiving of transient errors. A 503 or 429 that would be retried by Googlebot may cause an AI crawler to permanently discard the URL, especially if the error persists across multiple fetch attempts. OpenAI’s GPTBot documentation states that repeated 5xx responses can lead to deindexing.

Redirect chains break citation attribution. A 301 redirect to a different domain or a 302 temporary redirect that resolves to a non-canonical page confuses AI models trying to attribute a statement to a stable source. Perplexity’s citation algorithm prefers direct 200 responses from authoritative domains.

Stale content (410 Gone, 404 Not Found) is penalized more heavily. AI models cache content for weeks or months. If a page returns 410 after being cited, the model may still reference the old URL, but future citations will be blocked. Google’s AI Overviews documentation notes that removed pages can cause “citation gaps” that reduce answer quality.

ChatGPT: Getting Cited

ChatGPT (via GPTBot) uses a combination of crawl frequency, content freshness, and structured data to decide which pages to cite in its answers.

Return 200 with a Last-Modified header. GPTBot respects the Last-Modified and ETag headers. Set them to the actual date of the most recent substantive update. A page with a stale Last-Modified (older than 6 months) is less likely to be cited for current events.
Avoid 429 for GPTBot. OpenAI’s crawler respects Crawl-Delay in robots.txt but will back off if it receives 429. However, if you return 429 too aggressively, GPTBot may stop crawling altogether. Set a reasonable rate limit (e.g., 10 requests per second) and use 200 for the first few requests.
Use 301 redirects sparingly. If you must redirect, ensure the target URL returns 200 and has the same canonical domain. GPTBot follows at most one redirect hop; a chain of two or more will cause the page to be ignored.
Serve structured data on 200 pages. Include Article, FAQPage, or HowTo schema (see Schema Markup section). GPTBot extracts these to build answer snippets.

Perplexity: Citation Patterns

PerplexityBot is known for aggressive crawling and high sensitivity to response codes. Perplexity’s citation algorithm favors pages that return 200 quickly and have a clear “source” footprint.

Return 200 in under 2 seconds. Perplexity measures Time to First Byte (TTFB). Pages that take longer than 2 seconds are deprioritized. Use a CDN and server-side caching to keep TTFB < 500ms.
Never return 404 for a previously indexed page. Perplexity caches URLs for up to 30 days. If a page that was previously cited returns 404, Perplexity will remove the citation and may penalize the domain’s overall trust score. Use 410 Gone only if the content is permanently removed and you want to signal “do not re-crawl.”
Use 200 for all subpages of a topic cluster. Perplexity’s citation algorithm prefers pages that are part of a well-linked silo. If a supporting page returns 503 during a crawl, the entire cluster may be skipped.
Include a citation_author meta tag. Perplexity parses <meta name="citation_author" content="..."> to attribute quotes. Combine this with a 200 response to increase citation likelihood.

Claude: Knowledge Graph Positioning

ClaudeBot (Anthropic) uses a knowledge graph approach: it maps entities and relationships from pages that return 200 with high-quality schema.

Return 200 with Entity schema. ClaudeBot prioritizes pages that define entities (people, organizations, products) using schema:Person, schema:Organization, or schema:Product. A 200 response with a missing @id field reduces citation probability.
Avoid 302 temporary redirects. ClaudeBot treats 302 as “may change” and does not store the content in its knowledge graph. Use 301 for permanent moves and ensure the target returns 200.
Serve a 200 with a sameAs property. ClaudeBot uses sameAs to disambiguate entities. If your page returns 200 and includes "sameAs": "https://www.wikidata.org/wiki/Q...", Claude is more likely to use your page as the canonical source.
Rate-limit ClaudeBot with 429 only as a last resort. Anthropic’s documentation states that ClaudeBot respects Crawl-Delay but may stop crawling for 24 hours after receiving a 429. Prefer robots.txt rate limiting.

Schema Markup for AI

AI crawlers extract structured data from 200 responses. Below are JSON-LD examples optimized for each AI engine.

Article Schema (for ChatGPT and Perplexity)

{
 "@context": "https://schema.org",
 "@type": "Article",
 "headline": "AI Crawler Response Codes: A Technical QA Guide",
 "author": {
 "@type": "Person",
 "name": "Your Name"
 },
 "datePublished": "2025-03-15",
 "dateModified": "2025-03-20",
 "mainEntityOfPage": {
 "@type": "WebPage",
 "@id": "https://example.com/ai-crawler-response-codes"
 },
 "publisher": {
 "@type": "Organization",
 "name": "Your Company"
 },
 "description": "How to optimize HTTP response codes for GPTBot, ClaudeBot, and PerplexityBot."
}

FAQPage Schema (for AI Overviews)

{
 "@context": "https://schema.org",
 "@type": "FAQPage",
 "mainEntity": [
 {
 "@type": "Question",
 "name": "What HTTP status code should I return for GPTBot?",
 "acceptedAnswer": {
 "@type": "Answer",
 "text": "Return 200 with a Last-Modified header. Avoid 429 unless rate-limiting is necessary."
 }
 },
 {
 "@type": "Question",
 "name": "How does Perplexity handle 404 responses?",
 "acceptedAnswer": {
 "@type": "Answer",
 "text": "Perplexity caches URLs for 30 days. A 404 after indexing will remove the citation."
 }
 }
 ]
}

HowTo Schema (for step-by-step guides)

{
 "@context": "https://schema.org",
 "@type": "HowTo",
 "name": "Optimize Response Codes for AI Crawlers",
 "step": [
 {
 "@type": "HowToStep",
 "position": 1,
 "text": "Check your server logs for AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot)."
 },
 {
 "@type": "HowToStep",
 "position": 2,
 "text": "Ensure all important pages return 200 with a Last-Modified header."
 }
 ]
}

Citation Strategy

To get picked by AI models, you must control what response code the crawler sees at the exact moment it requests your page.

Serve 200 for all canonical URLs. AI crawlers often request the exact URL shown in search results. If that URL redirects (301) or returns 404, the citation is lost. Use rel="canonical" to consolidate signals.
Use 410 for permanently removed content. A 410 tells the crawler “do not re-crawl.” This prevents AI models from wasting crawl budget on dead pages and improves the freshness of your indexed content.
Implement conditional 200 with If-Modified-Since. If your content hasn’t changed, return 304 Not Modified. AI crawlers respect 304 and will keep the cached version. This reduces server load and keeps your citation active.
Avoid 503 during peak hours. If your site goes down, AI crawlers may deprioritize your domain for weeks. Use a maintenance page that returns 503 with a Retry-After header (e.g., 3600 seconds). Most AI crawlers will retry once.
Monitor 429 responses per crawler. Set up separate rate limits for each AI crawler in your web server config. For example, in Nginx:

if ($http_user_agent ~* (GPTBot|ClaudeBot|PerplexityBot)) {
 set $ai_crawler 1;
}
limit_req zone=ai_crawler burst=5 nodelay;

Case Studies

Case Study 1: SaaS Documentation Site (ChatGPT Citations)

A B2B SaaS company noticed its API docs were not appearing in ChatGPT answers. Analysis showed that GPTBot was receiving 302 redirects from http://docs.example.com to https://docs.example.com. After implementing a 301 permanent redirect and ensuring all pages returned 200 with Last-Modified, citation frequency increased by 340% over 8 weeks. The site also added Article schema with dateModified set to the actual update date.

Case Study 2: News Publisher (Perplexity Citations)

A regional news site saw a 60% drop in Perplexity citations after a server migration that caused intermittent 503 errors. PerplexityBot had cached the old URLs and received 503 on three consecutive crawls. The site implemented a CDN with origin shielding and returned 200 for all article pages. Within 4 weeks, citations recovered and exceeded pre-migration levels by 20%. The publisher also added citation_author meta tags to each article.

How to Optimize Your Site’s Response Codes for AI Crawlers

Follow these steps to ensure AI crawlers see the right response codes at the right time.

Audit your server logs. Extract requests from GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Count how many times each returned 200, 301, 404, 429, 503. Use a tool like grep or a log analyzer.

Fix all 404s for AI-crawled URLs. For each URL that returned 404, either restore the content (return 200) or implement a 301 redirect to a relevant live page. Do not leave 404s for URLs that were previously cited.

Replace 302 redirects with 301. AI crawlers treat 302 as temporary and may not follow them. Use 301 for all permanent moves. Test with curl -I -A "GPTBot" https://example.com/page.

Set Last-Modified and ETag headers. Configure your web server to emit these headers for all HTML pages. For dynamic sites, set Last-Modified to the last database update timestamp.

Implement conditional 304 responses. Enable If-Modified-Since handling. AI crawlers that receive 304 will keep their cached copy, reducing server load and preserving citation status.

Rate-limit AI crawlers with robots.txt first. Add Crawl-Delay: 10 for GPTBot, ClaudeBot, and PerplexityBot. Only use 429 as a fallback if they exceed the delay.

Add structured data to all 200 pages. Use the JSON-LD examples above. Validate with Google’s Rich Results Test or schema.org validator.

Monitor response codes weekly. Set up alerts for any AI crawler that receives a 5xx or 429. Investigate and fix within 24 hours.

Frequently Asked Questions

What HTTP status code should I return for AI crawlers if my page is temporarily down?

Return 503 with a Retry-After header set to at least 3600 seconds. Most AI crawlers will retry once. Avoid 429 for temporary outages because it signals rate limiting, not unavailability.

Does returning 304 help AI crawlers?

Yes. AI crawlers that support conditional requests (GPTBot, ClaudeBot) will keep the cached version and not re-download the content. This saves bandwidth and keeps your citation active without a full 200 response.

How many redirect hops do AI crawlers follow?

GPTBot follows at most one redirect. PerplexityBot follows up to two. ClaudeBot follows one. Always use a single 301 redirect to the final URL.

Should I block AI crawlers with 403 Forbidden?

No. A 403 tells the crawler the content exists but is forbidden. AI models may still attempt to cite the URL and show an error. Use robots.txt disallow instead.

Can I use 410 Gone for old content that I want AI models to forget?

Yes. 410 signals permanent removal. AI crawlers will remove the URL from their index and stop crawling it. This is better than 404 because it explicitly tells the crawler not to re-check.

What is the best response code for a paywalled article?

Return 200 with the full article content, but use structured data to indicate a paywall (e.g., hasPart with isAccessibleForFree: false). AI crawlers may still index the abstract. Returning 401 or 403 will cause the page to be ignored entirely.

Sources

Checklist: AI Crawler Response Codes Optimization

Audit server logs for AI crawler user agents and count response codes.
Fix all 404s for URLs that have been cited or linked from AI answers.
Replace 302 redirects with 301 for all permanent moves.
Set Last-Modified and ETag headers on all HTML pages.
Enable conditional 304 responses for AI crawlers.
Add Crawl-Delay in robots.txt for GPTBot, ClaudeBot, PerplexityBot.
Implement rate limiting with 429 only as a last resort.
Add Article, FAQPage, or HowTo schema to all content pages.
Use 410 Gone for permanently removed content.
Monitor response codes weekly and set up alerts for 5xx and 429.