nqzaiBlogTag archive
A precise, source-backed guide to reading 200, 301/302, 403, 404, 429, and 5xx responses through the lens of GPTBot, ClaudeBot, and PerplexityBot — plus the QA process to catch AI-crawler access failures before they cost you visibility.
A precise, source-backed framework for telling apart robots.txt disallows, WAF/firewall rules, rate limiting, and CDN bot-management blocks — with the exact crawler identities, IP verification methods, and diagnostic steps to confirm which one is stopping GPTBot, ClaudeBot, or PerplexityBot from reaching your pages.
A practical, consolidated crawl-and-retrieval audit for AI search engines — robots.txt, rendering, status codes, structured data, sitemaps, canonicals.
A single robots.txt check tells you what you asked crawlers to do, not what they actually did. Here's a three-layer audit method — rules, enforcement, and evidence — for finding out whether AI crawlers can really reach your site.
A working robots.txt for AI crawlers isn't a file you write once — it's a policy you maintain as OpenAI, Anthropic, Perplexity, Google, and Apple keep splitting one bot into three with different rules for training, search, and live answers.
A technical breakdown of ClaudeBot, Claude-User, and Claude-SearchBot, how robots.txt controls each one, and what's actually verified about how Claude finds and cites content.
How Perplexity actually retrieves the live web, how PerplexityBot and Perplexity-User differ, and what's documented (and disputed) about its citations.
Canonical tags tell Google which URL wins a duplicate-content contest, but several AI crawlers skip that step entirely — so ungoverned utm_*, session, and filter parameters can splinter one page into competing, uncredited copies.
