nqzaiBlogTag archive
A precise, source-backed guide to reading 200, 301/302, 403, 404, 429, and 5xx responses through the lens of GPTBot, ClaudeBot, and PerplexityBot — plus the QA process to catch AI-crawler access failures before they cost you visibility.
A precise, source-backed framework for telling apart robots.txt disallows, WAF/firewall rules, rate limiting, and CDN bot-management blocks — with the exact crawler identities, IP verification methods, and diagnostic steps to confirm which one is stopping GPTBot, ClaudeBot, or PerplexityBot from reaching your pages.
llms.txt isn't a one-time SEO file. Here's how to govern it on an ongoing basis — and what the adoption data actually shows in 2026.
A single robots.txt check tells you what you asked crawlers to do, not what they actually did. Here's a three-layer audit method — rules, enforcement, and evidence — for finding out whether AI crawlers can really reach your site.
A working robots.txt for AI crawlers isn't a file you write once — it's a policy you maintain as OpenAI, Anthropic, Perplexity, Google, and Apple keep splitting one bot into three with different rules for training, search, and live answers.
A precise guide to filtering server logs for GPTBot, ClaudeBot, and PerplexityBot traffic, verifying which requests are real, and tracking the metrics that actually predict AI-search visibility.
A technical breakdown of ClaudeBot, Claude-User, and Claude-SearchBot, how robots.txt controls each one, and what's actually verified about how Claude finds and cites content.
CSS, fonts, images, and blocked scripts don't stop AI crawlers from fetching a page — but they can stop that page from being read correctly. Here's what actually breaks retrieval, backed by real crawler data, and how to audit for it.
Beyond "does the sitemap validate" — how to audit orphan URLs, lastmod accuracy, and segmentation at scale, and why AI crawlers make sitemap hygiene matter more than it used to.
