nqzaiBlogTag archive
A precise, source-backed framework for telling apart robots.txt disallows, WAF/firewall rules, rate limiting, and CDN bot-management blocks — with the exact crawler identities, IP verification methods, and diagnostic steps to confirm which one is stopping GPTBot, ClaudeBot, or PerplexityBot from reaching your pages.
A working robots.txt for AI crawlers isn't a file you write once — it's a policy you maintain as OpenAI, Anthropic, Perplexity, Google, and Apple keep splitting one bot into three with different rules for training, search, and live answers.
A technical breakdown of ClaudeBot, Claude-User, and Claude-SearchBot, how robots.txt controls each one, and what's actually verified about how Claude finds and cites content.
