Robots.txt Rules for AI Crawlers: A Governance Guide
A working robots.txt for AI crawlers isn't a file you write once — it's a policy you maintain as OpenAI, Anthropic, Perplexity, Google, and Apple keep splitting one bot into three with different rules for training, search, and live answers.
