By 2026, AI crawlers like GPTBot, PerplexityBot, and anthropic-ai have spread across the entire web. Yet many websites have unintentionally blocked these crawlers through robots.txt, leaving their content unindexable by AI and completely "invisible" in AI search.
Who this is for: Anyone running a website who wants their content to be picked up by AI search engines. What you'll learn: How to identify and fix accidental blocks of AI crawlers in robots.txt; how to configure Allow rules correctly; and how to use llms.txt to proactively guide AI crawlers. The main AI crawlers to know:
- GPTBot (OpenAI/ChatGPT)
- PerplexityBot (Perplexity AI)
- anthropic-ai (Claude)
- Googlebot (Google AI Overview)
- bingbot (Bing Copilot)
Configuration example (recommended approach):
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: anthropic-ai
Allow: /
Note: If you're concerned about server load, you can add Crawl-delay: 10 to throttle the request rate.