Is Your robots.txt Blocking ChatGPT and Claude From Your Site?
Yes, this happens more often than you would expect — usually by accident. A generic "block all bots" rule added for an unrelated reason (stopping scrapers, reducing server load) can quietly disallow GPTBot, ClaudeBot, Google-Extended, and PerplexityBot along with everything else, making your site invisible to every major AI assistant without anyone realizing it.
The crawlers that actually matter
Each AI platform uses its own named crawler, and robots.txt rules are matched per user-agent, so blocking one does not automatically block another:
- GPTBot, OAI-SearchBot, ChatGPT-User — OpenAI / ChatGPT
- ClaudeBot, anthropic-ai, Claude-Web — Anthropic / Claude
- Google-Extended — Google Gemini (separate from regular Googlebot)
- PerplexityBot, Perplexity-User — Perplexity
- CCBot — Common Crawl, which feeds many AI models indirectly
How to check your own site in under a minute
Visit yourdomain.com/robots.txt directly in a browser. Look for any "User-agent: *" block with a blanket "Disallow: /" — that blocks everything, including every AI crawler above, unless a more specific rule exists for a given bot elsewhere in the file. Then look specifically for any User-agent line naming one of the bots above, followed by Disallow rules.
A commonly missed detail: meta robots and X-Robots-Tag
robots.txt is not the only way to block access. A page can carry a <meta name="robots" content="noindex"> tag, or an X-Robots-Tag HTTP header with the same effect, and both are honored independently of robots.txt. A site can have a perfectly open robots.txt and still be invisible because every page individually says "do not index me."
How to fix it
Remove or narrow any Disallow rule that catches the AI crawler user-agents unless you have a specific, deliberate reason to block a given one (some site owners do intentionally block GPTBot to opt out of AI training, which is a legitimate choice — the key is that it should be a decision, not an accident). If you want AI systems to read your site, make sure nothing in robots.txt, meta robots, or X-Robots-Tag contradicts that intent.
Frequently asked questions
If I block GPTBot, does that also block ChatGPT-User?
No — they are separate user-agents with separate purposes (GPTBot is used for training data collection, ChatGPT-User for live browsing during a conversation). Each needs its own explicit rule if you want to control them independently.
Is it ever a good idea to block AI crawlers on purpose?
Yes, for some sites — particularly ones with a business model built on the content itself, or specific legal/licensing concerns. The point of this article is not that blocking is always wrong, but that it should be an intentional choice, not an accidental side effect of a generic bot-blocking rule.
How can I check every AI crawler at once instead of manually reading robots.txt?
RankLLM's free scan checks robots.txt, meta robots, and X-Robots-Tag against every major AI crawler in one pass and tells you plainly which ones are blocked and why.