Why isn't my website showing up in ChatGPT responses?

A blanket Disallow rule in robots.txt, often written years ago to block spam crawlers, can block AI crawlers too unless they're explicitly allowed.

Which AI crawlers should be explicitly allowed in robots.txt?

GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are the key ones to check for explicit Allow rules.

How do I check if my robots.txt is blocking AI crawlers?

Visit yourdomain.com/robots.txt directly and look for a blanket Disallow rule with no explicit Allow rules for AI-specific user-agents.

← All Quick Wins

AEO & GEO

Website Missing From ChatGPT? Check Your robots.txt

A robots.txt rule written years ago to block old spam crawlers often uses a blanket Disallow: / under User-agent: * — which blocks every crawler, AI systems included, unless specific bots are explicitly allowed underneath it.

The fix:

go straight to yourdomain.com/robots.txt and check for a blanket disallow with no explicit Allow rules for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot underneath it. If that's what you find, that's very likely why AI systems can't see your site at all.

Comments

No comments yet — be the first.