Why Isn't My Site Showing Up in ChatGPT? Here's What I Fixed

The setup

I run a site for a niche B2B services business — decent traffic, ranked reasonably well on Google for its core terms, nothing broken about the SEO. Then a client mentioned they'd asked ChatGPT for a recommendation in that exact niche, and our client's competitor came up. Not us. Not even close — not on the first several follow-up prompts either.

That got me checking. I ran a dozen realistic prompts a potential customer might actually type into ChatGPT, Perplexity, and Claude. Zero mentions of the site across all three, despite ranking on page one of Google for nearly the same queries.

Where I went looking first

My first assumption was thin content — maybe the pages just weren't detailed enough for an LLM to want to cite them. I rewrote two of the strongest service pages to be more comprehensive, waited a couple weeks for re-crawling, and tested again. Still nothing. That ruled out "the content itself isn't good enough" and told me the problem was upstream of content quality — something about whether the content was even reachable or parseable in the first place.

What was actually happening

I checked robots.txt next, mostly on a hunch, and found a block right away:

User-agent: GPTBot Disallow: /

User-agent: CCBot Disallow: /

User-agent: * Allow: /

At first I assumed someone on the team had added this deliberately, maybe out of AI-training concerns. Turned out that wasn't it — the block matched, line for line, the default AI-crawler exclusion list shipped by a security plugin the site had installed over a year earlier, for unrelated bot-mitigation reasons. Nobody had opted into blocking AI crawlers specifically. The plugin's default config did it silently, months before any of this AEO stuff was even a consideration, and it had just been sitting there ever since. That's the part that actually got me — this wasn't a decision anyone made. It was a default nobody looked at.

I pulled server logs to confirm the practical effect — filtered for AI-crawler user agents over the prior 30 days:

const aiCrawlerHits = logs.filter(l => ['GPTBot', 'CCBot', 'ClaudeBot', 'PerplexityBot'].some(bot => l.userAgent.includes(bot) ) ); // aiCrawlerHits.length: 0

Zero hits, in a month, from any AI crawler. Not "rarely." Zero. Google ranks the site because Googlebot was never touched by this plugin's list. ChatGPT can't cite the site because GPTBot never even got in the door. Two completely separate crawler ecosystems, and the site had quietly opted out of one of them without anyone deciding to.

The fix

1. Fix robots.txt first — and isolate it. Removed the disallow rules for GPTBot, CCBot, ClaudeBot, and PerplexityBot, left them explicitly allowed rather than just deleting the block, and then deliberately shipped only this change and waited two weeks before touching anything else. I wanted to know what the crawler-access fix alone was worth before adding more variables.

User-agent: GPTBot Allow: /

User-agent: ClaudeBot Allow: /

User-agent: CCBot Allow: /

User-agent: PerplexityBot Allow: /

Within 48 hours, server logs showed GPTBot and ClaudeBot hits for the first time in the entire 30-day log window I'd checked. After the two-week isolation period, re-running the same dozen test prompts got the site mentioned in 4 of 12 — up from 0, from this change alone.

2. Then add structured data. With access confirmed, I added FAQPage schema to the core service pages, since FAQ-formatted content maps closely onto how people phrase questions to ChatGPT:

{ "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{ "@type": "Question", "name": "Does [service] work for small businesses?", "acceptedAnswer": { "@type": "Answer", "text": "Yes — most clients in this size range see results within..." } }] }

3. Publish an llms.txt file, as a low-cost bet, not a proven fix. I'll say this plainly: I don't have clean evidence this one moved the number on its own, and I'm skeptical anyone does yet — adoption across AI tools is inconsistent and still early. It's a plain-text summary at the site root pointing crawlers to the pages worth reading, similar in spirit to a sitemap. It cost almost nothing to add, so I added it, but I'm not claiming it's why the numbers improved further.

Result

After all three changes, the same test prompts got the site mentioned in 7 of 12 — up from the 4 of 12 the robots.txt fix alone produced. That gap is most likely the schema change, since it's the only other change with a plausible mechanism; I can't cleanly separate its effect from llms.txt since they shipped close together, which is exactly why I isolated the first fix and wouldn't claim the same precision for the other two.

What I took away from this

Traditional SEO and AI-answer visibility look similar on the surface — both are about being findable — but they run on separate infrastructure, and the block on this site wasn't even a deliberate one. It was a plugin default that predated anyone caring about AI crawlers at all, sitting untouched for over a year. That's the actual lesson: these blocks are rarely a decision. They're usually a default nobody audited, from a tool installed for a completely different reason. If your content isn't showing up in AI answers despite ranking fine on Google, don't start by rewriting content. Start by checking exactly what's in robots.txt right now, and ask whether anyone actually chose it — or whether it just came with something else you installed.