Is your site invisible to ChatGPT? Check one file.

For fifteen years, "being found" meant one thing: ranking on Google. That's changing fast. A growing share of people now ask ChatGPT, Perplexity or Google's AI answers what to buy, where to go, who to hire — and the answer names a handful of sources. If your site isn't one of them, you're not on page two. You're not in the room.

Here's the part that surprises people: the most common reason a site gets left out isn't weak content. It's that the site is accidentally blocking the AI crawlers — and nobody realises it, because it's one quiet line in a file called robots.txt.

How AI answers actually find you

AI answer engines read the web with named crawlers, each with its own user-agent — the same mechanism Google has used for decades. The ones that matter right now:

• GPTBot and OAI-SearchBot — OpenAI (ChatGPT and its search)
• ChatGPT-User — fetches a page when a ChatGPT user's question needs it live
• ClaudeBot — Anthropic
• PerplexityBot — Perplexity
• Google-Extended — Google's control for AI training and grounding
• CCBot — Common Crawl, a dataset many models are built from

Your robots.txt — a plain text file at yoursite.com/robots.txt — tells each of these whether it's allowed in. Block the crawler, and you can't appear in that engine's answers. It's that direct.

The accidental block

I see two versions of this constantly.

The over-eager "Google only" rule. Someone wanted to keep scrapers out and wrote a policy that allows Googlebot and disallows everything else. That "everything else" now includes every AI crawler. The site meant to protect itself and instead made itself invisible to the fastest-growing source of discovery on the web.

The 2023 panic block. When AI training crawlers first appeared, a lot of teams rushed to block them on principle — fair enough at the time. But that decision got made once, in a hurry, and never revisited. Two years later the same rule is quietly keeping the site out of answers its owner would now love to be in.

Either way, the fix is to decide on purpose instead of by accident.

Blocking them can be the right call — if it's a choice

To be clear: keeping AI crawlers out is a legitimate decision. If your value is your content and you don't want it feeding a model for free, blocking the training crawlers is reasonable. The point isn't "always let them in." The point is know which you're doing.

And it's not all-or-nothing. You can allow the crawler that fetches a page to answer a user's question right now while blocking the one that scrapes for training. Being cited when someone asks about you is a different thing from having your archive vacuumed into a dataset — and you're allowed to want one without the other.

The truth about llms.txt

You'll have seen advice to add an llms.txt file to "get cited by AI." Let me save you some false hope.

llms.txt is a proposed format for handing AI models a clean, curated map of your site. The idea is reasonable. The reality, as of 2026, is that no major AI crawler officially uses it. Google has publicly said it doesn't support it and has no plans to — one of its search engineers compared it to the long-dead keywords meta tag. OpenAI's own guidance points you back to robots.txt for controlling crawlers. Studies watching hundreds of millions of AI-bot visits find the file barely gets requested.

So: it's cheap to add and does no harm, and if the standard catches on you're early. Add it if you like. But don't believe anyone who tells you it's the lever that gets you cited — because today it isn't. The thing that actually decides whether an AI can see you is much more boring: can its crawler reach your pages, and is your content clear enough to quote.

How to check yours

Open yoursite.com/robots.txt in your browser right now and read it. Look for any Disallow rule sitting under User-agent: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — or a blanket User-agent: * Disallow: / that catches all of them. If you find one you didn't mean, that's your invisibility, in writing.

If reading crawler rules isn't how you want to spend the afternoon, checking AI-crawler access is one of the things a scan does for you — it reports which AI engines your site currently lets in and which it turns away, so the decision is at least yours to make. Being cited by AI starts with the unglamorous step of not being invisible to it.