Most people block the wrong robot

Every company running an AI product runs several bots. They do different jobs, they answer to different rules, and the difference between them is where most published advice in this field is simply wrong.

Every row in this table comes from the operator’s own documentation.

Two corrections that matter more than the rest of the table.

Google-Extended never kept you out of AI Overviews. It has been recommended for that purpose in guide after guide since 2023. Google’s own documentation on its bots is explicit that it governs Gemini app training and Vertex AI grounding. For AI Overviews, Google asks only that a page be “indexed and eligible to be shown in Google Search with a snippet”. Nothing more. Until mid-2026 there was simply no way to appear in Google Search and not in AI Overviews. There is one now, in Search Console, as of 31 August 2026. Google-Extended was never it.

OAI-SearchBot is the opposite mistake, and it is more expensive. It is not a training bot. OpenAI’s documentation says sites opted out of it “will not be shown in ChatGPT search answers.” Anyone who added it to a blocklist while trying to keep their content out of training has removed themselves from ChatGPT altogether. We have found this on live sites, added by well-meaning developers who read a list of AI bot names and blocked every one.

Now the trap that is not in your instructions file at all.

There is a third category of bot: the ones that fetch a page because a person asked. OpenAI’s ChatGPT-User, Perplexity’s Perplexity-User, Meta’s meta-externalfetcher. Their documentation states that the rules in that file may not apply, because a human initiated the request rather than a scheduled sweep of the web. OpenAI’s own wording: “Because these actions are initiated by a user, robots.txt rules may not apply.”

Which means your instructions file is not what stops them. Your security layer is.

The service protecting your site from attacks, the one that turns away hostile programs and caps how fast anyone can request pages, frequently cannot tell an assistant fetching a page for a customer from someone scraping your content, so it blocks both. It is the single most common invisible failure we find, it does not appear in any file you can read, and the person who configured it usually left two years ago.

(Anthropic is the exception worth noting: it commits to honoring the instructions file for Claude-User as well.)

Does blocking even work? Less reliably than you’d think. An analysis of four million AI citations looked at major news sites that were blocking a given bot: between 7 and 9 in 10 of them still turned up in citations anyway. Treat that number carefully, because it only says the two things go together, not that one causes the other. News sites get cited because they are news sites, and a citation can come from somebody else’s summary or from a stored copy rather than from a live visit. But it does knock down the easy promise that one line in that file makes you disappear.

The practical position for a business that wants to be found, which is almost every business reading this, is short:

Allow the bots that fetch pages and the ones that answer searches. Allow the ones a person sets off, in your firewall as well as in the instructions file. Block the training bots only if you have a real reason to. “I don’t want AI to have my content” is a real reason, but be clear that it costs you nothing in visibility and gains you nothing either, because the models were trained before you decided.

And check the service that protects your site. Not the instructions file. That.

One documented gap, flagged rather than glossed. Claude in Chrome became available to everyone on 26 August 2026: an agent that reads pages, clicks, and fills in forms using a person’s existing logins. Anthropic publishes no name you could recognize it by, no controls for site owners, and no guidance on the instructions file. If it visits your site you currently have no documented way to identify it. Nobody has solved this yet.