Most people block the wrong robot

Every company running an AI product runs several crawlers. They do different jobs, they answer to different rules, and the difference between them is where most published advice in this field is simply wrong.

Every row in this table comes from the operator’s own documentation.

Two corrections that matter more than the rest of the table.

Google-Extended never kept you out of AI Overviews. It has been recommended for that purpose in guide after guide since 2023. Google’s own crawler documentation is explicit that it governs Gemini app training and Vertex AI grounding. For AI Overviews, Google’s requirement is that a page is “indexed and eligible to be shown in Google Search with a snippet” — nothing more. Until mid-2026 there was simply no way to appear in Google Search and not in AI Overviews. There is one now, in Search Console, as of 31 August 2026. Google-Extended was never it.

OAI-SearchBot is the opposite mistake, and it is more expensive. It is not a training crawler. OpenAI’s documentation says sites opted out of it “will not be shown in ChatGPT search answers.” Anyone who added it to a blocklist while trying to keep their content out of training has removed themselves from ChatGPT altogether. We have found this on live sites, added by well-meaning developers who read a list of AI user-agents and blocked all of them.

Now the trap that isn’t in your robots.txt at all.

There is a third category of crawler: the ones that fetch a page because a person asked. OpenAI’s ChatGPT-User, Perplexity’s Perplexity-User, Meta’s meta-externalfetcher. Their documentation states that robots.txt rules may not apply, because a human initiated the request rather than a scheduled crawl. OpenAI’s wording: “Because these actions are initiated by a user, robots.txt rules may not apply.”

Which means your robots.txt is not what stops them. Your security layer is.

The service protecting your site from attacks — the WAF, the bot manager, the rate limiter — frequently cannot distinguish an assistant fetching a page for a customer from a scraper, and blocks both. It is the single most common invisible failure we find, it does not appear in any file you can read, and the person who configured it usually left two years ago.

(Anthropic is the exception worth noting: it commits to honouring robots.txt for Claude-User as well.)

Does blocking even work? Less reliably than you’d think. An analysis of four million AI citations found that among top news sites blocking a given bot, 70.6% to 92.3% still appeared in citations anyway. That data is correlational and heavily confounded — news sites are cited because they’re news sites, and a citation can come from a third party’s summary or an index snapshot rather than a live fetch. But it does falsify the simple promise that a line in robots.txt makes you disappear.

The practical position for a business that wants to be found, which is almost every business reading this, is short:

Allow the retrieval and search crawlers. Allow the user-triggered fetchers, at the firewall as well as in robots.txt. Block the training crawlers only if you have an actual reason to — and “I don’t want AI to have my content” is a reason, but be clear it costs you nothing in visibility and gains you nothing either, because the models were trained before you decided.

And check the edge. Not the file. The edge.

One documented gap, flagged rather than glossed. Claude in Chrome became generally available on 26 August 2026 — an agent that reads pages, clicks and fills forms using a person’s existing logins. Anthropic publishes no user-agent for it, no site-owner controls, and no robots.txt guidance. If it visits your site you currently have no documented way to identify it. Nobody has solved this yet.