Most sites that blocked an AI bot blocked the wrong name

1,091 words · about 5 min

If you run a website, you can put a file at yoursite.com/robots.txt that tells bots what they may read. It is how you say no to AI companies.

We read that file on 1,020 small websites that make their money from ads.

Most of them, 776, say nothing about AI at all. This is about the 244 that tried.

Of those 244 sites, 167 blocked a name that the company behind it does not list anywhere. They sat down, decided they did not want AI reading their work, typed a name in, and picked one the company does not use.

The clearest example

Anthropic makes Claude. It has a page listing the bots it sends out. We read it on 2026-09-03. There are three:

ClaudeBot … helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.

Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.

Claude-SearchBot navigates the web to improve search result quality for users.

That is the whole list, from Anthropic's own page. Two names are missing from it: anthropic-ai and claude-web.

Those are the two that site owners actually block.

126 sites block anthropic-ai or claude-web, and never mention ClaudeBot. That is just over half of everyone who wrote an AI rule. Only 101 sites out of 1,020 block ClaudeBot — the one Anthropic says takes content for training.

So the most common way to block Anthropic is to type a name Anthropic does not use, and leave out the one it does.

We are not saying those names are dead

This part matters, and it is where it would be easy to go too far.

No company puts out a notice when it stops using a bot name. So nobody can say for certain that these names do nothing. The people who track bots are careful about it too. Here is how one bot directory describes anthropic-ai:

a unconfirmed agent possibly used by Anthropic to download training data for its LLMs

It also still lists Claude-Web. Unconfirmed is not the same as dead, and we are only claiming the first thing.

What we can show is simple: the company does not list it. If you block a name your vendor does not publish, you cannot tell whether it works. You will never find out either, because the file never answers back. That is the real problem with robots.txt as a way to say no. It is a note left on the door.

The five names

We checked the Anthropic names against Anthropic's own page, quoted above. We have not done the same check for Cohere or Webz.io. Those rows just count what site owners wrote.

Where the wrong names came from

Of the 167 sites using an unlisted name, 111 also carry a block of text that a tool wrote for them — 105 with Raptive's, 6 with Yoast's.

Be careful with that number, and we will be too. It shows the two things turn up together. It does not prove the tool wrote the AI lines: a Yoast block often has no AI rules in it at all.

The plain explanation is the likely one. These lists get passed around. Someone wrote a good block list, it was right at the time, and it has been copied ever since. Nobody went back to check it against the companies' own pages.

Why it costs you something

Blocking a name that nothing answers to is free. What it costs you is the decision you think you already made.

If you believe you blocked Anthropic, you filed it away. You will not look again. If you later want to trade AI access for something, you think you have a switch to flip, and it is not wired to anything. And if you wanted your work kept out of training, it is not.

The same confusion runs the other way. We measured a matching mistake with Google, where 89 of these sites block Google-Extended believing it keeps them out of AI answers. Google's own page says it controls Gemini training and "does not impact a site's inclusion in Google Search". Same shape: a name copied off a list, doing something other than what the owner wanted.

There is a worse version of this. Across all 1,020 sites, owners block the bot that reads a page to answer someone's question and links back far more often than the one that takes work for training. That is the block that costs you readers instead of protecting your work.

What to do, in four minutes

1. Open your own file. Go to yoursite.com/robots.txt and read it. Not your plugin's settings screen — the file. What is in there is what bots follow. 2. Check each name against the company's page. Anthropic's is here. OpenAI and Google publish theirs too. If a name is not on the company's own list, you cannot tell that blocking it does anything. 3. Treat training and answers as two separate choices. ClaudeBot and GPTBot take content for training. Claude-User and ChatGPT-User fetch a page because a person asked a question, then link back to you. Blocking the second one takes you out of answers that would have sent you readers. 4. Check the order of the lines. If a file lists thirty bot names in a row and then allows them, it has allowed all thirty. Names stacked together share one rule. That is how the standard works, and it catches people out.

Our free tool reads your file and tells you which of these apply to it. Nothing you type leaves your browser.

Check this yourself

Every number here comes from a table we publish in full — all 1,020 sites, one row each, showing the names in every file and which ones are actually blocked:

- The data, as a CSV, free to reuse - Archived with a DOI: 10.5281/zenodo.22280370

We do not publish which site is which. Everything the measurement used is there, so you can redo the sums, but you cannot work out whose site a row is.

Tracked as AI-CRAWLER-POLICY-VENDOR-DEFAULT. If a number here looks wrong, tell us and we will check it against the source.