Publishers block the AI visitor that would have sent them a reader at least ten times more often than the one that only files them away

2,283 words · about 10 min

Watch it plot 65s · narrated · the same figures as above

Every website carries a small text file called robots.txt. It sits at the front of the site, anyone can read it, and its whole job is to tell automated visitors which parts of the site they may fetch. Those automated visitors are called crawlers, and each one gives its name when it arrives.

Small publishers are not deciding what AI companies may do with their sites. Most have no rule at all, and among those who do, the commonest single act is to block the crawler that would have sent them a reader.

11.9% of the sample blocks ChatGPT-User. 0.4% blocks OAI-SearchBot.

Those two crawlers belong to the same company. OpenAI's own documentation says ChatGPT-User is used "for certain user actions in ChatGPT and Custom GPTs", where when users ask a question "it may visit a web page with a ChatGPT-User agent" — it is the visit that happens because someone asked a question your page answers, and it is how the answer ends up carrying a link back to you. OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features", and of that one OpenAI adds: "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file."

So one of them can send you a reader and the other one files you away. Blocking the first while allowing the second is backwards, and it is not plausible that this many publishers meant it.

How much more often, exactly

121 of 1,020 files against 4 of 1,020. Divide those and you get 30, which is the best single guess and is nowhere near as solid as it looks, because it rests on a denominator of four files.

So here is what we can actually stand behind. Take the 1,020 files, draw 1,020 of them back out at random with repeats allowed, count both crawlers again, and do that twenty thousand times. That is a way of asking what this measurement would have said if a different thousand publishers had been picked, and it assumes nothing about the two counts moving together — which matters, because they are counts over the same files, and three of the four sites that block OAI-SearchBot block ChatGPT-User as well. Nineteen times in twenty, the answer lands between 14.5 and 134. In one resample in fifty, no site at all blocks OAI-SearchBot and the multiple has no top end.

A second method, which shares no assumption with the first: take the range each percentage is separately known to and divide the low end of one by the high end of the other. That is deliberately too wide, and it gives 9.9 to 127.

Both floors clear ten. So the honest sentence is "at least ten times more often", and the two percentages carry the finding better than any multiple does. What this note will not do is state 30× as though it were measured.

What the whole sample looks like

And by crawler:

The last column is the range each figure is really known to. It is the reason the bottom four rows should be read as "almost nobody", and not as an ordering.

What re-measuring changed, and what it did not

This was first published on 2026-09-01 from 106 files, fetched one at a time. Since then our own crawler has been reading the same kind of file every night, so the same question could be asked again of nearly ten times as many publishers without asking anybody for anything new.

Nothing turned over. The headline share — publishers with no rule at all — moved from 76% to 76.1%. The two figures that moved most, supplier wording from 34% to 29.9% and blocking a citing crawler from 14% to 12.1%, both moved less than the first sample's own margin of error, which was about eight points wide.

What did change is precision. At 106 files these shares were known to within roughly eight points; at 1,020 they are known to within two or three.

And the multiple at the top of this note is gone. It read 15× — 15 files of 106 against 1 of 106 — which is the right division of the right counts and still should not have been printed, because the denominator was a single file. At 1,020 files the point estimate is 30 and the range runs from about 10 to well over 100, so 15 sits outside what the larger sample supports. A number that survives being divided correctly can still be a number nobody should have stated, and the tell was in the first sample all along: 1 of 106 has a range of 0% to 5%, and no ratio built on it could have been worth two significant figures.

The check we had to do first, and nearly got wrong

A bigger sample is only better if it is a sample of the same thing.

Our crawler works through its list least-recently-tried first, so reading "whatever it has collected so far" is a sample of whoever it happened to reach, which is not random with respect to anything. An earlier pass on this same store held 597 publisher files where all 597 came from one of the two ad networks and none from the other. That is invisible in a total. It would have published one network's customers as the whole list.

So this measurement does not read everything in the store. It reads only the 1,200 domains that were drawn once, in advance, with each network given its share of the list, and only the files collected on a single day when all 1,200 were tried. Eleven other files in the store are readable and were left out because they are not part of that draw. All eleven are from one network, and all eleven say nothing about AI crawlers — so including them would have nudged the headline upwards using files that nothing chose at random.

It matters here more than it usually would, because the two networks' publishers genuinely differ: 88% of files from one of them say nothing about AI crawlers, against 60% from the other. Re-weighting so each network counts for its share of the full list, rather than its share of the files that answered us, moves every figure in the table above by less than half a point.

Why the numbers come out this way

Nearly a third of the files carry a tell-tale sign that a supplier wrote the AI section — an ad network putting a block into its publishers' files, or a WordPress plugin generating a list. The publisher did not choose it and in most cases has never worked out what it does.

The lists these templates carry are also out of date. 16.4% of the sample names at least one crawler that appears in no company's current documentation. 2.3% of the files block only names that no longer exist, which means they exclude nothing at all while looking, to anyone who opens them, like a policy.

The part that is easy to get wrong

Naming a crawler is not blocking it.

The file is read in blocks. A block is one or more lines naming crawlers, followed by the rules that apply to them — and the rules only start once the run of names ends. The standard that defines this, RFC 9309, puts it plainly: a block is "One or more user-agent lines that are followed by one or more rules. The group is terminated by a user-agent line or end of file."

So a file that stacks thirty crawler names on top of one permissive rule has permitted all thirty. It reads as a wall and works as an open door. 2.7% of this sample does exactly that with at least one crawler.

That is why the figures above are worked out rather than counted. Searching the files for User-agent: GPTBot would have reported those files as maximally strict when they are the opposite — and it is the mistake anyone checking their own file by eye will make.

What to do about it, if anything

There is a real decision here and it is not the one the templates make.

Blocking a crawler that collects pages to train a model is a decision about licensing your work. Your material is used to build a model and you are not paid for it; declining is coherent, and 21.2% of this sample declines.

Blocking a crawler that answers a question and links back is a decision about traffic, and it runs the other way. It does not protect your content — that content is already in the training sets, or it is not. What it does is take you out of an answer that would have named you and linked to you.

Whichever you want, the thing worth doing is opening the file and reading it as a machine would. Ours is one place to do that, and so is anyone else's.

Limits

This is 1,020 readable files, from a sample of 1,200 drawn with a fixed starting word from a list of 16,076 domains that two ad networks publish as their own publishers. It describes small, ad-funded content businesses and nothing else — the large publishers with signed AI deals behave completely differently, and they are where every other published figure on this subject comes from.

180 of the 1,200 gave us nothing we could read. They are left out rather than counted as permitting everything, because a refusal — an HTTP 403, which 107 of them gave — is a fact about a firewall and not about a policy, and counting them the other way would have moved the headline in the direction that flatters it. Ten of the 180 answered normally with an empty file; those are left out too, on the same rule the first measurement used, and counting them as silent instead would put the headline share at 76.3% rather than 76.1%.

The list we drew from is publishers on two ad networks, not the web. A publisher with no ad network is not in it.

What we measured is what the files say, not what happens next

This counts rules written in files. It does not measure whether any AI company obeys them, and one of them says plainly that it may not.

OpenAI's own documentation for ChatGPT-User, read again on 2026-09-02:

"When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." … "Because these actions are initiated by a user, robots.txt rules may not apply."

That qualifies the consequence, and it is worth being exact about which part it touches and which part it does not.

It does not change the finding. At least ten times as many publishers wrote a rule against the crawler that fetches a page to answer somebody's question as against the one that files pages away for search. That is a fact about what is in the files, most of those lines were written by a plugin or an ad network, and it is the same fact either way.

It does change what follows from it. We wrote that blocking the first crawler takes a publisher out of an answer that would have linked back to them. If the rule may not be applied, that consequence is weaker than the sentence implies. The honest version: they have written a rule that asks to be left out of it, and whether the request is honoured is not something we measured and not something we can see from a file.

Two more limits from the same page, both of which cut against reading too much into either number:

"ChatGPT-User is not used for crawling the web in an automatic fashion." "ChatGPT-User is not used to determine whether content may appear in Search."

So the two agents are not two settings on one dial. Comparing how often each is blocked is still the right comparison — they are the same company's crawlers and one cites while the other does not — but it is a comparison of what publishers asked for, not of what they received.

Download the data behind this 1,020 rows · CSV · CC BY 4.0 · no account, no email

Tracked as AI-CRAWLER-POLICY-VENDOR-DEFAULT. If a number here looks wrong, tell us and we will check it against the source.