89 publishers blocked Google-Extended to stay out of AI answers. Google says it does not do that

697 words · about 3 min

Google-Extended is the name publishers add to robots.txt when they want out of Google's AI. Across 1,020 small ad-funded publishers, 89 of them — 8.7% — block it. Google's own documentation says it controls whether your pages train future Gemini models, and that it 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.' AI Overviews are part of Google Search. So a publisher who blocked Google-Extended to stay out of the answer box has not left the answer box; they have opted out of training and kept everything else. Nothing tells them, because a robots.txt file never reports back.

How it was measured: We read 1,020 robots.txt files and worked out what each one actually permits at the site's front door, following RFC 9309 group rules rather than searching for crawler names. 89 of those files disallow Google-Extended at the root. What the token does is quoted word for word from Google's own crawler documentation, fetched on 2026-09-03. We did not measure whether any of those 89 publishers intended what we describe — that is the part we cannot see, and the note says so.
Last checked: 2026-09-03

Listen to this Playing read from this page, nothing added

There is a line publishers add to robots.txt when they decide they do not want Google's artificial intelligence using their work. It names Google-Extended. It is the line most commonly recommended for the job, and 89 of the 1,020 publishers we read have it.

It does not do the job.

What Google says it does

From Google's own list of its crawlers, read on 2026-09-03:

Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models.

And then, in the same entry, the sentence that matters:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

AI Overviews — the answer Google sometimes writes at the top of the results page — are part of Google Search. So blocking Google-Extended does not take a site out of them. It takes the site out of Gemini's training, which is a different thing, decided at a different time, with different consequences.

What that means for the 89

A publisher who added that line to stay out of the answer box is still in the answer box. They have given up something — their pages no longer train Gemini — and received nothing they were aiming for.

We cannot see what any individual publisher intended. A file does not record a reason. Some of the 89 may have wanted exactly what they got: out of training, in search. That is a coherent position and it is the one the token is designed to express.

But it is worth knowing which door you closed. Nothing about robots.txt tells you what happened after you wrote in it. There is no confirmation, no report, and no way to check from the outside whether the effect you wanted is the effect you got.

The pattern this belongs to

We have measured this shape before on the same 1,020 files. Publishers block the crawler that would have sent them a reader eleven times more often than the one that only files pages away for search. Three in ten of the files carry wording a plugin or an ad network put there rather than a decision the owner made.

The common thread is not carelessness. It is that this file has no feedback. You write a rule, and nothing ever tells you whether it did what you meant. Every other lever a publisher pulls reports back — traffic moves, revenue moves, a dashboard changes. This one is silent, so a wrong belief about it can sit in place for years.

What to do with your own file

Open it. It is at your domain followed by /robots.txt, and anyone can read it, including you.

If you find Google-Extended there, decide which thing you actually wanted. If it was "do not train on my work", it is doing that. If it was "keep me out of the AI answer at the top of the results", it is not, and Google publishes no token that does — their documentation says the way to control what appears in those answers is the same as for any other search result.

Our free reader will work out what your file permits, resolved the way a machine reads it rather than by searching for names, and tell you where you sit against these 1,020 publishers. It stores nothing.

Limits

This counts what 1,020 files say, drawn from a frame of 16,076 domains that two ad networks publish as their own publishers. It describes small ad-funded content businesses and nothing else.

We did not measure intent. We measured that 89 files block a token, and we quote what Google says that token does. Whether those publishers believed it did something else is not visible in a file, and this note does not claim it is. The finding is the gap between what the token is widely used for and what its owner documents it doing.

We also did not measure whether Google honours the token. That is not visible from the outside either.

Tracked as AI-CRAWLER-POLICY-VENDOR-DEFAULT. If a number here looks wrong, tell us and we will check it against the source.