Dataset

Which AI crawlers 1,020 small ad-funded publishers name and block in robots.txt

What the people who run small, advertising-funded websites have actually told the AI companies they may read. Every site can publish a file called robots.txt naming which automated readers may fetch its pages. We read 1,020 of those files, drawn at random from the 16,076 domains that two large advertising networks name as their own customers, and resolved each one the way a crawler has to: by the rules that apply at the site root, not by searching the text for a crawler’s name. Three quarters of the files say nothing about AI crawlers at all. Among those that say something, the name shut out most often belongs to a crawler that fetches a page because a person asked a question it answers, and then links back — so the block that costs a publisher readers is far commoner than the block that withholds their work from training. Every share this site publishes about AI-crawler policy — overall, crawler by crawler, and network by network — recomputes from this table, which is the point of publishing it: our aggregate becomes something you can falsify with a spreadsheet instead of having to take on trust.

• 16,076 domains in the frame
• 1,200 domains drawn
• 1,020 readable files
• 9 AI crawlers resolved against every one

Coverage

2026-09-02

Last measured

2026-09-02

Licence

CC BY 4.0

DOI

10.5281/zenodo.22280370

What a row is

One publisher’s robots.txt file: which AI crawlers it names, which it actually blocks, which names it carries that no vendor documents, and whose template it was written from.

Subjects

robots.txt, AI crawlers, GPTBot, ChatGPT-User, ClaudeBot, CCBot, Google-Extended, crawler blocking, AI training opt-out, web publishers, RFC 9309

Download the summary 1,020 rows · 8 columns · CSV · no account, no email

Read the finding this came out of Publishers block the AI visitor that would have sent them a reader at least ten times more often than the one that only files them away

How to cite this

Use it commercially, change it, build on it. The only condition is credit, and a link satisfies it.

What you should know about this data

• These are small, advertising-funded content businesses on two advertising networks. They are not the web. A publisher with no ad network is not in the frame at all, and the large publishers that every other published figure comes from behave completely differently.
• A file that could not be fetched is a fact about a firewall, not a policy. The 180 domains that refused are left out of the denominator rather than counted as silent.
• The categories are not a partition and must not be added up. A file that blocks a crawler which would have cited it also blocks at least one crawler, and often carries names somebody else’s template wrote. Adding them gives a number larger than the sample.
• “No vendor documents it” is not “it is dead”. We checked each name against its supposed operator’s own published crawler list. Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot, and neither anthropic-ai nor claude-web is among them — but no vendor publishes a retirement notice, so what we can show is that a publisher is naming something the operator does not list, not that nothing answers to it.
• A vendor marker says part of the file was not written by the publisher. It does not say the AI rules in particular came from the vendor — a Yoast block commonly contains no AI rules at all.
• The domain is not published. Every field the measurement used is, so all of the figures can be recomputed; which site each row is cannot be recovered from it, and the sample can be redrawn from the seed and the method published alongside it.

How it was measured

1,200 domains were drawn once from the 16,076 the two networks publish about themselves, ranked by a hash of a published seed and the domain name, so that anyone can redraw the same sample. The nightly crawler fetched each robots.txt on a single complete pass; 1,020 answered and 180 refused or could not be reached, and the ones that could not be reached are excluded from the denominator rather than counted as saying nothing, which would have moved the headline in the direction that flatters it. Each file was then resolved rather than searched: the rules that reach a named crawler at the site root, worked out by the grouping rules the robots.txt standard sets out, because a file that stacks thirty crawler names above one permissive block has authorised all thirty. Rows are in the order the source holds them, which is alphabetical by domain with the domain removed — unrelated to anything measured, so any run of rows from it is a fair slice of the whole.

The summary

The first 60 of 1,020 rows. The download has all of them. Free to use with credit, commercially included.

What each column means

network

The advertising network whose published customer list the domain was drawn from

names_crawlers

How many current AI crawlers the file names at all, whether or not it blocks them

blocks_crawlers

How many of them it actually blocks at the site root, once the file is resolved

named_tokens

The crawlers the file names, separated by semicolons

blocked_tokens

The crawlers the file blocks, separated by semicolons

blocks_citation_crawler

yes when at least one blocked crawler is one that fetches a page to answer a question and links back

retired_tokens

Names in the file that no vendor currently documents, separated by semicolons. Not a claim that the crawler stopped running — only that its supposed operator does not list it

vendor_template

The tool or network whose marked block appears in the file, where one does

← Every dataset