Plain Reckoner

Who reads your site

Are you unusual in what you let the AI companies read?

Measured against a 16,076-domain frame that two ad networks name as their own customers, from which we read the crawler rules of 1,020 publishers the data

Your own file

Read your site from the overview and we will mark your own rows below.

Your robots.txt is a short file at the front of your site that tells automated visitors what they may read. Each of those visitors is called a crawler and arrives under its own name. Nothing about yours is sent anywhere: we read it in your browser from the reading kept on this device.

What 1,020 publishers’ files say

We drew 1,200 domains from the 16,076 that two ad networks name in their own published lists of the publishers they sell for, and read each one’s robots.txt. 180 gave us nothing we could read and are left out of the count rather than counted as permitting everything. These 7 rows overlap. A file that blocks a crawler which would have cited it is also a file that blocks at least one — so they are 7 ways of looking at 1,020 files, not 7 groups of files, and adding them up gives a number bigger than the sample.

  1. 776 of 1,020 Say nothing about AI crawlers at all. Every AI company that respects the file is free to read the site — both to train a model on it and to quote it in an answer.
  2. 216 of 1,020 Block at least one AI crawler from the front page.
  3. 123 of 1,020 Block at least one crawler that would have sent them the reader — the kind that fetches a page because somebody asked a question it answers, and then links back.
  4. 28 of 1,020 Write a crawler’s name down and let it through anyway. The file is read in blocks — names first, then the rules that apply to them — so a list of names sitting above one rule that allows everything permits every name on it.
  5. 167 of 1,020 Name at least one crawler that appears in no company’s current documentation, so that line keeps nothing out.
  6. 23 of 1,020 Name only crawlers that no longer exist. Nothing at all is excluded, and the file reads as protection to anyone who opens it.
  7. 305 of 1,020 Carry a section a tool or an ad network wrote and signed with its own name, so those lines are not the publisher’s wording.

Blocked, crawler by crawler

Out of the same 1,020 files. Which crawler a publisher blocks is a different decision from how many: one kind collects pages to train a model, which is a decision about licensing the writing, and the other fetches a page because somebody asked a question it answers, which is a decision about traffic.

The two lines worth reading together are OpenAI’s. Its ChatGPT-User fetches a page because somebody asked a question it answers, and the answer links back; OAI-SearchBot only files pages away for search. 121 of these files block the first and 4 block the second: 11.9% against 0.4%. That is at least a tenfold difference and the sample cannot say much more than that: on four files, the multiple could honestly be anywhere from about ten to well over a hundred.

  1. GPTBot Collects pages to train a model 206 of 1,020
  2. CCBot Collects pages to train a model 201 of 1,020
  3. ChatGPT-User Answers a question, and links back 121 of 1,020
  4. Google-Extended Collects pages to train a model 89 of 1,020
  5. ClaudeBot Collects pages to train a model 74 of 1,020
  6. PerplexityBot Answers a question, and links back 7 of 1,020
  7. OAI-SearchBot Answers a question, and links back 4 of 1,020
  8. Claude-User Answers a question, and links back 2 of 1,020
  9. Claude-SearchBot Answers a question, and links back 2 of 1,020

What this is evidence about

1,020 files is 1,020 files. They are small, ad-funded content businesses on two ad networks — not the web, and not the large publishers with signed AI deals, who behave completely differently and are where every other published figure on this subject comes from. A publisher with no ad network is not in the list we drew from at all. The draw used a fixed starting word (plainreckoner) rather than chance, so anyone can repeat it and get the same domains, and each network was given the same share of the sample as it has of the list. Measured 2026-09-02.

This was first measured on 2026-09-01 against 106 files and re-measured here against 1,020. Nothing turned over: saying nothing about AI crawlers was 76% and is 76.1%. Carrying somebody else’s template was 34% and is 29.9%. What changed is how well each figure is known: at 106 files a share of this size could be read only to within about eight points either way, and at 1,020 it is about two or three. The rarest rows are still the least certain — the bottom of the crawler list above rests on two files and on four, and no sample size we have makes those precise.

One qualification, and it comes from the company itself. Of the crawler that fetches a page because somebody asked a question, OpenAI writes: Because these actions are initiated by a user, robots.txt rules may not apply. So the counts above are what the files SAY, which is what we measured and all we claim. Whether every company acts on what a file says is a separate question, and the honest answer is that we did not measure it.

The two networks are not alike, which is why the sample is drawn to match the list rather than by simply taking what a crawler reaches first. Raptive is 43.5% of the list and 42.4% of what we read, and 60% of its files say nothing about AI crawlers. Mediavine is 56.5% of the list and 57.6% of what we read, and 88% of its files say nothing about AI crawlers. Weighting each network to its share of the list rather than its share of the files that answered moves every figure above by less than half a point.

What each file permits is worked out rather than searched for. The file is read in blocks — one or more lines naming crawlers, then the rules that apply to them — so a file stacking thirty names above a single rule that allows everything has permitted all thirty. Counting the names instead would report that file as the strictest in the sample.