Searching for the thing you are measuring will manufacture your result
474 words · about 2 min
If you build your list of businesses by searching for "X monthly rates", you have picked them BECAUSE they publish rates, and your answer is near-certain before you start. Drawing the same sample from OpenStreetMap — a map that knows nothing about prices — gave 66.7%.
- How it was measured
- We built the list from OpenStreetMap, filtered only on the business having a website at all, then fetched each business's own site directly. We never used a comparison site or a directory.
- Last checked
Listen to this Playing read from this page, nothing added
Watch it plot 28s · narrated · the same figures as above
We needed to know what share of a kind of small business publishes its prices online. The obvious method is to search for the prices and see what comes back.
That method cannot produce an answer. Searching "Arizona RV park monthly rates" returns parks because they publish rates. The list you are measuring has been chosen by the very thing you are trying to measure, and the answer will come out near 100% whatever the truth is.
The list of things you draw your sample from has a name — it is called the sampling frame — and this is the way it most often goes wrong.
Drawing a list that cannot know the answer
We used OpenStreetMap instead, the public map anyone can edit. We asked it for campsites that had a website recorded, across three rectangles of the map — in the map's own terms, tourism=caravan_site carrying a website tag, over three bounding boxes. The map knows nothing about prices, which is exactly the property that makes it a fair list to draw from — being in it depends on somebody having mapped the business, not on what the business publishes.
Then each business's own website was fetched directly.
Result: 28 of 42 scored, 66.7%. Counting every site we could not reach as a failure still gives 54.9%. It clears half either way, and that is what makes it usable — the verdict does not depend on how you treat the ones that blocked us.
The number that mattered was the second one
Of the businesses publishing any price, 96.6% published the monthly one. Of the fourteen publishing no monthly price, thirteen published nothing at all.
That is a different finding from the headline, and a more useful one: there is no meaningful group that publishes some prices and hides this one. Businesses either post a price list or they do not.
Where else this bites
Any question of the form "what share of X do Y" is exposed, and the failure does not show up in the result. It looks like data. It has a sample size and a percentage and a margin of error, and it means nothing.
The tell is simple: ask what decided whether something got into your sample. If the answer involves the thing you are measuring, start again.
One honest limitation. We later found a bigger measurement of the same question by a different route — a trade association's own directory listings — giving 42.8% from 278 of them. It measures something slightly different, since those listings are filled in by the businesses themselves rather than read off their sites, so it does not overturn ours. But it points the same way as our harsher count of 54.9%, and the honest read is that 66.7% is the optimistic end of a 43-67% band.
Tracked as SAMPLING-FRAME-BIAS, RV-HIT-RATE. If a number here looks wrong, tell us and we will check it against the source.