Three times in one week, comparing two dates hid a sudden jump

506 words · about 2 min

Comparing a number on one date with the same number on another cannot tell growth apart from a one-off jump, or from someone changing what they count. We hit it three times in a week: a marketplace category, a map query, and a figure published by a software vendor.

How it was measured: Each one was caught by pulling every point in between instead of just the two ends. In every case the full run of figures showed something the two-date comparison could not.
Last checked: 2026-08-29

Listen to this Playing read from this page, nothing added

Comparing a number in year A with the same number in year B feels like measurement. It is not, and here is what it hides.

One: a category that did not exist

Sales of YouTube-earning businesses through one marketplace went from 0 in 2022 to 28 across 2025-26. We were one sentence away from reporting that as where the audience had moved to.

Then we pulled every year in between: 0, 0, 0, 0, 0, 6, 16, 12.

A run of zeros followed by a sudden start is a new label, not growth. The broker began tagging listings as YouTube businesses in 2024. The two-date comparison could not possibly have shown that, because both dates sat on the same side of the change.

Two: a query that threw away two-thirds of the data

We built a sample of RV parks from OpenStreetMap, the public map anyone can edit, and found 101 across three states — against an expected total of 1,000-2,500. That looked like a finding about how little of this data exists, and it was reported as one.

It was a bug. The rectangles we searched covered parts of states rather than whole ones, and — the real error — the query asked only for points. A campsite can be recorded on the map either as a single point or as an outline of the site, and most are outlines: worldwide there are 11,818 points against 24,233 outlines. Checking one state by hand gave 198 where we had reported 27. A 7x undercount, presented as a discovery that would have changed our plans.

Three: a figure published by a vendor

A search-analytics company reported one of its numbers rising from 8.15% to 18.57% over twelve months. We quoted it.

Then we found the thirteen monthly figures behind those two. They show one jump, in April-May 2025, then seventeen months flat or drifting down, with the number peaking three months after the jump. Nothing in the company's write-up mentions the jump, and nothing rules out the possibility that they changed how they classify things.

What to do instead

Always pull every point. If a data feed or a report hands you two dates, ask for the points between them; if you cannot get them, treat the comparison as a hypothesis rather than a result. Our own reading of AI Overviews is what that correction looks like in practice: the two dates said one thing and the thirteen points said another.

Watch for numbers that start from nothing. A clean run of zeros before a sudden start is almost always a change in definition — a new label, a new field, a new way of tracking — and almost never something real appearing out of nowhere.

The third one is worth sitting with. It happened after the first two had been written down, to somebody who had read them, on the item flagged as most important. Knowing about a trap does not protect you from it. Only going back to the data does.

Tracked as TWO-ENDPOINT-TRAP, YOUTUBE-SIGNAL-ARTEFACT, FRAME-QUERY-BUG. If a number here looks wrong, tell us and we will check it against the source.