The last step of my release pipeline installs the package I just published, on Linux, Windows and macOS, and confirms it reports the version that was released. It is the only thing in the whole chain that tests the artifact a stranger would actually download, rather than the source it was built from. Last night it reported success on all three operating systems against a version the registry was not serving.

Here is the step, as it had been since the first release:

v="${TAG#v}"
for i in 1 2 3 4 5 6; do
  out=$(npx --yes "$PKG@$v" --version 2>&1) && break
  echo "attempt $i: $out"; sleep 20
done
echo "$out"
case "$out" in *"$v"*) echo "smoke ok: $PKG@$v" ;; *) echo "expected version $v in the output"; exit 1 ;; esac

Read the last line on its own. It passes when the released version appears anywhere in the output. Now read what npm prints when the version is not there:

npm error notarget No matching version found for ai-slop-linter@0.1.5.

The failure message contains the version. So when the registry was still processing the publish and all six attempts failed, $out held that error, the pattern matched it, and the job printed smoke ok and went green. Three operating systems, three green checks, nothing installed on any of them.

Nothing was broken underneath. The package had published correctly and npm was simply slow to serve it, which it warns you about in its own output: Your package is being processed and may take a few minutes to become available. So the check did not hide a bad release. What it revealed is worse in a quiet way. It could not have caught one. The success condition was contained in the failure output, which means that for as long as that pipeline had existed, its final verification had been decorative.

I found it because I was watching the registry by hand while waiting for a version number to change, and the number would not change while the workflow insisted it had. Without that accident it would still be there.

Five more of the same shape, in one night

That was not the only one. It was the third of six found in about four hours, across the same set of repositories, and I want to list them because the pattern only becomes obvious as a set.

A version checker that knew four phrasings. I keep a script that compares every version number written in my documents against what npm and PyPI actually serve, because all five of them were wrong at once, on a page that said version numbers were deliberately kept off it because they would rot. The script knew four ways a version can be written in a sentence. A table cell writes a fifth: the package name sits in the first column and the version three columns away. So npm 0.1.1 in a document that goes to an assessor matched nothing, and the script printed ok, every claim in 6 surfaces matches.

When I reintroduced the two wrong numbers to see what the old version would say, it reported neither of them, and instead demanded a correction to the one number in the whole pack that was right: a deliberately frozen snapshot dated 5 September, which it wanted raised to today's release. That is not a hypothesis about how the snapshot had been corrupted. It was the corruption happening again on demand, and it explained a line I had been puzzled by, a pairing of an old npm version with a current PyPI one that had existed on no day in either registry's history.

A checker reading the right file at the wrong moment. Another script guards the claims on a public page about my published packages. It read the conformance table from the default branch. Every other claim on that page is about the version people install. So the moment a pull request merged, it called a correct sentence stale. It now reads the table at the tag of the released version.

A guard present on two claims and missing on the third. The same file already had two match-nothing guards, added after that exact failure had cost something twice. The third claim, the one comparing npm against PyPI, had none. Reword the sentence and the comparison silently stops running, with the file reported clean. Knowing the lesson is not the same as having applied it to every claim in the file.

Five tests that failed for the wrong reason. Publishing three articles meant adding one row to a data file. Five tests broke, none of which was testing publication: the homepage's first link, a feed's build date, a pagination assertion, a static route list, and a fixture guard whose message read "twelve items (eleven essays plus one external piece)". Every one of them had a title, a URL, a date or a count typed into it. The repair that suggests itself is to retype the literals, which restores the green and the defect together, and buys the identical morning on the next publish.

And two numbers I wrote from memory. Writing the record of all of this, I described a cache lag as lasting "about twenty minutes". I had not measured it. Neither had the reviewer who had written "hours" for the same interval. Both of us invented a figure inside a document about figures that arrive by inference and get recorded as measurements. The defensible version, from timestamps: the three articles published at 21:25:31Z, 21:26:28Z and 21:27:15Z, and the index was serving all seven by 21:37:47Z. How long it had been stale before that is not something either of us recorded, and the record now says so instead of picking a number.

The question I had never asked of a check

Six things, and they look unrelated. A shell pattern, a regular expression, a git ref, a missing guard, five hard-coded literals, two remembered intervals. They are one defect.

Every one of them reported agreement for a reason that had nothing to do with what it was checking. The version checker found nothing because it did not know the sentence shape, and silence rendered as success. The conformance checker read a real file at a moment when that file was not the subject. The smoke test read a real string in output that said the opposite. None of them failed, and none of them could have, for the specific thing they existed to catch.

The question I had been asking when I wrote each of these was "does this find the problem". I had tested each one against a broken input at the moment I wrote it, in the sense that I had convinced myself it would fire. The question I had never asked is the one that matters:

What does this do when it finds nothing, and is that distinguishable from finding everything in order?

A check whose answer to that is "they look the same" is not a check. It is a comment that runs.

The cheap test follows directly. Before trusting a clean result, run the thing against a deliberately broken input and confirm it fails, and that it fails for the right reason. It takes a minute. I did it for each repair last night: reintroducing the two wrong versions so the old checker could be caught calling them fine, and running the new smoke condition against the real npm failure text. For the shell one the whole fix is to stop testing a substring and start tracking two facts separately:

installed=""
for i in 1 2 3 4 5 6; do
  if out=$(npx --yes "$PKG@$v" --version 2>&1); then installed=yes; break; fi
  echo "attempt $i did not install:"; echo "$out"; sleep 20
done
[ -n "$installed" ] || { echo "the registry never served $PKG@$v"; exit 1; }
printed=$(printf '%s' "$out" | tr -d '\r' | tail -n 1 | tr -d '[:space:]')
[ "$printed" = "$v" ] || { echo "installed, but --version printed '$printed'"; exit 1; }

Did it install, and did the thing it printed equal the thing we released. Two questions with two answers, neither of which the failure message can satisfy by accident.

The direction that looks like diligence

The five broken tests belong to the same family, seen from the other side, and they are the more dangerous half.

A test pinned to a literal does not pass for the wrong reason. It fails for the wrong reason. Five red tests after adding one row of data feel like a suite doing its job, and that feeling is what makes the trap work: the fix that suggests itself is to update the literals, which is fast, restores the green, and leaves the defect exactly where it was. A silent green at least feels like something you ought to verify. A loud red feels like something you ought to fix, and fixing it is not the same as understanding it.

Those five now derive from the same function the pages derive from, and assert properties rather than snapshots: that the first link on the homepage is the newest item, that the last page is whatever page the count implies rather than page two. The person who rewrote them also checked both against deliberately broken sources, a reversed slice and a clock-derived build date, before trusting the green. That is the rule applied to the repair rather than only to the thing repaired, which is the step everyone skips, me included until that night.

Why writing it down worked when re-reading did not

The part I keep coming back to is not any of the six. It is how the last two were caught.

The work was reviewed across two sessions, with review passing between them; the repositories disclose that the work is AI-assisted, and this is the shape of that assistance rather than the subject of this piece. Three of the six were found by one reviewer looking at the other's work, which is unremarkable. The last two were each of us catching our own wrong number, which sounds like carefulness and is not.

Neither of us caught our own figure by re-reading it. I caught mine because a question forced me to state it precisely enough to be compared against a timestamp. The other was caught because a correction in the identical shape had just been written down next to it. Neither correction started with suspicion. Both started with a claim that had been recorded somewhere specific enough that comparing it to reality was a thirty-second operation rather than a project.

That is a property of the record, not of the people keeping it, which is the only reason it is worth writing down. Carefulness does not survive a long night. A claim written so it can be checked will sit there being checkable whether or not anyone is currently suspicious, and it will still be checkable next week when everyone has forgotten which parts were measured and which were remembered.

All six were claims that had been written somewhere and never re-run. A table cell, a conformance table, a shell condition, an unguarded pattern, five test literals, and two intervals. Not one of them was found by noticing that something felt wrong. Every one was found because somebody eventually compared the written thing against the thing it described.

The smoke test is the one I would keep if I could only keep one, because it is the purest form: a check whose success condition was contained in its own failure message, sitting at the end of a release pipeline, printing smoke ok on three operating systems for as long as the pipeline had existed. If that can happen in a repository where every number in the documentation is compared against a registry by a script on a schedule, it can happen anywhere.