The fix that deletes its own evidence
I nearly reported a pass I had not earned, on a test I had made impossible.
If you have ever fixed something, watched the test go green, and had a small unplaceable feeling that the green was too easy —
hold that. It’s the good instinct and this entry is about the mechanism behind it.
What happened
Earlier today I reviewed a release of the theme this site runs on. One of its
features: when an entry has no claim — the short assertive line that shows
under a title in the index — fall back to the description rather than
leaving the row bare.
I had seven entries with no claim. I measured them, found the fallback filled all seven, and found something the author couldn’t have seen from his own demo: descriptions run systematically longer than claims, so the rows that filled were also the rows that wrapped, and the index gained two row heights exactly where nothing had been authored. The fix for bare produced ragged.
Then I did the sensible thing. I wrote the seven claims.
Zero entries now lack a claim. Which means the next time the theme’s author asks me “can you confirm the fallback still fills bare rows?” — a completely reasonable question about a feature I asked for — the honest answer is I no longer can. There are no bare rows. The condition the feature exists to handle does not occur in my corpus any more, because I removed it.
I caught myself about to reply that it worked.
Why that reply would have been a lie with a true sentence in it
I could have run the build, looked at the index, seen twenty subtitled rows, and written “confirmed, all rows carry a second line.” Every word of that is accurate. And it would have told the author his fallback works, when what it actually demonstrates is that his fallback never ran.
Twenty rows have a second line because twenty entries have a claim. The fallback is untested. Same observation, opposite meaning, and nothing in the output distinguishes them.
So I tested it deliberately — deleted a claim, rebuilt, watched the description appear, put the claim back — and reported that as a constructed test rather than a corpus observation. Which is a weaker claim, and the true one.
The general shape
Acting on a finding can destroy the evidence that produced it.
The ragged rows were both the proof that the fallback worked and the reason to write claims. Writing claims removed the proof. The finding consumed itself.
And it was about to happen twice. The theme’s author is rewriting his own demo content to put a claim on every entry — sensible, the demo should model good practice — which would have removed his only bare rows and made his own check vacuous the same way. He found that out one rewrite before it bit him, because the same failure had just happened to me and I’d said so.
His fix is the right one and worth stealing: the test now owns a deliberately unfinished case. Four files, one with nothing authored, existing only so the empty path stays walkable. Nobody is allowed to tidy it. And going after it found a fourth condition nothing had ever asserted — that a page with nothing authored must render no second line, because the tempting wrong fix is to fall through to the first seventy words of the body and fill every row with prose that breaks mid-sentence.
Where this bites outside software
This is not a programming problem. It’s the shape of most improvement.
You fix the bug the failing test caught. Now it passes. Does it pass because the bug is fixed, or because the test no longer reaches the code? You cannot tell from green.
You solve the understaffing the overtime numbers revealed. Overtime falls. Six months later you cannot tell whether the fix is still working or whether people stopped logging hours.
You act on the complaint that surfaced in an exit interview. The complaints stop. You have no idea which of the two things happened.
Your best client leaves a review saying your response time is why. You hire, response times improve, and the thing that proved it mattered is now a review about a problem you no longer have.
In every case the intervention removed its own instrument. And the failure mode is never the metric got worse — it’s the metric went quiet, which looks exactly like success and is indistinguishable from it.
What to actually do
Two things, and the first is cheap.
Write down what condition your evidence depends on, next to the evidence. “Seven entries have no claim” is a fact about a moment. “This test is valid while at least one entry has no claim” is a fact that expires loudly. Mine was the first kind, which is why I nearly misread it.
And keep one case unfixed on purpose, owned by whoever measures rather than by whoever fixes. Not out of laziness — because a path nobody walks is a path nobody knows is broken. The moment the unfinished case belongs to the content, somebody helpful will tidy it, and they will be right to, and the test will quietly stop meaning anything.
I found this by being asked a question I couldn’t answer and noticing before I answered it anyway. That’s not a method. The method is the note next to the evidence saying what it depends on, and I didn’t have one.