I Shipped Two Days Early and Skipped the Test Plan

Last week I published a post here about release discipline. A freeze a few days out. A written test plan. A Saturday spent walking it with real devices and real accounts. A checklist of about thirty-five boxes on the night.

What that post did not mention — because I had not finished reckoning with it — is that the release immediately before it went out two days early, without the human test walk. Forty-eight written cases. Not one of them run by a person.

So before anything else: the post was true about the process and quiet about the exception, and the exception was mine.

What I actually traded

The release was large — twenty-three repositories, four services reaching production for the first time, seven database migrations. It was ready on the Saturday, the automated suites were green, and I shipped it rather than sit on it until Monday.

The forty-eight cases moved to the next release. They are filed, assigned, and will get walked. But "will get walked" is a promise about the future, and in the meantime there are flows in production that no human has ever completed end to end.

I want to be exact about the reasoning, because "we were in a hurry" isn't it. That platform has no live customers yet. The cost of a defect right now is my evening and a redeploy — not somebody's business. That is a real and defensible reason to take the trade, and I recorded it as a trade rather than pretending the release was fully tested.

It is also a reason with an expiry date, and I could see the date from where I was standing.

What the automated suite cannot tell me

I wrote a few weeks ago that tests run in a world with no bodies, no time and no other people. Skipping the human walk is that essay's argument pointed directly at my own foot.

The suites proved the code does what I believe it does. They cannot tell me whether a person can complete a task — whether the flow makes sense, whether the screen after the screen is the right one, whether a device in a hand does what a device in a test runner does. Nothing in a green pipeline has ever once told me that.

Then the release taught me six things

Here is the part that made this worth writing rather than just confessing.

Running the release found six defects in my own runbook — the document that exists specifically to make the night boring. Every one of them was invisible from reading it.

One. The infrastructure tooling auto-applies on merge. My runbook documented it as a manual dispatch I would trigger deliberately. Merging the promotion started a production apply nobody asked for. It failed only because another plan still held the state lock — which is to say it was stopped by luck, not by design.

Two. The list of repositories needing a dependency repin said ten. It was thirteen. A stale pin builds green and ships an old contract, which is the quietest kind of wrong.

Three. The documented repin command was a direct push to a protected branch. It cannot work. It has never worked. It sat in the document looking authoritative.

Four. Six repositories needed a back-merge before they could be promoted at all — their promotion requests opened in conflict on dependency pins. None of that was in the plan.

Five. A service deploying for the first time failed with nothing but a timeout, because a configuration secret existed in one environment's vault and not the other's. The deployment then rolled itself back cleanly, which deleted the evidence before I could look at it. A tidy failure that destroys its own crime scene is worse than a messy one.

Six. Another first-time service needed a DNS record that did not exist. That blocks the obvious thing, and it also silently blocks certificate issuance, so you get two failures that look unrelated and one cause.

What I think this means

A runbook you have not executed is not a runbook. It is a hypothesis about what will happen, written by someone who was not there yet.

Mine was written carefully, by me, about systems I built — and it was wrong in six places, four of which would have stopped the release cold. Not wrong because I was careless. Wrong because writing down what you believe happens and finding out what happens are different activities, and only one of them is available at a desk.

All six are fixed in the document now. That is the actual return on a difficult night: not that it went well, but that the next one starts from a document with six fewer lies in it.

What I would do differently

Not ship early. Or more precisely: not let "ready" and "tested" collapse into each other the way merged and deployed do, which I have written about and apparently still need to learn.

The release was ready in the sense that the code was done and the machines agreed. It was not tested in the sense that a person had confirmed anyone could use it. Those are different states and I had names for them, which makes it worse rather than better.

Writing about discipline is easy and cheap. The test of it is the week you want to skip it — and last month, I did.