The Fix Took Production Down for Three Days

In August I wrote here about fourteen services that came up unable to talk to anything while reporting perfect health. The cause was an injection webhook set to failurePolicy: Ignore, which told the cluster that if the injector could not be reached, pods should start anyway. They started. They were deaf.

I fixed it by setting the policy to Fail.

That fix is what took production down for three days and thirteen hours last week.

Why the fix was right

Fail-open is a strange promise when you think about it. It says that when the component responsible for making a pod functional is unavailable, the correct response is to create the pod anyway and let it be broken in a way nothing will report.

Fail-closed says the opposite. If the injector cannot vouch for a pod, the pod does not get created. You trade availability for correctness, and for something like sidecar injection that is the right trade. A service that does not exist is a problem you will notice. A service that exists and cannot speak is a problem you will notice three hours into an incident.

I still think flipping it was correct. What I had not thought through was the obligation that came with it.

What fail closed actually costs

Last week the cloud provider's auto-upgrade replaced every node in the cluster. That is routine, and it is supposed to be survivable.

The injector was running as a single replica with no disruption budget, on a pool that was rolling. It went down with its node, which is exactly what you would expect. What happened next is the part worth sitting with.

The injector's own webhook intercepts pod creation across the cluster. So when the scheduler tried to create the replacement injector, the webhook was consulted, the webhook tried to reach the injector, the injector did not exist, and fail-closed did precisely what I had asked it to do. It refused.

Nothing could be created anywhere in that cluster. Not the injector, not anything else. Fifty-two of fifty-seven services stayed down until I labelled a namespace by hand four days later.

There is an escape hatch, and it is the only reason this is survivable at all. The webhook skips namespaces marked as control plane. The injector had been installed with a command that creates its namespace without labelling it, so the one exemption that would have let it heal itself had never been applied.

Nothing was in Pending

This is the detail I would most like other people to take away, because it is quietly in a lot of tooling.

My instinct during an outage is to look for pods that are not Running. There were none. Every check of that shape came back clean while the entire platform was down.

Pods were being refused at admission, which means they were never created. A pod that does not exist cannot be Pending, cannot be CrashLoopBackOff, cannot appear in any list of unhealthy things. The absence looked like health.

What actually showed it was counting deployments with zero available replicas, and reading events for creation failures. Those are different questions than "is anything broken", and I did not have them wired to anything.

The bill I did not pay

Here is the thesis, and it took an outage to see it.

Fail-open and fail-closed are not correct and incorrect. They are a choice about which failure you would rather have, and each one comes with an obligation you take on at the moment you choose it.

Choose fail-open and you owe yourself detection, because you have just accepted that broken things will start and look fine. That was the August lesson.

Choose fail-closed and you have created a component that can stop everything, so you owe that component real availability. More than one replica. A disruption budget. An exemption so it can always recreate itself. Somewhere to run that is not the thing it is guarding.

I flipped the policy in August and paid none of that. The setting was right and the surrounding work was missing, which felt like a fix and was actually a transfer of risk from one failure mode to another.

The honest footnote

Nothing broke for anyone outside, because that platform still has no live customers. Three and a half days of total outage cost me a weekend and a considerable amount of dignity.

I would rather learn this now. The version of this post where venues are scanning badges at a door while I work out which namespace needs a label is one I would very much prefer not to write.