49 Microservices, One Invisible Failure: When “Running” Doesn't Mean Working

Last October I confessed, right here, that I had knowingly overengineered a side project into twenty-five microservices. Time for an update: the last count was forty-nine.

I wish I could tell you that was a plan. It wasn't. Something that was so simple in my head — a small prototype, a handful of services — kept meeting the real world, and the real world kept asking for things: ticketing, payments, access control, notifications, reporting. Somewhere along the way it stopped being "the prototype" and became a platform that runs actual live events, with actual people at the door. The overengineering post was about ego and learning. This one is about what happens when the thing you built for fun starts carrying weight — and one of its invisible pieces quietly fails.

Friday, 17:33

Picture the scene: a live event, attendees arriving, staff scanning tickets at the entrance. My platform in the middle of it. And then, on a Friday evening at exactly 17:33 UTC, logins stop working.

No deploy had gone out. No one had touched anything. We had even frozen infrastructure changes for the week of the event, specifically to avoid this kind of moment. But the cloud doesn't sign your freeze calendar: our provider decided the cluster's nodes needed a forced maintenance redeploy, and nodes started rolling — taking every pod on them along for the ride.

That, by itself, should have been fine. Kubernetes reschedules pods all day; that's the whole point. The platform is designed to survive nodes dying.

It didn't. And the reason is the most instructive failure I've had all year.

The invisible piece

Every service in the platform runs with a sidecar — a companion container (Dapr, in my case) that handles service-to-service calls, state, pub/sub. The application container without its sidecar is deaf and mute: the process runs, but it can't talk to anything.

Sidecars don't appear by magic. A cluster component — an injection webhook — attaches them to every pod at birth. And that injector is itself just another pod... which means it was also rolling when the nodes went down.

Here's the trap. The webhook was configured with failurePolicy: Ignore — which translates to: "if the injector isn't reachable when a pod is born, let the pod start anyway." During the minutes the injector was down, fourteen pods were born without sidecars. Fourteen services, running, healthy-looking, and completely unable to speak.

"Running" is not "working"

This is the part that stayed with me. Every dashboard was green.

Kubernetes reported the pods as Running — true. Health probes passed — also true, because the probes asked the app process "are you up?", and it was. Nobody had asked the question that mattered: "can you actually reach the things you depend on?"

The system didn't lie to us. It answered exactly the questions we had configured it to answer. We just hadn't configured the right questions. The first honest signal we got wasn't from observability at all — it was users failing to log in, during a live event, while every chart said everything was fine.

If you read Saturday's post, you've seen this shape before: a static host serving a "page not found" with a healthy HTTP 200. Same disease, bigger blast radius. A check that verifies the wrapper and not the content is worse than no check — it manufactures false confidence.

The fix took minutes. The knowing took longer.

Once we understood, recovery was almost embarrassingly simple: delete the half-born pods. The scheduler recreated them, the injector — back on its feet by then — did its job, sidecars attached, logins returned. Minutes of work.

The expensive part was everything before that: the window where the only symptom was "users can't log in" and every green dashboard was actively arguing against the truth.

One line

The permanent lesson fits in one line of configuration: failurePolicy: Fail — "if I can't inject the sidecar, don't let the pod start at all."

Look at the trade-off, because there genuinely is one. Fail means that if the injector is ever down, pods won't schedule — your cluster loudly refuses to make progress. Ignore means pods always start — some of them broken in a way nothing reports. Both are failures. But one failure pages you, and the other one smiles at your dashboards while your users suffer.

I'll take the loud failure every time. A system that refuses to run is a problem you fix in minutes. A system that runs half-alive is a problem you discover — usually from the outside, usually at the worst possible moment. 👉 Fail loud, fail early, fail in a way that points at itself.

What 49 services actually taught me

Not that microservices are bad — regular readers know I built this on purpose. What the incident taught me is that at forty-nine services, the system's behavior lives in the seams, not the services. Every service had tests. Every service had probes. The failure lived in a webhook none of my services knew existed, controlled by one default nobody had questioned.

So the real checklist that came out of that Friday:

• Ask your health checks the question that matters: not "is the process up?" but "can it do its job?"
• Find the pieces whose absence is silent — injectors, admission hooks, cert renewals, DNS — and make each one fail loud.
• Assume the cloud will pick the worst week to do maintenance, because it will.

The platform kept growing after that Friday. So did my respect for every quiet little component I'd never bothered to look at.

Thursday's post pulls on a different thread from the same platform: authorization — and why hiding a page from someone is not the same as taking away their permission. ✨ See you then.