Nothing Was Broken. It Cost $900 a Month.

For months I was paying about nine hundred dollars a month for compute that nothing was using.

Not misconfigured. Not crashed. Not degraded. It was running perfectly — containers healthy, platform green, zero errors — and doing essentially nothing, at full price, every hour of every day.

No alert fired. No dashboard turned red. No user complained, because from a user's point of view nothing was wrong. The system that eventually told me was an invoice, which is the slowest and most expensive monitoring tool ever built.

Why nothing caught it

Here's the uncomfortable design flaw in how most of us monitor: every instrument we install measures health. Almost none of them measure worth.

Think about what's actually wired up in a typical system. Uptime — is it responding? Latency — is it fast? Errors — is it failing? CPU and memory — is it saturated? Every one of those questions is about whether the thing is working.

Not one of them asks whether the thing is needed.

So a service with zero traffic and a full allocation of compute looks exactly like a healthy service. Better, in fact: no errors, flat latency, low CPU. If you built an alert on those signals, idle waste is the ideal state. My monitoring wasn't broken — it was faithfully reporting that a very expensive nothing was in excellent health.

The anatomy of the waste

It wasn't one dramatic mistake. It was the ordinary kind, which is worse, because the ordinary kind doesn't feel like a decision at all.

Capacity provisioned for a shape I'd outgrown. I was running on a managed container platform with per-service allocations, sized during a phase when I was guessing. Then the architecture changed underneath it. Services split, traffic patterns shifted, some things stopped being used at all — and the allocations never moved, because nothing in the system had any opinion about whether they should. Cloud platforms are extremely good at giving you resources and completely indifferent to whether you still need them.

Environments nobody was in. Non-production ran the way production ran, around the clock, for the benefit of exactly zero people at three in the morning.

And my favorite one: I was paying to collect telemetry nobody ever read. Logs and metrics streaming into an ingestion pipeline at real cost per gigabyte, retained, indexed, billed — and never once queried. That one deserves a line of its own, because it's the purest version of the whole problem: observability you don't look at isn't observability. It's a subscription.

What the fixes actually saved

Moving to managed Kubernetes was the big one. It wasn't a cost decision when I started it, but the shape change is what made the waste visible: instead of paying per service for capacity I couldn't see in aggregate, I was paying for a cluster whose utilization I could actually read in one place. You can't right-size what you can't total up.

Trimming the ingestion pipeline — dropping the telemetry I had never once queried, keeping what I actually use for debugging — took about $136 a month off the bill, permanently, with no loss of anything I'd ever looked at.

And once traffic had a floor I could prove rather than guess, I committed to it: a three-year reservation on the baseline compute, 62% off, about $201 a month for capacity I verified was 100% utilized before signing. That last clause is the whole trick. A reservation is a bet that you'll still want this in three years — safe when you're buying your floor, reckless when you're buying your ceiling.

Cost is a signal, not an accounting problem

The reframe that changed how I work: your bill is telemetry. It's just telemetry with a thirty-day sampling interval, delivered by the finance department.

Which means it should be treated like any other signal:

• Put spend on a dashboard next to latency and error rate. Not in a billing portal you visit when something feels wrong — in the place you already look.
• Alert on deltas, not absolutes. A number you have to remember is a number you'll ignore. But "spend rose 20% while traffic didn't" is as diagnostic as a latency spike, and it's the exact shape idle waste makes.
• Ask a question no health check asks: for each running thing, when did it last do work? Not "is it up" — is anyone using it. Most stacks can answer this and almost nobody wires it up.
• Give non-production a bedtime. The cheapest infrastructure is the kind that isn't running.

The honest trade-offs

I'd rather not pretend this is free advice.

Reservations buy discounts with flexibility. Mine locks a machine family and a region for three years — I saved 62% and gave up the right to change my mind cheaply. That's correct for a floor and wrong for anything speculative, and the only reason I could tell the difference was having enough real traffic data to know which was which.

Aggressively trimming telemetry risks the moment you desperately want a log you no longer keep. I've accepted that risk deliberately, for data I had provably never queried — but "provably" is doing heavy lifting there, and the honest version is that some future incident may make me regret one of those decisions.

And scale-to-zero trades money for a cold start. Fine for a development environment. A very different conversation in front of users.

The pattern, three times now

I've written this same failure twice already and hadn't noticed until I wrote this one.

Pods reported Running while being unable to talk to anything. This blog answered "page not found" with a 200 OK for four days. And infrastructure ran flawlessly for months while being worth nothing.

Three different systems, three green dashboards, three kinds of nothing happening. In every case the instrument was working exactly as designed and answering exactly the question it was configured to answer — and in every case the question that mattered was one nobody had thought to ask.

The money was gone before I noticed, and no amount of good intentions about "keeping an eye on costs" would have caught it. Only a number, on a screen I already look at, with an alert on the delta.

Saturday: what it actually takes to run the door at a real event — including the parts that went wrong. ✨