Your Alert Cleared Itself

On Saturday morning my production cluster lost the ability to create anything at all. Seven alerts fired within minutes, and every one of them was delivered to my phone.

Two hours and forty-five minutes later, the monitoring sent a message saying telemetry had been restored.

Nothing had been restored. Fifty-two of fifty-seven services stayed down until Wednesday evening.

How an alert talks itself down

Both rules were written the same way. Each compares the services currently reporting telemetry against the services expected to be reporting, and "expected" was defined as whatever had reported in the previous two hours.

Read that definition again with a long outage in mind. After two hours of silence a dead service is no longer expected to report anything. By the time the outage was three hours old, enough services had aged out of the roster that the gap being measured had closed on its own. The condition stopped being true, the alert auto-resolved, and it told me so.

The outage lasted long enough to become the baseline.

I want to be precise about why this is worse than having no alerting at all. An unmonitored system leaves you uncertain, and uncertainty sends you to go and look. A system that alerts and then reports its own recovery leaves you confident instead. Confidence is what stops you looking. I received a green message on Saturday morning and did not open a dashboard again until Tuesday.

The watchman was inside the building

A second failure had the same shape.

There was a synthetic probe built for exactly this situation, and it ran inside the cluster. Since the failure blocked everything from starting, the probe could not start either. Its silence looked identical to its usual quiet success.

Any check sharing infrastructure with the thing it checks will fail at the same moment, for the same reason, and will fail silently. That reads as obvious written down. It is very easy to miss while building, because next to the service is the convenient place to put the probe.

What I changed

Availability checks now run from outside the cluster, against the public hostnames, with no shared dependency on the thing being measured.

The roster of expected services is fixed rather than derived from recent activity. A service that has been dead for a week is still expected to be alive.

Blackout conditions no longer resolve themselves. When telemetry stops arriving, the alert stays open until a person closes it.

One more thing belongs in this post. Nothing broke for anybody outside, because that platform has no live customers yet. Three and a half days of downtime cost me a weekend of false confidence and nothing more. The next time will be different, and that difference is the entire reason to write this down now.