Who Checks the Outside Check?

Last week I wrote about an alert that cleared itself while production stayed down, and about moving availability checks outside the cluster. A reader asked the most direct question anyone could: how do you ensure those outside checks remain reliable over time?

The honest answer is partly. Before getting to why, I owe a correction.

A correction first

That post said blackout alerts no longer resolve themselves and stay open until a person closes them. I went and checked the actual rules before writing this, and that sentence is wrong. Auto-resolve is still switched on.

What did change is the list of services the alert expects to hear from. It used to be whatever had reported in the last two hours, which is how dead services quietly aged out and the alert declared victory. That list is now fixed. A dead service stays expected, so the alert can no longer close falsely. It will still close on its own when telemetry genuinely comes back. The dangerous behaviour is gone, and I described the fix inaccurately.

What exists

Three public hostnames are tested from four locations every five minutes, by a service that does not run inside the cluster it is testing. Each test is wired to an alert. If the site goes down, something outside the failure notices.

That closes the gap from the outage. It does not answer the question, because the question is about the next link in the chain.

How an outside check fails

Nothing about moving a check outside makes it immortal. It just gives it different ways to die. A test can be disabled during some unrelated change and never re-enabled. The alert can route to a notification channel that stopped delivering months ago. A credential somewhere expires. The monitoring provider has its own bad afternoon.

Every one of those produces the same result, which is silence. And silence is what a healthy system sounds like. It is the failure from the last two posts, one level further out.

Right now nothing watches the checks. That is the gap your question found.

The answer I would give

Invert what silence means. Have the check send a heartbeat on a schedule to something separate, ideally on a different provider with different credentials, and alert when the heartbeat stops arriving. Then silence is the alarm rather than the all clear.

Beyond that, the most reliable habit is the least technical one: break it on purpose now and then and confirm the alarm actually fires. If you have never seen an alert go off, you do not know that it can.

I won't pretend this ends cleanly. Something always has to watch the watcher, and in practice the chain stops at a third party whose entire business is staying up, plus a person who checks occasionally. You pick where it stops. You don't get to make it stop on its own.

Thank you for asking. It was the right question, and the true answer was more useful than the one I would have given without checking.