Your Tests Run in a World That Doesn't Exist

Every test was green. The event surprised me five times anyway.

Nine days of running access control at a real fair produced exactly zero failing builds and five things I had not accounted for. None of them were bugs in the ordinary sense. Nothing threw. Nothing returned a 500. No assertion I had ever written was violated. The suite wasn't wrong — it was answering a different question than the one production was asking.

Here's the thing I keep coming back to. A test suite runs in a world you build for it, and that world is missing almost everything. It has no time, no bodies, no other people, no landlord, no neighbours. It's a clean room, and it's very good at what clean rooms are for: proving that what you already understand still works. What it cannot do is surprise you, because everything inside it was put there by you.

Five categories. One receipt each, all from the same nine days.

No bodies

The main door reader slowed down after thirteen hours on battery.

That's the entire failure. A device that had been awake since morning began throttling itself, and no amount of backend engineering had anything to say about it.

No CI runner has a battery. Nothing in your test environment gets warm, gets carried around a fairground, gets left where the afternoon sun hits it, or runs out of power at hour thirteen because somebody unplugged it to charge a phone. Your laptop is plugged in and has been awake for forty minutes. Every device you actually ship to lives a completely different physical life, and the test suite has no vocabulary for any of it.

This is the category people find easiest to dismiss as "not really software." It cost me the busiest door of the day.

No time

From 19:00 onward, median read time went from 0.7 seconds to 2.1 seconds and stayed there until closing.

The strange part is the direction. This happened as traffic fell. The busiest hour of the day was among the fastest; the emptiest hours were the slowest.

That's backwards from the mental model most of us carry, where slow means overloaded. It's the signature of the opposite problem: connections, caches and pooled resources quietly expiring between requests, so each new scan pays setup costs the previous one had already paid.

Your test suite runs its cases back to back, in seconds, forever. Nothing in it is ever idle, so nothing in it ever goes cold. An entire class of behaviour — everything that only happens after a gap — is invisible by construction. You cannot write the test "and then nothing happened for forty minutes" without making your build take forty minutes.

No other people

Roughly 15% of all reads were rejections, and almost all of them said the same thing: already inside.

Not attackers. Not bugs. People walking out a side door for lunch without scanning out, coming back, and being told — correctly — that the system already had them inside.

The state machine was right. The users hadn't read it.

Tests encode the behaviour you designed. Production runs the behaviour people improvise. Every test I write has a well-behaved actor in it, because I'm the one writing the actor, and I already know the rules. Nobody at a fair is holding your presence model in their head. They're thinking about lunch.

No landlord

On the Friday, at 17:33, the cloud provider rolled the nodes underneath me. Not a request I made, not a deploy I ran — routine maintenance on infrastructure I rent.

Your test environment has no landlord. Nothing reboots it on someone else's schedule, nothing migrates it to different hardware mid-run, nothing decides Tuesday is a good day for maintenance. In tests you are the only agent in the universe. In production you're a tenant, and the building has its own plans.

No neighbours

Every service in the platform had a passing suite that Friday. Fourteen of them came up unable to talk to anything while reporting perfect health.

The failure wasn't inside any service. It lived in the space between them — an injection webhook, configured to let pods start even when it couldn't reach them, doing exactly what it was told at the worst possible moment. No service owned that behaviour. No service's tests could have covered it, because from inside any one of them nothing was wrong.

Unit tests verify a part. Integration tests verify a seam you already thought of. The failure came from a seam none of the parts knew existed.

The thing tests are actually for

None of this is an argument against testing, and I want to be careful here, because "tests didn't catch it" is a sentence people use to justify not writing them.

My suite has caught real regressions and will catch more. But a test suite is a ratchet, not a telescope. It stops you sliding backwards on behaviour you already understand. It has nothing to say about the behaviour you haven't imagined yet — and it never will, because you'd have to imagine it first in order to write the test.

So the practice I've settled on isn't more tests. It's this: before anything new goes live, one human walks the entire flow end to end, in the real environment, on the real device, on the real network. Not a staging approximation. The actual thing. It takes an afternoon and it finds a category of problem that no amount of coverage will.

Tests tell you that you haven't broken what you understand. Only reality tells you what you never understood. Both are worth knowing — and only one of them can be automated.