What It Takes to Run the Door at a Real Event

Feria Arequipa ran for nine days this month, and my platform ran the door.

Not a demo. Not a pilot with a friendly client who'd forgive a bad afternoon. Real attendees, real staff, real turnstiles at a real fair — where "the system is down" doesn't mean a red dashboard, it means a line of actual human beings not moving.

They said yes to that. This is the honest accounting of what happened, including the parts I'd rather leave out.

Day one, in numbers

The first full day looked like this:

• 1,846 credential reads at the doors
• 1,001 distinct badges — roughly a thousand different people
• 4 readers in operation
• 0.7 seconds median read time
• peak hour at 11:00, with 239 reads in sixty minutes

That 0.7s number is the one I care about. Access control has a brutal, non-negotiable requirement that most software doesn't: there is a person standing in front of the reader. Every extra second is a second of someone waiting, multiplied by everyone behind them. A web request that takes three seconds is annoying. A badge scan that takes three seconds, at 239 reads an hour, is a queue.

Then Friday happened

At 17:33 on a Friday evening, during the event, logins stopped working.

I've written the technical autopsy already — a forced infrastructure maintenance rolled the cluster's nodes, the component that attaches sidecars to new pods was itself rolling, and its configuration said "if I'm unavailable, let pods start anyway." Fourteen services came up unable to talk to anything, reporting perfect health.

Here's the part I care about for this post: the doors kept scanning.

Only new logins broke. The readers and staff already held valid tokens, and the scan path doesn't depend on the login path — so the thing with a queue in front of it kept working while the thing behind it was on fire. Recovery took minutes once we understood it: delete the half-born pods, let them be recreated properly.

Now the honest half. That was design and luck, and I'm not going to pretend it was all design.

The design part is real: authentication that doesn't sit in the critical path of every scan is a deliberate choice, and it earned its keep that evening. The luck part is equally real: it was the opening day, traffic was thin, and the tokens in circulation hadn't expired. Move that same failure to Saturday at 11:00 — 239 reads an hour, staff needing to re-authenticate — and I'd be writing a very different post.

That counterfactual is why the permanent fix matters. The configuration now says the opposite: if the sidecar can't be attached, don't start the pod at all. Refuse loudly rather than proceed silently. It's better to have a pod that won't start than a pod that lies about being ready.

The problem nobody noticed

Going through the data afterward, I found something no one reported and no alert caught.

From 19:00 onward, median read time went from 0.7 seconds to 2.1 seconds — and stayed there until closing. Three times slower.

The strange part: this happened as traffic fell. The busiest hour of the day was one of the fastest. The slowest hours were the emptiest ones. That's backwards from how you expect a system to behave, and it's the signature of things going cold — connections, caches, pooled resources quietly expiring between requests, so that each new scan pays for setup the previous one had already paid for.

Nobody complained, because at 50 reads an hour nobody feels 2.1 seconds. But run that same degradation through the 11:00 rush and it's a visible queue at the main door. I found a problem that hadn't hurt anyone yet — which is the only comfortable time to find one.

The lesson that isn't software at all

The main door reader slowed down after thirteen hours on battery.

That's it. That's the whole failure. A device that had been awake since morning began throttling itself, and no amount of backend engineering had anything to say about it. It's the kind of thing you cannot test, cannot simulate, and cannot predict from a laptop — because your laptop is plugged in and has been awake for forty minutes.

Conventions need power planning: packs at every door, a charging rotation, battery-saver modes explicitly disabled on the reader devices. Nightlife venues don't have this problem — a bar runs four hours, not thirteen. I only know the difference because I stood in one and watched it happen.

And the humans did what humans do

Roughly 15% of all reads were rejections, and almost all of them said the same thing: already inside.

Not attackers. Not bugs. People walking out a side door for lunch without scanning out, then coming back in and being told, correctly, that the system already had them inside.

The state machine was right. The users hadn't read it. That gap is not a defect you can fix in code — it's signage, staff prompting at the exits, and designing for the fact that people at a fair are thinking about lunch and not about your presence model.

What I'd change

• Fail loudly, not silently. Done — the sidecar configuration is inverted, permanently.
• Ask health checks the question that matters. Not "is the process up" but "can it reach the things it needs." A pod reporting Running while deaf is worse than one that's clearly down.
• Watch latency against load, not just against a threshold. Getting slower while getting quieter is a signal, and no alert I had was shaped to catch it.
• Bring power. Unglamorous, decisive, and the only fix on this list that costs nothing to implement.
• Prompt the scan-out. Most of that 15% is solvable with a sign and a person saying "scan before you step out."

The honest summary

Four things went wrong across nine days: an outage during the event, a threefold slowdown nobody noticed, a reader throttled by its own battery, and a rejection rate driven entirely by human behaviour.

And the doors kept moving. A thousand people got in on day one, at sub-second scans, and the failures either stayed behind the scenes or were absorbed by choices made months earlier.

I could have written this post as a victory lap. But "nothing went wrong" is a claim nobody experienced believes, and it's not what happened. What happened is that things went wrong and the event didn't feel them — which is a smaller, truer, and much harder thing to build.

Thanks to Feria Arequipa for the trust. Tuesday: what tests can't show you — which, as you may have guessed, is mostly this post's greatest hits. ✨