From Backup Chaos to a Tested Runbook
Resolution time dropped from hours to minutes; junior staff handle most incidents without escalation
The situation
When I joined, the backup recovery process lived inside one senior engineer's head. Every failure triggered the same scramble: page the one person who knew the system, wait for them to respond, watch them work through a diagnosis that looked different every time. Junior staff couldn't help. The playbook was "ask the senior engineer" — which meant the senior engineer got paged at 9 PM.
This is a pattern the SRE discipline has named. The senior engineer isn't reliable infrastructure — they're a single point of failure wearing a reputation. A bus factor of one. When they're unavailable, traveling, burned out, or eventually gone, the organization discovers that the recovery process left with them.
The failure modes compounded each other. Because the process was undocumented, every incident started from zero. Because every incident started from zero, recovery time was inconsistent — the same failure mode could take twenty minutes or three hours depending on who was on-call and what they remembered. Because there was no post-incident record, the organization couldn't improve. The same failures repeated. The same engineer got paged.
Meanwhile, the pressure accumulated. Alert fatigue set in. Junior staff stopped trying to help because they knew they couldn't — not because they lacked ability, but because they lacked information. The senior engineer's on-call quality of life deteriorated. And underneath all of it, the organization was exposed in a way it hadn't quantified: every unplanned recovery event carries a cost measured in minutes of downtime, and the industry median puts that cost in the hundreds of thousands of dollars per hour for systems that matter.
What I built
An eight-page runbook. Numbered. Linked. Tested.
The structure. Each entry followed a consistent format: Failure mode — what broke, described in the language of the alert or symptom Diagnostic step — how to confirm it was actually this failure and not something adjacent Resolution path — what to do, in the order you do it, with commands written out verbatim Verification step — how to confirm the fix worked before closing the incident Escalation trigger — the specific condition that meant stop trying and page up
The command verbatim requirement matters. "Dump the database" is useless at 2 AM. The exact command, with flags, is what an on-call engineer needs when they're tired and the clock is running.
Where it lived. The runbook went into the incident response system — not a wiki folder nobody opened, not a Confluence page last touched two years ago. It lived where engineers actually looked during active incidents, linked directly from the alerts that triggered them.
How it was tested. After drafting, the runbook was tested by a junior engineer who had never worked through those failure modes before. This is the critical step most documentation skips. If a junior engineer can execute the runbook cold, it's a runbook. If they need to ask questions, it's a draft.
The update contract. Every post-incident review included a check against the runbook. If the resolution path looked different from what was documented, the document changed. Runbooks without a living-document contract decay. The goal wasn't a perfect artifact — it was a maintained one.
What changed
Resolution time — dropped from hours to minutes for documented failure modes; the MTTR gap between automated-runbook teams and manual-response teams is measurable in hours, not minutes Escalation rate — junior staff resolved most incidents without paging up, because the information they needed was now accessible On-call quality of life — the senior engineer stopped getting paged at 9 PM for incidents that the runbook could handle Onboarding — new hires onboarded against the runbook instead of someone's memory; documented teams see new hire ramp times drop by months, not weeks Repeatability — the runbook format became the standard for other operational processes that had the same problem
The lesson
The SRE discipline calls this the hero engineer antipattern. It feels like reliability — one person who always knows what to do — but it is the opposite of reliable. It is a team that has organized itself around a single point of failure and called it institutional knowledge.
The fix is not glamorous. It's an eight-page document with numbered steps and verbatim commands. It is not a technical achievement. It is a discipline problem masquerading as a complexity problem, and the discipline required to solve it is not particularly difficult — it's just writing things down.
Most operational chaos isn't a complexity problem. It's a documentation problem dressed up to look complicated. The fix isn't smart. It's disciplined.