The SRE Book, Ten Years On — Part 6: How to Fix a Broken System
When the page fires, two kinds of engineers reveal themselves: the ones who start fixing, and the ones who start understanding. Effective troubleshooting is a defence of the second kind, and its first rule is the one everyone gets wrong under pressure.
What follows is a small number of habits, not a personality trait. Each has a story attached, and the last two turn out to be the same discipline practiced by companies that have never spoken to each other.
In this part
- Triage first, root-cause later — stabilise the system before you understand it, because MTTR is what users feel.
- Follow the loop, not your instincts — the hypothetico-deductive method, and the guardrails that keep it honest under stress.
- Publish negative results — an unrecorded dead end is a tax on whoever gets paged next.
- Drill for failure before it finds you — DiRT, Chaos Monkey, and why an unexecuted runbook is only a hypothesis.
Principle 1: Triage First, Root-Cause Later
The opening move in an emergency: fly the airplane. Before you understand the problem, you stabilise the system — roll back the last change, shed load, fail over.
The instinct to find the root cause first is a luxury the users don’t share: they want the service back, and the mean time to recovery (MTTR) is the metric that matters — reliability is a function of MTTF and MTTR, and MTTR is what emergency response moves.
Principle 2: Follow the Loop, Not Your Instincts
Only after triage comes the actual method — the hypothetico-deductive loop:
- Observe — what are the symptoms? What changed recently? (the sharpest question to ask: “what touched it last?”)
- Hypothesise — propose a cause that explains the symptoms.
- Test — design an experiment that would falsify the hypothesis, and run it.
- Treat — fix it, or refine and loop.
Figure: the loop's discipline is the hypothesis — every action tied to a testable belief, every negative result recorded.
The loop sounds like common sense, which is precisely why it’s worth formalising: under stress, engineers skip the hypothesis step and start treating randomly — restarting things, flipping flags, poking the system into a worse state. The discipline is to keep every action tied to a testable belief.
Two epigraphs worth carrying into every incident: John Allspaw’s “ways in which things go right are special cases of the ways in which things go wrong,” and the reminder that expertise isn’t knowing how a system is supposed to work — it’s understanding why it doesn’t.
The toolkit: simplify and reduce (remove variables until the system is minimal, ideally by injecting known test data and watching what breaks), bisect (binary-search the change history or the request path one hop at a time), and ask what, where, why — what’s broken, where in the stack, why does it behave this way, an approach that traces to the same lineage as Taiichi Ohno’s “five whys.”
Two guardrails for the hypothesis space
- When you hear hoofbeats, think horses, not zebras — most failures are the common failure, not the exotic one.
- Hold that against Hickam's dictum, the doctor's counter-rule: a patient — or a system — can have several common problems at once rather than one rare one explaining everything.
- Correlation is not causation, even at r² = 0.94 — the classic example is US computer-science PhDs awarded tracking per-capita cheese consumption almost perfectly, without either causing the other. Coincidence, shared hidden cause, or real causation: telling the three apart is the actual job.
The postmortem is the loop’s final step, and skipping it means the loop never closes.
A classic case study — the “Shakespeare” search service, alerting on a black-box probe failure — is a clean demonstration of the method: a curl against the failing endpoint returns an HTTP 502 carrying an X-Request-Trace header listing every backend the request touched, letting the engineer bisect straight to the failing tier without guessing.
The eventual root cause, once bisected down to a single function, is a regular expression with catastrophic backtracking — fixed by switching to Google’s own RE2 engine, which guarantees linear-time matching by construction rather than hoping the input never triggers exponential blowup.
It’s a small story, but it’s the whole method in miniature: observe (502, trace header), hypothesise (a specific backend), test (bisect the stack), treat (swap the regex engine) — and the tool that made the bisection possible, Dapper, is Google’s internal ancestor of the distributed tracing now standardised as OpenTelemetry (see Practical OpenTelemetry Part 6: Tracing, Supercharged Structured Logs for the modern, vendor-neutral version of exactly this idea).
Principle 3: Publish Negative Results
The most memorable aphorism in this space, and the one most teams violate daily:
Publish what didn’t work. Negative results are magic — they’re still progress.
A failed experiment eliminates a hypothesis; if you don’t record it, the next engineer re-runs it at 3 a.m. The App Engine case study makes the point: an investigation into mysterious request anomalies dragged on because correlation kept masquerading as causation — the smoking gun looked like a security scanner, and the real cause (whitelist objects interacting with the scanner) only emerged once the team documented what they’d ruled out.
The lesson
Debugging is a team sport, and an unwritten negative result is a tax on your future self — the same hypothesis gets re-tested by whoever's paged next.
Principle 4: Drill for Failure Before It Finds You
The case for drills is blunt: “until your system has actually failed, you don’t know how it fails.” So Google makes it fail on purpose. DiRT (Disaster Recovery Testing) runs scheduled, planned breakage — including in production — because a runbook that has never been executed is a hypothesis, not a plan.
Three real emergencies make the case, and they share a shape worth naming:
| Trigger | What happened | Why it spread |
|---|---|---|
| Test-induced | Access to one test database was blocked | A client library assumed the database would always be reachable — one block became a company-wide access failure |
| Change-induced | A Friday config push broke every external-facing system | It also broke the internal tools needed to fix it, forcing engineers into physical “panic rooms” with backup production access |
| Process-induced | Diskerase (from Part 3), retold from the responder’s side | Automation read an empty response as “delete everything” — and complied |
Figure: three different incidents, the same underlying grammar — and it recurs at Meta's scale below.
The right tone under fire is deliberately calm — the opening advice is always don’t panic, you aren’t alone, the sky isn’t falling — but the actual discipline is “cast your net farther”: if you can’t find a fix, pull in more people, page the whole company if you have to, because the highest priority is restoring service, not being the hero who solved it alone.
And the favourite pre-mortem exercise is a set of blunt hypotheticals every team should be able to answer before they’re tested for real: what if the building loses power? What if the primary datacenter goes dark? What if someone compromises your web server? If the honest answer is “I don’t know,” that’s the backlog item.
Beyond Google: Netflix’s Chaos Monkey Is DiRT, Independently Invented
Google isn’t the only company that concluded a runbook you’ve never executed is a hypothesis, not a plan. Netflix arrived at the same insight from a different direction, building the Simian Army — Chaos Monkey, which randomly terminates production instances during business hours specifically so a service that can’t survive losing a server finds out while engineers are awake, not at 3 a.m. — and, at the more dramatic end, full simulated region failures to prove failover actually works rather than merely existing on a diagram.
It’s the same DiRT philosophy, arrived at independently. Two companies with almost no engineering overlap converged on “deliberately break production to find out how it breaks” as the only honest way to know.
Netflix’s reliability organisation — internally called CORE — has more recently pushed the same logic one layer further into culture: a 2025 shift toward a “paved road” for incident declaration itself, so any engineer, not just a central on-call team, can raise their hand and say “this is an incident” without waiting for permission. It’s the “declare early, declare cheaply” principle, generalised from a rule for SREs into a right for the whole engineering org.
Ten Years On: The Modern Diskerase
The modern descendant of the “change-induced emergency” pattern is the most instructive outage of the decade: Meta’s October 4, 2021 disappearance. During routine maintenance, a command intended to assess global backbone capacity unintentionally disconnected everything — and an audit-tool bug let the command run instead of stopping it.
DNS servers then withdrew BGP routes, and the company that runs some of the largest private networks on Earth lost the network for about six hours, including the tools it would have used to fix itself. It is, precisely, all three shapes from the diagram above at once: a bad change executed by a bad process whose test — the audit — had silently broken. Diskerase’s lesson, replayed at the scale of a trillion-dollar company: automation amplifies bugs, and the empty set is still the most dangerous input.
Ten Years On: The 2026 Diagnosis Stack
The method aged flawlessly; the tooling got spectacular. The 2016 debugger’s tools — logs, dashboards, intuition — now include eBPF and continuous profiling (bpftrace, Parca, Pyroscope): flamegraphs of a production process while it misbehaves, no instrumentation required.
The 2026 diagnosis stack has a new tier entirely: autonomous AI SRE agents that don’t just summarise logs and draft runbooks, but — per the industry’s own framing — have moved from “AI that suggests” to “AI that investigates,” running the hypothetico-deductive loop itself.
Systems like the research prototype STRATUS coordinate specialised sub-agents for detection, diagnosis, and mitigation as a pipeline. The underlying “RC-LLM” line of research fuses traces, metrics, and logs through the same kind of causal reasoning a human troubleshooter has always had to do by hand, correlating a symptom back through the stack to a probable cause.
That same guardrail applies with new urgency, and the published research on these systems is honest about it: root-cause accuracy is substantially higher when the failure resembles something the model has seen before, and drops hard on genuinely novel failure modes — correlation-shaped pattern-matching, not the causal reasoning it’s often marketed as. An LLM’s confidently wrong hypothesis is the App Engine story again, at machine speed, with the zebra-versus-horses guardrail now needing to be enforced on a system that has no instinct for which is which.
Treat the AI’s suggestion as a hypothesis to falsify, not a diagnosis to trust.
Key takeaways
- Stabilise before you understand. Root-causing first is a luxury users don't share — reliability is a function of MTTF and MTTR, and emergency response only moves the second one.
- Every action needs a testable belief behind it. Under stress engineers skip the hypothesis step and start treating randomly; the loop exists precisely because common sense fails at 3 a.m.
- Negative results are progress, and unwritten ones are a tax. The App Engine investigation only broke open once the team documented what they'd ruled out.
- An unexecuted runbook is a hypothesis, not a plan. Google's DiRT and Netflix's Chaos Monkey are the same conclusion reached independently: you don't know how a system fails until you've made it fail.
- Outages rhyme — bad test, bad change, bad process. Meta's 2021 outage was all three at once, and a drilled team recognises the shape long before it works out the details.
Next: Part 7: Run the Incident, Don’t Let It Run You — command, postmortems, and the data that makes incidents useful.
References
- Google SRE Book — Effective Troubleshooting and Emergency Response
- Meta engineering — October 4th outage details
- 2021 Facebook outage (Wikipedia)
- How Netflix Uses Fault Injection to Truly Understand Their Resilience — Coralogix
- Empowering Netflix Engineers with Incident Management — Netflix TechBlog
- What Is AIOps in 2026? Event Intelligence Explained — Augment Code
- STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds (NeurIPS 2025) · RC-LLM: Root Cause Analysis Method Based on Large Language Models with Residual Connection Structures
Have thoughts on this?
I read every email. If something resonated, felt wrong, or made you think — I'd love to hear from you.
Comments