The SRE Book, Ten Years On — Part 7: Run the Incident, Don't Let It Run You
Incident response covers the part of the job most engineers secretly enjoy: the incident. But the real subject isn’t heroics — it’s structure. Three things form one closed loop, and skipping any step makes the loop leak.
In this part
- Give the incident a structure — ICS roles, the incident-state doc, and why the commander doesn't touch the keyboard.
- Make the postmortem blameless — what that actually sounds like in writing, and when one is required.
- Aggregate the data, don't just archive it — how a pile of postmortems becomes organisational statistics instead of anecdotes.
The Mary Story: What an Unmanaged Incident Costs
The composite horror story here is an engineer — “Mary” — handling an incident alone, in chat, while everyone else watches. She has no authority, no defined roles, no one tracking the timeline; people freelance fixes in parallel; the incident stretches for hours while nobody can say what’s been tried. The diagnosis has three named failure modes: sharp technical focus with poor communication, and freelancing. Mary is technically excellent. The structure around her is missing.
The managed version of the same incident takes a fraction of the time, because Google’s incident response imports a ready-made structure: Incident Command System roles adapted from emergency services.
Principle 1: Give the Incident a Structure
The roles, adapted from the US Incident Command System:
- Incident Commander (IC) — owns the incident: priorities, decisions, the clock. Explicitly not the person fixing things; their job is to keep the response coherent.
- Operations lead (ops) — applies operational tooling, executes the actual mitigation.
- Communications lead (comms) — the single voice to the outside: users, management, status pages.
- Planning lead — maintains the incident-state doc: timeline, what’s been tried, what’s next, who owns what.
The live incident-state doc is the quiet superpower here: one shared document as the team’s working memory, so the IC can hand off mid-incident with “You’re now the incident commander, okay?” — and nothing is lost, and the handoff isn’t complete until the outgoing IC hears that acknowledgment back.
The command channel matters too: Google leans on IRC precisely because it’s boring and reliable — a durable, bot-loggable record of who said what, independent of whatever chat tool the outage might itself be affecting — with a “War Room” as the physical or virtual fallback for anyone who needs to be in the same room as the decision.
And the separation of roles is deliberately recursive: if the Ops lead gets overwhelmed, they don’t quietly struggle, they ask the Planning lead for more hands and delegate a subincident, so the structure scales down to a two-person page and up to a whole-company outage without changing shape.
The rule: when to declare an incident
- Do you need to coordinate a response across more than one person?
- Do you need to control the message to users or management?
- Is this work's priority above everything else right now?
A yes to any one is enough. Declaring is cheap; pretending an incident isn't an incident is expensive.
Principle 2: Make the Postmortem Blameless
Blameless postmortem culture is the idea that changed industry culture. Google operates under a blame-free postmortem culture: the goal is exposing faults and engineering fixes, not finding culprits. The case for it is ruthlessly practical — blame makes people hide mistakes; hiding mistakes guarantees they recur.
A postmortem is a document with structure (what happened, the timeline, the root causes, the action items), and the review culture around it (postmortems reviewed by peers, shared org-wide, the “postmortem of the month” read-aloud) is what makes them stick.
The optics problem has a real answer too: at a 2014 company-wide all-hands, an engineer described a release that took down a critical service for four minutes despite thorough testing — and rather than facing consequences, he received two peer-nominated cash bonuses and applause from an audience that included the company’s founders, specifically for the quality of his rollback and his postmortem.
The lesson
Reward the handling, not the luck. Whether an outage happens is often luck; how well it's handled and written up never is — and that's the only part a culture can actually reinforce.
What blameless actually sounds like in writing is just as concrete. Contrast “we need to rewrite the entire complicated backend system, I’m sure we’re all tired of fixing things onesy-twosy” (blame, dressed as frustration) against “an action item to rewrite the backend might actually prevent these pages, and the maintenance manual is too long to fully train up on” (the same underlying complaint, rewritten around the system instead of the people in it) — the difference is not tone, it’s subject.
Postmortems aren’t written for every hiccup — a concrete set of triggers decides when one is required:
Write a postmortem when
- User-visible downtime crosses a threshold.
- There's any data loss.
- An on-call engineer had to roll back or reroute traffic.
- Resolution time crossed a threshold.
- A monitoring failure meant a human found the problem before the alerting did.
And the culture is actively cultivated, not assumed: a monthly “postmortem of the month” newsletter, postmortem reading clubs (old incidents, revisited with snacks, sometimes years later), and the Wheel of Misfortune — a past incident re-enacted live with a cast of engineers playing each ICS role, the original incident commander attending to keep it honest. None of this is unique to Google, either: Etsy open-sourced its own postmortem-tracking tool, Morgue, years before most companies had a name for the practice at all.
Principle 3: Aggregate the Data, Don’t Just Archive It
Closing the loop means going beyond individual postmortems. They answer “what happened here?” but can’t answer “what keeps happening everywhere?” — postmortems alone under-count because they only fire above a severity threshold, missing the frequent-but-small failures that quietly cost the most in aggregate.
Google’s answer is a pair of tools with a clean division of labour:
| Tool | Layer | What it does |
|---|---|---|
| Escalator | Mechanical | Tracks whether a page was acknowledged, and re-routes it up the chain if not — primary to secondary, and beyond |
| Outalator | Analytical | Links related alerts into one incident, tags them with a self-organising vocabulary, and rolls the tagged history into trend analysis |
The Outalator’s tagging is deliberately lightweight: colons imply hierarchy, so cause:network can specialise into cause:network:switch without anyone designing a taxonomy up front. The trends that fall out are the payoff — which services page most, which failure modes keep recurring, which fixes actually stuck.
Without it, the organisation learns anecdotes one postmortem at a time; with it, the organisation learns statistics — the same move from raw signal to organisational insight covered from the observability side in Observability Engineering Part 3: From Data to Insights.
Figure: command structure on top, the learning loop below — the postmortem and the Outalator are what make incidents compound into improvement.
Beyond Google: Amazon’s Correction of Error, Microsoft’s IcM
Google’s blameless postmortem is the best-known version of “write down what broke and why,” but it isn’t the only design. Amazon’s equivalent — the Correction of Error (COE) — is deliberately not a postmortem by another name: where a postmortem’s job is understanding, a COE’s stated job is generating corrective action items, and it’s wired directly into a feedback loop with Operational Readiness Reviews (ORRs), the pre-launch checklist every new AWS-style service goes through.
A COE’s own template asks, in effect, when the affected workload’s last ORR was and whether any of its recommendations would have prevented the event — turning Google’s separate “postmortem culture” and “launch checklist” (Part 11) into one closed loop by design rather than by convention.
Microsoft’s incident-management practice (IcM) draws its own version of the ICS lifecycle as four explicit phases — create, acknowledge, mitigate, resolve — a compressed, tooling-first restatement of the same declare-manage-handoff-resolve arc, now with Microsoft’s own AI operations agent plugged directly into the “mitigate” step for well-understood failure classes.
Figure: same underlying discipline, three different companies, three different emphases — understanding, corrective action, and tooling.
Ten Years On: The Tooling Arrived, the Debate Got Sharper
The ironic, satisfying arc: in 2016 this was all bespoke internal machinery (the incident-state doc, the Outalator) that didn’t exist commercially. In 2026 it’s a crowded category: incident.io (founded 2021 by ex-Monzo engineers, literally built around ICS roles), FireHydrant, Rootly, Blameless, Transposit — all productising exactly this workflow.
The newest entrants have gone further than “AI drafts the timeline”: the 2026 wave of AI SRE products bill themselves as triaging alerts, investigating the failure, and generating the postmortem draft with minimal step-by-step direction — collapsing the four ICS roles and the postmortem write-up into one automated first pass, with a human reviewing rather than authoring. The tooling this discipline always needed became a market, and the market is now trying to automate the market.
And the philosophical debate matured. The word “blameless” itself has been stress-tested by a decade of criticism: taken to an extreme, it can shade into consequence-free, where repeat offenders never feel accountability. The modern synthesis — visible in the best public postmortem cultures, Cloudflare’s and GitHub’s among them — is blameless but accountable: no individual witch-hunts, but real ownership of action items, real tracking that fixes land. This was anticipated from the start: blamelessness was never the goal; learning was. Blame is just the thing that most reliably prevents it.
The closing lesson is the one Mary’s story teaches: an incident is a social event. The technical problem will yield to Part 6’s loop; the organisational problem — who decides, who talks, who remembers — is what determines whether the next hour is fifty minutes or five.
Key takeaways
- An incident is a social event, not just a technical one. Mary was technically excellent — what was missing was structure: who decides, who talks, who remembers.
- The commander owns the clock, not the keyboard. Separating command from execution is what stops freelancing, and the roles recurse cleanly from a two-person page to a whole-company outage.
- Declaring an incident is cheap; pretending it isn't one is expensive. A yes to any single trigger — coordination, comms, or priority — is enough.
- Blame makes people hide mistakes, and hidden mistakes recur. The modern synthesis isn't consequence-free: it's blameless but accountable — no witch-hunts, but real ownership that action items actually land.
- Individual postmortems produce anecdotes; aggregation produces statistics. Severity thresholds mean postmortems systematically under-count the frequent-but-small failures that cost the most in total.
Next: Part 8: Why Systems Fall Over — Load, Overload, and Cascading Failure, the physics of failure.
References
- Google SRE Book — Managing Incidents, Postmortem Culture, Tracking Outages
- incident.io · The SRE Workbook — Incident Response
- Amazon Correction of Errors Process — AWS · AWS Well-Architected — Operational Readiness Reviews
- Microsoft Azure Well-Architected — Incident Management (IcM)
- What is AI SRE? — incident.io
Have thoughts on this?
I read every email. If something resonated, felt wrong, or made you think — I'd love to hear from you.
Comments