The SRE Book, Ten Years On — Part 12: The Cockpit
The “management” side of SRE is the part people skip, and the part that aged best. It contains the training model, the embedding model, the communication rituals, and — most prescient of all — the quiet blueprint for what the industry would later call platform engineering.
This closing post ties together five loose threads.
In this part
- Training — how a team teaches someone into on-call instead of throwing them in.
- Interrupts — how it protects attention from the pages and tickets fighting for it.
- Embedding — how one SRE joins an already-overloaded team and changes it.
- The engagement model — how the whole organisation's approach evolved into something the industry would later rename platform engineering.
- Ten years on — what a decade of data says about the claims that were still unproven in 2016.
Training: No Trial by Fire
On-call readiness is taught, not survived, and five practices are concrete enough to copy wholesale.
Figure: five deliberate steps from "trained" to "trusted" — trial-by-fire only ever produces the first.
Reading and sharing postmortems treats a team’s own outage history as curriculum — some teams run standing “postmortem reading clubs” or “tales of fail” gatherings where the original author walks a room through what happened, on the theory that a postmortem’s most appreciative future reader is someone who hasn’t been hired yet.
Disaster role-playing — known internally as “Wheel of Misfortune” or “Walk the Plank,” and famously described by one Google SRE as “SRE Zork” — puts two trainees in front of the team as primary and secondary on-call while a “game master” unspools a real or hypothetical incident live, redirecting them away from red herrings and improvising as any escalation contact they’d need to page.
Break real things, fix real things goes further than a tabletop exercise: the Search SRE team runs a quarterly ritual nicknamed “Let’s burn a search cluster to the ground,” where the team predicts, out loud, what will break before they deliberately impair a live-but-diverted stack piece by piece and compare their guesses to what actually happens — a habit that reliably surfaces bugs in how the system claims it degrades versus how it actually does.
Documentation as apprenticeship turns the team’s on-call learning checklist into a living contract: a new hire is assigned to overhaul its most outdated section, paired with the senior engineer or developer listed as that section’s expert, and their revision has to survive peer review before it’s trusted.
And shadowing on-call early, capped by a technique called “reverse shadowing” — where a newly-minted on-caller takes the primary seat for real while their mentor silently shadows them, watching without touching anything, ready to help only if asked — is the deliberate last step between “trained” and “trusted.”
The deeper claim ties all five together: trial-by-fire produces engineers who know their war stories and nothing systematic; structured training produces engineers who know the system.
Interrupts: Humans Are Imperfect Machines
This is the most psychologically honest idea in the whole discipline, and it’s never gotten its due — mostly because it was never given its own post until now.
The taxonomy of operational load is simple but load-bearing: pages (production emergencies, minutes-scale SLOs), tickets (customer requests, hours-to-weeks-scale SLOs), and ongoing responsibilities (rollouts, ad hoc questions — toil with no SLO at all, but real interruption cost regardless).
The central idea borrows from psychology rather than engineering: cognitive flow state, the “in the zone” condition of clear goals, immediate feedback, and a sense of control that any focused, skilled work can produce — and, counterintuitively, that on-call itself can produce, when an engineer is fully devoted to it rather than trying to split attention with project work.
Figure: the difference isn't on-call itself — it's whether an engineer is asked to hold it alongside deep work in the same stretch of time.
The practical answer is polarizing time: never ask a person to be interruptible and deeply focused in the same period.
The takeaway
A person on-call for the week should have that week entirely written off for project purposes — not because on-call is less valuable, but because a single 20-minute interruption during deep work costs far more than 20 minutes: the estimate is a couple of hours of genuinely productive time per context switch.
The rule
- Stop randomly assigning tickets across an entire team. A dedicated rotation, sized to the actual volume, protects everyone else's flow time.
- Ask why the same class of ticket keeps recurring. A team that never does is running a gauntlet rather than doing engineering.
- Run a root-cause scrub: once an interrupt class is identified as fixable, silence the noisy alert or auto-triage rule until the fix lands — don't let it tax every future rotation indefinitely.
Embedding: The Three Phases
There’s a playbook for the most common SRE engagement: parachute one SRE into an overloaded team.
Figure: the embedding playbook in order — skipping straight to phase 3 is the most common way this engagement fails.
The two sharpest observations: the “Bad Apple” theory — that an overloaded team must contain bad engineers — is empirically false; and the diagnostic shortcut — “if you find a team without SLOs, start there.” An absence of SLOs is the canary for every other absence.
Communication as Infrastructure
A team’s information flow deserves the same treatment as a service’s data flow: as an interface, and a bad interface is painful to fix later. The concrete artifact is the production meeting — weekly, 30–60 minutes, chaired on rotation, working a fixed agenda (upcoming changes, metrics, outages, paging events, nonpaging events, prior action items) so that everyone leaves with the same idea of what’s going on, not just their own slice of it.
The best case study here, Viceroy, is a preview of what’s coming next in miniature: half a dozen teams independently built their own monitoring-console framework, discovered each other mid-flight, and spent two years merging the duplicates into one shared platform — the same “stop reviewing, start building shared infrastructure” instinct that becomes a doctrine next.
The Engagement Model: The Prophecy
This is the single most prescient idea in the whole discipline, and deserves unpacking in full: Google’s engagement with product teams evolved from PRR (Production Readiness Review) — a gate at the end, reviews at launch time — through early engagement — SREs involved during design — toward a third model: Frameworks and SRE Platform — where SRE stops reviewing individual services and instead builds shared infrastructure that makes every service reliable by default.
That third model is, word for word, the definition of platform engineering — the movement that, a half-decade later, produced Backstage, golden paths, the CNCF platform white papers, and the “SRE is dead, long live platform engineering” discourse from Part 1. Google didn’t predict the future; it wrote it, and the industry renamed it. The honest 2026 reading: classic per-service SRE didn’t die — it specialised. The judgment work (error budgets, incident command, postmortems) stayed human; the repetitive reliability machinery moved into platforms. Exactly as predicted.
Lessons from Other Industries
This is the humility section of the whole discipline: aviation simulators and lifeguard drills as DiRT’s ancestors (Part 6’s drills are a hundred-year-old idea), the nuclear Navy’s refusal to automate away human judgment, and the near-miss reporting systems that catch failures before they become incidents.
The punchline is the 747 cockpit: two pilots, a thousand failsafes — and the observation that the failsafes exist because the humans are fallible, not because they’re replaceable. This final image is the series’ thesis in miniature: reliability is a human system with engineering support.
Figure: the three engagement models — the third one, written in 2016, is what the industry spent the 2020s building.
Ten Years On: The Data Caught Up
The most satisfying coda: the people-and-culture ideas here were argued from experience, not evidence — and the evidence arrived.
Figure: what was argued from experience alone in 2016 has since been independently validated four separate ways.
Project Aristotle, Google’s own multi-year study of 180 teams (findings public from 2015–2017), found psychological safety to be the single strongest predictor of team effectiveness — the empirical vindication of the blameless-postmortem culture already covered, from the same company, published in the same era.
DORA’s Accelerate (Forsgren, Humble, Kim, 2018) turned the release-engineering intuition above into the four key metrics — deployment frequency, lead time, time to restore, change failure rate — now five, with deployment rework rate added. The claim was “velocity and safety are the same investment”; DORA spent a decade proving it with data.
Google’s own two follow-ups extend the same thesis outward and sideways:
- The Site Reliability Workbook (2018) is full of outside companies’ hard-won case studies of adopting SRE practice on their own infrastructure — proof the model exports past Google’s walls.
- Building Secure and Reliable Systems (2020) makes the harder argument: security and reliability were never actually separate disciplines, just separately staffed ones. The same blameless, evidence-driven culture already described for outages turns out to be exactly what a security incident needs too.
The decade’s least comfortable question, though, belongs squarely here. If AI-SRE agents — the ones Observability Engineering’s chapter on LLMs and AI agents covers from the tooling side — absorb the known-pattern half of the operational-load taxonomy (the repetitive tickets, the pages that match a historical incident closely enough for an agent to triage and even remediate on its own), the honest reading isn’t simply “less toil, more flow time.”
The uncomfortable part
The satisfying tier of on-call — the "Angry Birds" flow state described above, the pattern-matching, closing-tickets, watching-the-graphs-recover kind of engagement — is disproportionately the tier most easily automated away, while the genuinely novel failures (the ones this whole series has repeatedly found sit exactly where AI root-cause analysis is weakest) are what's left for humans to carry alone.
Meta’s own push toward AI-augmented production-engineering workflows is an early, live experiment in exactly this trade-off — asking, in practice, whether removing the easy tier of interrupts leaves humans with more flow time, or just leaves them holding only the hardest, least satisfying tier of the job.
So where does the decade leave SRE? The founding insight — operations is engineering, and the pager is a design problem — is no longer a Google secret; it’s the water the industry swims in. The 50% cap became platform teams. The error budget became SLO platforms. The blameless postmortem became a product category.
And the one thing that couldn’t be outsourced survives unchanged, AI-SRE agents included: somebody still has to want the system to work, with enough craft to know when it’s lying, and enough humility to drill for the failure they haven’t imagined yet. That person is the SRE — whether the job title survived the decade or not.
The Series in One Paragraph
Reliability is a product decision wearing engineering clothes (Part 2), bought with the 50% of SRE time that isn’t toil (Parts 1, 3), guarded by monitoring that respects humans (Part 4) and pagers with budgets (Part 5), defended by method under pressure (Part 6) and structure during incidents (Part 7), against the physics of cascades (Part 8), the mathematics of consensus (Part 9), and the fragility of data (Part 10), shipped through gates earned by past disasters (Part 11), and held together by the culture that trains, protects flow time, embeds, and platforms the whole thing (Part 12). Hope is not a strategy — but ten years of practice since 2016 is a pretty good one.
Key takeaways from the whole series
- Reliability is a product decision, not a technical one. 100% is the wrong target for almost everything; the error budget is what turns that fact into an incentive system both teams optimise together.
- Toil is a bug, and the 50% cap is what keeps it from winning. Without an enforced ceiling on ops work, an SRE team decays into an ops team with a better title — and the sublinear scaling that justified the whole model disappears with it.
- Every mechanism here exists because something specific broke. Checklist questions earned by past disasters, canaries earned by bad rollouts, soft deletion earned by lost data, consensus earned by split brains. The ones you skip are the ones nobody has been burned by yet.
- Structure beats willpower, every time. Blameless postmortems, incident command roles, alerting that only pages on symptoms, drills that rehearse failure before it happens — all of it exists because tired humans under pressure need scaffolding, not exhortation.
- AI changes the tooling, not the judgment. Agents are strongest exactly where failures resemble something already seen, and weakest on the novel ones — so the work that's left for humans is the hardest tier. Somebody still has to want the system to work.
References
- Google SRE Book — Accelerating SREs to On-Call, Dealing with Interrupts, Embedding an SRE, Communication and Collaboration, The Evolving SRE Engagement Model, Lessons Learned from Other Industries, Conclusion
- The Site Reliability Workbook · Building Secure and Reliable Systems
- Google re:Work — Project Aristotle
- DORA — Four Keys · Accelerate (Forsgren, Humble, Kim) · Deployment Rework Rate — the 5th DORA metric
Have thoughts on this?
I read every email. If something resonated, felt wrong, or made you think — I'd love to hear from you.
Comments