The SRE Book, Ten Years On — Part 5: On-Call and the 25% Rule
Every pager rotation contains a hidden payroll problem. Treating human attention as the scarcest resource in the system is where Google’s on-call practice shows its receipts, and the psychology of interruption fills in the rest. Together they answer a question most orgs never ask: how much of a human should a pager be allowed to consume?
Four things determine the answer.
In this part
- The explicit budgets — the 25% rule, two incidents per shift, and the staffing arithmetic underneath both.
- The neuroscience of a 3 a.m. page — what stress hormones do to the exact reasoning an incident needs.
- The risk of paging too little — why operational underload is as dangerous as burnout, and quieter.
- The true cost of an interrupt — what it takes from everything around it, not just the minutes it consumes.
The Budgets of the Pager
On-call runs on two explicit budgets, and their precision is the point:
- The 25% rule: on-call work should consume at most 25% of an engineer’s time — half of the 50% ops cap from Part 1, because on-call is only half of ops work.
- The quality budget: a healthy on-call shift handles at most 2 incidents per 12-hour shift. The arithmetic: a typical on-call incident — triage, root-cause, remediation, and the postmortem — takes about 6 hours end to end, so two per shift is a ceiling, not a target. The median should be closer to zero: if a component pages every single day, something else in the system is about to break too, and you’re already over-budget without knowing it.
- A concrete reaction-time floor: response time ties directly to the SLO. A user-facing service targeting four nines (99.99%) has roughly 13 minutes of downtime to spend per quarter — which means the on-call engineer’s acknowledgment window has to be on the order of minutes (Google’s own defaults: 5 minutes for time-critical services, 30 for everything else). Loosen the SLO and the human gets more slack; tighten it and the pager gets faster, whether or not the org has hired for it.
The arithmetic behind the rotation is equally concrete: a single-site team needs 8 engineers minimum (2 per shift, 4 shifts a day… actually: one primary + one secondary per shift), and “follow the sun” across two sites needs 2×6.
If your org has three engineers and a pager, the honest answer is not “suck it up” — it’s that you’re under-staffed for the reliability level you’ve chosen, and the fix is either hiring or reducing the SLO (Part 2’s lever, again).
The recurring theme: pager fatigue is a system failure, not a personnel problem. A burned-out on-call engineer doesn’t care about your error budget. They care about the next page. Burnout is a reliability defect with a measurable root cause, not a personality problem.
Feeling Safe: The Cognitive Science of a 3 a.m. Page
The most underrated part of on-call design isn’t arithmetic at all — it’s neuroscience. Daniel Kahneman’s dual-process model (the “thinking fast and slow” framework) explains why a page forces a choice between two modes of thought: intuitive, automatic, rapid action, or rational, focused, deliberate analysis.
Under a real outage, the first mode usually wins, and the mechanism is well understood: stress hormones — cortisol and corticotropin-releasing hormone — measurably impair the cognitive functions the second mode needs, pushing a stressed engineer toward unreflective heuristics exactly when the incident most needs the opposite.
The classic trap: a page fires for the fourth time this week, and the previous three were caused by the same upstream infrastructure fault. The tempting move — confirmation bias — is to assume cause four is cause one again, without checking. Sometimes it is. Sometimes it’s the new bug hiding behind the old, familiar noise.
The takeaway
The real counter to confirmation bias isn't willpower; it's structural: clear escalation paths, well-defined incident procedures, and a blameless postmortem culture (Part 7) exist specifically so an exhausted engineer has a scaffold to lean on instead of a heuristic to gamble on.
Operational Underload: The Failure Mode Nobody Anticipates
The most counterintuitive part of on-call design is the warning that underload is dangerous too. An on-call rotation that never pages lulls the team into complacency — the engineers forget the runbooks, the instincts decay, and the first real incident in a year hits a team with zero live-fire practice. Underload is “a treacherous enemy” — quieter than burnout, but it produces the same bad outcome: a team that can’t respond when it matters.
Google’s answer is the Wheel of Misfortune (and its bigger sibling, DiRT — Disaster Recovery Testing): scheduled drills where engineers respond to injected failures, purely for practice. Part 6 covers DiRT’s philosophy; the point here is that on-call is a skill that requires rehearsal, and a silent pager is a rehearsal skipped.
Figure: follow-the-sun passes the pager around the globe so nobody owns the 3 a.m.; the budgets and the underload cure sit on the sides.
The Cost of a Context Switch
Engineering is flow work, and every interrupt costs the team not the minutes of the page but the lost flow around it — the half-hour to reload context, the day’s momentum broken. The countermeasures are organisational, not personal:
- Polarised time: block the calendar into full-interrupt periods and full-project periods, so nobody oscillates between modes all day.
- The push-manager: a single person whose job is to absorb interrupts from outside the team and triage them — shielding everyone else’s flow.
- Ticket scrubs: batch-review tickets so the small stuff never fragments the day.
And the sharpest edge: “give back the pager.” When a service generates too much operational load, the overflow goes back to the development team — not as punishment but as feedback. The people who build the system must feel its operational weight; otherwise the 50% cap (Part 1, given its full depth in Part 3) has no teeth. It’s the mechanism by which operational pain becomes engineering motivation.
Beyond Google: Uber’s OODA Loop and the “omg” CLI
Google isn’t the only company that turned on-call into an engineering discipline rather than a chore roster. Uber’s approach leans on a different vocabulary borrowed from military doctrine: the OODA loop — observe, orient, decide, act — as the explicit mental model for how an on-call engineer should move through an incident, paired with the same golden-signals instinct (traffic, errors, latency, saturation) as the starting point for observation.
The more interesting artifact is tooling built to reduce cognitive load rather than add to it. Uber built a command-line tool, internally called omg, specifically so an engineer paged into an unfamiliar service doesn’t have to remember a dozen different dashboards and runbook locations — one interface, consistent across services, for the first sixty seconds of response, when working memory is at its most expensive.
It’s the “push-manager” idea taken to its tooling conclusion: if you can’t eliminate the interrupt, at least standardise the response so the interrupt costs less each time.
Ten Years On: The Industry PagerDuty Built
The most visible change since 2016 is that the pager became a product category. PagerDuty (founded 2009) and its peers — Opsgenie, incident.io — turned the rotation arithmetic into software: scheduling, escalation policies, on-call compensation tracking.
Google’s own compensate-with-time-or-cash policy became the reference point for a decade of industry debate about paying engineers for the pager, and PagerDuty’s annual State of Digital Operations reports document the after-hours burnout that on-call practice predicted. The Site Reliability Workbook, published two years later, doubled down on the same arithmetic with real customer data rather than just Google’s — proof the 25%/2-incident budget wasn’t a Google-only artifact of Google’s scale.
The deeper shift is structural. In 2016, “give back the pager” was a safety valve inside Google’s model. In 2026, you build it, you run it is the default for most startups and half of the large companies — the dev team never gives the pager away in the first place.
The insight survives the inversion: whatever the org chart says, operational pain must flow back to the people who create it, or the system never improves. Central SRE, embedded SRE, platform teams — the labels differ, but the pager arithmetic (25%, 2 incidents, postmortems) is the same, and it’s still under-staffed wherever you look.
The newest wrinkle is the pager itself getting an assistant. The 2026 generation of AI SRE agents — Microsoft’s Azure SRE Agent and AWS’s DevOps Agent both reached general availability in March 2026 — sit in front of the human on-call, triaging alerts before a page ever lands: correlating metrics, traces, and logs, surfacing a probable cause with historical precedent, and for well-understood failure classes, executing a pre-approved runbook without waking anyone up.
That’s a genuine dent in the quality budget — fewer of the two incidents per shift are ones a human needs to touch at all.
The caveat is precisely the one this whole framework anticipated: these agents are reliable in proportion to how much the failure resembles something seen before, and current research on LLM-based root-cause analysis (the “RC-LLM” approach to fusing traces, metrics, and logs into a causal explanation) is explicit that accuracy drops hard on genuinely novel failure modes.
The takeaway
An AI triage agent inherits the same confirmation-bias trap that was always a risk for a tired human — it just does it with more confidence and less hesitation. The pager got quieter; the two incidents that still reach a human are, increasingly, the two hardest ones in the shift.
Key takeaways
- The budgets are arithmetic, not preference. 25% of an engineer's time on-call, at most two incidents per 12-hour shift, and an acknowledgment window your SLO dictates — a four-nines service has roughly 13 minutes of quarterly downtime to spend.
- Under-staffing shows up as a reliability choice. If three engineers carry a pager, you're not under-resourced by accident — you've implicitly picked a lower reliability target, and the honest fix is hiring or lowering the SLO.
- Stress degrades exactly the reasoning an incident needs. Cortisol pushes a tired engineer toward fast heuristics and confirmation bias, so the counter has to be structural — escalation paths, written procedures, blameless review — not willpower.
- A silent pager is a rehearsal skipped. Operational underload decays runbooks and instincts quietly, which is why drills like the Wheel of Misfortune exist: on-call is a skill that needs practice, not just coverage.
- Interrupts cost far more than their minutes. A single 20-minute interruption can cost hours of genuinely productive time, which is why the answer is polarising time — never asking one person to be interruptible and deeply focused in the same stretch.
Next: Part 6: How to Fix a Broken System — Troubleshooting and Emergency Response, on the method behind the pages.
References
- Google SRE Book — Being On-Call and Dealing with Interrupts
- The SRE Workbook — On-Call
- PagerDuty (Wikipedia)
- Azure SRE Agent GA announcement (March 10, 2026) · AWS DevOps Agent GA announcement (March 31, 2026)
- AI in Incident Response 2026: The Data — Traversal · RC-LLM
- Inside Uber’s Incident Response Plan — Interview with an Expert
Have thoughts on this?
I read every email. If something resonated, felt wrong, or made you think — I'd love to hear from you.
Comments