The SRE Book, Ten Years On — Part 3: The War on Toil

More than any other, one concept defines the discipline of site reliability engineering: toil. It’s a word that didn’t exist in the operations vocabulary before 2016, and it’s now the single most useful diagnostic term for explaining why an engineering org is drowning.

The definition is strict, but not a checklist. Toil is work that tends to carry these six qualities:

Not every task deemed toil carries all six — the closer a task matches these descriptions, the more likely it is toil.

The carve-outs matter as much as the definition: overhead (meetings, HR, reviews) is annoying but not toil; grungy work that creates enduring value (writing docs, cleaning up a legacy config) is not toil either, even though it feels similar in the moment.

Toil is specifically the work that shouldn’t be done by humans — and in an SRE org, it’s tracked, measured, and treated as a bug to fix, not a cost of doing business.

The 50% Cap Is a Promise to Hires

Part 1 mentioned the 50% cap. Here’s what gives it teeth.

Google caps the aggregate ops work for all SREs at 50%, measures where time actually goes, and redirects overflow back to the development teams when the cap is breached.

In practice the average lands around a third — Google’s own quarterly surveys put it at roughly 33%, well under the 50% ceiling, though the average hides real spread: some SREs report 0% toil, others 80%, and the right response when that happens isn’t to average it out but to look for a manager failing to spread the load.

There’s also a hard floor nobody can automate away: primary and secondary on-call shifts alone impose a minimum, because someone has to be reachable. In a 6-person rotation that floor is 2/6 ≈ 33%; in an 8-person rotation it’s 2/8 = 25% — which is exactly the arithmetic Part 5 covers when it gets to on-call staffing.

It’s worth stating plainly: without constant engineering, operations load increases and teams need more people just to keep pace — “the team tasked with managing a service needs to code or it will drown.”

The cap is also a hiring promise: come work here, and half your time will be engineering. It’s the reason SRE can hire actual software engineers instead of burned-out operators.

And excess toil costs more than the individual’s time: it stalls careers, drops morale, sets a precedent where dev teams keep shifting more ops work onto SRE because SRE keeps absorbing it, and eventually drives out exactly the engineers a team can least afford to lose.

Everything from here is really one question, answered five different ways: how do you actually get toil down, and keep it down? Not one trick, but five habits that compound — each with a real story behind it.

Principle 1: Climb the Automation Ladder

Automation isn’t binary. It’s a ladder of five levels, from no automation at all to systems that run themselves:

1. No automation — a human does everything.
2. Home-directory scripts — automation that lives on one person’s machine and dies with them.
3. Generic automation — a real system, run by many, but not designed with the service in mind.
4. Built-in automation — the system is designed for automation; state is visible and actionable.
5. Autonomous systems — the system self-heals: automatic, not just automated.

The blunt truth: the biggest gap is between level 2 and level 3. Home-directory scripts are where automation goes to be forgotten; the jump to a shared, generic system is the moment automation stops being a personal hack and starts being infrastructure.

Google’s own Ads Database is the clearest proof of what climbing that ladder is worth. It ran on MySQL from 2005, with an SRE team doing manual master failovers that took 30–90 minutes each — level 1 on the ladder, a human doing everything.

Migrating onto Borg (the cluster scheduler that later spawned Kubernetes) meant tasks could be rescheduled onto different machines once or twice a week — intolerable at that failover latency, and the error budget required failovers under 30 seconds, a number no human-driven process could hit.

The team built an automated failover daemon, nicknamed Decider — level 4, built-in automation — that got failovers under 30 seconds in 95% of cases. The payoff compounded: operational maintenance work dropped by roughly 95%, and consolidating instances onto shared Borg machines freed up 60% of the team’s hardware. That’s the sublinear-scaling curve from Part 1, made concrete — one rung climbed didn’t just remove toil, it freed the engineering time that then climbed the next rung.

Climbing the ladder gets you most of the way there. But two teams can both sit at the same rung and get wildly different results, because how well the automation works is a separate question from how much of it there is.

Principle 2: Automation Quality Is a Three-Way Tradeoff

Three axes are what automation quality actually moves on, and they matter more than the ladder itself: competence (does it do the job right?), latency (how fast does it run once triggered?), and relevance (does it cover what this specific service actually needs, including as that service changes?).

Google’s own cluster-turnup automation lived through all three failure modes in sequence. It started competent, high-latency, and relevant — service owners ran their own scripts, slowly, but they knew exactly what their service needed.

To cut latency, Google centralised the work into a dedicated “turnup team” — and latency dropped, but competence and relevance dropped with it, because the people running the automation were no longer the domain experts who understood why each step existed.

The eventual fix wasn’t more centralisation or less; it was service-oriented turnup, where each team owns an API (an “Admin Server”) that a central trigger calls — competence and relevance come back because the owning team maintains it, latency stays low because triggering is centralised.

The lesson generalises past cluster turnup: automation quality isn’t a knob you turn up, it’s a three-way tradeoff you have to keep re-solving as ownership shifts.

Both principles so far assume the automation does what it’s supposed to. The next one is about what happens when it doesn’t — and it might be the best engineering story in all of SRE lore.

Principle 3: Verify Before You Trust It

The real story is a decommissioning workflow for third-party colo racks — the CDN edge nodes Google uses to terminate connections closer to users, and the events below actually happened during that process.

Diskerase, the workflow’s last step before a rack is decommissioned, wipes every disk, then verifies.

One run of the decommission automation failed partway through, after Diskerase had already completed on that rack. When the process was restarted from the beginning to debug the failure, the automation computed the set of machines still needing Diskerase — correctly, this time, the set was empty, because the previous run had already finished it.

But the empty set had been special-cased to mean “everything.” Within minutes, nearly every machine across all of Google’s CDN colo racks had its disks wiped.

The team’s own datacenters kept serving users throughout — the failure was caught fast enough, and capacity planning was generous enough, that the externally visible effect was a small latency bump most users never noticed.

The internal cost was two days spent physically reinstalling the affected racks, followed by weeks auditing every other piece of decommission automation for the same class of bug, adding rate limiting, and making the whole workflow idempotent so a restart could never again re-interpret “already done” as “do everything.”

The empty set is the most dangerous input in computing.

Diskerase was an accident. Netflix’s Chaos Monkey is the same principle, done on purpose: it automates the injection of failure instead of just the response to it, on the theory that untested automation is just automation you haven’t found the bug in yet.

Chaos Monkey and the broader Simian Army it spawned randomly terminate production instances during business hours, on the premise that a service that can’t survive losing one server will find out while the engineers who can fix it are awake, not at 3 a.m. Netflix’s reliability team (now called CORE — Cloud Operations Reliability Engineering) has since pushed the same idea to simulating entire AWS regions going dark — verification, run continuously instead of hoped for.

Principle 4: Keep It Simple, or Toil Regenerates

Toil keeps regenerating even after you’ve automated it away, and simplicity is the reason why: complexity is what creates new toil faster than automation can retire the old kind.

“Boring” is a compliment for production software — Google engineer Robert Muth’s line, “unlike a detective story, the lack of excitement, suspense, and puzzles is actually a desirable property of source code,” captures the whole idea in one sentence.

The useful distinction underneath it, borrowed from Fred Brooks, is essential complexity (inherent to the problem, can’t be removed) versus accidental complexity (a byproduct of how you happened to build it, and fully removable).

What happens when you don’t: Knight Capital lost $440 million in 45 minutes in 2012 after a deployment re-activated an old, dead flag-gated code path in production.

Minimal APIs, versioned protobuf schemas, and small, isolated releases are the same instinct applied at different layers: every one of them is a way of keeping the surface area small enough that a human can still hold the whole system in their head, which is what makes the next automation effort tractable instead of a rewrite.

Principle 5: Build Products, Not Scripts

Auxon is the proof that SREs build products, not scripts. Google’s capacity planning had become a swamp: every team asking for hardware, every request hand-tuned. Auxon flipped the model: specify the requirements, not the implementation. Teams declared intent — “N+2 redundancy per continent” — and the system computed what hardware that implied, across every data centre.

The principle generalises far beyond capacity: intent-based systems separate what you want from how it’s achieved, and let the machine own the how. Kubernetes manifests, Terraform, and every declarative tool since are the same idea.

That’s the whole set: climb the ladder, treat quality as a tradeoff, verify before you trust it, keep it simple, and build for intent rather than implementation. None of them are AI-specific — which makes them the right yardstick for what’s actually changed.

Ten Years On: Can the LLMs Eat Toil?

The obvious 2026 question: with AI everywhere, is toil solved? The competence/latency/relevance framework answers it, even though it predates AI entirely. Apply those same axes from the turnup-team story to today’s AI-SRE agents and the verdict is structurally the same as it was for the centralised turnup team: fast is easy to get, competent and relevant is the hard part.

The shift industry-wide in 2026 has genuinely been real, though — from AI that suggests to AI that investigates. Microsoft’s Azure SRE Agent and AWS’s DevOps Agent both reached general availability in March 2026, and both go further than a chat-based copilot: they read telemetry, query metrics and traces, correlate against historical incidents, and either execute a pre-approved runbook automatically or page a human with the investigation already done.

Research systems like STRATUS go further still, coordinating multiple specialised agents — one for detection, one for diagnosis, one for mitigation — rather than one model doing everything.

A long-standing scepticism about “magic systems that try to learn thresholds or automatically detect causality” — the exact phrase that anticipated what became the AIOps category a decade later — turns out to be the single most useful lens for evaluating these agents honestly.

A 2026 research architecture called RC-LLM reframes root-cause analysis as causal reasoning over fused trace, metric, and log data, rather than the correlation-based pattern-matching most AIOps tools still ship. But the practitioner-relevant caveat, confirmed across the current generation of AI-SRE tooling, is that accuracy is substantially higher for incidents that resemble a known historical pattern than for genuinely novel failure modes — the correlation-to-causation gap flagged a decade ago hasn’t closed, it’s just gotten a better-marketed name.

LLM-assisted incident summarisation and runbook drafting are genuinely useful, reviewed toil rather than eliminated toil; LLM-decided alerting recreates the same email-driven chaos golden-signal monitoring buried, with a fancier subject line.

DORA’s own 2026 research lands on the same conclusion from the delivery-metrics side: AI is “an amplifier, not a fix” — it multiplies the effectiveness of an already-disciplined engineering org and multiplies the dysfunction of one that isn’t, which is exactly what you’d expect if AI is retiring toil at the level automation always has, rather than replacing engineering judgment. (The Observability Engineering series’ Part 7 covers the same LLM-agent-in-the-loop question from the instrumentation side — what an agent needs to actually see before it can investigate anything.)

And the org-scale answer to toil is exactly what the discipline’s own engagement-model thinking predicted: platform engineering — golden paths, Backstage, paved roads — the Frameworks/SRE Platform model going mainstream. Toil didn’t disappear; it moved to the platform layer, and now increasingly to an AI agent sitting in front of that platform, where one fix compounds across every team that hasn’t had to touch the underlying automation at all.

Next: Part 4: Four Signals and a Pager — Monitoring That Respects Humans, on the four golden signals and the philosophy that pages must be actionable.

References

• Google SRE Book — Eliminating Toil, The Evolution of Automation at Google, Simplicity, Software Engineering in SRE
• The SRE Workbook — Toil
• Netflix — Chaos Engineering and the Simian Army
• Azure SRE Agent GA announcement (March 10, 2026) · AWS DevOps Agent GA announcement (March 31, 2026) · AI SRE explained (incident.io)
• STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds (NeurIPS 2025) · RC-LLM: Root Cause Analysis Method Based on Large Language Models with Residual Connection Structures
• DORA — State of AI-assisted Software Development 2025