The SRE Book, Ten Years On — Part 2: 100% Is the Wrong Target
If you take exactly one idea from site reliability engineering, take this one: 100% is the wrong reliability target for basically everything. (The standard exceptions: pacemakers and anti-lock brakes.)
The reasoning is brutally practical. Your users cannot tell the difference between 100% and 99.999% availability — because between them and your service sits their laptop, their WiFi, their ISP, and the power grid, all of which are far less available than 99.999%. The marginal 0.001% you’d grind for gets lost in the noise of everyone else’s unavailability. You’d pay an enormous cost for a benefit nobody receives.
Everything else is built on top of that one observation.
In this part
- The risk continuum — why reliability cost is exponential, and why the target is a ceiling as much as a floor.
- SLI → SLO → SLA — the vocabulary, and the acid test that separates a real agreement from a wish.
- The error budget — how
1 − SLOturns a number into an incentive system that dissolves the dev/ops fight. - Ten years on — burn-rate alerting, what the industry industrialised, and where the original framing fell short.
Figure: how the risk continuum and the SLI/SLO/SLA vocabulary turn "100% is the wrong target" into a single number — the error budget — that both teams optimise together.
The Risk Continuum
Reliability isn’t free, and the shape of the cost curve is specific: it’s not linear, it’s exponential. An incremental improvement in reliability can cost 100x more than the previous increment — the cost of redundant compute to buy headroom, plus the opportunity cost of engineers who are hardening a system instead of shipping the features users actually asked for.
The goal is “reliable enough, but no more reliable than it needs to be” — the availability target is both a floor and a ceiling: overshoot it and you’ve wasted the exact same engineering hours you’d waste falling short.
This is a product question, not a technical one. Three things actually determine the right target:
- What level of availability will users be happy with, given how they use the product?
- What alternatives exist for users who are dissatisfied?
- What happens to usage at different availability levels?
The cost/benefit math is concrete rather than hand-wavy. Take a service earning $1M in revenue where every request has equal value: going from 99.9% to 99.99% availability (a 0.09-point improvement) is worth, at most, $1M × 0.0009 = $900 in additional revenue. If the engineering cost of buying that extra nine is under $900, do it. If it’s over, don’t — no matter how good the extra nine would feel.
When there’s no clean revenue-per-request line to draw, the fallback is to compare against the background error rate of consumer ISPs (measured at 0.01–1%): push your own error rate below that floor and the remaining errors are indistinguishable from the noise the user’s own internet connection already produces.
Real Google examples show the same math cutting opposite ways:
| Service | Target set | Why the product answer differed |
|---|---|---|
| YouTube (acquired 2006) | Lower availability than Google’s enterprise products | Young, consumer-facing — rapid feature development was worth more than an extra nine |
| AdWords | Tight latency budget | It couldn’t be allowed to slow down search results |
| AdSense | Hundreds of ms slower allowed | It only had to avoid slowing the third-party page it was embedded in — which let Google consolidate serving into far fewer datacenters and cut operational cost |
Same company, same underlying tradeoff, opposite answers — because the product answer was different, not the technology.
Nines, SLIs, SLOs, SLAs
The vocabulary the whole industry now uses:
| Term | What it is | Example |
|---|---|---|
| SLI — Service Level Indicator | A measurement: latency, error rate, throughput, availability | Success rate of requests — not raw uptime, because users experience requests, not wall-clock minutes |
| SLO — Service Level Objective | The target for an SLI | “99.9% of requests succeed within 200ms, over a quarter” |
| SLA — Service Level Agreement | The SLO plus a consequence | Rebates, credits, contract language |
The acid test is still the best one in print: if nobody pays a penalty when the target is missed, you’re looking at an SLO, not an SLA.
Each extra nine of availability costs roughly an order of magnitude more engineering, and the raw numbers make the stakes concrete:
| Target | Downtime per year | Engineering cost vs. previous nine |
|---|---|---|
| 99.9% | ~8.77 hours | baseline |
| 99.99% | ~52.6 minutes | ~10x |
| 99.999% | ~5.26 minutes | ~100x |
There’s a subtlety most summaries drop, though: at Google’s scale, a time-based availability metric barely means anything, because a globally distributed service is almost always serving some traffic somewhere. So the real metric isn’t uptime-minutes, it’s request success rate — the proportion of well-formed requests that succeed over a rolling window. A service handling 2.5M requests a day at a 99.99% target can serve up to 250 failed requests and still hit the number for that day.
This is also why percentiles beat means: averaging latency hides the tail, and the tail is where users actually feel pain. A system with a 50ms median where 5% of requests take 20x longer shows how an averaged dashboard can look flat all day while the tail latency is actively getting worse.
And the counterintuitive advice: don’t overachieve. The Global Chubby story is worth telling in full, because it’s the sharpest illustration anywhere of a target as ceiling, not just floor.
Chubby is Google’s global lock service, and it was so reliable that service owners quietly started building hard dependencies on it never going down — the kind of unstated assumption that turns a rare failure into a catastrophic one.
SRE’s fix was almost perverse: deliberately synthesize a controlled outage any quarter where a real failure hadn’t already used up the budget, specifically to flush out those unreasonable dependencies while the blast radius was still small and expected.
The takeaway
Over-achieving your SLO isn't a bonus. It's a silent liability accumulating in every team that built on top of you.
Figure: the error budget burns slowly during normal operation and cliff-dives during an outage; at zero, releases stop.
The Error Budget: An Incentive System, Not a Metric
Here’s the masterstroke. The error budget is 1 − SLO — the permitted unavailability. A 99.99% SLO gives you 0.01% of the quarter to spend. And you want to spend it:
The development team wants to launch features. Ideally, we would spend all of our error budget taking risks with things we launch in order to launch them quickly.
An outage stops being a moral failure and becomes an expected part of the process of innovation — something both teams manage rather than fear. SRE’s goal is no longer “zero outages.” It’s “spend the budget on maximum feature velocity.” That single reframe dissolves the dev/ops trench warfare from Part 1: both teams now optimise the same number.
The enforcement mechanism is deliberately crude: when the budget hits zero, releases stop. This is “bang-bang control,” and it has known weaknesses — freezing the entire quarter because of one bad week is harsh. But harsh beats ambiguous: the freeze makes the budget real in a way no dashboard ever could.
And because it hurts, teams learn to spend budget safely — phased rollouts, 1% experiments, canarying — to stretch the same budget further. There’s a softer alternative too: instead of a hard on/off freeze, some teams slow the release cadence or roll back automatically as the budget gets close to zero, rather than waiting for it to hit the floor.
It’s worth being precise about why this dissolves Part 1’s trench warfare rather than just papering over it: both sides are now optimising the same number, in the same direction, with the same instrument. Product development can’t accuse SRE of blocking launches for no reason (the budget says exactly how much risk is left); SRE can’t accuse product development of recklessness (the budget says exactly how much risk was authorised).
The cleanest summary of this is the simplest: an error budget aligns incentives and emphasises joint ownership, letting two teams reach the same conclusion about production risk without rancor — and, tellingly, without a manager in the room refereeing it.
Ten Years On: The Budget Grew Teeth
This idea was right, and the industry spent a decade industrialising it in three ways: the alerting got precise, the discipline spread past “reliability” as a category, and a couple of the original assumptions turned out to be slightly wrong.
Burn-rate alerting, worked out in detail. Alerting on the budget line itself pages too late: by the time the budget is visibly depleted, you’re already frozen. The SRE Workbook’s take on “Alerting on SLOs” fixed this with multiwindow, multi-burn-rate alerting — the mechanism is worth spelling out because it’s now the industry-standard pattern.
Figure: a genuine incident shows up in both windows at once; a transient blip only trips the short one and self-clears.
In 2026 this exact pattern is a checkbox, not a project: Nobl9 (founded by ex-Google SREs), the vendor-neutral OpenSLO spec, Pyrra/Sloth for Prometheus, and built-in SLO monitoring in Datadog and Grafana Cloud all ship it out of the box. What required a home-grown Borgmon rule set in 2016 is now a form you fill in.
The discipline scaled down and outward. The error-budget idea has outgrown “reliability” as a category entirely: DORA — the research group behind the Accelerate work referenced across this series — added a fifth metric, Deployment Rework Rate, to its original four delivery metrics in 2024, specifically to catch work that reverts, patches, or redoes a deployment already counted as “shipped.”
By its 2025/2026 research (rebranded from State of DevOps to the State of AI-assisted Software Development), DORA leans on that fifth metric hard — because the classic four alone become misleading once a meaningful share of committed code is AI-generated. A team can look fast on lead time and deployment frequency while quietly spending that speed back on rework.
The throughline is direct: once you’ve accepted that velocity and stability are the same investment, any change to how code gets written — AI-assisted or not — has to be measured against both sides of that ledger, not just speed.
Where the early framing fell short. The nines framing is time-based in places, when the better advice is to measure request success rates — and the modern consensus (the SRE Workbook included) settled firmly on the latter. Percentiles over means, error budgets over uptime.
Also: DORA’s own research later showed change failure rate and recovery time matter as much as raw availability, a nuance the original change-management thinking only half-anticipated. For how those SLOs actually get wired into an alerting storage layer at a level below Google’s proprietary Borgmon, the Observability Engineering series’ Part 4 covers SLO-based alerts and the storage systems built to scale them.
The takeaway
Reliability targets are a business decision wearing engineering clothes. If nobody in your organisation can state why 99.9% instead of 99.99% — you don't have a target, you have a superstition.
Key takeaways
- Pick the target from the product, not the platform. Users can't perceive the marginal nine, and each one costs roughly 10x the last — so the right number comes from what users need and what alternatives they have, not from what the infrastructure could theoretically hit.
- The target is a ceiling as well as a floor. Chubby was so reliable that teams built hard dependencies on it never failing — which is why deliberately synthesising an outage was the fix. Over-achieving quietly transfers risk to everyone downstream.
- The error budget converts a target into an incentive. Once
1 − SLOis a shared allowance, an outage becomes an expected cost of shipping rather than a moral failure, and both teams start optimising the same number. - Enforcement has to bite to be real. Bang-bang control — releases stop at zero — is crude and harsh, but harsh beats ambiguous: it's what teaches teams to spend the budget safely through canaries and phased rollouts.
- Measure requests, not minutes. Time-based availability means little for a globally distributed service; request success rate and percentiles tell you what users actually felt, where an average hides the tail entirely.
Next: Part 3: The War on Toil — Why Ops Work Is a Bug, on what happens to the 50% of engineering time the error budget buys.
References
Have thoughts on this?
I read every email. If something resonated, felt wrong, or made you think — I'd love to hear from you.
Comments