SherlockLiu Logo SherlockLiu
Back to all posts
Engineering

The SRE Book, Ten Years On — Part 11: Ship It Without Sinking It

SL
• Sep 05, 2026 • 18 min read
The SRE Book, Ten Years On — Part 11: Ship It Without Sinking It

Part 1 opened with the bluntest observation in the whole discipline: most outages are caused by change. Release engineering, testing for reliability, and reliable launches are the answer to that fact — the machinery for shipping change safely, at speed.

This is really one pipeline in three gates — and skipping any one of them means the other two can’t save you, which is exactly what one 2024 disaster proves at the end of this post.

In this part

  1. Build it so it can ship safely — release engineering: hermetic builds, self-service releases, and an audit trail of what actually shipped.
  2. Verify it before it goes out the door — testing, and why a canary bug's order decides whether you can catch it at all.
  3. Clear it for the actual launch — the checklist, and the one brutal rule that stops it becoming theatre.

Together they’re the SRE take on the oldest tension in software: the faster you ship, the more carefully you must ship — and the real argument is that these aren’t in tension at all once the machinery is built correctly.

Release Engineering: Velocity as a Feature

The philosophy here is four principles, and each closes a specific failure mode the others leave open.

Principle What it means The failure mode it closes
Self-service Developers push the button themselves A human gatekeeper doesn’t scale past a handful of teams
High-velocity Frequent releases, deliberately — some teams build hourly and cherry-pick from the pool; others run “Push on Green,” deploying the instant a build passes every test A smaller diff is easier to test, easier to attribute a regression to, and easier to roll back. Batch size shrinks until “release” stops being an event and becomes a background process
Hermetic A build depends only on pinned, versioned tools and libraries — never on whatever happens to be installed on the machine that ran it Rebuilding an old release to backport a fix (cherry-picking) reproduces that release’s original toolchain, not today’s
Gated access + audit trail Approving a source change, cutting a release, and deploying it are enforced separately, with a record of which changes shipped in which release Troubleshooting a bad release starts from a known list of suspects instead of a guess

The concrete machinery is worth naming because it’s a template, not a Google-only artifact:

Source to Rollout, Four Named Tools Blaze / Bazel compiles from pinned build targets Rapid cuts a branch, builds and tests in parallel MPM versions, signs, labels dev / canary / prod Sisyphus drives the rollout — instant to multi-day crawl A canary "becomes" the release when MPM moves a label — nobody touches a deploy script.

Figure: the four-tool pipeline from compiled source to a rolled-out release — the pattern generalises even where the specific tool names don't.

Google’s own branching model is deliberately boring: everything lands on the mainline, but releases branch from the mainline and never merge back — bug fixes are cherry-picked onto the release branch instead — precisely so that “what’s in this release” stays a closed, auditable set rather than whatever happened to be on the mainline the moment someone pressed build.

Configuration gets the same hermetic treatment as code once it matters enough, and there are four increasingly rigorous ways to manage it:

Approach What you get — and what it costs
Edit the mainline directly Simplest — but breeds skew between what’s checked in and what’s actually running
Bundle config into the binary’s package Simple — but couples config and binary tightly
Config as its own versioned MPM package, labelled to a binary release Fix a bad flag with a config-only rebuild, no binary rebuild required
Read it live from an external store (Chubby) For anything that must change while the binary keeps running

The insight that aged best across all of it: release velocity and release safety are the same investment. The org that makes shipping boring makes it safe.

Testing for Reliability: Production Is the Only Honest Environment

One epigraph doubles as the thesis here — “if you haven’t tried it, assume it’s broken” — and the most useful structural idea is the split between traditional tests (unit, integration, system: run against a hermetic sandbox before anything ships) and production tests, which are candidly a form of black-box monitoring in disguise.

Configuration tests compare a service’s actual, live configuration against what source control says it should be — not hermetic by nature, since they deliberately look outside the sandbox, but exactly the tool that catches “the config in prod lags what’s checked in” skew before a user does. Stress tests find the cliff, not the slope: most components don’t degrade gracefully near their limit, they fall off it, so the useful question isn’t “how slow does it get” but “at what point does it stop answering at all.”

The canary test produces the most genuinely surprising math here. A canary isn’t really a test — it’s structured user acceptance, exposed to unpredictable live traffic rather than a deterministic assertion — but its failure signal can be decomposed with real precision.

As a rollout grows exponentially, the rate at which a rare bug’s symptoms get reported climbs in a way that reveals the bug’s order:

A Canary Bug's Order Tells You How to Catch It Order 1 — most bugs A single request breaks on its own, self-contained Fix: replay the unusual response as a regression test Order 2 — dangerous A broken request corrupts data a future request reads Invisible if you omit any prior request while debugging Order 3 — worst Corrupted data becomes a valid-looking identifier followed downstream — same trap, one hop further Watching the failure-rate curve during the canary window catches all three; watching pass/fail alone only catches order 1.

Figure: higher-order bugs are rarer and invisible to the debugging trick that catches order 1 — a request that only fails after every prior request also ran will suddenly "pass" the moment you skip one to isolate it.

Most bugs are order 1. Order-2-and-up bugs are the dangerous minority precisely because they’re invisible to the obvious debugging trick: a request that only fails if every prior request in the sequence also ran will suddenly pass the moment you omit one of them for isolation, which is exactly backwards from how a normal bug behaves. Catching the higher orders during the canary window, rather than after full rollout, is the entire point of watching the rate curve instead of just the pass/fail count.

It’s worth being candid, in a way many engineering texts aren’t, about run-to-run noise: with roughly 21,000 simple tests gating a single service, and wanting to accept 99 patches out of 100 as clean, the individual test’s own flake rate has to sit above 99.9999% — the 42,000th root of 0.99, since every patch reruns the whole suite before and after. That number is a quiet indictment of “just add a flaky retry” as a testing culture; at scale, flakiness this small still adds up to real, unnecessary developer pain.

And for genuinely chaotic failure modes — the ones no deterministic test can enumerate in advance — two techniques are still in daily use a decade later: Chaos Monkey for distributed-state fault injection, and Jepsen (the same partition-testing project Part 9 covered) for consensus correctness.

Neither produces a repeatable pass/fail — rerunning a statistical test doesn’t prove a fix worked. But logging the random seed that drove a failing run turns it into a release test worth keeping, and a fault that resurfaces with a more severe expression on a later run is itself a signal worth escalating.

The Launch Checklist’s One Brutal Rule

One story captures this better than any definition could: on Christmas Eve 2011, Google’s Keyhole imagery service — the one that renders the satellite view behind Maps and Earth — took roughly 25 times its normal peak traffic, over a million requests a second, because Google had partnered with NORAD to track Santa’s flight in real time for millions of watching kids. There was no moving the deadline. The team’s kill switches for the experience had an internal nickname that says everything about the stakes: “make-children-cry switches.”

That’s the job Google’s Launch Coordination Engineering (LCE) team exists for — SREs who function as a dedicated, cross-product consultancy: auditing a launch against reliability best practices, acting as the neutral liaison between engineering, product, and marketing, and holding final sign-off on whether a launch is safe to proceed.

Their signature artefact is the launch checklist, and its governing rule is the real lesson here.

The rule

  • Every question on the checklist must be justified by a specific past disaster. No hypothetical anxieties allowed.
  • Growing the checklist is itself a risk to defend against. An unbounded checklist becomes theatre nobody actually reads.

How Google applies it: adding a new question to the checklist required a vice president’s sign-off — the checklist’s own growth was something to actively gatekeep, not something that happened by default.

The categories that survived that filter map a launch’s real risk surface with striking economy:

  • Architecture and dependencies — what this launch is standing on.
  • Integration with shared infrastructure — what else it can take down with it.
  • Capacity planning — Google products have seen launch spikes 15 times their forecast; public interest is notoriously hard to predict.
  • Failure modes — single points of failure, degraded-mode behaviour, DoS resilience.
  • Client behavior — exponential backoff and jitter, treated as a reliability requirement on code you don’t even control once it ships to a user’s device.
  • Processes and automation — the unglamorous section whose real subject is admitting that humans are themselves a single point of failure, and documenting the manual runbook accordingly.

The story of launching Android is the sharpest illustration of what happens when the checklist meets something genuinely new: Google had never before shipped client-side logic it couldn’t patch within hours the way a JavaScript bug in Gmail could be. The existing checklist’s assumptions simply didn’t hold, and the LCE team’s real skill wasn’t following the checklist — it was returning to the first principles behind each question and re-deriving which ones still applied before making the checklist concrete again for a genuinely new kind of launch.

The Staged Rollout 1% watch SLOs 10% watch errors 50% watch saturation 100% baked, shipped Dark launch flag on later any gate fails → rollback, cheap and boring

Figure: staged rollout with observation gates at each step — and a rollback path that's cheap by design.

Ten Years On: CrowdStrike, the Counterexample That Proved the Point

The decade’s definitive release-engineering incident is CrowdStrike, July 19, 2024. A Rapid Response Content update — the channel meant for quick tuning, deliberately exempt from the full sensor validation that code changes receive — shipped a Channel File 291 update with an out-of-bounds read. The result: 8.5 million Windows machines blue-screened and boot-looped, airlines grounded, hospitals disrupted; recovery required manual per-machine intervention; damages estimated in the billions, £1.7–2.3B in the UK alone.

All three gates, missed at once

A release path with no validation gate. A client-side payload shipped to millions with no canary. A checklist question this exact scenario would have written — that nobody had, because this specific channel had never disastered yet. Every warning above, stacked in one incident.

The fix CrowdStrike itself announced — staged content deployment, more validation, customer control of rollout timing — is the launch checklist, implemented post-hoc.

The industry’s answer went further than any one company’s internal tooling: progressive delivery became open tooling, precisely because most organizations will never build their own Rapid, Sisyphus, or LCE team. Argo Rollouts and Flagger brought canary analysis to Kubernetes as a declarative primitive; LaunchDarkly and the OpenFeature standard turned feature flags into infrastructure any team can adopt without inventing MPM-style labeling from scratch.

And the supply-chain layer that didn’t exist a decade ago — SLSA provenance levels, sigstore signatures, SBOMs — exists because in 2026 the release path includes other people’s code, dependency-shaped risk the hermetic-build principle above never had to model in 2016.

The clearest 2026 validation of the core claim here — that release velocity and release safety are the same investment, not opposing forces — comes from DORA’s own research, and it lands with a twist nobody could have anticipated in 2016.

DORA’s January 2026 ROI of AI-Assisted Software Development report models a roughly 39% first-year return (~$11.6M against an $8.4M investment for a 500-person engineering org, an ~8-month payback) from AI-assisted development — but explicitly only when the underlying release-engineering foundations above are already sound: platform quality, clear workflows, team alignment. AI amplifies whatever discipline already exists; it doesn’t manufacture it.

DORA’s report is blunt that classic deployment-frequency and lead-time metrics become misleading, not just insufficient, once AI generates 30–70% of committed code — the exact blind spot Deployment Rework Rate, the fifth metric DORA added to its original four in 2024, was built to catch, and the reason DORA’s own annual flagship report has since been rebranded from “Accelerate State of DevOps” to the State of AI-assisted Software Development.

A team that can’t answer “what shipped in this release” the way Rapid’s audit trail always could is, in 2026, in a strictly worse position to reason about AI-assisted velocity than a team that can — the hermetic, auditable release pipeline described above turns out to be exactly the prerequisite the AI-era metrics-literature is rediscovering the hard way.

Key takeaways

  • Velocity and safety are the same investment. Frequent releases are safer than rare ones, because a smaller diff is easier to test, to attribute a regression to, and to roll back. The org that makes shipping boring makes it safe.
  • Hermetic builds are what make history reproducible. If a build depends on whatever was installed on the machine, you can never rebuild an old release to backport a fix and get the same thing back.
  • A canary bug's order decides whether you can catch it. Order-1 bugs fail on their own and replay cleanly as regression tests; order-2 and higher vanish the moment you omit a prior request to isolate them — which is why you watch the failure-rate curve, not the pass/fail count.
  • Every checklist question needs a specific past disaster behind it. The checklist's own growth is a risk: an unbounded list becomes theatre, which is why adding a question at Google needed a VP's sign-off.
  • CrowdStrike missed all three gates at once — no validation on that release path, no canary for a client-side payload shipped to millions, and no checklist question covering a channel that had never disastered before. 8.5 million machines, billions in damage.

Next: Part 12: The Cockpit — People, Culture, and the Future of SRE, the final third and its most prescient ideas.

References

Comments

Related Blogs