SherlockLiu Logo SherlockLiu
Back to all posts
Engineering

The SRE Book, Ten Years On — Part 8: Why Systems Fall Over

SL
• Sep 02, 2026 • 13 min read
The SRE Book, Ten Years On — Part 8: Why Systems Fall Over

The physics of failure under load is one dense, unified subject, and it resolves into three questions asked in sequence. Load balancing, provisioning, and the cascade — read together, they’re the closest thing to a unified theory of why large systems die.

In this part

  1. How do you spread load evenly in the first place? — why Round Robin wastes half your capacity, and the ladder that fixes it.
  2. What do you measure to know you're overloaded? — why "queries per second" is the wrong unit, and what to provision against instead.
  3. What happens once you're overloaded anyway? — the cascade loop, its amplifiers, and where to break it.

Principle 1: Spread the Load Evenly, or Don’t Bother

Google’s frontend — DNS-based load balancing, virtual IPs, DSR, GRE encapsulation, GSLB — reads like a museum piece in 2026, because the ideas survived into tools you buy: anycast edges (Cloudflare), managed LBs, Envoy-based meshes. The one idea worth extracting from the archaeology: one consistent hashing function keeps connections stable when backends churn. That’s it — that’s the insight. Everything else is packaging.

The real content lives one level down, and the headline is a counterintuitive performance result: simple Round Robin wastes up to half your capacity. The reason is skew — a heavy request lands on a server that already has heavy requests, and while some servers idle, others are saturated.

The progression from Round Robin through Weighted Round Robin, “least-loaded” variants, and deterministic subsetting (each client talks to a fixed subset of backends, bounding connection counts) is the design ladder every service mesh still climbs today. Plus the operational nicety: the lame-duck state, where a server is told “stop taking new work, finish what you have” before it’s removed — so draining a server never drops a connection.

Principle 2: Measure What the System Actually Consumes

The single most valuable line in this whole area, full stop:

Provision by CPU and measured resource cost, not by “queries per second.”

The argument: QPS varies wildly in cost — a cached read and a 40-table join both count as one query, and a system provisioned for its cheap queries collapses the moment the traffic mix shifts to expensive ones. Resource-based provisioning (CPU, memory, IO per request class) is the only metric that survives a traffic-mix change. It’s the same lesson as Part 4’s “symptoms, not causes”: measure the thing the system actually consumes, not the label on the request.

The overload toolbox

  • Adaptive client-side throttling — clients reject requests locally when requests > K × accepts, giving the server an implicit signal about client demand without a round trip.
  • Criticality classes — shed the least important traffic first, not the traffic that happens to be first in line.
  • Retry budgets — a global cap on retries so a slow dependency doesn't get hammered into a dead one.

Underneath all of it sits a distinction worth keeping separate: process health checking (“is this binary responding at all?”, the cluster scheduler’s concern) versus service health checking (“can this binary answer this class of request right now?”, the load balancer’s concern) — conflate the two and you get a system where a scheduler restarts healthy-but-busy tasks for failing a check that was never meant for it, manufacturing the very overload it’s trying to relieve.

Principle 3: Understand the Cascade, Because It Will Happen

The cascade loop is the masterpiece of this whole area, and it’s the failure mode that takes down almost every large system eventually. The starting point is a taxonomy of exactly what runs out first, and each resource fails a different way:

Four Ways to Run Out CPU queueing delay grows nonlinearly, well before 100% graceful, then sudden Memory an out-of-memory kill is instant and total no warning, just dead Threads pool exhaustion silently blocks new work behind stuck work quiet, then total File descriptors a slow leak, invisible for weeks, catastrophic in a day slow burn, fast death Different signatures, same next step: the loop below.

Figure: each resource fails on its own schedule, but every one of them can trigger the same cascade.

Each resource is capable of triggering the same loop:

Overload → crash → load shifts to the remaining servers → they overload → crash…

Around that loop orbit the amplifiers: retry storms (every layer retries, multiplying load geometrically), deadline propagation (a request’s remaining time budget passed down through every hop, so a downstream service doesn’t keep working on something the caller already gave up on — the same context-carrying problem Practical OpenTelemetry Part 5: Context, Baggage, and Propagators covers from the tracing side), bimodal latency (queues that are either empty or catastrophically full), and cold caches (an outage that empties a cache is followed by an outage caused by repopulating it).

The cures are each loop-breakers: load shedding, graceful degradation, backoff with jitter (so retries spread out instead of synchronising into a thundering herd), and the blunt rule — don’t retry at multiple layers; pick one.

Finding your breaking points before production does is equally non-negotiable:

  • Load-test each component past failure, and watch how it fails — a well-behaved component sheds a few requests and survives; a fragile one crashes or error-storms.
  • Test gradual ramps and sudden impulses separately, because caching behaves differently under each.
  • Test your popular clients — do they queue politely while you’re down, or hammer you the instant you recover?
  • Test your noncritical backends — does a spelling-suggestion service that never responds quietly stall the whole request, or does the frontend correctly give up on it?

The last two are the easiest to skip and the most likely to bite.

The “Shakespeare” service, again, ties every thread together as a worked example: a documentary drives a traffic surge into one datacenter at the exact moment a major update is rolling out there; graceful degradation (drop the illustrations, keep the text) and randomized-backoff retries hold the line for a while, but tasks still fail one by one until GSLB reroutes the overflow to neighbouring datacenters and autoscaling — turned on as a direct result of the postmortem — closes the loop for next time.

The Cascade Loop, and Where to Break It Overload Crash Load shifts More overload Amplifiers retry storms · cold caches bimodal queues Loop-breakers load shedding · degradation backoff + jitter · retry budgets

Figure: the cascade is a loop with known amplifiers; each cure is a deliberate break in the loop.

Beyond Google: The Ideas Became Open Source

The load-balancing ladder isn’t Google-specific theory that happened to leak out — most of the industry now runs on direct descendants of it, built by other companies that hit the same wall. Envoy, originated at Lyft and now the substrate under most service meshes (Istio included), implements the same least-loaded and subsetting ideas as configurable load-balancing policies rather than bespoke infrastructure.

Kubernetes’ Horizontal Pod Autoscaler and cluster autoscaler are the “provision by resource cost, add capacity dynamically” instinct, shipped as a default rather than something every company reimplements.

The most quietly influential idea in this whole area — retry budgets and jitter as a global policy rather than a per-call decision — is now table stakes in every major service mesh and most cloud SDKs. An engineer joining the industry today learns “add jitter to your retries” as received wisdom, without ever knowing where the argument was first written down.

Ten Years On: Fastly’s Forty-Nine Minutes

The definitive modern cascade is Fastly’s June 8, 2021 outage. A bug had been deployed on May 12; on June 8, one customer’s legitimate config change triggered it, and within minutes 85% of the network was returning errors — taking down gov.uk, Stripe, Amazon, Twitch, and half the internet’s checkout buttons with it.

Detection took a minute; within 49 minutes 95% of the network was normal. It’s a one-change cascade exactly as the theory predicts: a single valid input landed on a latent bug, and the blast radius was every service that had centralised its edge on one provider.

Two 2026 takeaways this half-anticipates. First: the frontend architecture is now a dependency you rent, which concentrates the very failure modes this warns about into a handful of companies — the counter-argument being that those companies now run drills and postmortems at a maturity most in-house teams never reach. Second: the loop-breakers went mainstream, as the previous section describes — yesterday’s cures are now defaults, not decisions.

The newest layer is automated intervention itself. The same 2026 AI-SRE agents reshaping troubleshooting (Part 6) and incident response (Part 7) are starting to close the cascade loop without a human in it at all — detecting the early signature of an overload-crash-shift cycle from metrics and traces, and triggering a pre-approved load-shedding or degraded-mode response before a human is even paged.

That’s a genuine win when the cascade matches a known shape, exactly the “Shakespeare” case study’s structure. It’s also exactly where the oldest warning here cuts hardest: without proper care, changes meant to help the steady state can expose the service to a bigger failure — and an autonomous agent making that trade-off at machine speed, on a novel failure shape it hasn’t seen, is the 2026 version of the same risk.

The lesson

When your system falls over, it will almost always fall over the same way — so understand the pattern, or make sure your agent has, before the incident, not during it.

Key takeaways

  • Round Robin wastes up to half your capacity. Skew is the reason — heavy requests pile onto servers already carrying heavy requests while others idle.
  • Provision by resource cost, never by QPS. A cached read and a 40-table join both count as one query; a system sized for its cheap queries dies the moment the traffic mix shifts.
  • Every cascade is the same loop — overload, crash, load shifts, overload again — and every cure is a deliberate break in it: load shedding, degradation, backoff with jitter, and retrying at exactly one layer.
  • Your untested failure modes are the ones that take you down. Load-test past failure, ramp and spike, and don't skip your popular clients or your noncritical backends.
  • Renting your frontend concentrates the risk. Fastly's 49 minutes showed one latent bug plus one valid config change can take out a slice of the internet — the trade is that providers drill harder than most in-house teams ever will.

Next: Part 9: Agreeing to Agree — Consensus and the Distributed Cron, on the ticking bomb you need to defuse.

References

Comments

Related Blogs