The SRE Book, Ten Years On — Part 10: Never Trust a Backup You Haven't Restored

Two ideas that almost nobody enjoys learning, and that every surviving company eventually wishes it had internalised sooner: data processing pipelines, and data integrity. The first explains why batch jobs are secretly distributed systems with their own thundering-herd and split-brain-adjacent failure modes; the second delivers the rule that’s been tattooed on every DBA since 2016: you don’t want backups — you want restores.

Both halves of what follows are really the same warning, wearing different clothes: a system can look fine from where you’re standing and still be quietly wrong.

A pipeline can run green while sibling pipelines it can’t see conspire to overload the resource underneath it. A byte can rot in storage for months before anything reads it back and notices.

The fix, in both cases, is the same shape — instrument for the interaction you can’t observe directly, and treat “we have a backup” as an unverified claim until someone has actually pulled the trigger on a restore.

Pipelines: The Moiré Problem

The headline insight here is visual, best shown with a graph rather than a definition. When multiple periodic pipelines read from the same shared resource — a daily export, a weekly rollup, a monthly report — their load curves overlap and interfere, producing a beating pattern known as the Moiré effect: interference between overlapping periodic patterns.

Plot three pipelines’ resource usage separately and each looks tame; stack them and the peaks align in ways no single pipeline’s author could ever have predicted, because none of them can see the other two.

The aggregate load isn’t the sum of the averages — it’s the sum of the peaks. “Just add more workers” makes it worse: one documented case found that adding workers to one stage of a chained pipeline decreased end-to-end throughput, because downstream stages couldn’t absorb the burst those workers created.

The other structural insight here is the hanging chunk problem, and it’s a trap with real teeth. Big Data pipelines cut work into “embarrassingly parallel” chunks, but the chunks are rarely even — if you shard by customer, the largest customer’s chunk sets the runtime floor for the entire batch, because most pipeline code waits for every chunk before advancing to the next stage (sorting, in particular, forces this).

One slow or stuck chunk stalls the whole run. The instinctive fix — kill the job and restart it — is usually the wrong one: without checkpointing, killing a hung job throws away every other chunk’s completed work too, wasting exactly the compute and human attention the restart was meant to save.

Google’s actual answer, Workflow (built in 2003, still running the vast majority of Google’s continuous data processing today), reframes the whole problem as a distributed-systems design pattern borrowed, oddly, from Model-View-Controller: a Task Master holds all job state in memory for speed while synchronously journaling every mutation to disk (the model); stateless workers pull tasks, do the work, and write results (the view); an optional controller handles scaling, snapshotting, and rollback.

The correctness guarantees stack four deep, and each one closes a specific hole the previous three leave open:

None of these four is sufficient alone — it took all four, discovered the hard way, to make “exactly once” actually mean exactly once. For true business continuity across whole datacenters, Workflow’s global variant leans on the same primitives Part 9 covered in the abstract: journals in Spanner for globally consistent low-throughput storage, and Chubby (Google’s consensus-backed lock service) to elect which Task Master gets to write — distributed consensus, reappearing exactly where you’d expect it to.

Data Integrity: Whatever Users Think It Is

Data integrity starts with a definition that refuses to stay put: “data integrity is whatever users think it is.” A user who sees an empty mailbox for four days doesn’t distinguish “your data is gone” from “your data is fine but inaccessible” — both destroy trust identically.

That reframing produces the sharpest quantitative argument here, and it’s worth sitting with because the numbers are genuinely counterintuitive: a service with 99.99% uptime loses only about an hour a year, a bar most users would call excellent. A service with 99.99% good bytes in a 2GB artifact has up to 200KB of silent corruption — random opcodes in executables, unloadable databases.

The same-looking number means “basically fine” for availability and “catastrophic” for integrity, because corruption doesn’t distribute evenly across a file the way downtime distributes across a year. The conclusion is the one line worth memorizing: the secret to superior data integrity is proactive detection and rapid repair, not prevention — an artifact corrupted once a year but caught and fixed within 30 minutes is, from the user’s perspective, still effectively 100% intact.

A study of 19 real Google data-recovery efforts found the hardest variant wasn’t the dramatic one. It was low-grade corruption from a software bug, discovered weeks or months after the bug shipped — by which point the corrupted data has already propagated into derived tables, caches, and every backup taken since.

This is why replication is not recovery: a replicated datastore faithfully copies a bad delete to every replica, usually before anyone notices there was a problem to protect against.

The defence-in-depth answer layers three mechanisms, each worth understanding for why it exists rather than just what it’s called:

One line puts the last one best: “in theory, Paxos ignores failed nodes and makes progress with a quorum. In practice, ignoring a failed node corresponds to timeouts and retries beneath the implementation” — trust the algorithm’s proof, but verify the actual bytes it produced.

At Google’s own scale the arithmetic gets absurd enough to need its own fix: validating 700 petabytes at a naive 300MB/s takes roughly eight decades. The answer is trust points — treat data as immutable once it’s settled, verify and back up only what’s changed since the last trust point, and shard the verification job across enough parallel workers that wall-clock time drops from decades to hours.

Restoring is the same problem in reverse: a full restore that has to replay a thousand sequential incremental backups since the last full backup is its own reliability risk, which is exactly why “how many backups deep is our restore path” is a metric worth watching on its own, not just “do we have backups.”

Ten Years On: The Two Postmortems Everyone Should Read

These rules were proven the hard way by companies whose names everyone knows.

GitLab, January 31, 2017. An engineer, intending to work on the secondary database, wiped ~300GB from the primary — db1 — while tired at 23:00. Then the layers fell, one by one, exactly the way a defence-in-depth model warns they will: pg_dump backups had been silently failing (a Postgres version mismatch plus email alerts swallowed by DMARC), the S3 backup bucket was empty, Azure snapshots weren’t enabled for database servers, and replication was broken.

The rescue came from an LVM snapshot taken ~6 hours earlier — a layer nobody was managing as a layer. GitLab lost ~6 hours of data (about 5,000 projects, 5,000 comments, 700 users), live-streamed the entire recovery, and published one of the best postmortems in industry history.

The lesson, verbatim from their own incident doc: “out of 5 backup/replication techniques deployed, none are working reliably or set up in the first place.” Five layers, zero restores — because nobody had done the one thing that matters: practiced the restore, not just configured the backup.

Atlassian, April 2022. A maintenance script deleted data for 775 customers — wrong IDs (site IDs instead of app IDs) and wrong mode (permanent delete instead of mark-for-delete). The recovery took two weeks, rebuilding tenants from backups in batches of 60, because no per-tenant point-in-time recovery existed. The “soft deletion first” rule, ignored at scale, cost 775 organisations their Jira for a fortnight.

And the 2026 addition nobody could have predicted in 2016: ransomware. The new data-loss vector doesn’t come from bugs but from attackers who specifically target backups first, which reorders the recovery priorities entirely.

That makes the modern equivalent of tape — immutable, versioned object storage (S3 Object Lock, GA since November 2018) — a defence-in-depth layer against a threat model that simply didn’t exist a decade ago: an adversary with valid credentials trying to delete the backup on purpose.

Every tier of the recovery pyramid now needs a “can an attacker with my credentials delete this?” audit alongside its original “how fast can we restore” question.

AI-assisted incident response reaches this territory from an unexpected angle: the hardest failure mode above — low-grade corruption discovered months after the triggering bug shipped — is precisely the pattern that trace-and-log fusion approaches like RC-LLM are best suited to, if the out-of-band validators described above are emitting the telemetry to fuse in the first place.

An AI investigator correlating “this table’s row count started drifting from its historical baseline eleven weeks ago” against a deploy log is doing, faster and continuously, exactly what “trust points” and out-of-band checks were manually designed to catch.

The prerequisite is the same one Practical OpenTelemetry’s discussion of the five clocks inside your MTTR makes explicit: you can’t detect a drift you never instrumented — and the earlier “the point where the bug shipped” and “the point where a human clocked it” converge, the smaller the eventual restore has to reach back.

Next: Part 11: Ship It Without Sinking It — Releases, Tests, and Launches, where the change finally reaches production safely.

References

• Google SRE Book — Data Processing Pipelines and Data Integrity: What You Read Is What You Wrote
• GitLab — Postmortem of database outage of January 31, 2017
• Atlassian — Post-incident review, April 2022 outage
• AWS S3 Object Lock
• RC-LLM: Root Cause Analysis Method Based on Large Language Models with Residual Connection Structures