Observability Engineering — Part 4: SLO-Based Alerts and Storage That Scales

Somewhere around 2 a.m., an alert fires: your error budget is burning fast. But why does it say “fast”? Fast compared to what? And once you’re awake, will the system that paged you also help you find the broken thing — or will you be tailing logs by hand until sunrise?

This post covers the more technical middle stretch of Part IV of Observability Engineering. Chapter 12 explains the arithmetic behind SLO alerts — how a system decides an error budget is in danger before it’s actually gone. Chapters 13 and 14 tackle a harder problem: what kind of database can answer the open-ended questions engineers ask during an incident, at real production scale. Honeycomb and ClickHouse offer two different, genuinely valid answers to that same problem, which makes comparing them worthwhile.

Error Budgets: The Currency of Reliability

An SLO (service level objective) sets a target, like “99.9% of requests should succeed.” Whatever sliver is left over — 0.1% in this case — is your error budget. It’s the amount of badness your business has agreed to tolerate before customers really start to notice.

The practical question isn’t “did we hit our target this month?” It’s “are we currently on a trajectory to blow through our budget, and if so, how soon?” That’s what a burn-rate alert answers, and the book walks through three ways to build one, each more sophisticated than the last.

Threshold-crossing alerts are the blunt instrument: page someone once the remaining budget drops below a fixed line, say 30%. It works, but it just moves the “zero” goalpost — your team reacts the same way once you’re past it, having quietly given up runway you meant to protect.

Relative burn alerts compare your recent error rate against what your budget can proportionally sustain, instead of reacting to a raw error count. A service failing 4x as fast as its budget allows over a 15-minute window is a real problem; one failed request out of 20 overnight is not. Because it’s scaled to expected traffic, it won’t cry wolf during quiet periods.

Predictive burn alerts ask “if this keeps up, when do we run out?” That takes two windows: how far back you look to measure the current rate (baseline window), and how far forward you forecast (lookahead window). The book’s rule of thumb: don’t stretch the forecast more than about 4x beyond the baseline, or you’ll chase noise instead of signal — a 4-hour forecast built from the last hour of data, a 24-hour forecast from the last six hours.

A worked example from the book makes this concrete. Say a service handles 43,800 requests a month and targets 99% success, so 438 failures is the whole month’s budget. In the last six hours it saw 50 requests, 25 of which failed. Naively extrapolating that raw count forward looks almost fine. But scale it proportionally against typical daily volume — accounting for traffic not being flat around the clock — and that 50% failure rate, if it holds, works out to roughly 720 failures over the next 24 hours: well past the month’s budget. The lesson: raw counts lie, proportional extrapolation doesn’t.

A pricier variant, the context-aware burn alert, factors in the entire history of good and bad events across the whole SLO window instead of just the recent baseline. It’s more accurate but compute-heavy — Honeycomb notes that a few customer SLO configurations quietly ran up thousands of dollars a day in serverless costs before caching was added. For most teams starting out, the simpler short-term model with the 4x heuristic is the sane default.

Why Wide Events Make Better SLOs Than Metrics

Here’s where storage and alerting connect. If SLO math is built on metrics — counters bucketed into, say, five-minute windows — each window gets judged entirely “good” or entirely “bad.” A rough five-minute stretch where only 94% of requests were fast enough can burn a quarter of an entire month’s budget for a strict SLO, because the whole window counts as a loss even though 94% of it succeeded.

Judge success at the individual request level instead, and that same 94%-good window only costs 6% of the budget, because you’re counting actual failures, not writing off the whole bucket. The book’s framing: modern outages are rarely full blackouts, they’re partial “brownouts,” and measuring at the event level is what lets you tell the difference and buy proportionally more time to react.

What a Datastore Needs to Actually Support This

None of the above works if storage can’t answer the questions an engineer asks mid-investigation. Chapter 13 lays out demands quite different from a typical metrics or logging database — four things that must be true at once:

• Speed — results in seconds, not “long enough to make coffee,” or nobody will use it mid-incident.
• No privileged fields — you can’t know in advance which of the (possibly thousands of) attributes will matter to a given investigation, so every field must be roughly as fast to query as any other. Indexing everything is prohibitively expensive; being fast without indexes is the real goal.
• Freshness — queryable within seconds of being generated, or you’re debugging a stale picture.
• Durability — losing data mid-investigation is close to the worst possible failure mode.

A normal time-series database can’t just be repurposed here because of what the book calls cardinality explosion. A TSDB is fast because it assumes distinct tag combinations stay few and get reused constantly — creating a new series is expensive, but appending to one is nearly free. Structured events break that assumption: nearly every event carries some unique combination of fields (a trace ID, a specific error message), so instead of appending to existing series, you’re constantly creating brand-new ones that get written to exactly once. The database ends up doing its expensive operation over and over, defeating its own design.

The book’s answer, borne out by two independent real-world implementations, is a hybrid: partition data by time first, then store it in columns within each partition — columnar speed for filtering arbitrary fields, with time partitioning keeping any single query from scanning the whole dataset.

Case Study: Honeycomb’s Retriever

Retriever is Honeycomb’s homegrown storage engine, and Chapter 13 walks through it in real detail, since the authors built it themselves.

The core idea: incoming trace spans get grouped into time-bounded “segments” — append-only chunks that close once they hit an age, size, or record-count limit. Within a segment, each field is written to its own file, so a query needing three fields out of two thousand only reads three files. Values get compressed with dictionary encoding (repeated values become short references) and bitmasks marking which rows even have a value for a sparse field.

Because the schema is essentially unknown at write time, Retriever pushes almost all real computation to query time instead. Segments older than a few hours ship off to S3 to keep local storage cheap, and querying that data fans work out across a swarm of serverless functions scanning segments in parallel, merged into one result. To handle the reality that a few parallel workers always straggle, Retriever races a second copy of the slowest 10% of requests rather than waiting — whichever finishes first wins. The numbers in the book are striking: millions of trace spans ingested per second, data queryable within milliseconds, query latency around 50 milliseconds even scanning hundreds of millions of records.

Case Study: ClickHouse’s Counter-Approach

Chapter 14, written by ClickHouse’s own engineering team, shows a different path to the same destination. Where Retriever splits ingestion, indexing, and querying into separate services, ClickHouse keeps things simple: every node runs the same single binary and handles both reads and writes.

Its storage engine, MergeTree, writes each batch of inserted rows as an immutable “part,” sorted by whatever columns form the primary key. Rows within a part are grouped into granules (blocks of a few thousand rows), and a sparse index remembers only the first key value per granule — small enough for memory, specific enough to skip huge swaths of irrelevant data before reading a column file. Background processes keep merging smaller parts into larger ones, hence the name.

For schema flexibility, ClickHouse spans a spectrum: dedicated columns for commonly filtered fields (fastest), a Map type for less-common attributes (flexible but slower), and a newer JSON type that auto-promotes frequently-seen fields into real columns while keeping rare ones tucked away. It also offers skip indexes (Bloom filters that let a query cheaply guess “not here, skip it”) and materialized views that pre-compute rollups on insert instead of at query time.

Scaling follows “grow up before you grow out”: push one well-provisioned node as far as possible before sharding, since inter-node network hops are usually the real bottleneck. When you must scale out, replication via a Raft-based coordinator called Keeper guards against node loss, and a newer variant called SharedMergeTree separates storage from compute entirely — nodes stop holding their own data copies and all read from shared object storage, the same instinct behind Retriever’s S3 tiering, reached from a different direction.

Here’s how the two stack up:

The book is refreshingly non-dogmatic here: it says outright that Honeycomb’s architectural complexity doesn’t make sense for small deployments, and that ClickHouse’s single-binary design is the better starting point unless you’re already at serious scale.

Enabling high-cardinality and exploratory observability is a solved problem.

That’s the authors’ own summary — worth sitting with, given that a decade ago people doubted this query pattern could work economically at all.

Key Takeaways

• Compare error rate against what the budget can proportionally sustain, not a raw threshold — keep predictive forecasts within about 4x of the baseline window used to build them.
• Judging SLOs per-request instead of per-time-bucket turns partial “brownouts” into proportionally smaller budget hits, buying real response time.
• A time-series database breaks down on structured events because nearly every event creates a brand-new, never-repeated tag combination — the opposite of what a TSDB optimizes for.
• Observability storage needs speed, freshness, durability, and equal performance across every field — commonly solved with columnar storage partitioned by time, computation deferred to query time.
• Retriever and ClickHouse solve the same problem with opposite architectures: many specialized services with serverless fan-out, versus a single flexible binary scaled up before out.
• Neither is objectively “better” — the right choice depends on your actual scale and appetite for operational complexity, not on which sounds more advanced.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• ClickHouse — the open-source column-oriented database discussed in Chapter 14, contributed by members of the ClickHouse engineering team.
• Honeycomb’s Retriever — the proprietary columnar storage engine detailed in the Chapter 13 case study.
• HyperDX — the open-source, MIT-licensed visualization layer built for ClickHouse, mentioned as a query interface option.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 4 of a 10-part series on Observability Engineering. Continue to Part 5: Sampling, Pipelines, and a Shared Language.