Observability Engineering — Part 3: From Data to Insights

Collecting good telemetry is the easy part, relatively speaking. The harder question is what you do with it at 2 a.m. when something breaks and nobody knows why. Part III of Observability Engineering is about that gap — turning a mountain of structured events into a fast, repeatable path to “here’s what happened.”

This part covers four things: a step-by-step method for investigating problems without guessing, a case for baking observability into everyday coding and deployment rather than bolting it on after incidents, a guest chapter on where AI agents genuinely help, and a rethink of alerting that swaps noisy thresholds for something tied to what users actually feel.

Debugging Without a Map

The senior engineer who’s seen a system break a dozen ways glances at a dashboard and just knows where to look. To everyone else that looks like magic. It isn’t — it’s pattern memory built from living through enough failures. That kind of knowledge doesn’t scale, so teams try to bottle it into runbooks cataloging every remembered failure. But distributed systems rarely break the same way twice, and a stale runbook is worse than none — people trust it right up until it lies to them. The book argues the only documentation worth maintaining is ownership, escalation paths, dependencies, and a few good starting queries; everything else calcifies too fast to stay reliable.

The alternative is a repeatable loop that works whether you joined yesterday or built the service five years ago, and it treats every hypothesis as provisional until data confirms it:

1. Start with the signal. A complaint, an anomaly, an alert — what exactly did it tell you?
2. Confirm it’s real. Check the telemetry for a genuine shift instead of assuming one.
3. Compare the weird part against normal. Which fields show up disproportionately inside the anomaly versus the baseline?
4. Decide if you know enough. If not, narrow your view and repeat step 3.

That last point is what makes the loop democratic: a newcomer can debug as effectively as a veteran, because it rewards curiosity over institutional memory.

This works by hand, but manually diffing fields against a baseline is grinding work computers do better than people. Automating it means computing every dimension’s frequency inside the anomaly versus outside it and ranking the differences, so you start with a short list of suspects. Honeycomb’s BubbleUp feature is the book’s example: draw a box around the weird part of a graph, and it ranks which fields — an availability zone, an endpoint — show up disproportionately inside it. In one real incident, that surfaced that nearly all the slow requests shared a single cloud availability zone, which turned out to be a genuine outage on the provider’s side.

That raises the question of how far to push automation with generative AI. The book sorts AI assistants into three relationships with the investigator: a copilot alongside you, only as good as the questions you steer it toward; a commander that takes a fuzzier goal like “why is my service slow?” and runs with more independence, the way Cursor or Claude Code do; and a caretaker operating unsupervised on standing rules like “after every deploy, check the new feature flag.” The further toward autonomy, the faster a small misunderstanding compounds, since the agent has no tribal memory to fall back on.

Building Observability Into How You Ship Code

It’s tempting to think of observability as something that kicks in once code is running in production. Chapter 9 pushes back, proposing that instrumentation belongs in the feature itself, not bolted on afterward — a practice it calls observability-driven development, echoing test-driven development (TDD) while covering different ground.

TDD proves your code does what the spec says, inside a sandbox where data resets and dependencies are mocked into predictable behavior. That predictability is also its limit: production is nothing like a sandbox, and real traffic produces a long tail of conditions no test suite anticipated. TDD proves correctness against a spec; it says nothing about whether the code behaves well once real users touch it.

So the practical habit is: whenever you write a pull request, ask “how will I know if this works once it ships?” and add the instrumentation to answer that before you merge, not after something breaks. One trick the book highlights: route the on-call page for a fresh deployment straight to whoever wrote it, at least for the first half hour — feeling your own code misbehave, even briefly, teaches more than a postmortem written weeks later by someone else.

There’s a business case too, drawn from the research behind Accelerate (Forsgren, Humble, and Kim): the old assumption that speed and quality trade off is false for the best-performing teams — they get faster and more reliable together, since small, well-instrumented changes are easier to reason about and roll back than big, opaque ones. That’s what turns production from a “glass castle” everyone fears touching into a place engineers can work confidently.

On the AI side, the advice is grounded: coding assistants will happily add tracing the moment you ask, but “working” telemetry and “useful” telemetry aren’t the same thing. Left alone, agents invent attribute names, duplicate signals across metrics and logs, and bury you in traces nobody asked for. The fix is writing your team’s instrumentation conventions down once, somewhere the agent’s configuration can reference, so it has something concrete to follow instead of guessing.

What AI Agents Are Actually Good At

Chapter 10 hands the mic to guest author Boris Tane, founder and CEO of Coreplane Labs, who deliberately avoids predicting which AI tools will matter in a year — the space moves too fast. Instead he zeroes in on something durable: agents, like the humans they assist, are only as good as the context they’re given.

That’s easy to underestimate. An experienced engineer investigating an incident carries months of knowledge about which service names are really aliases for each other, which dependency has been flaky since last quarter’s migration, and what “normal” looks like on a random Tuesday. Hand an agent raw query access to your telemetry without any of that, and it produces answers that are technically accurate and practically useless, because it can’t tell a meaningful correlation from a coincidental one.

Where agents genuinely earn their keep is narrower: running the tedious first minutes of incident response (pulling dashboards, checking recent deploys, mapping affected services) so a human starts reasoning sooner; translating a confusing stack trace or trace waterfall into a plain-language explanation a junior engineer can act on immediately; and auditing instrumentation or alerts for gaps and drift — unglamorous, ongoing work human attention tends to slide off of.

The chapter’s real thesis is that all of this depends on a mental model that used to live entirely in senior engineers’ heads — what calls what, which names are aliases, which incidents happened recently and why. That map is getting harder to maintain, not easier, as more code ships with less sustained human attention on any one part of the system. If nobody actively curates that context, agents — and new engineers, for that matter — are left reasoning about a system nobody fully understands.

Why Thresholds Fail and SLOs Don’t

Chapter 11 turns to a problem almost every engineer knows: a pager that goes off constantly for things nobody needs to act on. Threshold alerts — CPU over 80%, memory under 10%, error rate over some percentage — measure whatever’s convenient, not whatever predicts a bad time for users. A CPU spike might be a real outage, or a backup job, or garbage collection doing its job. The alert can’t tell the difference, so it fires anyway, and engineers learn to shrug it off.

That habitual shrugging has a name borrowed from outside software: normalization of deviance, a term from the investigation into the Challenger disaster, describing how a team that mutes enough false alarms stops noticing it’s doing so — until the one alert that mattered gets lost in the same noise.

Threshold alerts aren’t useless — they suit known-unknowns: failure modes you can name in advance, like a disk nearing capacity or a certificate about to expire. The problem is that distributed systems mostly fail in ways nobody enumerated ahead of time — unknown-unknowns — and you cannot write a threshold for a failure mode you haven’t imagined. Trying anyway just piles up brittle checks that still don’t tell you whether a customer is actually suffering.

Service level objectives (SLOs) reframe the question. Instead of watching internal system metrics, you define a target tied to a real user journey — “checkout should complete successfully and quickly” — measured with a service level indicator (SLI) that scores each event good or bad. The book favors event-based SLIs (what fraction of requests met the bar) over time-bucketed averages, since averaging over five-minute windows hides real bad experiences. Every bad event spends a little of your error budget — the allowed failure within your target — and an alert only fires when that budget is genuinely at risk, not whenever some unrelated number twitches.

There’s a real cost to getting this right, illustrated through a firsthand account from Mat Vine, Cloud, Quality, and Reliability Lead at ANZ Bank. His team launched a working set of SLOs in twelve weeks against an original estimate of twelve to eighteen months, not because SLOs are quick to write but because they’d already invested in standardized OpenTelemetry instrumentation. That left the remaining work purely organizational: agreeing which handful of user journeys actually mattered and getting genuine buy-in, rather than fighting inconsistent data formats.

Drafting SLOs is often blocked by ambiguity rather than technical difficulty, which is where the book brings in generative AI again. LLMs are useful for turning a vague target (“checkout should be reliable”) into a concrete first draft — a reliability percentage with a business rationale, or a working SLI query once you supply the schema details. It won’t replace the judgment call about what your business can tolerate, but it removes the blank-page problem that stalls a lot of SLO efforts before they start.

“Observability is the telescope that helps you aim the microscope.”

That line, from the book’s discussion of how observability and debuggers relate to each other, sums up this whole part well. Observability doesn’t replace judgment, testing, or a good old-fashioned debugger — it tells you where to point them.

Key Takeaways

• Debug from first principles, not intuition: hypothesize, slice telemetry to compare the anomaly against a baseline, and repeat — it works even on a system you’ve never seen fail before.
• Runbooks that try to catalog every failure go stale fast; stick to ownership, dependencies, and a few good starting queries, and let live telemetry do the rest.
• Treat instrumentation as part of the feature, not an afterthought — asking “how will I know this works?” before merging shortens the gap between shipping and understanding.
• AI agents are only as useful as the context you give them; the same model produces shallow guesses with thin context and useful answers with rich, well-maintained context.
• Threshold alerts suit known failure modes but can’t cover the unknown-unknowns that dominate distributed systems — that’s the gap SLOs, measured through event-based indicators and error budgets, are built to fill.
• Getting SLOs adopted is usually an organizational challenge more than a technical one, and AI can help produce a usable first draft to argue over.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Boris Tane, Founder and CEO of Coreplane Labs — guest contributor for the chapter on AI agents in observability workflows.
• Mat Vine, Cloud, Quality, and Reliability Lead at ANZ Bank — guest contributor for the SLO adoption case study.
• Murphy, Niall Richard, Betsy Beyer, Chris Jones, and Jennifer Petoff. Site Reliability Engineering. O’Reilly Media, 2016.
• Hidalgo, Alex. Implementing Service Level Objectives. O’Reilly Media, 2020.
• Forsgren, Nicole, Jez Humble, and Gene Kim. Accelerate: Building and Scaling High Performing Technology Organizations. IT Revolution Press, 2018.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 3 of a 10-part series on Observability Engineering. Continue to Part 4: SLO-Based Alerts and Storage That Scales.