Observability Engineering — Part 1: What Observability Actually Means

Ask five engineers what “observability” means and you’ll probably get five different answers — a dashboard, a vendor category, “basically logging but fancier.” That vagueness isn’t an accident. When a word can mean anything, every monitoring tool on the market gets to claim it.

Observability Engineering (2nd Edition), by Charity Majors, Liz Fong-Jones, George Miranda, and Austin Parker, spends its opening chapters trying to undo that fog. The core claim: observability isn’t a feature you buy — it’s a property a system has more or less of, the same way it has more or less reliability. This post covers those first three chapters: where the term actually comes from, why the industry’s default way of collecting telemetry works against that definition, and why the authors think AI-written code is about to make all of this unavoidable.

A Word Borrowed From Control Theory

“Observability” didn’t originate in software at all. It comes from control theory, the branch of engineering math concerned with steering systems — thermostats, cruise control, guided missiles — toward a target state even when you can’t see everything happening inside them. Engineer Rudolf E. Kálmán introduced the term in 1960, in a paper now treated as a foundational text of the field.

Kálmán’s definition was precise: a system is observable if you can fully reconstruct what’s happening inside it just by watching what comes out of it. No peeking under the hood — just outputs. If some internal state can never be reconstructed from what the system exposes, that information is permanently lost, because it was never captured in the first place.

Software obviously isn’t the tidy mathematical system Kálmán had in mind. But the book argues the underlying question still applies, translated into something concrete: can you figure out what your system is doing right now using only the data it already emits — without writing new code, without SSHing into a box, without guessing? If yes, you have observability. If the honest answer is “not unless Dave is on call, because only Dave remembers how this part breaks,” you don’t — no matter how much telemetry you’re collecting or how much the platform on top of it costs.

Observability as a Property of Dependable Software

The book connects this to an older idea from systems engineering: dependability, meaning how much a system can be trusted to behave the way it’s supposed to. In 1995, researcher Jean-Claude Laprie proposed measuring dependability across six qualities — things like availability (does it respond when asked), reliability (does it do the right thing when it responds), and maintainability (can it be safely changed and repaired).

That framework held up fine when failures were mostly binary: the site was up, or it was down. Modern distributed systems break that assumption — failures today are more likely to be partial and strange, like a load balancer that’s degraded but not dead, or a dependency three services away misbehaving for one subset of users. Laprie’s six qualities say nothing about whether you can actually find and explain that kind of failure. The authors propose observability as a seventh property of dependability: a system’s capacity to have any of its internal states understood, by anyone on the team, using only what it emits.

The Three Pillars, and Why They Box You In

For most of the past decade, “observability tooling” has meant three separate categories: metrics (aggregated numbers over time, like request-count-per-minute), logs (timestamped text records of individual events), and traces (a record of how one request traveled through multiple services). Each usually lives in its own storage system with its own query language.

The book’s objection isn’t that these three things are useless — infrastructure teams managing systems they didn’t write (databases, load balancers, third-party services) are genuinely well served by this model. The objection is narrower: for the code your own team writes, deciding in advance to split every event into three disconnected shapes quietly limits what you’ll be able to ask later.

Here’s why. A metric is usually a single number with a handful of fixed labels; slicing it by a specific customer ID or build number is often impossible, because that would blow up the size of the metrics store. That constraint has a name: cardinality, meaning how many distinct values a field can take. A true/false field has low cardinality; a user ID has extremely high cardinality. High-cardinality fields are exactly what let you pinpoint this request rather than a vague aggregate — and metrics systems are generally built to reject them. Logs avoid the cardinality problem but don’t preserve connections between related events. Traces solve that for one request but don’t correlate with everything else.

The book’s alternative is to stop splitting things up. Every event gets recorded once, as a single “wide” structured record — a request, a job, a database call — carrying as many relevant fields as possible: user ID, build number, active feature flags, region, query duration. Because nothing has been pre-aggregated or discarded, you can generate a metric-like view, a log-like view, or a trace-like view from that same record afterward, on demand — including views nobody thought to ask for when the event was first recorded.

The book calls out a combinatorial effect here: each additional field attached to an event doesn’t just add one more thing to filter by — it multiplies the number of combinations you can slice by. Four fields give you eleven possible combinations; thirty fields give you over a billion. That’s the real argument for keeping data wide and un-aggregated: value compounds, and pre-aggregating throws that compounding away permanently. Once a metric has been averaged or bucketed, the original detail is gone for good.

Why This Is Suddenly Urgent: AI-Generated Code

Chapter 2 turns to an older organizational habit: the wall between the people who write code and the people who run it. In many companies, developers hand code off at deploy time and rarely look at it again in production — the tooling is unfamiliar, and the payoff has traditionally felt too small to justify the friction.

The book argues AI finally makes that wall untenable. When a human writes code, there’s someone who can explain the intent behind it. When an AI agent generates it, that mental model doesn’t exist in the same way — the only way to check whether generated code does what was intended is to watch what it does once it’s running for real. Preproduction testing still matters, but it can never be complete, because production is never fully reproducible in advance. That’s the sense in which the book calls production telemetry the last line of defense: the only remaining checkpoint once writing code stops being the bottleneck.

The chapter also distinguishes disposable code (cheap to regenerate from scratch, so not worth maintaining carefully — a one-off script, a prototype) from durable code (code you’ll patch and evolve in place for years, where every change needs to be small and reversible). How central a piece of code is to the business — its “critical path” — determines how much rigor it deserves. Code that runs constantly and touches money, safety, or reputation earns the highest bar; scaffolding that fails quietly and harmlessly doesn’t.

To validate code responsibly as it crosses into production, the book lays out practices that build on each other — a flywheel, because each one makes the next more effective:

Concretely: instrument code as you write it and look at it in production right away, before you’ve forgotten your own intent. Keep a real, if narrower, preproduction test suite whose job isn’t to prove correctness but to stop old mistakes from repeating. Separate deploying code (an engineering decision) from releasing a feature to users (a product decision) using feature flags, so a bad change can be switched off instantly instead of requiring a fresh rollback. And roll changes out gradually — to one server, one region, one slice of traffic — while automated checks compare that slice against the rest of production and revert if something looks wrong. The book also mentions a few specialty techniques: capture-and-replay for testing database upgrades against real recorded traffic, traffic splitting for diffing an old and new service side by side, and the “strangler fig” pattern (coined by Martin Fowler) for incrementally replacing legacy systems instead of rewriting them in one risky leap.

None of these practices are new. What the book stresses is that they only work as well as the data feeding them — a canary comparison is worthless if your telemetry can’t isolate that traffic with precision, and an automatic rollback is dangerous if the triggering signal is noisy. That’s why “build rich observability first” is Practice 0, not an afterthought.

Charity Majors on How the Term Got Away From Its Own Inventors

Chapter 3 switches voice: it’s written directly by Charity Majors, cofounder and CTO of Honeycomb, recounting how she and cofounder Christine Yen ended up in the middle of this debate. Both had been engineers at Facebook, using an internal analytics tool called Scuba that let anyone — junior or senior — explore huge amounts of raw event data on the fly, without knowing in advance what they were looking for. Missing that tool enough to try rebuilding it is what eventually led them to found Honeycomb.

They struggled to explain what made their product different from a logging tool or a monitoring dashboard, until Majors came across Kálmán’s control-theory definition and, in her words, it “clicked.” From there, she and Yen worked backward to a specific technical bar: rich, wide events; unrestricted cardinality; no pre-aggregation at write time; fast, exploratory queries instead of static dashboards.

Majors is candid that they lost the fight to keep that definition intact. Once “observability” started generating buzz around 2017-2018, existing monitoring and logging vendors simply relabeled their products as observability tools rather than rebuild them to meet the technical bar. By the time analyst firm Gartner added an “observability” market category in 2022, the term had largely finished its slide toward meaninglessness — which is why she now distinguishes between an old, siloed “Observability 1.0” and the wide-event “Observability 2.0” the book advocates (noting that newer labels, like Hazel Weakly’s proposed “Observability 3.0” for data-lakehouse approaches, are already appearing).

She offers a memorable analogy for why real observability costs more than plain monitoring: monitoring is fine for a personal bank account, where checking your balance occasionally is enough — but a bank has to trace every transaction, on demand, for every account, or risk real legal consequences. As she puts it in the book:

“Does observability cost more than monitoring? Yes. Because it does more.”

Key Takeaways

• Observability isn’t a tool category — it’s a property of a system, meaning how well you can understand any internal state purely by watching what the system outputs, without shipping new code first.
• The term comes from 1960s control theory (Rudolf E. Kálmán), and the book proposes adding it as a seventh quality alongside older dependability measures like availability and reliability.
• Splitting telemetry into metrics, logs, and traces at write time throws away combinations of data you might need later; storing one wide structured event and generating views from it afterward preserves that flexibility.
• High cardinality (many unique values, like user IDs) and high dimensionality (many fields per event) are what make an event useful for pinpointing a specific problem — and they’re exactly what traditional metrics systems are built to discard.
• AI-generated code removes the human developer’s mental model of “what this should do,” which is why the book treats production observability as the last remaining checkpoint for validating that code behaves as intended.
• The test-in-production flywheel (instrumentation, preproduction tests, feature flags, canaries, automated rollback) only works when it’s grounded in rich, high-cardinality telemetry — otherwise every step in the flywheel is running on noise.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Kálmán, Rudolf E. “On the General Theory of Control Systems.” IFAC Proceedings Volumes 1, no. 1 (August 1960): 491–502. (The paper that introduced the mathematical definition of observability the book builds on.)
• Laprie, Jean-Claude. “Dependability — Its Attributes, Impairments and Means.” In Predictably Dependable Computing Systems, edited by Brian Randell, Jean-Claude Laprie, Hermann Kopetz, and Bev Littlewood. ESPRIT Basic Research Series. Springer, 1995. (Source of the six-quality dependability framework the book extends.)
• Honeycomb — the observability company cofounded by Charity Majors and Christine Yen, whose founding story anchors Chapter 3.
• Fowler, Martin. The “strangler fig” pattern for incrementally migrating legacy systems, referenced in Chapter 2.
• Cantrill, Bryan — credited in the book with an early 2003 internal presentation at Sun Microsystems that helped introduce the term “observability” into systems engineering circles.
• Weakly, Hazel — credited in the book with proposing “Observability 3.0” to describe newer data-lakehouse-based approaches.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 1 of a 10-part series on Observability Engineering. Continue to Part 2: The Art of Instrumentation.