Practical OpenTelemetry — Part 1: The Five Clocks Inside Your MTTR

Here’s an uncomfortable question for anyone who owns a production system: when something breaks, which clock are you actually racing?

Most teams answer “MTTR” — mean time to recovery. It’s the one metric everyone tracks. But Practical OpenTelemetry by Daniel Gomez Blanco opens with an argument that MTTR is a lie of aggregation: it’s five different clocks hiding inside one number, and the industry has spent twenty years optimizing one of them while the others barely moved.

This post — the first in a series walking through the book — covers why observability matters, where the real gap is, and why context propagation (the thing OpenTelemetry standardizes) is the difference between a five-team incident call and a single engineer finding the root cause in minutes.

A note on the series: the Observability Engineering series on this blog covered the philosophy — SLOs, debugging workflows, the culture. This series is the implementation companion: the OpenTelemetry project itself, its APIs and SDKs, the Collector, and the protocols. If you haven’t read that series, this one stands alone; if you have, think of this as “the part where the standards actually get built.”

What “Observability” Actually Means

Before the book gets anywhere near MTTR, it defines the term precisely — and the definition predates software entirely. Observability is a control-theory concept: the degree to which a system’s internal state can be inferred purely from its external outputs. Rudolf E. Kálmán introduced it in 1960, studying control systems, not computers.

Why that matters here: observability is what makes a system controllable. You can only close the loop between “I changed something” and “here’s the effect it had” if you can actually see enough of the internal state to tell what changed. A system you can’t observe is a system you can only guess at — which is the whole problem this post is about.

Failure Is Not an If

The book starts from a position most teams accept intellectually but rarely act on: failure is constant. Component changes, unstable infrastructure, human error — Gomez Blanco adds “or even cosmic rays,” citing research on radiation-induced bit flips. Regressions will happen.

If failure is constant, then the thing worth optimizing isn’t avoiding failures — it’s the speed at which you recover from them. Reliability work should focus on reducing the time to debug and fix, not just the time to prevent.

That reframing matters because it changes what you measure. And what you measure changes what you build.

Five Clocks Hiding Inside “MTTR”

The book spends real effort separating two things most teams conflate:

• MTTRec (Mean Time to Recovery) — incident start to the system returning to normal operation. This can happen before the root cause is found: an N+1 redundant system can fail over and self-heal while engineers are still scratching their heads.
• MTTRes (Mean Time to Resolution) — incident start to the root cause being identified and the fix deployed.

These decouple. You can recover without resolving. And MTTRes itself decomposes into four sub-clocks:

(A terminology note worth flagging once: reliability engineering usually reserves MTTF for Mean Time To Failure — the interval between failures of a non-repairable component. The book overloads the abbreviation for Mean Time to Fix instead. Different concept, same four letters. We’re following the book’s usage for the rest of this series — just don’t carry it into a conversation about hardware reliability without checking which one someone means.)

Here’s the thing the book hammers home: the industry spent decades optimizing MTTD and MTTV. Alerting got faster, dashboards got prettier, alert fatigue got worse. Microservices and continuous deployment made rollbacks nearly instant, which crushed MTTF. But MTTK — the time from “something’s wrong” to “I know why” — barely moved. Teams still debug the way they did in 2005: open a dashboard, grep a sea of logs, page someone who remembers how this part breaks.

The Two Questions Observability Must Answer

The book reduces observability to two questions:

1. “Is my system behaving as expected?” — targets MTTD and MTTV.
2. “Why is my system not behaving as expected?” — targets MTTK.

These require considerably different telemetry. Question 1 wants stable, cheap, always-on aggregates — metrics, essentially. Question 2 wants high-granularity, per-transaction detail you can drill into when something goes wrong — traces and correlated logs.

Historically, nearly all effort went into question 1. That’s why we have vast data volumes feeding dashboards and alerts while debugging workflows remain “rudimentary at best” — component-specific dashboards and unstructured log dumps.

Context: The Missing Ingredient

The book’s most memorable passage is a worked scenario that will feel familiar to anyone who’s been on call:

A Payment Service experiences a latency regression. It’s only 0.1% of the traffic to a downstream Config Service. Config Service’s 95th percentile looks fine — no alerts fire. The real culprit is slow database queries through a Config Store Proxy caused by poor indexing.

Without correlation, this incident is a multi-team scavenger hunt. Someone has to manually align graphs from three different systems, page the Config Service team, and argue about whose dashboard is wrong. The author’s aside is painfully relatable: “I have personally had to move graphs around in the screen to check that they visually align more often than I’d like to admit.”

With context propagation — the ability to pass a transaction ID through every hop of a request, including non-direct dependencies — the same investigation collapses. You propagate a transaction ID, find patterns within the affected transactions only, and the root cause shows itself conclusively, DB queries and all.

This is what distributed tracing gives you: supercharged, standardized, structured application logs. Every meaningful operation timed as a span, correlated across services and replicas, drillable by one person with no prior knowledge of the system.

Why “Store Everything” Doesn’t Work

There’s a tempting counterargument: storage is cheap, so why not just keep every log line and every request with full attributes, forever?

It’s the pattern most teams actually run today — dashboards built on Brendan Gregg’s USE method (Usage, Saturation, Errors), backed by whatever got logged, on the theory that if you store enough of everything, the answer is in there somewhere. The book’s answer is blunt math. Naively storing every client/server interaction with full context is prohibitively expensive — more than 95% of that data would be of no debugging interest whatsoever. Most requests succeed. Most log lines describe happy paths. You’d be paying to store and query a warehouse of noise.

And here’s the counterintuitive part: naive mitigation (log only errors and slow requests, plus a random sample) doesn’t work either. Because a single service replica cannot know whether its request belongs to a “good” or “bad” transaction — it only sees its own boundaries. The payment service’s 404 might be noise; the same 404 inside a transaction that eventually fails is gold. That decision requires context that crosses service boundaries.

This is exactly what observability tooling enables: transaction-aware sampling decisions. Store the interesting transactions, discard the noise — but make that decision with the full picture, not a single service’s limited view.

The book closes the chapter with a lovely analogy: we’d all rather drive a car that shows our current speed 99.9% of the time than one that shows it 100% of the time with a ten-minute delay. Observability data wants freshness over completeness; auditing data wants the opposite. Conflating the two — pushing everything through the same pipeline — is the root of a lot of telemetry cost.

Why OpenTelemetry Exists to Close That Gap

Chapter 1’s thesis in one sentence: monitoring optimized “is something wrong?”; observability must optimize “why is it wrong?” — and that requires standardized, correlated, context-rich telemetry across every service in the transaction path.

That word “standardized” is the whole rest of the book, and the whole rest of this series. Context propagation only collapses a five-team incident call into a one-person investigation if every service along the path agrees on how to pass, name, and export that context — which is a project-wide problem no single team can solve by instrumenting harder. Part 2 covers the standard built to solve it: how OpenTelemetry merged two competing, incompatible projects into one, and why that merger is the reason “instrument once, export anywhere” stopped being a slogan and started being something vendors had to build around.

Next: Practical OpenTelemetry — Part 2: Open Standards and the Vendor Shift

References

• Practical OpenTelemetry (Apress, 2023) — Daniel Gomez Blanco, the book this series follows chapter by chapter
• Kálmán, “On the General Theory of Control Systems” (1960) — the control-theory origin of “observability”
• Brendan Gregg, “The USE Method” — the dashboard pattern behind most “store everything” observability setups