Observability Engineering — Part 2: The Art of Instrumentation

Part 1 of this series made the philosophical case for observability: the ability to ask new questions about your system without shipping new code first. Part 2 is where the book gets its hands dirty. If you buy the argument that observability matters, the very next question is mechanical: what do you actually put in your telemetry, and how?

That’s what chapters 4 through 7 answer. They cover the basics of instrumentation, a genuinely useful reframe of what “telemetry” even is, a long practical list of what to capture from a guest contributor who’s clearly done this for real systems, and a tour of OpenTelemetry — the open-source project that has become the default way to do all of this in practice.

Instrumentation Is Something You Design, Not Something You Bolt On

Instrumentation just means writing code whose job is to report on what your other code is doing. Every timer around a database call, every log line, every counter is instrumentation. It sounds simple, and it is — but treating it as a first-class part of the system, rather than a debugging afterthought, changes how well it works.

For a long time, the default pattern was reactive: ship the feature, and only add logging once something breaks in production and someone needs to reconstruct what happened. That produces exactly the telemetry you’d expect from a rushed, panicked addition — inconsistent, incomplete, and shaped around whatever the last incident happened to need rather than the incident you’re actually having right now.

The book argues instrumentation belongs to the engineers writing the feature, not a separate operations team working after the fact — they’re the ones who understand the business logic and the code paths that tend to misbehave. In mature teams this becomes a habit: think about telemetry while designing a feature, check for it in code review, and after every incident ask what would have made the investigation faster and add that.

Two flavors of instrumentation work together here. Automatic instrumentation comes largely for free — libraries that hook into your web framework, database driver, or RPC layer and report on technical mechanics like request durations and error codes. It gives broad coverage instantly but has no idea what your business does. Custom instrumentation is what you write yourself to capture what automatic tooling can’t know — that this database call was part of a signup flow, or that this request came from your highest-value pricing tier. Good systems lean on both: automatic instrumentation for the skeleton, custom instrumentation for everything that makes your product yours.

None of this is free — every field you capture costs storage, processing, and often a literal vendor bill. The book’s answer isn’t to instrument less, it’s to sample deliberately rather than accidentally. Sampling before data is emitted (keeping only some fraction of requests, or pre-aggregating everything) is cheap but blind: a rare error that shows up 0.1% of the time can vanish if you’re only keeping 1% of traffic. Sampling after data is emitted — looking at a whole record and then deciding whether it’s worth keeping — costs more upfront but lets you keep the boring majority cheap without losing the rare, interesting cases.

One Rich Record Beats Three Separate Silos

This is where the book makes its central argument, and it’s worth sitting with. For decades, engineers have treated logs, metrics, and traces as three fundamentally different kinds of data, each with its own storage system, query language, and mental model. Metrics give you a fast pulse check. Logs give you a narrative. Traces give you the shape of a request as it hops across services. The book’s claim is that this split isn’t a law of nature — it’s a historical accident, inherited from an era when computers had a fraction of today’s memory and bandwidth.

Why does that matter beyond being mildly annoying? Whatever shape you commit your data to when you collect it is the only shape you’ll have later. If you only ever stored an averaged response time per minute, you can never go back and ask what the P99 was — that information was gone the moment it got averaged away. Deciding the data’s shape in advance quietly decides which questions you’re allowed to ask during an investigation.

“The shape of the data you collect constrains the questions you can ask later.”

The book’s fix is to stop treating logs, metrics, and traces as separate species and instead treat them as views generated on demand from one underlying thing: a structured event, a flat record of key-value pairs describing everything that happened during one unit of work — a timestamp, which service handled it, how long it took, which user was involved, whether it errored.

Seen this way, all three “pillars” are the same substance in different packaging. A metric is a structured event that’s been aggregated and stripped of detail to save space. A log is a structured event with a message field. A trace is a set of structured events (spans) sharing a trace ID, with a parent-child relationship showing how one span nested inside another. These aren’t separate technologies — they’re different queries over the same data.

The practical payoff is that you don’t need to have predicted your question in advance. If you captured a wide event with duration, region, device type, and a dozen other fields, you can compute a P99 latency metric today, and next week compute the exact same P99 broken down by region — without touching your instrumentation code or waiting for new data to accumulate. You’re not choosing the shape of your data before the investigation; you’re choosing it during the investigation.

What Belongs in the Event: A Field Guide

The book’s Chapter 6 — written by guest contributor Jeremy Morrell, a Principal Engineer at Cloudflare — turns the abstract “structured event” idea into a genuinely practical checklist. His pitch is straightforward: most teams underestimate by an order of magnitude how many fields a well-instrumented request should carry. A dozen attributes is a start; a mature system typically has hundreds.

Rather than reproduce his list field-by-field, here’s how the categories map onto situations you’ll actually be debugging:

The common thread across all of these is that the fields aren’t just describing what happened — they’re capturing enough surrounding context that why becomes answerable later, without having to reproduce the problem or dig through five different tools by hand. Morrell also flags a small but genuinely useful convention: tag your intentionally wide, catch-all events with a single boolean field (he calls his main) so you can cleanly filter them out from all the other, narrower logs and spans your system still emits for other purposes.

OpenTelemetry: The Standard Way to Actually Build This

Knowing what to capture doesn’t tell you how to wire it up without locking yourself into one vendor’s proprietary agent. That’s the problem OpenTelemetry (often shortened to OTel) solves. It’s an open-source, vendor-neutral project — under the Cloud Native Computing Foundation — that gives you a common API for emitting telemetry and a common wire format for shipping it, so you can point the same instrumentation at any backend, or several backends at once, without rewriting application code.

OTel is described as “trace-first,” meaning context does the heavy lifting. A trace ID and span ID get created for a unit of work and carried along everywhere it goes — across an HTTP call via request headers, or through a single process via thread-local or async-local storage. Because that identifier travels with the work, every log line, metric, and span touched along the way ties back to the same request — the mechanism that makes “one wide event, many views” actually work in a live distributed system.

A recurring question teams get wrong is when to wrap something in its own span. The book offers a simple two-part test: is this operation interesting (does its duration or failure rate actually vary and matter), and is it aggregable (would grouping a thousand of these together tell you something useful)? A database query passes both. A getter or private helper passes neither. Get this wrong in either direction and you pay for it — too many spans buries every trace in noise, too few collapses a whole request into one opaque black box.

Not every system fits the clean request-in, response-out shape traces were built for. Streaming pipelines, fan-out/fan-in jobs, and long batch work rarely have one obvious root span — the book’s guidance is to lean on span links and a shared correlation ID passed through each stage, instead of forcing everything into one ever-growing parent span. For code you genuinely can’t touch, it also points to eBPF: programs that run inside the Linux kernel and can observe network activity and propagate trace context without touching application source at all.

AI Agents Change the Mechanics, Not the Judgment

The book closes this section with a candid look at AI coding assistants — tools like Claude Code, Cursor, and Gemini CLI are named as current examples. Ask one to “add OpenTelemetry to this service” and it will often produce something that runs and emits data. The catch: telemetry that compiles and telemetry that’s actually useful are two different bars to clear. An agent has no idea which code paths matter to the business unless you tell it.

The book’s advice is to use agents in two modes. In planning mode, you design span boundaries, names, and attributes with the agent before any code gets written — reviewing a spec, not a diff. In execution mode, you give it explicit conventions to follow, especially in older codebases, since an agent will otherwise copy whatever pattern already exists in the repo, good or bad. Keeping changes small enough to review, and writing your team’s naming conventions somewhere the agent can read them, matters more with AI-generated instrumentation than with hand-written code, not less.

Key Takeaways

• Instrumentation is a design decision owned by the engineers writing the feature — not paperwork added by a separate team after something breaks.
• Logs, metrics, and traces aren’t fundamentally different data types; they’re different views generated from one underlying structured event, and the shape you commit to early limits the questions you can ask later.
• A well-instrumented request typically needs far more context than intuition suggests — service and build metadata, per-stage timings, user and account context, and a unique slug per error location, so a slow request or partial failure points straight at a cause.
• OpenTelemetry’s value is context propagation: a trace ID that travels with a request everywhere it goes is what lets every signal along the way get tied back together.
• Not every function needs its own span — ask whether the work is genuinely variable and worth grouping in aggregate before instrumenting it.
• AI coding assistants can generate working telemetry quickly, but only explicit, written conventions turn “it compiles” into “it’s actually useful during an incident.”

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• The OpenTelemetry project (opentelemetry.io), the CNCF open-source observability standard.
• Jeremy Morrell, Principal Engineer at Cloudflare, credited guest contributor for the book’s practical instrumentation chapter (Chapter 6) and a major contributor to Chapter 5.
• Young, Ted, and Austin Parker. Learning OpenTelemetry. O’Reilly Media, 2024 — recommended in the book as a deeper, dedicated guide to OTel.
• Honeycomb, referenced for its BubbleUp analysis feature as an example of tooling built around wide structured events.
• Hannah Moran, Anthropic engineer, credited in the book for a widely cited 2025 definition of what an AI agent is.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 2 of a 10-part series on Observability Engineering. Continue to Part 3: From Data to Insights.