Observability Engineering — Part 5: Sampling, Pipelines, and a Shared Language

Once a system is generating good telemetry, a new problem shows up: there’s too much of it. This part of the book is about the plumbing that keeps that flood manageable — deciding what to keep, deciding where it goes, and making sure everyone agrees on what the data actually means.

Three chapters carry this section. The core authors cover sampling — cutting event volume without losing the ability to debug. Two guest chapters follow: Mike Kelly, founder and CEO of Bindplane, on telemetry pipelines, the routing layer between everything that emits data and everywhere it needs to go; and Frank Chen, CTO at Balto Energy and a designer-in-residence at UC San Diego’s Design Lab, on ontologies — shared definitions that keep “uptime” or “error” from meaning different things to different teams, which matters even more once AI agents start acting on your telemetry.

Why You Can’t Keep Everything

At small scale, storing every event, every log line, every trace span is fine. At large scale, it isn’t — the infrastructure needed to store and process all your telemetry starts rivaling the size of the production system it’s watching, before you’ve gotten any debugging value out of it.

The instinct here is to reach for aggregated metrics: roll everything up into a handful of counters and gauges. But pre-aggregation throws away the ability to slice data by arbitrary fields later. Once ten thousand requests collapse into one average, you can never ask “was this only happening to customers on the free tier?”

Sampling is the better lever. Instead of compressing every event into an aggregate, you keep a subset of the actual, individual events — untouched, fields intact — and use math to make sure that subset still represents the whole. A sampled event isn’t a lossy summary; it’s a real event standing in for itself and a number of statistical twins.

The Sampling Toolkit

The simplest version is fixed-rate sampling: keep 1 out of every N events, discard the rest. It’s easy to reason about, and with enough volume, any error that matters will show up often enough to survive the cut. But it breaks down when you care much more about failures than successes, when one customer sends orders of magnitude more traffic than another and you still need to catch their problems, or when a traffic spike would otherwise spike your observability bill too.

Dynamic, traffic-based sampling fixes the spike problem by adjusting the keep-rate on the fly — sample generously when quiet, back off when hammered. The catch: your backend can no longer multiply every kept event by one constant to reconstruct totals, since the multiplier now changes over time. Each event has to carry its own sample rate as a tag, or the math quietly breaks. A naive median of three raw values might read as one number, while the true median — once each value is correctly expanded by how many similar events it represents — comes out completely different.

Content-based (per-key) sampling varies the rate by what’s in the event: keep all the errors, keep a healthy slice of slow requests, keep only one in a thousand routine successes. This works well as long as the key space is small and the mix of “boring” versus “interesting” traffic stays stable — worth checking explicitly, since if errors ever become the majority case, a rate built for “errors are rare” lets too much through.

The two techniques combine naturally: pick a key, then adjust that key’s specific rate based on how often it’s shown up recently. A rare combination gets kept almost every time; a constant one gets thinned out. This is target-rate sampling applied per key instead of globally, and it’s what most production systems actually want, since it guards against both “too much data” and “missed the rare thing.”

Traces Add a Timing Problem

All of this works fine for single, self-contained events. Distributed traces complicate it: a trace is a chain of spans generated by different services, each possibly running its own sampling logic independently. If every service flips its own coin, the odds that every span belonging to one trace survives are terrible — you end up with fragments instead of a story.

There are two ways to make a sampling decision for a trace, and they differ in when it happens.

Head-based sampling decides at the very start, using only what’s known up front (an endpoint, a customer ID). That decision rides along with the request as a header, and every downstream service honors it — the whole trace is kept or dropped together. Simple and cheap, but blind: it can’t react to something discovered later, like an error or a slow downstream call.

Tail-based sampling waits until the request finishes, then decides based on outcome — this trace had an error, keep the whole thing. More useful, but not free: every span must be buffered somewhere until the trace resolves, so the decision usually has to be made by a separate collector process rather than in application code.

Most real systems combine both: head-based sampling handles the everyday baseline consistently, while tail-based logic reaches back and rescues traces that turned out to be interesting after the fact. Libraries like Go’s dynsampler package — and increasingly OpenTelemetry itself — implement this so teams don’t have to hand-roll it, but the mechanics still matter when deciding which knob to turn.

Telemetry Pipelines: Routing Before Storage

Sampling reduces volume at the source. Mike Kelly’s chapter tackles a different piece of the puzzle: once telemetry exists, how does it reach the right place, in the right shape, without every application needing to know about every destination?

Kelly’s framing is a utility grid: you don’t wire every appliance straight to a power plant, you route through substations. A telemetry pipeline plays the same role for observability data — applications plug in once, and the pipeline handles getting the data where it needs to go.

He breaks a pipeline into five stages, wrapped in two supporting concerns:

• Collect — gather logs, metrics, and traces from everything, modern and legacy. OpenTelemetry has become the common language here, with well over a hundred built-in receivers covering everything from OTLP to Prometheus to JMX.
• Normalize and secure — turn messy formats into one common shape, fix timestamps, and strip sensitive data before it travels further, to satisfy rules like GDPR or FedRAMP.
• Enrich — attach context the raw signal lacks: which Kubernetes pod or region it came from, which customer or environment it belongs to.
• Reduce — filtering, deduplication, aggregation, and sampling all live here, cutting volume before it reaches a paid destination.
• Route — fan the same clean stream out to however many destinations need it: an observability backend, a security tool, cold storage, an analytics warehouse.

Underneath all five, a pipeline needs resilience (buffering and retries so a backend outage doesn’t mean lost data) and its own observability — dashboards on throughput, error rate, and queue depth, since a pipeline is infrastructure too and can fail silently.

Architecturally, this usually means lightweight agents near each source, forwarding to centralized gateways that do the heavy lifting, with a control plane on top to configure and monitor the fleet. Kelly notes the reduction step alone routinely saves 50% or more of vendor ingest costs, since a meaningful share of collected data — in one case study, around 40% of SIEM data — is never actually queried.

For adoption, Kelly recommends starting small: a single greenfield application on the new pipeline, then redirecting legacy streams into it for centralized handling, and only replacing the oldest agents last — some fragile systems may stay on an intermediate step indefinitely, which is a fine trade-off.

Ontologies: Making Sure Everyone Means the Same Thing

Frank Chen’s chapter opens with a scenario every on-call engineer recognizes: middle of the night, pager going off, the infrastructure dashboard says everything is fine — servers up, latency low, status codes green — while the business side is on fire, because customers are being billed incorrectly and an AI agent has started proposing actions that don’t make sense.

Nothing crashed. Nothing errored. “Healthy” meant something different to the infrastructure dashboard than it did to the business. That’s a semantic failure, not a technical one, and it gets worse as an organization grows and as decisions get delegated to AI systems that can’t tell a slightly-wrong field from a correct one.

Chen’s fix is an ontology: an explicit, agreed-upon map of a domain’s core objects, how they relate, and the rules that must always hold true. In observability terms this becomes semantic conventions — standardized field names every service uses the same way, so “customer_id” means the same thing everywhere instead of also surfacing as “acct_num” or “user_ref” elsewhere with slightly different rules.

Chen walks through building one for a small energy-billing system, and the pattern generalizes well. First, name your core entities — the “nouns,” like a raw usage reading, the pricing logic applied to it, and the resulting bill. Then define your invariants — facts that must always hold regardless of what generated them: every timestamp maps to exactly one pricing window with no gaps, a bill’s total equals the sum of its parts, and the same inputs always produce the same output.

Those invariants matter most once an AI agent enters the picture. You can’t test every way a language model might go wrong, but you can constrain what it’s allowed to produce. Chen’s systems force any AI-generated proposal into a structured schema — an “action plan” — that gets simulated against the deterministic rules before anything executes. If the simulated outcome doesn’t hold up arithmetically, the proposal is rejected before a person ever sees it.

He calls this an “AI sandwich”: deterministic, rule-based logic on either side of the nondeterministic AI in the middle, so creative-but-unreliable output is always checked against boring-but-trustworthy math before reaching a user. The rate at which proposals get rejected becomes a first-class reliability signal — and rejected cases feed back into the test suite, so production failures become new test fixtures rather than one-off incidents.

The failure is invisible to your standard observability tools because it isn’t a failure of infrastructure; it’s a failure of semantics. — Frank Chen, Observability Engineering

Key Takeaways

• Sampling should preserve real, full-fidelity events rather than collapsing them into aggregates — the sample rate travels with each event so the original distribution can be reconstructed later.
• Fixed-rate sampling is simple but blind to rare, important events; dynamic and per-key sampling adapt the rate to traffic and content, at the cost of more bookkeeping.
• For distributed traces, head-based sampling decides upfront and keeps a trace consistent; tail-based sampling waits for the outcome and catches anomalies head-based sampling would miss, but needs buffering.
• A telemetry pipeline (collect → normalize → enrich → reduce → route) decouples every data source from every destination, so adding a new tool becomes a config change instead of a redeploy.
• Reduction at the pipeline layer is often the biggest cost lever available — a meaningful share of collected data is frequently never queried at all.
• Ontologies and semantic conventions keep terms like “uptime” or “customer” consistent across teams, and give you a way to mathematically constrain what an AI agent is allowed to do before its output reaches a user.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Mike Kelly, Founder and CEO at Bindplane, and the Bindplane telemetry pipeline project.
• The OpenTelemetry (OTel) project, including its Collector and semantic conventions work.
• Frank Chen, CTO at Balto Energy and Designer-in-Residence at the UC San Diego Design Lab.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 5 of a 10-part series on Observability Engineering. Continue to Part 6: Observability Across CI/CD, Mobile, and Performance.