Practical OpenTelemetry — Part 7: Metrics and Cardinality

Metrics are the oldest telemetry signal and the one most likely to already exist in your stack — which is exactly why OpenTelemetry’s metrics design has to solve a harder problem than tracing’s: compatibility with the tools people already run, without inheriting their flaws.

Chapter 7 of Practical OpenTelemetry covers the Metrics API and SDK in full: six instrument types, aggregation, Views, exemplars, and the temporality question that confuses everyone who moves between Prometheus and OTLP.

Measurements, Metrics, Time Series

Three precise definitions set up the chapter:

• A measurement is a single observation — a value plus attributes.
• A time series is a succession of data points over time, identified by a unique combination of name + attributes.
• A metric is a group of time series with common properties.

The subtle part: a time series alone carries no semantics. A rising series could be a counter or a gauge — the instrument type is what gives it meaning. OTel fixes a historical coupling problem here: pre-OTel clients baked both the transport and the aggregation function into the code. StatsD metrics needed an adapter to reach Prometheus; a histogram couldn’t be reconfigured by the app owner. OTel separates measuring (the API) from aggregating (the SDK) — and lets operators override aggregation with Views at configuration time.

Cardinality: The Metric Killer

Before the instrument types, the book’s most important warning: cardinality — the number of unique attribute combinations — is the limiting factor and cost driver for every metrics backend.

The classic mistake: add the raw http.url as an attribute on a metric. Every unique URL is a new time series. Every new time series costs memory in Prometheus, rows in your TSDB, and query latency everywhere. The book’s rule of thumb: if an attribute combination isn’t plottable or alertable, it’s too granular — that’s what tracing is for.

Metrics answer “how much”; traces answer “which one.” Conflating them is how teams end up with million-series Prometheis that answer neither well.

Getting a Meter, and the Duplicate-Registration Trap

Before any instrument, a Meter:

Meter meter = openTelemetry.meterBuilder("my-meter")
    .setInstrumentationVersion("0.1.0")
    .build();

Instrument identity is Name (max 63 case-insensitive ASCII chars) + Kind + Unit (max 63 chars, case-sensitive — kb ≠ kB) + Description. Register two distinct instruments under the same name and instrumentation scope, and the SDK logs a “duplicate instrument registration” warning — a working instrument is still returned, but the book is explicit that this can produce semantic errors, and in Java the two are treated as genuinely separate counters, both exported under the same name.

The one case that isn’t a conflict, and is worth knowing precisely because it looks like one: two meters with different instrumentation scope (here, differing only by version) never collide, even registering the identical instrument name:

Instrument={InstrumentationScope={name=my-meter, version=1.0.0}, name=my-counter, description=random counter, type=LONG_SUM}
Instrument={InstrumentationScope={name=my-meter, version=2.0.0}, name=my-counter, description=random counter, type=LONG_SUM}

Different exporters represent that split differently — OTLP as separate metric streams, Prometheus as separate label sets on the same metric name:

my_counter_total{otel_scope_name="my-meter", otel_scope_version="1.0.0"} 1707.0 1665253818117
my_counter_total{otel_scope_name="my-meter", otel_scope_version="2.0.0"} 345.0 1665253818117

Six Instruments, One Decision Tree

The Metrics API exposes six instrument types, and the book’s guidance for choosing between them is cleaner than most:

The sync/async split answers one question: does the application report values in its own code (sync), or does the SDK call a callback at collection time (async)? Async instruments exist because some values — JVM GC stats, for instance — are already being tracked somewhere, and wrapping them in a callback is cheaper and less error-prone than instrumenting the thing that computes them.

The monotonic split answers another: can this value only go up? Monotonic counters answer “how much in total”; non-monotonic ones answer “what is the current state.” One gotcha the book flags: with delta temporality, even a monotonic counter’s exported values can decrease, because each export only covers the interval since the last one.

Two practical rules for async callbacks: report the total, not the delta (the SDK computes deltas itself), and never record the same measurement twice per callback.

Real code for the three instruments you’ll reach for most. A Counter, incremented in application logic:

LongCounter counter = meter
    .counterBuilder("tickets.sold")
    .setDescription("number of tickets sold")
    .build();

counter.add(12);
counter.add(8, Attributes.of(stringKey("myKey"), "myValue"));

Negative increments log a warning and get dropped — a Counter can only go up. An ObservableCounter, reporting a pre-computed total via callback — note the callback reports the total, never the delta, exactly as the rule above states:

ObservableLongCounter counter = meter
    .counterBuilder("mycounter")
    .buildWithCallback(measurement -> {
      measurement.record(2, Attributes.of(AttributeKey.stringKey("myKey"), "foo"));
      measurement.record(5, Attributes.of(AttributeKey.stringKey("myKey"), "bar"));
    });

The returned object is auto-closable — close() de-registers the callback. And a Histogram, Double-typed by default in Java because of its statistical use case:

DoubleHistogram histogram = meter
    .histogramBuilder("http.client.duration")
    .setUnit("milliseconds")
    .build();

histogram.record(121.3);

Histogram values must be zero or greater — the same monotonicity assumption the aggregation strategy below depends on.

Aggregation: What the SDK Computes In Memory

Between measurement and export, the SDK aggregates. The book covers five strategies:

• Sum — the default for counters: total of all measurements in the interval, per attribute set.
• Last value — the default for gauges: the most recent value per interval.
• Explicit bucket histogram — pre-defined buckets ([0, 5, 10, 25, 50, 75, 100, 250, 500, 1000, 2500, 5000, 7500, 10000] by default), each a monotonic counter. Percentiles come from interpolation. Bucket choice is the accuracy-vs-cost tradeoff.
• Exponential histogram (OTEP-149) — buckets on an exponential scale with a resolution parameter; more accurate with fewer buckets, and solves merging of overlapping buckets. Not required for spec compliance.
• Drop — discard matching measurements entirely (configured via Views).

Views: The Operator’s Escape Hatch

Views are where OTel’s “defer decisions to configuration time” philosophy lands for metrics. A View selects instruments (by name, kind, or meter, with wildcards) and overrides:

• Aggregation — turn a histogram into a sum, or drop it entirely
• Name/description — rename a metric to fit your org’s conventions
• Attribute keys — aggregate only across listed keys, ignoring the rest

The primary real-world use: reducing the cardinality of auto-instrumented metrics. Auto-instrumentation ships rich attributes by default; Views let you prune the expensive ones without touching the library.

Exemplars: Metrics With Trace Links

Exemplars are the bridge between Part 6 and Part 7: high-granularity samples attached to metric data points, carrying the trace and span IDs active when the measurement was recorded. The payoff: from a slow histogram bucket on your dashboard, jump directly to the individual trace that populated it.

Two hooks control them: a Filter (always sample / never sample / sample only if the active trace is sampled) and a Reservoir (bounded storage per period). Sync instruments get exemplars; async instruments can’t — no context at collection time.

Aggregation Temporality: Cumulative vs. Delta

The final concept explains a lot of cross-backend confusion. Temporality is the time period a data point covers:

• Cumulative: the sum of all measurements since the instrument was created. Prometheus’s model. Pros: scrape gaps get fixed at query time (you only need the first and last points). Cons: producer keeps memory per attribute combination.
• Delta: the sum for a single collection interval. Pros: less producer memory, no counter resets to reason about. Cons: requires start timestamps, needs queuing to survive interruptions.

OTLP supports both, and temporality is a preference, not a guarantee — because non-monotonic instruments (gauges, UpDownCounters) must always use cumulative. Delta only applies to counters and histograms. Collectors can transform between the two (with min/max loss going cumulative→delta).

The practical consequence: a team moving from Prometheus to an OTLP-native backend keeps cumulative temporality and nothing changes; a team going the other way flips the exporter preference and lets the Collector bridge.

Next: Practical OpenTelemetry — Part 8: Logging: Enrich, Don’t Replace

References

• Practical OpenTelemetry (Apress, 2023) — Daniel Gomez Blanco
• OpenTelemetry Metrics Specification
• Exponential Histogram OTEP