Practical OpenTelemetry — Part 9: The Collector and OTLP

If the API/SDK split (Part 3) is OpenTelemetry’s brain, the Collector is its central nervous system. Chapter 9 of Practical OpenTelemetry covers the two halves that make the project deployable at scale: the OTLP protocol that moves telemetry around, and the Collector that processes it in transit.

The book’s nickname for the Collector — “the Swiss Army knife of the observability engineer” — undersells it, honestly. It’s closer to universal plumbing: receive anything, transform it, ship it anywhere.

OTLP: A Protocol Built for Telemetry

OTLP (OpenTelemetry Protocol) is the vendor-agnostic format for encoding and transporting telemetry. The design goals tell you what the project cares about:

• High throughput in high-latency networks (cross-datacenter shipping)
• Batching, compression (Gzip), encryption (TLS)
• Reliability: acknowledgments, retryable error codes, backpressure signals
• Minimal serialization/GC overhead — the Collector does fast pass-through and enrichment, and parsing cost matters
• Layer-7 load balancing — rebalance traffic between batches to avoid hot collectors

Semantically it’s at-least-once delivery: if an acknowledgment is lost in a network interruption, OTLP prefers redelivery over silence — meaning unacknowledged requests may be retried, and you should design for possible duplication. Payloads are defined as Protobuf schemas.

Two transports exist for different environments:

OTLP/gRPC (the recommended default): native flow control and backpressure via gRPC; clients send sequentially (low-latency scenarios) or concurrently (high throughput). Server responses include a partial success state that reports rejected data points — e.g., “no timestamp” — surfaced as metrics and not retried. Retryable gRPC codes include UNAVAILABLE, RESOURCE_EXHAUSTED (with RetryInfo), DEADLINE_EXCEEDED; retries use exponential backoff.

OTLP/HTTP: for environments where gRPC isn’t feasible — browsers being the canonical case. HTTP/1.1 and HTTP/2, binary Protobuf or JSON encoding, fixed paths (/v1/traces, /v1/metrics, /v1/logs). Everything except HTTP 400 retries; backpressure arrives as a Retry-After header on 429/503.

Configuration is one env var: OTEL_EXPORTER_OTLP_PROTOCOL=grpc|http/protobuf|http/json, with signal-specific endpoints for split pipelines.

The Collector: One Config File, Three Component Types

The Collector’s architecture is the cleanest part of the chapter. A single YAML file defines pipelines built from exactly three component types:

• Receivers — pull (Prometheus scrape, filelog) or push (OTLP, Zipkin) telemetry in
• Processors — filter, aggregate, transform, enrich in transit
• Exporters — push or pull to backends

A pipeline wires them in order: receivers → processors → exporters, and one pipeline handles exactly one signal type (traces, metrics, or logs). Multiple pipelines can share receivers (fan-out) and the same component definition can be reused with different names (otlp and otlp/differentport). Here’s a real, complete config from the book — not a fragment, the whole file:

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 10

exporters:
  otlp:
    endpoint: otel-collector:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

extensions:
  health_check:
  zpages:

service:
  extensions: [health_check, zpages]
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp, prometheusremotewrite]

Every named block under service.pipelines is one pipeline; note the traces pipeline and metrics pipeline reuse the exact same otlp receiver and the exact same [memory_limiter, batch] processor chain, but export to different destinations. That reuse is why the naming scheme matters — you can register more than one instance of the same component type, each with its own config, and reference them by name. Two otlp receivers on different ports, one with TLS and one without:

receivers:
  otlp:
    protocols:
      grpc:
        tls:
          cert_file: server.crt
          key_file: server.key
  otlp/differentport:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4319

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp/differentport]
      processors: [memory_limiter, batch]
      exporters: [otlp]

The same name / name/qualifier pattern works identically for processors, extensions, and exporters — it’s the one mechanism behind every “run two of these with different settings” scenario in the Collector.

Two processor behaviors deserve memorizing, because they prevent the most common production incidents:

memory_limiter — the OOM guard. Configure a soft limit (limit_mb minus spike_limit_mb); at the soft limit the Collector returns errors to the previous component, which refuses data, which pushes backpressure all the way to clients. At the hard limit it additionally forces GC. This is the component standing between your Collector and the kernel’s OOM killer.

batch — groups telemetry into efficient payloads (send_batch_size, timeout). Placement matters: it must come after processors that need the originating request context (like k8sattributes, which uses the client IP to attach pod metadata) and after samplers, so batches are optimally sized.

The book also flags the exporter queue behavior bluntly: when an exporter’s retry queue fills, failed exports are dropped — permanent data loss. The queue depth is your telemetry’s high-water mark.

Core vs. Contrib

The Collector’s codebase lives in two repos, and the distinction matters for production confidence:

• Core (opentelemetry-collector): general-purpose processors, receivers/exporters for standard protocols. The default otlp receiver, memory_limiter, batch, logging/otlp exporters.
• Contrib (opentelemetry-collector-contrib): 77 receivers, 42 exporters, 21+ processors covering the long tail of open-source and vendor formats.

Crucially, status is per component and per signal — the OTLP receiver is stable for traces and metrics but beta for logs, and contrib components range from alpha to stable. The book’s advice: check component status before betting a production pipeline on it.

Deployment: Agent vs. Gateway

The Collector runs in two roles (the deployment topologies get their full treatment in Part 10):

• Agent mode — on the same host as the application (daemonset or sidecar), receiving locally and enriching with host-level context.
• Gateway mode — a standalone, horizontally scalable service providing a centralized view, central config, and authentication against backends.

Packaging is mature: Docker images, Helm charts (including the opentelemetry-operator, whose mutating admission webhook can auto-inject Collector sidecars into pods), and native packages for Linux, macOS, and Windows.

Self-Telemetry: The Collector Observing Itself

The last section is the one that saves you at 3 a.m. The Collector exports its own metrics, and the book names the ones that matter:

• otelcol_exporter_queue_size — near capacity means drops are imminent
• otelcol_exporter_enqueue_failed_* — queue full: irrecoverable drop happened
• otelcol_exporter_send_failed_* — retryable failures (not data loss unless the queue then fills)
• otelcol_receiver_refused_* — nonzero usually means memory_limiter is active and pushing backpressure

Reading these four numbers tells you the health of your entire telemetry pipeline before any user notices an incident. The Collector is the one component in the stack whose job is to watch everything else — watching it in turn closes the loop.

Next: Practical OpenTelemetry — Part 10: Sampling and Deployment Models — how to sample intelligently, and the four places a Collector can physically live.

References

• Practical OpenTelemetry (Apress, 2023) — Daniel Gomez Blanco
• OpenTelemetry Protocol Specification
• OpenTelemetry Collector Documentation
• OpenTelemetry Operator