Observability Engineering — Part 6: Observability Across CI/CD, Mobile, and Performance

Most of this book has been about production services — the APIs, the databases, the things customers touch directly. This part takes the same toolkit and points it somewhere less obvious: the pipeline that builds and ships your code, the app running on a stranger’s five-year-old phone, and the spreadsheet-shaped question of why your cloud bill keeps climbing.

None of these feel like “observability problems” at first glance. A slow CI pipeline feels like a tooling complaint. A crashy mobile app feels like a QA problem. A high AWS bill feels like a finance problem. But all three are really the same kind of problem: a complex system is behaving in ways nobody fully understands, and the fix is to start asking it questions with real data instead of guessing.

Your Build Pipeline Is a Production System Too

Here’s a reframing worth sitting with: every time a developer opens a pull request, they become a customer of your CI/CD system, and they’re waiting on a response. That response just happens to be “your tests passed” instead of “here’s your search results.” Treat it as a toy internal tool that nobody really owns, and it’ll behave like one — flaky, slow, and mysterious. Treat it like the production system it actually is, and you can debug it the same way.

This chapter, co-written by Hugo Santos (CEO of Namespace Labs) and Liz Fong-Jones, makes a case that build pipeline observability might have the best return on investment of anything in this book. A production API request might be worth a fraction of a cent. A slow or broken CI run wastes real developer time — and developer salaries are not fractions of a cent. A developer sitting idle waiting for a build, or context-switching to Slack because the wait is too long, is an expensive kind of waste that’s easy to overlook because it doesn’t show up on a dashboard anywhere.

The trick to applying observability here is recognizing that CI/CD systems already speak the same language as distributed tracing, if you squint. A full pipeline run is a trace. Each job inside it is a span. Each individual step inside a job is a span nested inside that span. You don’t need a new mental model — you need to point the one you already have at a new target.

Once jobs branch into parallel work — which happens naturally as any pipeline grows — you have to stop thinking about individual job speed and start thinking about the critical path: the longest chain of dependent jobs that actually determines how long a developer waits. If your Go build gets 20% slower but your Rust build already dominates the runtime, fixing the Go build changes nothing end-to-end. You have to trace the whole graph to know where your time actually goes, exactly the way you’d trace a slow API request through a dozen microservices.

Honeycomb’s own build times crept from 7 minutes to nearly 14 minutes between 2018 and 2019, and they only caught it because they had SLOs — service level objectives, meaning explicit thresholds like “95% of builds finish under 15 minutes” — tracked over time. Without that, slow creep is invisible; everyone just quietly gets used to it, the classic boiled-frog problem. Years later, digging back into the data with proper tracing, they found the real bottleneck wasn’t compilation at all — it was 3.5 gigabytes of intermediate files being shuffled between build steps, compressed with the wrong algorithm. The fix involved restructuring the build around Docker Bake (a way of chaining Docker image builds so independent steps run in parallel and only block when they truly need each other’s output) and consolidating jobs to avoid that data-shuffling tax. Build times came down from 14 minutes to under 7. They even stitched together their homegrown tracing with Docker BuildKit’s built-in OpenTelemetry support, so a single trace could show exactly which command, which upload, and which wait was eating the clock.

CI/CD pipelines are, at their core, production systems for developer feedback. — Hugo Santos and Liz Fong-Jones, Observability Engineering

The lesson generalizes past Honeycomb’s specific tools: instrument the start and end of every job and step, treat the resulting data as traces, define SLOs on build time the same way you’d define them for an API, and always ask “is this actually on the critical path?” before spending an afternoon optimizing something that won’t move the total.

Mobile and Frontend: Observability in Hostile Territory

If backend observability happens on turf you control — your servers, your network, your OS versions — mobile and frontend observability happens on turf you emphatically do not control. This chapter is a guest contribution from Hanson Ho, an Android architect, with an added section from Matt Klein (CTO at bitdrift), and it opens with a blunt admission: most mobile and web teams don’t really practice observability at all. What they have is crash reporting and basic session replay, which answers “did it crash” but almost nothing else.

The core problem is a kind of multiplication. On the backend, your fleet of servers is relatively uniform — same OS, same hardware generation, same network. On mobile, every user brings their own combination of device model, OS version, network quality, battery state, and background app noise, and none of those combinations are under your control. Multiply them together and you get a cardinality explosion — an enormous, ever-growing number of distinct operating conditions, most of which no single engineer will ever personally witness.

That multiplication has a nasty side effect: aggregate metrics lose their meaning. If 1% of your users are hitting something badly broken, that might not move your P50 or even your P99 latency at all — but if you have 100 million daily users, that “insignificant” 1% is a million people having a genuinely bad time. And because manually instrumenting a mobile workflow is hard — a single user journey might cross several screens, components, and threads with no shared execution context tying them together — huge swaths of app behavior simply never get recorded as telemetry in the first place. Teams end up boxed in by what’s easy to capture (crashes, startup time) rather than what actually matters to users.

Matt Klein’s section tackles a more mechanical version of the same problem: even when you do capture good telemetry, mobile devices can’t just stream it all to your servers in real time. Bandwidth is limited, connections are unreliable, and users notice (and resent) apps that burn through their data plan. His proposed fix is to stop treating the network as a live pipe and instead treat the device as a local recorder with a remote conductor. Telemetry gets written to a small on-device circular buffer — meaning old entries get overwritten once it fills up, so it never grows unbounded — and a server-side control plane sends the device precise rules about what to actually upload and when. A rule might read something like: watch for the user reaching the checkout screen, and if fifteen seconds pass without a completed purchase, send the full recorded session. The device logs generously by default; the network only carries what you’ve decided is worth paying for.

The other major idea in this chapter is a shift Hanson Ho calls user-focused observability. Instead of the usual approach — notice a technical anomaly, then try to figure out if it matters to anyone — you flip the order. Start by watching a metric that’s a direct proxy for whether users are getting what they came for, like the percentage of checkout attempts that actually complete. When that number moves, you know for certain that something real happened to real users, and only then do you dig into the technical telemetry to explain why. The book walks through a food delivery app example where this approach surfaces a “Submit” button that silently stopped working for a subset of users on a particular screen size — the kind of bug that would never trip an alert on server-side dashboards, because nothing on the server was actually broken.

Performance Engineering: Turning Guesswork Into Measurement

The last chapter in this stretch of the book pairs observability with a discipline that’s usually treated as its own separate specialty: performance engineering, the deliberate practice of making systems faster and cheaper using evidence rather than intuition.

It opens with a genuinely fun detective story. In late 2021, Honeycomb noticed something odd: their load balancer logs showed requests taking single-digit milliseconds, while their own application traces claimed those same requests finished in a fraction of a millisecond. Time was disappearing somewhere their tracing simply couldn’t see. Distributed tracing, for all its power, only shows you what you instrumented — it can’t see inside a black box. So the team reached for a different tool: CPU profiling, which works by periodically sampling exactly which line of code the processor is executing, building up a statistical picture over time of where cycles actually go.

The profile revealed that 17% of all CPU time was being spent recompiling a regular expression router on every single request, because the router object was being thrown away and rebuilt from scratch each time instead of being created once at startup. The fix was five lines of code. The larger point is that tracing and profiling are complementary lenses, not competing ones: tracing shows you which request is slow and roughly where, at a granularity of tens of milliseconds; profiling shows you which specific line of code is burning the cycles, at a granularity too fine for tracing to reasonably capture without slowing everything down itself.

The rest of the chapter is a practical tour of using this evidence-based mindset for cost, not just latency. Before optimizing anything, the book insists on gathering a baseline — a documented measurement of current performance and cost — so that any change can be judged against a real “before” number instead of a vibe. From there it walks through several angles: choosing the right compute purchasing model (owning hardware outright, reserved capacity, on-demand virtual machines, spot/interruptible instances, or serverless functions, each trading cost against control and reliability); squeezing an existing fleet by right-sizing CPU and memory allocations and testing on newer, more efficient chip generations; and Kubernetes-specific tricks like reading cluster load through heatmaps rather than percentiles, since a single percentile number can hide wildly uneven saturation across nodes.

The chapter’s closing example ties tracing and profiling together in one investigation: Honeycomb found a single customer’s unusually complex query was being serialized from one data format to another hundreds of thousands of times inside a hot code path, discovered only because profiling data could be indexed down to the level of an individual customer’s traffic. Tracing alone would never have caught it — no one traces every function call inside a tight loop — but profiling combined with the context tracing provides made the fix obvious: cache the serialization, and simplify the customer’s deeply nested query logic. The chapter’s real message is that performance work isn’t a one-time cleanup project; it’s a habit of continuously measuring, improving, and re-measuring, with small percentage gains compounding the same way technical debt compounds in the other direction.

Key Takeaways

• Your CI/CD pipeline is a production system for developer feedback — instrument it as traces (workflow → job → step), set SLOs on build time, and optimize the critical path rather than whichever job happens to look slow in isolation.
• Build observability tends to have outsized ROI because every wasted minute is expensive developer time, not a fraction-of-a-cent request.
• Mobile and frontend telemetry faces a genuine cardinality explosion — device, OS, network, and behavior combine into conditions no single engineer will ever personally reproduce, which makes aggregate percentiles misleading for any individual user.
• On-device local storage paired with a remote control plane (log generously, upload selectively) solves mobile’s bandwidth and reliability constraints without sacrificing depth when you actually need it.
• User-focused observability flips the usual order: detect real user impact first (like a drop in checkout completion), then dig into the technical telemetry to explain it — rather than chasing every technical anomaly hoping it matters.
• Tracing and profiling are complementary, not competing: tracing finds which request is slow, profiling finds which exact line of code is responsible, and combining them turns performance and cost optimization into measurement instead of guesswork.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Hugo Santos, CEO, Namespace Labs — co-author, Chapter 18 (Observability for CI/CD Pipelines)
• Liz Fong-Jones, Technical Fellow, Honeycomb — co-author, Chapter 18, and the Honeycomb build-time case study
• Hanson Ho, Android Architect, Embrace — guest author, Chapter 19 (Observability for Mobile and Frontend)
• Matt Klein, CTO, bitdrift — contributing author, Chapter 19, on local storage and real-time telemetry control
• Gregg, Brendan. BPF Performance Tools. Addison-Wesley, 2019.
• Gregg, Brendan. Systems Performance. Pearson, 2020.
• Ren, Gang, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus, and Robert Hundt. “Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers.” IEEE Micro 30, no. 4 (2010): 65–79.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 6 of a 10-part series on Observability Engineering. Continue to Part 7: LLMs, AI Agents, and the Fin Story.