Observability Engineering — Part 9: Making the Business Case and Driving Change

Every chapter so far in this series has made the technical case for observability. This one is about money, politics, and change management instead — because none of the technical practices matter if nobody signs off on the budget, or the org quietly reverts to old habits six months after the rollout party.

Chapters 26 through 28 tackle three practical problems in sequence: how to justify spending on observability, how to tell whether the money you’re already spending is doing anything, and how to push a real observability practice through an organization that’s structurally built to resist it.

Start With the Bottleneck, Not the Vendor

The book’s first piece of advice for building a business case is almost anti-climactic: don’t start by comparing vendor feature sheets. Start by figuring out which of two feedback loops is actually hurting you right now.

The operational loop is your safety net. It’s triggered by alerts, pages, and angry customers, and its job is to answer three questions fast: what’s broken, who’s affected, and what gets service restored soonest. It’s reactive by nature — by the time it fires, something has already gone wrong.

The developer learning loop sits upstream of that. It’s how engineers find out whether the thing they shipped an hour ago is actually doing what they intended, before a customer ever notices. It’s not just about catching bugs — it’s about understanding how code, infrastructure, and real users interact in production, which is messier and more interesting than any staging environment.

Most organizations need both, but they’re not interchangeable, and conflating them is a common mistake. Treating your whole telemetry budget as one undifferentiated pile of “observability spend” means you can’t tell which loop is actually getting stronger.

There’s a great line from a Honeycomb SRE, Fred Hebert, that captures the trap teams fall into when they over-index on the operational side:

“It’s great that you’re so good at firefighting, but maybe it’s time to read the fire codes?”

Being excellent at incident response is genuinely valuable. But if your org’s whole identity is built around fighting fires well, you’ve quietly accepted that improvement only happens after something breaks. The bigger lever is upstream: building the kind of feedback that stops fires from starting, not just crews that respond faster once they do.

Building the Case for Each Loop

The case for operational loops is the easier sell because it’s the traditional monitoring pitch: better uptime, faster incident response, fewer avoidable outages, and less burnout-driven attrition on your on-call rotation. If customers are finding your bugs before you do, or minor incidents are snowballing into major ones, this is where to point your investment first.

The case for developer learning loops is where the real leverage lives, and it’s a harder sell because the payoff is less visible on a slide. It has two halves. Externally, it’s about product quality — understanding not just whether something is “up,” but what users are actually trying to do and where your assumptions are wrong. The book cites some blunt numbers here: Amazon reportedly loses roughly 1% of sales per 100 milliseconds of added latency, and both Walmart and Staples saw conversion gains from shaving seconds off load time. Small performance differences move real revenue.

Internally, the case is about velocity as a competitive advantage. A recurring theme is that much of the existing tooling budget — CI/CD, feature flags, chaos engineering, progressive delivery — is already an indirect attempt to buy developer learning, and most of it underperforms because it doesn’t close the loop. Chaos engineering without clear attribution is just disruption. Feature flags without validation just add complexity. Observability is what turns those practices from activity into actual learning, by attributing cause to effect.

Worth remembering too: some of the best product pivots come from watching what users unexpectedly do with your software. The book points to Instagram starting as a check-in app before its founders noticed people cared about the photo feature, and to YouTube, Twitter, and Slack all pivoting after watching real usage. You only catch that signal if you’re able to look.

Cost Center or Strategic Investment?

Here’s the framing question that decides whether your budget survives the next planning cycle: is observability a cost center to be minimized, or an investment expected to compound?

The book’s answer is refreshingly not black-and-white: it’s both, depending on what you’re observing. Telemetry on raw infrastructure is a cost center, because infrastructure itself is a cost center — you don’t make more money running extra idle capacity or piling on unnecessary checks. But telemetry on revenue-generating product code, and on the internal tooling that determines how fast engineers ship, behaves like an investment, because better visibility there can directly translate into more revenue or more developer throughput.

The practical consequence is structural. If observability reports up through IT or ops and is managed to a strict cost-center budget, it gets optimized for minimal spend — quietly starving the parts meant to accelerate learning. The book argues observability should instead roll up through engineering leadership, with the team run more like an internal platform group serving customers than an ops function counting line items.

Diagnosing Whether the Investment Is Actually Working

Chapter 27 tackles a question a lot of engineering leaders are hearing right now: finance wants to know why the observability line item keeps climbing. Per Gartner, observability spend has grown roughly 40% a year for over a decade and is now the second-largest R&D software cost after cloud itself. That pressure is actually useful — it forces a harder question than “can we justify this bill”: is it buying the right capability?

The diagnosis starts with a distinction between activities and learning. Activities are what your teams do day to day; learning is the actual change in understanding that results. A lot of “productivity” investment quietly turns into activity without learning: chaos experiments that can’t say exactly what broke, feature flags nobody validates, progressive rollouts where regressions can’t be traced to a cohort. Observability is what closes that loop.

So how do you know if you’re getting real returns? For operational loops: time to identify known causes going down, alert quality improving, escalation rates dropping, recovery getting faster. For developer learning loops: engineers answering questions nobody anticipated, deploy frequency rising while batch sizes shrink, more than two or three specialists able to debug independently, and more issues caught internally instead of reported by customers.

Why Organizations Resist the Shift

Chapter 28 (contributed by Rick Clark) opens with a familiar scenario: you bought the tools, renamed the team, checked every box on the “three pillars” list — and war rooms still happen, correlation is still manual, and the same handful of engineers still get paged for everything. The label changed. The capability didn’t.

That gap triggers real resistance, worth understanding rather than resenting. Three groups tend to push back, each for rational reasons. Legacy vendors resist because their contracts renew automatically and sales teams are paid to sell more seats, not to make your debugging faster — expect upsells framed as fixes. Existing monitoring teams resist because professional identity is often tied to a specific tool; walking away from years of accumulated expertise can feel like starting over, and their skepticism is often earned from watching prior “transformation” efforts quietly die. Leadership resists because they remember signing the check for the current stack, and replacing it can feel like admitting the money was wasted.

None of this is solved by arguing — it’s solved with evidence: run honest diagnostics, collect real stories (the four-hour incident only two people could debug, the bug a customer caught first), and build a coalition around them — ideally including the people who kept the legacy systems running, since sidelining them turns institutional knowledge into an obstacle instead of an asset.

A sponsor saying supportive things in a meeting isn’t enough, either — that’s sponsorship without authority, producing polite nodding and no real change. What you need is a mandate: someone with the standing to make the new approach the default for new work, on a published timeline.

The Roadmap: Small, Paved, Proven, Expanded

Once you have a mandate, the rollout follows a fairly disciplined shape. Pick one team and one domain — not the most critical system (too much pressure), not the most neglected one (too much unrelated debt) — and instrument it completely rather than everywhere shallowly. Then make the new approach genuinely easier than the old one: good libraries, sane defaults, low friction, because adoption that requires heroics won’t happen. After some months, rerun your diagnostics to prove the gap closed, and use those results — not tool features — as your evangelism. Only then expand domain by domain, treating the effort as an ongoing capability rather than a project with an end date. The book also stresses building a dedicated team that owns the platform and developer experience rather than instrumenting every service itself — that doesn’t scale and just creates a new bottleneck.

There’s also an urgency argument the book raises that’s easy to overlook: AI-assisted development is going to generate code faster than teams can manually instrument or reason about it. Organizations that can’t ask novel questions about their own systems won’t get more productive from AI — they’ll just ship faster into the dark.

Key Takeaways

• Build your business case around whichever feedback loop is your real bottleneck — operational stability or developer learning speed — rather than around a vendor’s feature list.
• The operational loop protects you after something breaks; the developer learning loop catches problems before customers ever see them. Most orgs need both, funded and measured differently.
• Observability is a cost center where it touches raw infrastructure, but an investment where it touches revenue-generating code and internal developer velocity — treat it accordingly in reporting lines and budget.
• Before cutting spend, diagnose it: are you paying for observability capability but only getting monitoring outcomes? Track whether time-to-cause, alert quality, and independent debugging are actually improving.
• Organizational resistance from vendors, existing teams, and leadership is rational, not spiteful — counter it with evidence, a coalition that includes the people who kept the lights on, and a genuine executive mandate rather than polite sponsorship.
• Roll out change like a product launch: one team instrumented deeply, a paved path that makes the new way the easy way, proof before expansion, and patience for a multiyear capability rather than a one-time project.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Gartner — cited in the book for the statistic that observability costs have risen roughly 40% year over year for over a decade.
• Fred Hebert — SRE quoted on “reading the fire codes” instead of only celebrating firefighting skill.
• Charity Majors — quoted for her definition of infrastructure as “the code you have to run, in order to run the code you want to run.”
• Rick Clark, Global Head of Cloud Advisory at UST — contributed Chapter 28 on organizational change.
• Simon Wardley — his pioneers/settlers/town planners model, referenced for matching engineers to roles during the transition.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 9 of a 10-part series on Observability Engineering. Continue to Part 10: Build vs. Buy, Vendor Partnerships, and What’s Next.