Observability Engineering — Part 10: Build vs. Buy, Vendor Partnerships, and What's Next

We’ve spent nine posts on the mechanics of observability — what it means, how to instrument for it, how to make the case for it inside your org. This last post covers the two chapters closing out the book’s practical section, plus its final chapter: whether to build your own observability stack, buy one, or lean on open source, and how to actually work with a vendor once you’ve picked one. Then we’ll close the series out.

Observability Tooling Doesn’t Play by the Normal Rules

Most build-vs-buy decisions are just cost comparisons. Observability complicates that, because it’s the layer that watches everything else. If your observability stack runs on the same cloud, region, and infrastructure as the systems it’s watching, it will go dark exactly when you need it most — during the outage that’s also taking down your own infrastructure. Part of what you’re really paying a vendor for isn’t the dashboard, it’s a stack that fails independently of yours.

That doesn’t mean you have to buy. If your business genuinely tolerates lower availability — the high 90s is fine, and you mean it — running your own stack is a reasonable bet. The point is just that this decision carries different weight than a typical internal-tool call.

A Simple Map for Where Your Engineering Time Should Go

The book (crediting a framework originally developed by engineering leader Rick Clark) offers a clean way to sort any piece of your stack into one of four buckets, using two questions: does this create real business value — more revenue, more customers, a genuine edge over competitors — and is it a commodity problem every company already has, or something unique to you?

Plot those on a grid and four answers fall out. Genuinely unique and revenue-moving is worth building — that’s your edge. Common to every company and non-differentiating is a straightforward buy. Unique to you but not actually helping the business is the worst kind of technical debt — nobody else has this problem because it isn’t a real problem, and you should be looking for the exit. The fourth bucket — commodity-ish, but with real value at stake — is where the genuinely hard conversations live, and where domain experts, not managers, should be weighing in.

For most companies, plain observability infrastructure — storage engines, query backends, alerting pipelines — lands squarely in “buy it.” The exceptions are companies at genuine hyperscale, companies for whom observability tooling is the core product, and companies with regulatory constraints that rule out third-party vendors entirely. Everyone else is probably overestimating how special their situation is.

Open Source Isn’t Free, It’s Deferred

Open source doesn’t fit neatly into “build” or “buy” — you’re not writing it from scratch, but no one is running it for you either. It’s a third option that moves where the cost shows up: instead of a monthly invoice, you pay in engineering hours to evaluate, deploy, patch, and babysit it, plus the salary and recruiting cost of the people doing that work, plus what else those people could have been building instead.

OSS is “free as in puppies, not free as in beer.”

That’s developer advocate Heidi Waterhouse’s line, and it’s the whole argument in one sentence: a puppy is free to acquire and expensive to keep alive. The book backs it with a real case — an engineering manager running a self-hosted ELK stack (Elasticsearch, Logstash, Kibana) balked at an ~$80,000/month vendor quote, pointing proudly at his “free” setup. Add the dedicated hardware, three engineers hired specifically to keep the cluster alive, their recruiting costs, and ramp-up time, and that “free” stack was quietly costing over $2 million a year — more than double the quote he’d rejected. None of that spend showed up on a single line item, which is exactly why it went unnoticed.

Open source is often still the right choice. The honest way to evaluate it is to add up fully-loaded engineer time, infrastructure, the ongoing patching treadmill, and the opportunity cost of what those engineers aren’t building instead — then compare that number to the vendor’s invoice, not the license’s sticker price.

What Each Path Actually Costs You

Building your own gives you deep in-house expertise, integrations tailored to your systems, and — if it goes well — real savings or even something you can sell. But it requires product management discipline most infra teams don’t have (someone has to interview users, prioritize, and iterate, or the tool dies unused), and it needs a permanent owner, not just whoever built version one before switching teams.

Buying gets you a working product in hours instead of months, a more polished experience than most homegrown tools reach, and access to a vendor whose entire job is this one problem. The risk is the flip side: your org may never build its own deep understanding of the problem, you’re waiting on the vendor’s roadmap for features that matter only to you, and per-host or per-query pricing can turn painful once usage — and curiosity — grows, which is exactly what you want as your observability culture matures.

Buy the Foundation, Build the Last Mile

The honest answer for most organizations isn’t “build” or “buy” — it’s both, layered. Pay a vendor to handle what’s genuinely commodity: storage, query performance, uptime decoupled from your own infrastructure, the core feature roadmap. Then have your own team build the thin layer on top that makes it feel native — naming conventions, internal libraries wrapping the vendor’s SDK, dashboards and onboarding tuned to how your teams work.

The technical hedge that makes this work is standardizing on OpenTelemetry (OTel), the open standard for generating and shipping telemetry, rather than a vendor’s proprietary agent. Some sales teams insist their own agent is more mature than the OTel equivalent — and sometimes that’s true, because they’ve chosen not to invest equally in both. Either way, instrumenting against an open standard means switching vendors later is just replacing a config file, not re-instrumenting your whole codebase.

Running a Vendor Trial That Actually Proves Something

Once you’ve decided to buy, the proof-of-concept (POC) stage is where most evaluations quietly go wrong. A loosely scoped POC drags on for a year and proves nothing; a demo run entirely on the vendor’s handpicked sample data proves even less. A trial worth running is a stress test of your actual hardest problems, on real production traffic, with success criteria written down and agreed to before anyone starts.

A few details matter more than they seem. Let each vendor instrument your workload the way they’d recommend to a real customer, rather than forcing identical instrumentation on every contender — otherwise you’re comparing your own mediocre setup work, not the products. Get pricing against your real, growing usage, not a demo account, and ask what similar customers actually pay — a vendor’s list price is close to meaningless. And signing isn’t the finish line: successful migrations tend to move in three stages (prove it on one real case, do the messy bulk migration, then clean up service by service), and you’re not saving money until the old tool is actually turned off — the last 10% of any decommission is always the hardest, and it’s tempting to leave the old system limping along “just in case.”

Trust, Credibility, and the Long Game

The chapter opens with a story from co-author Charity Majors about Mark Callaghan, a Facebook engineer who sat quietly through a tense meeting between her team (then at Parse, running a crash-prone open-source MongoDB deployment) and MongoDB’s executives. Rather than venting like everyone else in the room, Callaghan asked a few sharp questions about MongoDB’s storage engine roadmap — and over the following months quietly brokered the introductions and commitments that led MongoDB to build a pluggable storage engine, eventually adopting the WiredTiger engine that still underpins the product today. He did it as an individual contributor, with no formal authority over either company, purely on technical credibility built over time.

That’s the model the book holds up for what it calls vendor engineering: influence without authority, exercised across company lines. Internally, it means mapping every stakeholder in a purchase — engineering (win them over by fixing something that actually hurts them), management (keep them updated so they can advocate upward), finance (translate your ask into revenue or cost impact), security (never accept a rumor of “they’ll say no” — go find out directly), and procurement (they’re not the enemy). It also means knowing the difference between credibility (you say something will happen, and it does) and trust (you act in someone’s interest even when it costs you) — and never letting a stakeholder get blindsided by bad news you already knew about.

With vendors themselves, the dynamic flips: they want to sell you something, you want what’s genuinely best for your company. The healthiest partnerships still get there — through reciprocity, straight answers about even the vendor’s own weaknesses, and service-level agreements (SLAs) that formalize what everyone’s actually promising, rather than a handshake that dissolves once the people who made it move on.

Closing the Series: Observability in an Age of Cheap Code

Back in Part 1, the case for observability rested on a simple idea: it’s not a tool you buy, it’s a property your system either has or doesn’t — the capacity to have any of its internal states explained by someone on your team, using only what the system already emits. Everything since then — instrumentation, high-cardinality data, SLOs, incident response, the business case, and now this build-vs-buy calculus — has been in service of that one property.

The book’s final chapter argues that property is about to matter more, not less, because the economics of writing code have flipped. Code used to be expensive to produce, so we protected it, changed it cautiously, and trusted it mainly because it had survived in production a long time. When code becomes cheap and abundant to generate, that trust model breaks down. Engineer Chad Fowler frames this well with a thought experiment he calls the “deletion test”: imagine deleting all your source code — what’s actually lost isn’t the code, but the unwritten knowledge of which behaviors matter, which failures are unacceptable, and how to tell a correct rewrite from a broken one. Those were never coding problems; they’re observability problems, because the only place that knowledge has reliably lived is in what the running system does in production, under real load, with real users.

The authors are careful not to oversell AI as some civilization-bending force — they note that weavers, scribes, and sysadmins all lived through similar waves of automation anxiety before adapting. But they do think the old wall between “writing code” and “running it in production” no longer makes sense, since production is the only place left to actually confirm what a system does. That’s a fitting note to end on: the tools for building software keep changing, but the need to understand what your systems are actually doing, right now, in production, is exactly where this series started — and it isn’t going anywhere.

Key Takeaways

• Build vs. buy vs. open source is a spectrum, not a binary — map each piece of your stack by business value and how unique it really is, and expect most observability infrastructure to land in “just buy it.”
• Open source shifts cost, it doesn’t remove it. Count the fully-loaded engineer time, hardware, and opportunity cost before comparing it to a vendor’s invoice.
• The strongest pattern for most teams is layered: buy the commodity foundation, build a thin, friendly integration layer on top, and instrument against OpenTelemetry so you’re never locked into one vendor’s agent.
• A proof of concept only proves something if it runs real production workloads against success criteria you set before the trial starts — not after the results come in.
• Vendor relationships work best as genuine partnerships built on reciprocity and honesty, not one-sided transactions — and internally, trust and credibility have to be earned separately with every stakeholder group.
• As AI makes writing code cheap, the knowledge that used to live implicitly in careful, hard-won code has to live somewhere else — and observability, watching what a system actually does in production, is where that knowledge moves.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Rick Clark — engineering leader (formerly Mastercard, VMware, Cisco, Pivotal Software), credited with the build-vs-buy framework discussed in this post.
• Heidi Waterhouse — developer advocate, source of the “free as in puppies, not free as in beer” line on open source.
• Peter Corless — credited with the expanded build/buy/open-source decision framework referenced in this post.
• Matthew Sanabria — Engineering, Oxide Computer Company, quoted on OpenTelemetry and cost modeling.
• Mark Callaghan — engineer (Facebook), central figure in the MongoDB/Parse vendor-engineering story.
• Chad Fowler — engineer and CTO, BlueYard, originator of the “Deletion Test” thought experiment.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 10, the final post in a 10-part series on Observability Engineering. If you’re just starting out, Part 1: What Observability Actually Means is the place to begin.