Observability Engineering — Part 7: LLMs, AI Agents, and the Fin Story

Part V of Observability Engineering closes with a hard question: what happens to instrumentation, traces, SLOs, and iterative analysis when the thing you’re observing doesn’t behave the same way twice? That’s the situation with large language models. Chapter 21 (Phillip Carter, principal contributor) tackles observability for LLM-powered features head-on. Chapter 22 hands the mic to Kesha Mykhailov, a staff product engineer at Fin, the AI customer-service agent built by the company formerly known as Intercom, who describes how his team used these ideas to fix a real, expensive, customer-facing problem. Together the chapters make one point at two altitudes: the fundamentals of observability don’t change just because a model has replaced deterministic code.

Nondeterminism doesn’t break the playbook

Traditional software is annoying in predictable ways: give it the same input twice and, bugs aside, you get the same output twice. An LLM breaks that assumption by design — the same prompt sent twice can come back with two different answers, and settings like temperature only shape the odds without guaranteeing a result. The range of things users might type at a model is also essentially infinite, and some users are actively trying to break it for fun.

Old proxies for user experience stop working cleanly, too. In ordinary web systems, latency is a solid stand-in for happiness — famous experiments at Amazon and Google found small delays cost real revenue and traffic. With an LLM, a snappy but wrong response doesn’t make anyone happy, and a slow response that nails the answer might be perfectly fine. Speed alone no longer tells you whether the product is working.

None of this breaks observability as a discipline — it just makes the opacity, unreliability, and cost-sensitivity distributed systems have always had more visible and more extreme. The fix is the one this book keeps making: instrument properly, and treat real production data as the source of truth instead of guessing.

Evals: a test suite that’s allowed to say “maybe”

If telemetry tells you what happened, you still need something to say whether it was good. That’s the job of evaluations, or evals. An eval looks like a unit test — give it an input, check the output — but with a twist: instead of pass/fail, a result can be pass, fail, or maybe, an honest acknowledgment that much of what a language model does can’t be judged as strictly right or wrong.

Honeycomb builds evals for Canvas, an AI feature that lets users ask open-ended questions about production telemetry, using two families. Question evals check whether the model can pull a known fact out of a dataset — say, which error type showed up most often — graded against the known answer. Task evals are fuzzier: an open-ended job, like investigating the root cause of a slowdown, with no single correct answer. Since you can’t grade those on the output alone, you grade the steps taken to get there — which only works if you already have detailed telemetry describing what happened along the way.

That’s the hinge the chapter turns on: evals are only as good as the telemetry feeding them, and telemetry only improves once evals show where it falls short. That’s the learning flywheel.

What actually belongs in the telemetry

A lot of LLM instrumentation looks familiar: spans covering requests, responses, and durations, much of it free from OpenTelemetry’s generative-AI conventions, which already define fields like token counts, model name, and sampling settings. The book flags a few decisions specific to this kind of application, though.

How much of the prompt and response do you capture? Payloads can be huge and contain anything a user pasted in, so one workable pattern is storing them in a separate system of record and attaching a reference to your span rather than the raw text — also a natural place to apply privacy rules.

How do you track cost? There’s no standard OTel field for it, but you don’t need one — cost is easy to calculate after the fact from model name and token counts, letting you compare spend across prompt versions or model upgrades.

How do you fold in what users actually thought? Thumbs up/down, retries, and abandoned sessions matter more here than for a typical web request, since errors and latency alone won’t tell you if people are satisfied.

Sometimes the fix isn’t a better prompt — it’s a few lines of code

One useful lesson here is a reminder not to treat prompt engineering as the only lever. Honeycomb’s Query Assistant, an early AI feature for building telemetry queries, initially produced an unusable response roughly a quarter of the time. Most failures weren’t the model misunderstanding intent — they were small, mechanical formatting mistakes, like attaching an extra field to an operation that didn’t accept one. Instead of prompting their way out of every edge case, the team wrote correction code that fixed the malformed structure before validation ran. That alone cut the error rate from roughly 25% to about 14%; prompt refinements then pushed it under 1%.

There’s a nice coda: some “wrong” queries turned out to reflect a real feature users wanted but the product didn’t yet support. The team shipped it, needing less correction code going forward — the telemetry hadn’t just caught bugs, it had surfaced a product gap.

Fin: turning a resolution-rate business into a latency story

Chapter 22 shows all of this running inside a real company under real pressure. Fin is an AI agent that answers customer-support questions on behalf of the businesses that use it. Its central efficiency metric is resolution rate — the share of conversations it handles without a human stepping in — because customers are only billed when a conversation actually gets resolved. That billing model aligns incentives: Fin’s team wants resolution rate to climb, and so does every customer feeding it better support content.

Since launching in 2023, resolution rate crept up by roughly a percentage point a month — not from one breakthrough, but from dozens of small, stacked, A/B-tested improvements to tool use and workflow rules. The catch: every added capability meant more processing steps, and by early 2025 customers were openly complaining Fin felt slow. Engineers were already optimizing individual services and had charts to prove it — but customers felt no difference, since none of those charts represented what a person experiences while waiting for a reply.

Time to first token: measuring what users actually feel

The team’s answer was to instrument a single end-to-end number, measured in the browser: the time between a user hitting send and the first character of Fin’s reply appearing on screen. They called it time to first token, and every change from then on had to move it in the right direction.

That number alone wasn’t actionable, though. Engineers could see the “what” but not the “why,” because existing traces didn’t capture where control handed off between the many services and model calls in one interaction. A naive mental model treats those steps as fully serial, one finishing before the next starts. The real system runs several in parallel, and some background work isn’t even on the path that determines when the first token appears. Without spans marking those hand-offs, there was no way to tell which optimizations mattered and which were wasted effort. The fix: rebuild the traces with that detail baked in, so engineers could finally see both the customer-facing number and the code behind it.

Eager requests: a two-second win with a hidden bill

With that visibility in place, the team found one especially large lever: eager requests. Since model calls were always the slowest part of any interaction, they started the LLM call early instead of waiting for routine setup work — saving the message, walking through the customer’s configured workflow — to finish first, betting that setup usually wouldn’t change what gets sent to the model. Most of the time the bet paid off, and by the time normal processing caught up, an answer was often already ready to stream back. That one change trimmed roughly two seconds off median time to first token.

It also created a blind spot. If a customer’s workflow routed a conversation straight to a human, skipping Fin entirely, the eagerly-started call still fired and its tokens went to waste. Because Fin was never officially “engaged” in those conversations, no latency metric recorded the waste — nobody in engineering saw it happening. Finance found it first, during a quarterly review, as an unexplained dip in gross margin. Engineering reused the same tracing infrastructure to find and patch the leak, then instrumented cost per interaction directly on the trace, alongside latency, with its own SLO so a future regression would surface immediately instead of a quarter later.

Three boundaries, one word: empathy

Mykhailov ties the story together with the Rasmussen safety model, a way of thinking about how organizations drift toward failure without anyone deciding to take a risky step. It frames every system as operating inside three boundaries: cross the boundary of acceptable performance and customers get unhappy; cross the boundary of economic failure and the business stops making sense; cross the boundary of unacceptable workload and engineers burn out. The danger is that these boundaries are usually invisible until you’ve already crossed one — exactly what happened when Fin’s cost leak went unnoticed until finance flagged it.

His closing argument: surfacing those boundaries in real time, tied to traces detailed enough to explain why a signal moved, is what turns observability from a cost center into something that visibly protects customers, revenue, and the team at once. He sums up the chapter’s spirit in one line:

As ironic as it sounds, in the age of AI, empathy is all you really need.

Key Takeaways

• LLMs are nondeterministic, but the observability playbook still applies: instrument well, treat production data as ground truth, build feedback loops.
• Evals have three outcomes, not two — pass, fail, maybe. Pair fact-based “question evals” with judgment-based “task evals” scored on the steps taken.
• The learning flywheel: telemetry surfaces real behavior, which becomes eval cases, which drive prompt or code fixes, which show up as better telemetry.
• Not every fix belongs in the prompt — Honeycomb cut Query Assistant’s error rate from ~25% to ~14% with a few lines of correction code alone.
• Latency needs a customer-perceived metric (time to first token) backed by traces detailed enough to show the real, overlapping critical path.
• Speed and cost trade off. Fin’s eager-request win saved ~2 seconds but wasted LLM spend until a cost-per-interaction SLO closed the gap.

Sources

• Majors, Charity, Liz Fong-Jones, George Miranda, and Austin Parker. Observability Engineering, 2nd Edition. O’Reilly Media.
• Kesha Mykhailov, Staff Product Engineer at Fin (formerly Intercom) — guest contributor, Chapter 22.
• Phillip Carter — principal contributor, Chapter 21.

This is an independent summary and personal commentary on the book — not affiliated with or endorsed by the authors or O’Reilly Media.

This is Part 7 of a 10-part series on Observability Engineering. Continue to Part 8: Why Learning Speed Is Your Biggest Bottleneck.