The Observability Stack for Agent Infrastructure: What You Actually Need to See

Table of Contents

• Why Agent Observability Is a Different Animal
• The Three Layers of the Stack
• Layer 1: Tracing and Spans
• Layer 2: Metrics, Cost, and Token Accounting
• Layer 3: Evaluation and Quality Signals
• The OpenTelemetry Question
• What to Instrument (and What Everyone Forgets)
• Buy, Adopt, or Build
• Observability and the Economics of GaaS
• Insights Most People Overlook
• References

Why Agent Observability Is a Different Animal

If you've run a web service, you know the classic observability triad: logs, metrics, traces. You watch latency, error rate, throughput. A request comes in, a response goes out, and the path between them is deterministic enough that a stack trace tells you most of the story.

Agents break that model in a way that isn't subtle. An agent run is non-deterministic, multi-step, and self-directed. The same prompt can take three tool calls one time and eleven the next. A request can "succeed", HTTP 200, no exceptions, clean logs, and still be completely wrong, because the agent hallucinated a refund policy or called the wrong API with confidently incorrect arguments. Your APM dashboard will show a beautiful green line the whole time.

This is the core reason agent observability deserves its own category. You're not just asking did it run? You're asking did it reason well, use the right tools, stay on budget, and produce an outcome a human would sign off on? Those are product questions wearing infrastructure clothing. And in a GaaS business, where you're often charging per task or per outcome, they're also revenue questions. If you can't see why an agent burned 40,000 tokens to answer a simple question, you can't price the task, and you definitely can't improve the margin.

There's a second wrinkle. Agents are compositional. A single customer-facing "agent" is frequently a supervisor coordinating sub-agents, each calling tools, each hitting a model that might be routed to a different provider depending on the model-routing layer you've built. Observability has to follow that whole tree, not just the top-level call. Lose the parent-child relationships and you've got a pile of disconnected model calls that tell you nothing.

The Three Layers of the Stack

I find it useful to think about the stack in three layers, because teams tend to nail one and ignore the other two, and the gaps are where the pain lives.

Layer 1: Tracing and Spans

This is the foundation: a hierarchical trace of a single agent run. The root span is the user's request. Child spans are the steps, each LLM call, each tool invocation, each retrieval, each sub-agent handoff. Every span should capture inputs, outputs, latency, token counts, the model and parameters used, and any errors.

The mental model that's won is the trace tree. A good agent trace looks like a flame graph where you can expand the supervisor's reasoning, see it decide to call a search tool, watch the tool return, and follow the model as it reasons over the result. Tools like Langfuse, LangSmith, and Arize Phoenix all render this, and the rendering matters more than it sounds, debugging an agent without a trace tree is like debugging a recursive function with no call stack.

The non-obvious requirement here is capturing the full prompt as actually sent. Not the template. The rendered prompt, after all the retrieval results, memory, and tool definitions got injected. Most "my agent went off the rails" incidents resolve the moment you see the actual context window the model received, usually because something stuffed it with garbage or blew past the context-window budget.

Layer 2: Metrics, Cost, and Token Accounting

Traces are per-run. Metrics are aggregate, and for GaaS they're where the money lives. You want time-series on: tokens per task (prompt vs. completion, broken out), cost per task by model and provider, tool-call counts and failure rates, latency percentiles (p50/p95/p99) at both the step and end-to-end level, and step counts per run.

Cost-per-task is the metric most teams under-instrument and later regret. When you're selling outcomes, an agent that occasionally loops, calling a tool, getting a vague result, calling it again, fifteen times, doesn't throw an error. It just quietly destroys your unit economics. The only way you catch it is a distribution of cost-per-task with a long right tail you're actively watching. The mean lies; the p99 tells the truth.

Layer 3: Evaluation and Quality Signals

This is the layer that separates agent observability from everything that came before, and it's the one most infrastructure teams treat as someone else's problem. Tracing tells you what happened. Evaluation tells you whether what happened was good.

Evals come in flavors. Offline evals run a fixed dataset of inputs through your agent and score the outputs, useful in CI before you ship a prompt or model change. Online evals score real production traffic, usually by sampling, using an LLM-as-judge, heuristic checks, or human review. The judge scores things like task completion, faithfulness to retrieved context, correct tool selection, and tone.

The trap is treating evals as a one-time launch checklist instead of a continuous production signal. Agent quality drifts, a provider silently updates a model, your retrieval corpus changes, user inputs shift. Without online evals wired into the same trace data, you find out about quality regressions from angry customers, which is the most expensive possible detection method. Anthropic's own guidance on building effective agents is blunt about measuring agent behavior empirically rather than trusting that a clever prompt will hold up.

The OpenTelemetry Question

You can't talk about this stack in 2026 without addressing OpenTelemetry. The big shift over the last two years is that OpenTelemetry's semantic conventions for GenAI became real, there's now a (still-evolving) standard for what a span representing an LLM call or an agent invocation should contain: the model, the token counts, the operation type, and so on.

Why should an operator care? Because it's the difference between vendor lock-in and portability. If your agent emits OTel-compliant spans, you can pipe that telemetry to Langfuse today, Datadog tomorrow, and your own ClickHouse cluster the day after, without re-instrumenting your application. The instrumentation lives in your code as standard OTel; the backend is swappable. Given how young and how venture-funded the LLM-observability vendor space is, betting on a standard rather than a single startup is the conservative, correct move.

The honest caveat: the GenAI conventions are not fully stable, and the LLM-native tools (LangSmith especially) still have richer proprietary formats for agent-specific concepts like multi-step reasoning and evals. A pragmatic pattern that's emerging is OTel for transport and the cross-system backbone, with a vendor SDK layered on for the agent-specific richness. This connects directly to the broader push toward telemetry standards for the agent stack that the whole infrastructure space is converging on.

What to Instrument (and What Everyone Forgets)

Here's a working checklist, ordered roughly by how often I see it skipped:

• The full rendered prompt and the full raw completion. Truncating these to save storage is the single most common false economy. You will need them at 2 a.m. during an incident.
• Tool call arguments and tool responses. Not just "called search", the actual JSON in and out. Most agent failures are bad tool arguments, and you can't see that without the payload.
• Retrieval results with scores. When an agent gives a wrong answer grounded in retrieved docs, you need to know whether retrieval handed it garbage or it ignored good context.
• The decision points. Why did the supervisor route to sub-agent A? Capturing the reasoning span here is what makes multi-agent coordination debuggable instead of a black box.
• Session and user IDs, propagated through the whole trace tree. Without correlation IDs that survive sub-agent handoffs, you can't reconstruct a customer's full session, and you can't tie a complaint to a trace.
• Token budget consumption over the run, not just the total. A run that hit 90% of the context window on step 2 is a ticking bomb even if it finished fine.

The thing nearly everyone forgets: instrument the human-in-the-loop checkpoints. When an agent pauses for human approval, log the suggested action, the human's decision, and the latency. That data is gold for figuring out which actions you can safely automate next, and it's invisible if you only trace the autonomous path.

Buy, Adopt, or Build

The market sorts into a few camps. The LLM-native specialists, LangSmith, Langfuse, Arize Phoenix, Braintrust, HoneyHive, are purpose-built for traces, prompts, and evals, and they'll get you to value fastest. Langfuse and Phoenix are open-source, which matters if you don't want your most sensitive prompt and customer data living in a third-party SaaS. The incumbent APM vendors, Datadog, New Relic, Grafana, are bolting LLM observability onto platforms you may already run, which is attractive if you want agents in the same pane of glass as the rest of your infrastructure, though their eval stories are generally thinner.

My blunt advice for most GaaS builders: adopt OpenTelemetry instrumentation in your code from day one, and start with an open-source LLM-native backend (Langfuse or Phoenix) so you're not locked in and your data stays yours. Resist building the whole thing in-house. A homegrown trace viewer is a deceptively large project, the storage layer, the trace-tree UI, the eval runner, and it's almost never your differentiation. Save the build energy for your agents. This is the same platform-versus-framework calculus that runs through the rest of the GaaS infrastructure conversation: own the layer that's your moat, rent the rest.

Observability and the Economics of GaaS

Tie this back to the business, because that's the whole point. In an outcome-priced or per-task GaaS model, observability isn't a nice-to-have ops function, it's the instrument panel for your gross margin.

Three concrete ways it pays for itself. First, cost attribution: per-task token and tool-call metrics let you actually price a task and spot the workflows that are silently unprofitable. Second, reliability evidence: enterprise buyers increasingly ask "how do you know your agent works?" and a real eval pipeline producing quality metrics over time is the answer that closes deals. McKinsey's research on scaling generative AI in the enterprise keeps landing on the same point, the gap between pilot and production is operational discipline, and observability is most of that discipline. Third, continuous improvement: the trace-plus-eval dataset you accumulate becomes the raw material for prompt tuning, model routing decisions, and eventually fine-tuning. Your observability data is a flywheel, not a cost center.

The teams that win the GaaS race won't necessarily have the cleverest agents. They'll be the ones who can see their agents clearly enough to make them cheaper, more reliable, and more trustworthy, one traced run at a time.

Insights Most People Overlook

• "No errors" is the most dangerous status in agent systems. A silent wrong answer costs more than a crash, because a crash gets attention and a confident hallucination gets shipped to a customer. Build your alerting around quality regressions and cost outliers, not just exception rates, the green dashboard is lying to you exactly when it matters most.

• The most valuable observability data is the prompts you're tempted not to store. Full rendered context windows are large and feel wasteful to retain. They're also the single artifact that resolves the majority of "why did my agent do that" incidents. Storage is cheap; a multi-hour incident with no prompt history is not.

• Evals are observability, not QA. Teams file evals under "testing" and run them before launch, then never again. But agent quality is a continuously drifting production property, model updates, corpus changes, input shifts. Online evals belong in your observability stack, sampling live traffic, the same way you'd never turn off latency monitoring after launch.

• OpenTelemetry adoption is a hedge against a vendor shakeout that's coming. The LLM-observability space is crowded and venture-funded; consolidation is inevitable. Instrumenting with OTel semantic conventions means a vendor's acquisition or pivot is a backend swap, not a re-instrumentation project. The least exciting architectural decision here is the one that ages best.

• Your human-in-the-loop logs are a roadmap for automation. Every approval checkpoint is a labeled example of a decision a human currently makes. Mine that data and you'll find the actions humans approve 99% of the time, your next candidates for safe full automation. Almost nobody instruments this deliberately, and it's sitting right there.

References

• Building Effective Agents, Anthropic
• OpenTelemetry Semantic Conventions for Generative AI
• The State of AI, McKinsey & Company (QuantumBlack)