Tracing a Multi-Step Agent Run End to End: The Observability Skill That Decides Whether Your GaaS Survives Production

Table of Contents

• Why a Single Agent Run Is Harder to Trace Than a Hundred Microservices
• What "End to End" Actually Means for an Agent
• The Anatomy of a Trace: Spans, Attributes, and the Causal Spine
• Walking Through a Real Multi-Step Run
• Instrumenting Without Drowning in Noise
• Reading a Trace to Find the Actual Failure
• Cost, Latency, and Token Attribution Per Step
• From Trace to Replay: Recreating the Exact Run
• Insights Most People Overlook
• References

Why a Single Agent Run Is Harder to Trace Than a Hundred Microservices

A web request through a microservice stack is, for all its complexity, deterministic and shallow. You hit the API gateway, it calls auth, auth calls the user service, the user service hits Postgres, and the whole thing returns in 200 milliseconds across maybe six hops. Distributed tracing was built for exactly this shape: a fan-out tree of services, each one stateless, each call reproducible if you replay the same inputs.

An agent run breaks every assumption in that sentence.

It is deep, not wide, a single task can chain twenty, fifty, a hundred LLM calls in sequence, each one feeding the next. It is non-deterministic: the same prompt produces a different plan on Tuesday than it did on Monday, which I'll come back to because it's the single most underappreciated property of agent observability. It carries state across steps through a context window and an external memory store, so a failure at step 40 can be caused by something the agent "remembered" wrong at step 3. And the unit of work that matters, "did the agent accomplish the user's goal?", isn't visible in any individual span. It only exists in the relationship between them.

This is precisely why traditional application performance monitoring falls flat here, a problem worth its own treatment (see #111 in this beat). APM tools assume the interesting failures are slow queries and 500 errors. With agents, the request returns HTTP 200, the latency looks fine, every tool call succeeds, and the agent still confidently did nothing useful. The failure is semantic, and semantic failures are invisible to infrastructure-level monitoring. You need a trace that captures reasoning, not just requests.

What "End to End" Actually Means for an Agent

"End to end" gets thrown around loosely, so let me pin it down. A complete agent trace runs from the moment the task enters the system to the moment a result leaves it, and crucially, it captures everything in between as a single connected object, not a pile of disconnected logs.

Concretely, an end-to-end trace should let you answer all of these from one view:

• What was the user's original request, verbatim, and what system prompt and tools was the agent given?
• What plan did the agent form, and did it revise that plan mid-run?
• Every LLM call: the exact rendered prompt (including injected memory and retrieved context), the model and parameters, the raw completion, and the token counts.
• Every tool call: which tool, the arguments the agent generated, the raw response, and whether it errored or retried.
• Every memory operation: what the agent wrote, what it read back later, and why.
• Every guardrail or verification check and its verdict.
• The final output, and a judgment, human or automated, of whether it was correct.

If your "tracing" captures the first and last bullet but not the middle, you have logging, not tracing. The middle is the whole point. The value of end-to-end visibility is that causality flows through it: you can put your finger on the output, walk backward through the causal chain, and arrive at the root cause without guessing.

The Anatomy of a Trace: Spans, Attributes, and the Causal Spine

The industry has more or less converged on a span-based model borrowed from distributed tracing, and the OpenTelemetry GenAI semantic conventions have started to standardize the vocabulary. Worth understanding the building blocks, because most observability tooling, the emerging category covered in #110, speaks this language.

A trace is the entire run: one task, one trace ID. Inside it lives a tree of spans, where each span is a single unit of work with a start time, end time, a parent, and a set of attributes. The nesting matters enormously for agents because it encodes the causal spine of the run.

A typical span hierarchy looks like this:

• Agent span (the root), wraps the whole task.
• Planner span, the LLM call that decides what to do.
• Step 1 span
  • LLM span, the reasoning call. Attributes: model, prompt tokens, completion tokens, temperature, the full prompt, the full response.
  • Tool span, e.g., search_invoices. Attributes: tool name, input args, output, error status.
• Step 2 span
  • Retrieval span, a vector lookup. Attributes: query, top-k results, similarity scores.
  • LLM span
• ... and so on.

The attributes are where the diagnostic power lives. A bare span tells you something took 800ms. A well-attributed span tells you it was a gpt-style reasoning call with 4,200 prompt tokens, that 3,000 of those tokens were retrieved documents, that the temperature was 0.7, and that the completion proposed calling a tool that didn't exist. That last detail, a hallucinated tool name, is one of the most common failure modes in tool-call chains, and it's invisible unless you log the raw completion. Debugging those specifically is a deep enough topic that it gets its own node (see #133 in this beat).

One non-obvious design choice: link your spans to the prompt template version, not just the prompt text. When you change a prompt and reliability shifts, you want to filter every trace by template v17 versus v18 and compare success rates. Teams that only log the rendered text lose the ability to do that cohort analysis, and they regret it the first time a "small wording tweak" tanks their numbers.

Walking Through a Real Multi-Step Run

Let me make this concrete with a vertical agent, say, an accounts-payable agent that processes a vendor's emailed invoice. Here's what the trace looks like when it works, and where it tends to break.

Step 0, Ingest. The trace opens. Root span attributes: the inbound email, the attachment, the tenant ID, the agent version. Already useful: if this agent misbehaves only for one customer, you'll filter traces by tenant and the pattern jumps out.

Step 1, Plan. An LLM span. The agent reads the email and forms a plan: extract line items, match to a purchase order, flag discrepancies, queue for payment. The completion is logged verbatim. If the agent later goes off the rails, this is the first place you check, did it even understand the task?

Step 2, Extract. A tool span calling an OCR/parsing service, then an LLM span structuring the result into JSON. Here's a real failure I've seen repeatedly: the OCR span succeeds (status OK), but the parsed total is $1,240.00 when the invoice says $12,400.00, a decimal-point error. The tool succeeded. Nothing erred. The trace is the only artifact that shows the parsed value diverging from the source document.

Step 3, Retrieve. A retrieval span pulls the matching purchase order from a vector store. Attributes capture the query and the top results with their similarity scores. When matching fails, those scores tell you whether the right PO was retrieved-but-ignored, or never retrieved at all. Completely different bugs, completely different fixes.

Step 4, Reconcile. An LLM span compares invoice to PO. This is the reasoning core. If the agent's context window is now stuffed with 8,000 tokens of retrieved POs, you can see in the span that the relevant line got pushed toward the middle of the context, the "lost in the middle" effect that Stanford's research on long-context retrieval documented, where models reliably miss information buried in the center of a long prompt.

Step 5, Act / Escalate. Either the agent queues the invoice for payment (a tool span) or escalates to a human (a different span). The escalate-to-human decision is itself a reliability feature worth designing deliberately (see #121 in this beat), and your trace should record why it escalated, not just that it did.

When this run fails in production, you don't read logs. You open the trace, scan the timeline, find the span where the parsed total first went wrong, and you're done in minutes. Without the trace, you're staring at a payment queued for the wrong amount with no idea which of five steps introduced the error.

Instrumenting Without Drowning in Noise

The naive approach, log everything, every token, every intermediate, produces traces so heavy nobody opens them and storage bills that make finance nervous. The discipline is in deciding what to capture at full fidelity, what to sample, and what to drop.

A few rules that hold up in practice:

Capture full fidelity on the spine, sample the bulk. Always record the full prompt and completion for planner and reasoning spans. For high-volume, low-information spans, a retry that succeeded on the second attempt, a routine retrieval, head-based or tail-based sampling is fine. Tail sampling, where you decide whether to keep a trace after you know the outcome, is especially powerful here: keep 100% of failed and escalated runs, sample 5% of clean successes. You learn almost everything from the failures anyway.

Redact at the instrumentation layer, not after. Agent prompts routinely contain PII, payment details, health records. Scrub it before it hits your trace store, not in a downstream job, because the moment it lands in your observability backend it's a compliance surface. Enterprise buyers will ask exactly this before they sign (a concern that surfaces in #150 of this beat).

Make trace IDs propagate across service boundaries. If your agent calls an external tool that itself is an agent, increasingly common in multi-agent systems, the child agent's trace should link to the parent's. Otherwise the chain breaks exactly where multi-agent reliability problems live, and a weak sub-agent failure looks like it came from nowhere.

Don't reinvent the wire format. Adopt OpenTelemetry's GenAI conventions even if you use a specialized agent-observability vendor on top. Frameworks like LangChain's LangSmith tracing and others emit OTel-compatible spans precisely so you're not locked into one backend. Vendor-neutral instrumentation is cheap insurance.

Reading a Trace to Find the Actual Failure

Capturing a trace is half the job. The skill that separates teams who ship reliable agents from teams who don't is reading one fast. Here's the mental algorithm I use.

Start at the output and ask: is it wrong, or merely different? Non-determinism means two correct runs can look nothing alike. Don't chase a "bug" that's just a valid alternate path.

If it's genuinely wrong, classify the failure shape first, because the shape tells you where to look:

• Wrong plan. The agent misunderstood the goal. Look at the planner span, the bug is upstream, in the prompt or the task framing.
• Right plan, wrong tool call. The agent knew what to do but called a tool with bad arguments or hallucinated a tool name. Look at the tool spans and the LLM span that generated their inputs.
• Right tool call, wrong data. The tool returned garbage, or returned good data the agent then misread. Compare the tool's raw output span to how the next LLM span used it.
• Right everything, wrong synthesis. Every step succeeded but the final answer is still wrong, the dreaded "silent failure" (its own topic at #113). This is the hardest, and it usually lives in the last reasoning span where the agent assembled its conclusion.

The single most valuable move is to diff the rendered prompt against what you thought the prompt was. Half of all "the model is dumb" complaints turn out to be "the prompt template injected stale memory" or "retrieval returned the wrong document and the model faithfully reasoned over it." The model did its job; the context was poisoned. You only see that with the full rendered prompt in the span.

Cost, Latency, and Token Attribution Per Step

A trace isn't only a debugging tool, it's your unit economics ledger, and for a GaaS business priced per task or per outcome, that's not a footnote. It's the model.

Because every LLM span carries token counts and you know your per-token rates, you can roll up the exact cost of a single agent run and, more usefully, see which step dominates it. I've watched teams discover that 70% of their per-task cost came from one over-eager retrieval step that stuffed 6,000 tokens of marginally-relevant context into every reasoning call. The fix was a tighter top-k. The trace made the waste obvious; without it, they were just staring at an aggregate cloud bill.

The same per-span data drives the latency-reliability tradeoff (explored in #137). Adding a verification pass, a second agent checking the first one's work, improves reliability but adds a span, latency, and cost. Whether that's worth it is a per-vertical judgment, and you can only make it with honest per-step numbers. A trace turns "agents are expensive" from a vibe into a line-item you can actually attack.

One practical note: attribute cost to the cause, not just the call. A retry that fired because a tool timed out should be attributed to that tool's unreliability, not lumped into generic LLM spend. When you tag spans this way, your cost dashboards start pointing at the actual culprits.

From Trace to Replay: Recreating the Exact Run

The highest form of agent observability is replay, taking a captured trace and re-running it to reproduce the exact behavior, optionally swapping one variable to test a fix. This is genuinely hard with agents (it has its own dedicated node at #134) precisely because of the non-determinism I keep flagging.

The trick is that a complete trace already contains everything you need to make a run deterministic if you choose to: the exact prompts, the tool inputs, and the tool outputs. If you record the tool responses and the model completions in your spans, you can replay the run by feeding those recorded values back instead of re-calling the model and the live tools, a fixture-based replay. Now you can change one thing, say, the system prompt, re-run, and see whether the failure disappears, with every other variable held constant.

This is the foundation that makes eval-driven development possible. A failed production trace becomes a regression test: capture it, freeze it, and assert that your fix turns it green without breaking the others. Teams doing this well treat their library of captured failure traces as a golden dataset that grows every time production surprises them, which is exactly how reliability compounds into a moat that competitors can't copy by spending more on a bigger model.

The path is consistent: trace everything, read traces fluently, attribute cost honestly, and replay the interesting failures. Do that, and the question "where did the agent go wrong?" stops being a mystery and becomes a query.

Insights Most People Overlook

The trace is your product's reliability story made auditable, sell it. Most teams treat tracing as internal plumbing. The smart GaaS vendors expose a sanitized version of the trace to the customer as a transparency artifact: "here's exactly what the agent did, step by step." Enterprise buyers trust an agent they can audit far more than one that's accurate-but-opaque. Your trace infrastructure is a sales asset, not just an engineering one.

Logging the tool output matters more than logging the tool call. Everyone logs which tool the agent invoked and with what arguments. Far fewer log the raw response, and that's where the bodies are buried. The most insidious failures are tools that succeed and return subtly wrong data that the agent then faithfully reasons over. If you can't see what the tool actually returned, you'll blame the model for the tool's sins every time.

Non-determinism means a single trace can lie to you. One trace shows you one path through a probabilistic system. A run that "worked" might have succeeded by luck and fails 30% of the time. Never conclude reliability from a single green trace, you need the distribution. The corollary: when reproducing a bug, capture the seed and parameters, because a bug you "fixed" might just be one you got lucky on once.

Memory reads are the most under-instrumented span in agentic systems. Teams diligently trace LLM and tool calls but treat the agent's memory store as invisible infrastructure. Yet a huge class of failures is "the agent remembered something wrong from a previous turn." If your trace doesn't show what was written to memory, what was read back, and when, you're blind to an entire failure category (which is why #132 treats memory observability as its own discipline).

The most expensive span is rarely the slowest one. Latency dashboards draw your eye to the slow spans, but cost and slowness decouple completely with agents. A fast, cheap-looking reasoning call that quietly carries 8,000 tokens of injected context can cost ten times more than the dramatic-looking tool call you keep optimizing. Sort your spans by cost, not just duration, and you'll find waste in places latency alone never reveals.

References

• OpenTelemetry GenAI Semantic Conventions, OpenTelemetry
• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., Stanford / arXiv
• Observability Concepts and Tracing, LangSmith Documentation