The Latency Budget: Where Agent Time Actually Goes

Table of Contents

• Why Latency Is an Economic Problem, Not a UX One
• The Anatomy of a Single Agent Step
• Time to First Token vs. Total Generation
• The Tool-Call Round-Trip
• Retrieval and Context Assembly
• The Multi-Step Multiplier
• A Realistic Latency Budget, Decomposed
• Where the Hidden Time Hides
• Queueing and Cold Starts
• Serial When It Could Be Parallel
• Context Bloat
• What You Can Actually Move
• Insights Most People Overlook
• References

Why Latency Is an Economic Problem, Not a UX One

In a chatbot, latency is a comfort metric. A response that lands in two seconds feels good; four seconds feels sluggish; ten seconds and people start clicking away. That framing carries over into agent work, and it's where a lot of teams get the priorities wrong.

When you're running an Agentic AI-as-a-Service product, agents sold per task or per outcome, latency stops being about feel and starts being about cost and throughput. A slow agent ties up a runtime slot longer, which means you serve fewer concurrent jobs per dollar of infrastructure. If your pricing is per-outcome, every extra second of wall-clock time is margin you're spending without charging for it. And if a customer's SLA says "resolved in under a minute," latency is the difference between a job you can sell and one you can't.

So the question "where does agent time actually go?" isn't idle curiosity. It's the same question as "where is my margin leaking?" You can't fix what you haven't measured, and most teams have never actually profiled a full agent run end to end. They have a vibe, "the model is slow", and they act on the vibe. The vibe is usually wrong.

The Anatomy of a Single Agent Step

Before you can budget a multi-step run, you have to understand one step. A single "turn" of an agent, one trip through the reasoning loop, is rarely just one model call. It's a small pipeline, and each stage has its own latency profile.

Time to First Token vs. Total Generation

The single most useful distinction in agent latency is between time to first token (TTFT) and total generation time. TTFT is how long the model takes to start responding after it receives the prompt; it's dominated by the prefill phase, where the model processes your entire input context. Total generation time is TTFT plus the per-token decode time for everything the model writes.

This matters because the two scale differently. TTFT grows with your input size, a bloated context window with 40,000 tokens of history and retrieved documents makes prefill slow even if the model only needs to emit a three-word tool call. Generation time grows with output size. An agent that reasons out loud for 600 tokens before deciding what to do pays for every one of those tokens at decode speed.

The practical takeaway, well documented in inference-optimization work like NVIDIA's writeups on LLM inference performance, is that prefill and decode are different bottlenecks with different fixes. Streaming hides generation latency from a human watching a UI, but it does nothing for an agent that has to wait for the complete structured output before it can act. For agents, the full generation time is what counts, not the perceived TTFT.

The Tool-Call Round-Trip

Here's the part people underestimate. When an agent decides to call a tool, query a database, hit an API, run a search, that round-trip is its own latency event, and it's often the slowest single thing in the step.

A model might decide on a tool call in 800 milliseconds. The tool itself, say, a third-party CRM API with its own cold caches and rate limits, might take 2 to 5 seconds to respond. Then the result has to be serialized back into the context, and the model has to be called again to interpret it. So a single "use a tool" action can be: model call, wait for tool, model call. Two inference round-trips bracketing a network call you don't control.

This is why infrastructure-layer reliability for tool calling matters so much. Slow or flaky tools don't just cause errors; they blow the latency budget even when they eventually succeed. A retry on a timed-out tool can silently double the cost of a step.

Retrieval and Context Assembly

If your agent uses retrieval, there's another stage hiding before the model even sees the prompt: embedding the query, hitting the vector store, ranking results, and stitching them into context. On a well-tuned setup this is fast, tens of milliseconds for the vector lookup. On a poorly tuned one, with a cold index or an over-eager reranker, it can add hundreds of milliseconds to every single step. Multiply that across a ten-step run and it's no longer a rounding error.

The Multi-Step Multiplier

Now assemble the steps. The defining feature of agent latency, the thing that separates it from plain LLM serving, is that steps are usually sequential and dependent. The agent can't decide step three until it sees the result of step two. That dependency chain is the heart of the problem.

If one step takes 4 seconds and a task needs eight steps, you're at 32 seconds before you've accounted for any overhead. And the step count itself is variable: a well-scoped task might resolve in three loops, while an ambiguous one sends the agent wandering through twelve. Latency in agent systems isn't a fixed number you can quote, it's a distribution with a long, ugly tail, and the tail is where your SLA breaches and margin disasters live.

This is the single biggest reason "just use a faster model" disappoints. Shaving 20% off per-step generation time on a workload that's 30% generation and 70% tool-calls-and-waiting moves your total by about 6%. The structure of the loop dominates the speed of any one component.

A Realistic Latency Budget, Decomposed

Numbers vary wildly by workload, but here's a representative decomposition for a moderately complex vertical agent task, something like "look up this customer, check three systems, and draft a resolution." Treat these as illustrative proportions, not gospel:

• LLM generation (all steps combined): ~25-35%. The actual token production. Real, but rarely the majority.
• Tool / API round-trips: ~30-45%. External systems you mostly don't control. Usually the single largest bucket.
• Network and serialization overhead: ~10-15%. Round-trips between your orchestrator, the model provider, and your tools. Each hop is small; there are many hops.
• Retrieval and context assembly: ~5-15%. Embeddings, vector lookups, reranking.
• Queueing, cold starts, and orchestration overhead: ~5-20%. The most variable bucket, and the one most likely to be invisible in your traces.

Add it up and the model, the thing everyone blames, is often a third or less of the wall-clock time. McKinsey's analysis of the economic potential of generative AI and agentic workflows makes the broader point that value comes from end-to-end workflow redesign, not from any single model capability, and latency follows the same logic. You optimize the workflow, not the component.

Where the Hidden Time Hides

The budget above lists the visible buckets. The dangerous time is the time your observability stack doesn't show you.

Queueing and Cold Starts

If you run agents on serverless infrastructure or autoscaling containers, cold starts are a latency tax that hits hardest exactly when you're scaling up to meet demand. A function that spins up in 2 seconds adds 2 seconds to the first request after a scale event, and under bursty load, that can be a meaningful fraction of your traffic. Likewise, if you're sharing model capacity, provider-side queueing during peak hours inflates TTFT in ways that never show up in your own application traces. You see "the model was slow"; you don't see "the model was queued."

Serial When It Could Be Parallel

A huge amount of recoverable latency comes from agents doing one thing at a time when they could do several. If step four needs the results of three independent lookups, a naive agent calls them one after another. A well-built one fires all three in parallel and waits for the slowest. The difference between 3 × 2 seconds and max(2, 2, 2) seconds is the difference between 6 seconds and 2. Most agent frameworks support parallel tool calls now; most agent implementations don't use them, because the default reasoning pattern is stubbornly serial.

Context Bloat

Every step, the full context gets re-sent and re-prefilled. If your agent accumulates history without pruning, dumping every tool result, every intermediate thought, every retrieved document into a growing context, prefill time climbs with each step. By step ten you might be paying to re-process 50,000 tokens to make a one-line decision. This is where prompt caching earns its keep: providers like Anthropic offer prompt caching that can cut latency and cost on repeated context, letting you reuse the prefill work for the stable portion of your prompt instead of re-paying for it every turn.

What You Can Actually Move

Given that breakdown, here's where the leverage actually is, roughly in order of bang-for-buck:

Parallelize independent work. Cheapest, biggest win for most teams. If your agent makes multiple independent tool calls, fan them out. This is an orchestration change, not a model change.

Cache aggressively. Prompt caching for stable context, result caching for deterministic tool calls, and embedding caching for repeated retrievals. The stable system prompt and tool definitions that ride along in every step are prime cache candidates.

Route to the cheapest viable model per step. Not every step needs your most capable model. A quick classification or routing decision can run on a small, fast model; reserve the expensive one for genuine reasoning. This cuts both latency and cost on the same lever.

Cut step count. Fewer loops is the most direct latency reduction available, because it attacks the multiplier. Better tool design, one tool that does the right thing instead of three the agent has to chain, and clearer instructions that prevent wandering both reduce the number of round-trips. This is underrated precisely because it requires understanding the task, not tuning infrastructure.

Tame your context. Prune aggressively. Summarize old turns. Don't carry forward what the next step doesn't need. Smaller context means faster prefill on every remaining step.

Notice what's not at the top of that list: switching to a faster model. It helps, but it's usually the smallest lever relative to the effort of re-validating quality on a new model. The real latency budget is spent in the architecture of the loop, and that's where you should spend your engineering attention too.

Insights Most People Overlook

Latency variance matters more than the average, and it's where SLAs die. Teams quote a median ("our agents respond in 8 seconds") and design pricing around it, then get wrecked by the p95 and p99. Because step count is variable and tool latency is long-tailed, agent latency distributions are heavily right-skewed. If you're selling per-outcome SLAs, you have to budget against the tail, not the median. A system that's fast on average but occasionally takes 90 seconds will burn through your SLA credits on the unlucky jobs.

Faster models can make total latency worse. Counterintuitive, but real: a cheaper, faster model that's slightly less capable may take more reasoning steps to reach the same outcome. Each step is faster, but you do more of them, and because of the multi-step multiplier, more steps usually loses. The fastest path to done isn't the fastest model; it's the model that nails the task in the fewest loops. Always measure total task latency, never per-step latency, when comparing models.

Streaming is a trap for agent UX metrics. Streaming makes a model feel fast to a human, and it's genuinely valuable for human-facing turns. But agents can't act on a half-streamed tool call, they need the complete structured output. Optimizing for TTFT because it looks good on a dashboard can lead you to ignore the total generation time that actually gates the agent's next move. Measure the metric that gates the work, not the one that feels good.

The slowest part of your agent is usually code you didn't write. Third-party APIs, the model provider's queue, the vector DB's cold index. Your own orchestration logic is often the smallest, fastest piece. This is humbling but liberating: it means latency work is mostly about managing external dependencies, timeouts, parallelism, caching, fallbacks, rather than optimizing your own hot loops. Build your reliability infrastructure (retries, circuit breakers, fallbacks) with latency budgets in mind, because a naive retry policy can quietly double your worst-case time.

Per-step latency reduction compounds, but only if the steps are on the critical path. Shaving 200ms off a retrieval that happens in parallel with a tool call that takes 3 seconds saves you nothing, the tool call was the long pole. Latency optimization without a real trace of the critical path is guesswork, and most teams optimize the wrong line item because they never profiled which one was actually blocking.

References

• Mastering LLM Techniques: Inference Optimization (NVIDIA Developer Blog)
• The Economic Potential of Generative AI: The Next Productivity Frontier (McKinsey)
• Prompt Caching (Anthropic Documentation)