The Cost of Context: Managing Token Budgets at Runtime

Table of Contents

• Why Context Is the Hidden Cost Center
• Where the Tokens Actually Go
• The Runtime Levers That Move the Needle
• Prompt Caching: The Cheapest Win
• Summarization and Context Compaction
• Selective Retrieval Over Dumping
• Model Routing by Token Tier
• Building a Token Budget That Holds at Runtime
• The Economics: Per-Task Pricing and the Margin Trap
• Insights Most People Overlook
• References

Why Context Is the Hidden Cost Center

Talk to anyone running an agent product in production and you'll hear the same confession eventually: the demo was cheap, and then the bill arrived. A single chat-style call looks trivial, fractions of a cent. But agents don't make single calls. They loop. A vertical agent handling, say, an insurance claim might fire twenty or thirty model calls, each one carrying the accumulated history of the task: the original request, the tool outputs, the intermediate reasoning, the retrieved documents. By the tenth step, you're paying to re-read the same growing transcript over and over.

This is the part that catches teams off guard. Token cost in an agentic system isn't linear with the amount of work, it tends toward quadratic, because the context typically grows with each step and gets resent on every step. Ten steps that each append a little context don't cost 10x the first step. They can cost far more, because step ten is paying input-token rates on everything that happened in steps one through nine.

For a Gaas provider, that math is existential. If you're charging per task or per outcome (the pricing models that define the category, covered more broadly in the GaaS infrastructure cost stack discussion), your revenue is fixed the moment the customer agrees to a price. Your token spend, by contrast, is a runtime variable that depends on how messy the task turns out to be. A clean task might cost you twenty cents in inference; a pathological one, an agent that gets confused, retries, and bloats its own context, might cost five dollars. The customer pays the same either way. Context management is the difference between those two outcomes, and it happens at runtime, not in the prompt you wrote three months ago.

Where the Tokens Actually Go

Before you optimize, instrument. The single most common mistake is optimizing the wrong thing, shaving the system prompt while a tool is quietly returning 8,000 tokens of raw JSON on every call.

In most agent traces, the token spend clusters in a few predictable places. System prompts and tool definitions are sneaky because they're fixed but get resent every turn; if you have forty tools each with a verbose JSON schema, that's a tax on every single call. Tool outputs are usually the biggest offender, an API that returns a full database row, an HTML page scraped wholesale, a search result with twenty hits when the agent needed two. Conversation history accumulates relentlessly. And retrieved documents, if you're doing RAG, can dominate everything else when someone sets the retrieval count to "top 20" and forgets about it.

The output side matters too, and it's priced higher. Reasoning models that "think" out loud generate large volumes of output tokens before producing an answer, and output tokens often cost several times what input tokens do. An agent that's prompted to deliberate verbosely at every step is burning the expensive kind of token. Anthropic's own guidance on context engineering for agents makes the point that treating context as a finite, curated resource, rather than a dumping ground, is what separates reliable agents from expensive ones. The framing is deliberate: context is something you engineer, not something you accumulate.

The practical takeaway: get per-component token accounting into your observability layer before you touch anything else. You want a breakdown, per task, of how many tokens went to prompts versus tools versus history versus retrieval. Without it, you're guessing, and you'll usually guess wrong.

The Runtime Levers That Move the Needle

Once you can see where tokens go, four levers do most of the heavy lifting. They're not mutually exclusive; the best systems stack all four.

Prompt Caching: The Cheapest Win

If you do nothing else, do this. Prompt caching lets the model provider store the processed form of a stable prefix, your system prompt, tool definitions, few-shot examples, a fixed knowledge block, and charge you a steep discount when that prefix repeats across calls. Cached input tokens typically run at roughly a tenth of the normal input price on major providers, and the latency drops too because the model skips reprocessing.

For agents, this is close to free money, because the agentic loop is repetition. The same system prompt and tool schema ride along on every one of those thirty calls. Structure your context so the stable parts come first and the variable parts come last, and you can cache the expensive front matter. The discipline here is ordering: anything that changes between calls has to live after everything that doesn't, or you break the cache. Teams that retrofit caching onto an existing agent often find the win is bottlenecked by a single dynamic timestamp jammed into the top of the prompt.

Summarization and Context Compaction

When a task runs long, the history has to be compressed or the cost runs away. The standard move is to summarize older turns: once the transcript crosses a threshold, an agent (often a cheaper model) condenses the earlier exchanges into a compact synopsis, and the full detail gets dropped from the active context. The original is preserved in external storage if it's ever needed again, which ties this directly to how stateful agents manage memory and persistence outside the window.

The art is in what you summarize and when. Summarize too aggressively and the agent forgets a constraint it needed. Summarize too late and you've already paid the bloated-context tax for several turns. A common pattern is to keep the most recent N turns verbatim (recency matters most for coherence) while compacting everything older. Some teams trigger compaction on a token threshold; others do it on logical task boundaries, which tends to produce cleaner summaries because you're compressing a completed sub-task rather than slicing mid-thought.

Selective Retrieval Over Dumping

RAG and agent retrieval are powerful, but they're also where token budgets go to die. The default instinct, retrieve a generous chunk of documents and let the model sort it out, is exactly backwards for cost. Every retrieved token is an input token you pay for, on this call and potentially on every call after if it sticks in the history.

The fix is to retrieve less but better: tighter chunks, reranking to surface only the genuinely relevant passages, and a hard cap on how much retrieved material enters the window. Just-in-time retrieval, where the agent fetches a document only at the moment it needs it, rather than front-loading everything it might need, keeps the baseline context lean. This is the same shift the agent-retrieval-beyond-basic-RAG conversation describes: moving from "stuff the context" to "fetch on demand."

Model Routing by Token Tier

Not every step in an agent's workflow needs the flagship model. Classifying an intent, extracting a field, summarizing a tool output, deciding which branch to take, these are often handled perfectly well by a smaller, cheaper model running at a fraction of the per-token cost. Reserve the expensive frontier model for the steps that genuinely require its reasoning, and route the rest down-tier.

This is its own infrastructure concern (the model-routing layer gets a full treatment elsewhere in this beat), but from a token-budget standpoint the principle is simple: the cost of a token depends on which model reads it, so part of managing the budget is managing which model sees which context. A well-routed agent might run 70% of its calls on a cheap model and only escalate the hard parts.

Building a Token Budget That Holds at Runtime

Levers are tactics. A budget is policy. The teams that keep agent costs predictable treat the token budget as an explicit runtime constraint the system enforces, not a hope.

In practice that means a few concrete things. First, a per-task token ceiling: a hard limit on total tokens an agent may consume before it must either finish, escalate to a human, or fail gracefully. This single guardrail prevents the catastrophic-tail cost, the runaway task that loops forever and eats your monthly margin in one afternoon. Tie it to human-in-the-loop checkpoints so a near-limit task gets handed off rather than killed.

Second, a context window allocation, where you decide ahead of time roughly how much of the window each component is allowed to occupy: so much for the system prompt, so much for history, so much for retrieval, so much reserved for the model's output. When a component wants more, something else has to give, retrieval gets reranked harder, or history gets compacted. This turns context management from reactive firefighting into a budget you allocate deliberately.

Third, runtime metering, where the orchestration layer tracks token spend live within a task and can change behavior as the budget depletes. Early in a task, the agent can afford to be generous with retrieval and exploration. As it approaches its ceiling, the system tightens, shorter outputs, less retrieval, more aggressive summarization. McKinsey's analysis of the economic potential of generative AI is bullish on the value agents create, but the value only survives contact with reality if the unit economics underneath hold, and at runtime, unit economics is token economics.

The orchestration layer is where all of this lives, which is why token budgeting is increasingly considered part of the agent runtime itself rather than something bolted on. You want the budget enforced by infrastructure, consistently, across every agent, not reimplemented by hand in each one.

The Economics: Per-Task Pricing and the Margin Trap

Here's the strategic point that ties the whole thing together. The GaaS model's headline promise is outcome-based or per-task pricing, you pay for results, not seats. That's compelling to buyers and it's the right long-term direction, as analyses like a16z's writing on the new business models AI agents enable have argued. But it quietly transfers a real risk onto the provider: cost variance.

Under seat-based SaaS, your costs barely move with usage. Under per-task agentic pricing, every task has a cost distribution, and the tail of that distribution can be brutal. Most tasks cost what you expect. A few cost five or ten times more because they hit edge cases, retried, or spiraled into context bloat. If you priced off the average and your distribution has a fat tail, a handful of pathological tasks can erase the margin from hundreds of clean ones.

Runtime token management is the lever that compresses that tail. Caching lowers the floor. Budget ceilings cap the top. Compaction and selective retrieval narrow the spread. The goal isn't just a lower average cost per task, it's a tighter cost distribution, because predictability is what makes outcome-based pricing survivable. A provider who can promise "this task type costs us $0.30 ± $0.05" can price confidently. One whose costs range from $0.20 to $6.00 is gambling on every contract.

That's why context discipline has graduated from a developer nicety to a board-level concern in serious agent businesses. It's not about saving pennies. It's about whether the core business model is structurally profitable, and at runtime, the token budget is where that question gets answered, one task at a time.

Insights Most People Overlook

• Quality often improves when you cut context, not just cost. There's a persistent assumption that more context equals better answers, so trimming feels like a tradeoff against quality. In practice, oversized contexts cause "lost in the middle" degradation and let the model latch onto irrelevant material. Aggressive, well-targeted context pruning frequently makes agents both cheaper and more accurate. The cost optimization and the quality optimization point the same direction more often than people expect.

• Output tokens are the silent budget killer, not input. Most teams obsess over shrinking prompts while ignoring that output tokens cost several times more per unit and that reasoning models can emit thousands of them per step. A verbose chain-of-thought at every turn is more expensive than a bloated system prompt. Capping and structuring output, asking for terse intermediate steps and verbose answers only at the end, often beats input optimization on dollars saved.

• Caching changes your architecture, not just your bill. Once prompt caching is in play, the cost-optimal context layout (stable-prefix-first, dynamic-suffix-last) can conflict with the layout a developer would naturally write. Teams that adopt caching seriously end up restructuring how they assemble prompts, and that constraint propagates into how tools, memory, and retrieval are ordered. Caching isn't a setting you flip; it's a design discipline.

• The cheapest token is the one you never send. Half of context optimization isn't compression, it's prevention. A tool that returns a 50-token summary instead of a 5,000-token raw payload solves the problem at the source, before any caching or summarization is needed. The highest-leverage work is often in tool design and output shaping upstream, not in the agent's context-management logic downstream.

• Per-task cost variance, not average cost, is the metric that matters. Teams report average cost per task and feel good about a low number. But the financial risk under outcome-based pricing lives in the variance and the tail. A higher average with a tight distribution can be a far healthier business than a low average with a fat tail. Instrument and manage the spread, not just the mean.

References

• Effective Context Engineering for AI Agents, Anthropic
• The Economic Potential of Generative AI: The Next Productivity Frontier, McKinsey
• AI Agents and the Future of Pricing, Andreessen Horowitz (a16z)