The Context-Window Economy: Managing What Agents Remember

Table of Contents

• Why the Context Window Became an Economic Problem
• The Three Cost Curves Nobody Warns You About
• What an Agent Actually Holds in Context
• The Core Strategies for Managing Agent Memory
• Compaction and Summarization
• Retrieval Over Retention
• Eviction and Windowing
• Prompt and Cache Discipline
• Context Rot: The Accuracy Tax of a Full Window
• How This Shows Up in GaaS Pricing
• Designing a Context Budget for a Production Agent
• Insights Most People Overlook
• Frequently Asked Questions
• Conclusion
• References

Why the Context Window Became an Economic Problem

For the first couple of years of the LLM era, the context window was something you bumped into. You pasted a long document, hit the ceiling, and got an error. The fix was always "use a bigger model." Then the windows got enormous, hundreds of thousands of tokens, then a million, and a lot of people quietly assumed the problem had been solved.

It hadn't. It had changed shape.

The moment you put an LLM inside an agent, a loop that calls tools, reads results, and decides what to do next, the window stops being a one-time constraint and becomes a running meter. A single customer-support agent resolving one ticket might make twelve model calls. Each call re-sends the accumulated conversation, the tool definitions, the system prompt, the retrieved documents, and the growing pile of tool outputs. By call number ten, you are paying to re-read the same history nine times over. The window isn't a wall anymore. It's a tab that keeps growing while the agent works.

This is the heart of what I'll call the context-window economy: in Agentic AI-as-a-Service, the thing you are really selling is the difference between the value of an outcome and the cost of the tokens it took to produce. Context management is where that margin lives or dies. It sits squarely in the infrastructure-and-orchestration layer of the GaaS stack, alongside the broader questions of memory systems and the architecture options behind them and state management for stateful agents, but it deserves its own treatment because it's the one cost line that scales super-linearly with task difficulty.

The Three Cost Curves Nobody Warns You About

When operators model agent costs, they usually multiply tokens by a per-token price and call it a day. That misses three curves that compound.

The quadratic re-read curve. In an agent loop, the conversation accumulates. If each step adds roughly the same amount of context and you re-send everything each step, total tokens consumed across an N-step task grow on the order of N². A ten-step task doesn't cost ten times a one-step task, it can cost closer to fifty, because the early context gets paid for again and again. This is the single most under-modeled line in agent economics, and it's why a task that looks cheap in a demo becomes alarming at scale.

The latency curve. Bigger contexts take longer to process. Time-to-first-token climbs with input length, and for a user-facing agent that delay stacks at every step. A bloated window doesn't just cost dollars; it costs the perceived responsiveness that determines whether anyone keeps using the product. This ties directly into where agent time actually goes in the latency budget.

The accuracy curve. This is the counterintuitive one. More context does not monotonically improve answers. Past a certain fill level, models start to lose track of what matters, a phenomenon documented in research on how models struggle to use information buried in the middle of long inputs, summarized well in the study Lost in the Middle: How Language Models Use Long Contexts. A window stuffed to 90% capacity often produces worse results than the same task run with a tight, curated 30%.

Three curves, all pointing the same direction: less context, managed deliberately, beats more context dumped in.

What an Agent Actually Holds in Context

Before you can manage memory, you have to know what's competing for space. A typical production agent's window at any given step contains a predictable mix:

• System prompt and policies, the agent's instructions, guardrails, persona. Static, but often bloated with rules that rarely fire.
• Tool definitions, JSON schemas for every tool the agent can call. These are sneaky-expensive; an agent with thirty tools can spend thousands of tokens just describing them before doing anything.
• Conversation history, the running back-and-forth, including the user's turns and the agent's own reasoning.
• Tool outputs, the results of API calls, database queries, file reads. These are the biggest and least predictable contributors. One verbose API response can swallow your whole budget.
• Retrieved knowledge, documents pulled in from a vector store or other source to ground the answer, which connects to the broader question of retrieval for agents beyond basic RAG.
• Scratchpad / working memory, intermediate notes, plans, and partial results the agent keeps for itself.

The discipline of deciding what goes into this mix, in what form, at each step, has acquired a name in practitioner circles: context engineering. Anthropic's own guidance frames it as a shift from one-shot prompt-writing to managing a dynamic state across a loop, see their note on effective context engineering for AI agents. The reframing matters. Prompt engineering optimizes a single message. Context engineering optimizes the entire window over time, which is a fundamentally harder and more economically loaded problem.

The Core Strategies for Managing Agent Memory

There are four families of technique that production teams reach for. Most mature agents use all four at once.

Compaction and Summarization

When the conversation gets long, replace the old, verbose history with a compressed summary. Instead of carrying twenty turns of raw dialogue, you carry a paragraph capturing the decisions made and facts established, plus the most recent few turns verbatim.

The art is in what you compress and when. Summarize too aggressively and you lose the specific detail an agent needs three steps later, the exact order ID, the precise error message. Summarize too late and you've already paid for the bloat. The best implementations summarize tool outputs immediately on receipt: a 4,000-token API response gets distilled to the 200 tokens the agent actually needs before it ever re-enters the loop. Done well, this single move can cut a long-running agent's token bill by more than half. It's the most direct lever in the cost of context, managing token budgets at runtime.

Retrieval Over Retention

The instinct to keep everything in the window is the expensive one. The cheaper pattern is to store information outside the window and pull it back only when needed. A customer's full history lives in a database; the agent retrieves the three relevant facts for this specific step rather than carrying the entire file.

This is where the relationship between context windows and external memory systems becomes a design decision rather than a default. Retrieval trades token cost for retrieval-system cost and a little latency, usually a good trade, because vector lookups and database reads are far cheaper than LLM token processing. The failure mode is over-retrieval: yanking in ten documents when two would do, which just relocates the bloat from history into retrieved-knowledge. Retrieval is a tool for precision, not volume.

Eviction and Windowing

Sometimes the right move is simply to drop things. Sliding-window approaches keep the last K turns and discard older ones outright. More sophisticated eviction scores each item in context by relevance and recency, evicting the lowest-value entries when the budget tightens, essentially a cache-replacement policy applied to agent memory.

Eviction is blunt but cheap, and it pairs naturally with persistence: what you evict from the live window can be written to durable storage, so it's recoverable via retrieval if it turns out to matter. That handoff between hot context and cold storage is one of the central problems of state management for stateful agents.

Prompt and Cache Discipline

The least glamorous and highest-ROI lever. Two specifics:

First, trim the static stuff. Tool definitions and system prompts are paid for on every single call. An agent that loads thirty tool schemas when the current task needs four is burning money on every step. Dynamic tool selection, exposing only the relevant tools for the current phase, can reclaim a surprising amount of budget.

Second, use prompt caching. Major providers let you cache the stable prefix of a prompt (system instructions, tool definitions, long static documents) so re-reads are billed at a steep discount. Structuring your context so the stable parts come first and the volatile parts come last turns the quadratic re-read curve from a liability into a largely-discounted line item. This is closely tied to the broader topic of caching strategies that cut agent costs, and it's often the first thing I'd check on an agent with a surprising bill.

Context Rot: The Accuracy Tax of a Full Window

It's worth dwelling on the accuracy point because it inverts the intuition most people bring.

The term making the rounds for this is context rot: as a window fills, the model's ability to reliably use any given piece of information in it degrades. It's not a hard cliff. It's a gradual erosion, the signal you care about gets diluted by everything else you crammed in. An agent that has accumulated fifteen tool outputs, three of them contradictory and most of them irrelevant to the current decision, is an agent that is more likely to make a mistake than one working from a clean, curated context.

This reframes context management from a pure cost play into a quality play. The same discipline that saves money, compress, retrieve narrowly, evict ruthlessly, also makes the agent smarter. That alignment is rare in engineering, where cost and quality usually trade off against each other. Here they pull the same way, which is exactly why context management deserves first-class attention rather than being treated as an optimization you bolt on later. When an agent's reliability mysteriously sags as conversations lengthen, context rot is the first suspect, and it connects directly to tool-calling reliability at the infrastructure layer, since a confused context produces malformed tool calls.

How This Shows Up in GaaS Pricing

For anyone selling agents as a service, the context-window economy is the difference between a healthy gross margin and a money-losing one, especially under outcome-based or per-task pricing, where you've fixed your revenue per task but your costs float with how much context the agent burns.

Consider per-outcome pricing: you charge a flat fee to resolve a support ticket. A well-managed agent resolves it in eight steps with a tight context and earns you margin. A poorly-managed one wanders for twenty steps, re-reading a swollen history each time, and the same flat fee now barely covers the inference bill, or doesn't. The variance between best- and worst-case context behavior is, in practice, the variance in your unit economics. This is one of the load-bearing realities in the GaaS infrastructure cost stack, decomposed.

The strategic implication: context management isn't a back-office optimization. It's a pricing prerequisite. You cannot responsibly quote per-outcome pricing until you understand and have bounded your context costs, because an unbounded context budget means an unbounded cost of goods sold. The providers who can offer aggressive outcome pricing are, almost always, the ones who've gotten ruthless about what their agents remember.

Designing a Context Budget for a Production Agent

Treat context like a financial budget with line items and a ceiling. A practical approach:

1. Set a token ceiling per step, well below the model's maximum, often 30-50% of capacity, both to control cost and to stay clear of context rot.
2. Allocate the budget across categories. Reserve fixed amounts for system prompt, tool definitions, retrieved knowledge, and working history. When a category exceeds its allocation, a policy kicks in, summarize, evict, or retrieve more narrowly.
3. Instrument everything. You cannot manage what you don't measure. Track tokens-per-step, where they're going by category, and how it correlates with task success. This is a core feature ask for the observability stack for agent infrastructure.
4. Test at length, not just at the happy path. Most context problems only appear in long-running tasks. An agent that's lean for three steps can be ruinous for thirty. Simulate the long tail before it hits production.

The teams that do this well stop thinking of context as something that simply happens and start treating it as something they design, a deliberate, budgeted, instrumented part of the agent runtime rather than an emergent side effect of the loop.

Insights Most People Overlook

Bigger context windows made the problem worse, not better. When the ceiling was low, the constraint forced discipline, you had to be selective. Million-token windows removed the forcing function, so teams stopped curating and started dumping, and their costs and accuracy quietly degraded. The expanded window is a loaded gun pointed at your margins. The discipline that scarcity used to impose now has to be imposed deliberately.

Tool definitions are the silent budget killer. Everyone obsesses over conversation history and retrieved documents. Almost nobody audits their tool schemas, which get re-sent on every single call and are often verbose, redundant, and full of tools the agent will never use in the current task. Dynamic tool exposure is one of the highest-ROI, least-discussed optimizations in the whole stack.

Cost optimization and quality optimization are the same project here. This is genuinely unusual. In most engineering, making something cheaper makes it worse. With context, the same moves, compress, retrieve narrowly, evict, both cut cost and reduce context rot. A team told to "improve agent accuracy" and a team told to "cut agent costs" should, if they're competent, end up writing nearly identical code.

The summary you write is a new attack surface and a new failure mode. Compaction quietly rewrites the agent's memory. If your summarizer drops a critical constraint, the agent forgets a rule it was supposed to follow, and you'll never see it in the logs, because the logs only show the summary. A bad summary doesn't error; it silently makes the agent wrong. Treat the summarization step with the same rigor you'd give any other reliability-critical component.

"Just use a bigger model" is often the most expensive possible answer. When an agent underperforms, the reflex is to upgrade to a larger, pricier model. But if the real problem is a rotted, bloated context, a bigger model just pays more to be confused by the same noise. Fixing context first is frequently cheaper and more effective than the model upgrade, and it's the move a cheap-model-when-you-can routing strategy depends on to work at all.

Frequently Asked Questions

Is context management still necessary now that windows hold a million-plus tokens? More than ever. Large windows removed the hard error but kept, and amplified, the cost and accuracy penalties. Filling a million-token window every step is financially ruinous and, thanks to context rot, often less accurate than a curated fraction of it.

What's the difference between context management and an agent's memory system? The context window is the working memory the model sees right now, paid for on every call. A memory system is the durable store that lives outside the window. Context management is the policy that decides what moves between the two, what gets loaded in, what gets written out, and when.

How much can good context management actually save? For long-running, multi-step agents, summarizing tool outputs and disciplined caching commonly cut token bills by half or more, sometimes far more. The savings scale with task length, because the quadratic re-read curve is exactly what these techniques flatten.

Does prompt caching make context management unnecessary? No, it makes one slice of it cheaper. Caching discounts re-reads of the stable prefix. It does nothing for the volatile, growing parts of context, which is precisely where bloat and rot accumulate. Caching and active management are complements, not substitutes.

Why would less context produce better answers? Because models lose the thread in large inputs, the relevant signal gets diluted by surrounding noise, and information buried mid-window is used less reliably than information at the edges. A tight, relevant context gives the model less to get distracted by.

How do I know if my agent has a context problem? Watch for three tells: token cost that grows faster than task complexity, accuracy that degrades as conversations lengthen, and latency that creeps up over the course of a task. Any of the three points back at the window. Instrumenting tokens-per-step by category usually surfaces the culprit fast.

Conclusion

The context window stopped being a wall and became a meter the moment we wrapped LLMs in agent loops. In Agentic AI-as-a-Service, that meter runs against your margin on every step, and it compounds, a quadratic re-read curve in cost, a creeping curve in latency, and a counterintuitive curve where a fuller window produces worse answers. The teams that win treat context as a scarce budget to be designed, allocated, and instrumented: compress aggressively, retrieve narrowly, evict ruthlessly, and cache the stable parts.

The deeper lesson is that context management is where cost and quality stop fighting each other. The same discipline that protects your unit economics also protects your accuracy, which makes it one of the few genuinely free lunches in the agent stack, free only if you do the work. As the broader GaaS infrastructure layer matures around orchestration, memory architecture, and runtime, the operators who understand the context-window economy will be the ones who can price per outcome and actually keep the difference. Everyone else will be selling tasks at a loss and wondering where the margin went.

References

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al.)
• Effective Context Engineering for AI Agents (Anthropic)