Caching Strategies That Cut Agent Costs (Without Wrecking Reliability)

Table of Contents

• Why Caching Is the Cost Lever That Matters Most for Agents
• The Three Layers Where Agent Caching Lives
• Layer 1: Provider-Side Prompt Caching (Start Here)
• How KV-Cache Reuse Actually Works
• Prompt Ordering: The Mistake That Kills Your Hit Rate
• Layer 2: Semantic Response Caching
• The Similarity-Threshold Trap
• Layer 3: Tool-Result and Retrieval Caching
• The Economics: What Each Layer Saves
• When NOT to Cache
• A Practical Implementation Order
• Insights Most People Overlook
• References

Why Caching Is the Cost Lever That Matters Most for Agents

If you sell agents on per-task or per-outcome pricing, your margin is the gap between what a task earns and what its tokens cost. The problem is that agentic workflows are token-hungry in a way ordinary chat completions never were. A single autonomous task might loop a dozen times, re-sending the full system prompt, tool definitions, and accumulated conversation on every turn. A ReAct-style agent that does eight tool calls can re-process the same 4,000-token system prompt eight times in one task. You're paying for the same input over and over.

This is why caching, not model selection, is usually the first place to look when an agent's unit economics are underwater. Switching from a frontier model to a cheaper one (covered in the model-routing layer discussions in this cluster) helps, but it's a blunt instrument that trades quality for price. Caching is different: done right, it cuts cost with no quality loss at all, because you're returning bytes you already computed rather than computing worse ones.

The numbers are not subtle. Provider prompt caching alone routinely cuts input costs on the cached portion by 50-90%. Anthropic's prompt caching documentation prices cache reads at one-tenth of base input tokens. OpenAI's automatic prompt caching, described in its prompt caching guide, discounts cached input by 50% with zero code changes. For an agent that re-sends a large stable prefix every turn, that discount applies to the bulk of every request.

The Three Layers Where Agent Caching Lives

The word "caching" gets thrown around as if it's one thing. In an agent stack it's at least three distinct mechanisms, each with its own savings profile, risk profile, and place in the architecture:

1. Provider-side prompt caching, the inference provider keeps the computed attention state (the KV cache) for a prefix you've sent before, so it skips recomputing it. Cheap, low-risk, and you should turn it on today.
2. Semantic response caching, you store full model outputs keyed by the meaning of the input, and serve a stored answer when a new request is close enough. High savings, real correctness risk.
3. Tool-result and retrieval caching, you cache the outputs of the tools and retrievals the agent calls, not the model's tokens at all. The most overlooked layer, and often the cheapest win.

Mixing these up is where teams go wrong. Prompt caching and semantic caching solve different problems and fail in different ways. Treat them as one knob and you'll either under-save or quietly serve wrong answers.

Layer 1: Provider-Side Prompt Caching (Start Here)

How KV-Cache Reuse Actually Works

When a transformer processes your prompt, it computes a key-value representation for every token, the "KV cache." For a 5,000-token prompt, that's a meaningful chunk of the compute bill. Provider prompt caching stores that computed state for a short window (typically 5 minutes to an hour, refreshed on each hit) and reuses it when your next request starts with the exact same tokens. You pay a small premium to write the cache, then a steep discount on every read.

The critical word is exact. KV-cache reuse is prefix-based and token-exact. The cache matches from the start of your prompt up to the first token that differs. Change one character near the top and everything after it is a cache miss. This single fact drives every prompt-caching best practice that follows.

For agents, this is a gift, because agents send enormous stable prefixes: a system prompt, a block of tool/function definitions, few-shot examples, maybe a chunk of retrieved policy documents. Those rarely change across the turns of a task. Mark them as cacheable and the agent stops paying full freight to re-read its own instructions on every loop.

Prompt Ordering: The Mistake That Kills Your Hit Rate

Here's the error I see most often: teams put the variable content, the user's latest message, a timestamp, a turn counter, at the top of the prompt, then the stable system instructions below. That destroys caching. Because matching is prefix-based, any variability near the start invalidates the cache for everything after it.

The fix is structural and free. Order your prompt most-stable-first, most-variable-last:

• System prompt and role definition (never changes), top
• Tool / function definitions (stable per agent version), next
• Few-shot examples and static context (stable), next
• Retrieved documents (stable within a task), next
• Conversation history (grows by appending, which preserves the prefix), next
• The current user turn (always new), bottom

Append-only conversation history is the subtle win: because you only add to the end, the previous turns remain a valid cached prefix. If you instead summarize or rewrite history mid-conversation, a common context-window management tactic, you break the prefix and force a full recompute. There's a real tension between trimming context to save tokens and preserving it to keep the cache warm; the right answer depends on whether your recompute cost exceeds the tokens you'd save by trimming. Measure it, don't assume.

Two more practical notes. First, cache windows are short and sliding, so caching helps high-frequency agents far more than ones that fire once an hour, an idle agent's cache expires before the next call. Second, on providers with manual cache breakpoints (Anthropic), you choose where the cacheable boundary sits; on providers with automatic caching (OpenAI), you control it purely through ordering. Either way, the discipline is the same: stable stuff first.

Layer 2: Semantic Response Caching

Prompt caching reuses computation within similar requests. Semantic caching skips the model entirely. You embed the incoming request, search a vector store of past request embeddings, and if you find a sufficiently similar prior request, you return its stored answer, no inference at all. The savings here aren't a discount; they're the entire request cost going to zero.

For high-volume agents fielding repetitive questions, a support agent answering "how do I reset my password" phrased four hundred different ways, this is transformative. Tools like GPTCache popularized the pattern, and most production setups now build it on a vector database (a topic the vector databases in the agent stack piece in this cluster digs into).

The Similarity-Threshold Trap

Semantic caching is where good intentions corrupt outputs, and the culprit is almost always the similarity threshold. Set it too loose and the cache returns the answer to a different question because the embeddings looked close. "How do I cancel my subscription?" and "How do I change my subscription?" can sit dangerously near each other in embedding space, yet demand opposite answers. A loose threshold turns your cost-saving cache into a confident-liar machine.

A few hard-won rules:

• Tune the threshold against real misfire data, not vibes. Pull actual query pairs, look at which ones your threshold would collapse together, and check whether collapsing them is correct. The right cutoff is empirical and domain-specific.
• Never semantically cache anything personalized, stateful, or time-sensitive. Account balances, order status, "what did I just ask you", these must miss the cache. The agent's own state makes the same surface question have different correct answers per user and per moment.
• Cache the retrieval-and-reasoning step, not the user-facing final answer, when you can. Caching an intermediate that still gets personalized downstream is far safer than serving a frozen final response.
• Add a cheap verification pass for high-stakes domains. A small, fast model checking "does this cached answer actually fit this question?" costs a fraction of a full generation and catches the worst misfires. This connects to broader agent reliability practices in the cluster.

The honest framing: semantic caching trades a correctness risk for a cost saving. That trade is great for FAQ-style breadth and terrible for anything where a wrong answer has consequences. Know which side of that line your agent is on before you ship it.

Layer 3: Tool-Result and Retrieval Caching

This is the layer most teams never instrument, and it's often the cheapest win of all. Agents don't just generate tokens, they call tools, hit APIs, run searches, and query databases. Each of those calls has latency and frequently a dollar cost (a paid search API, a metered data provider). If an agent calls the same weather API or runs the same database query three times in one task, you're paying three times for one answer.

Tool-result caching sits in the orchestration layer, between the agent and its tools, and it's standard cache engineering: key on the tool name plus its arguments, return the stored result if it's fresh. Because tool inputs are usually structured and discrete (get_weather(city="Denver")), you get clean exact-match keys, no fuzzy embedding similarity, no correctness ambiguity. The only real decision is the time-to-live, and that follows from how fast the underlying data changes. Stock prices: seconds. A product catalog: hours. A company's org chart: days.

A worked example. An agent that researches a company might call get_company_profile, get_recent_news, and get_financials. Across a multi-turn task it may revisit those tools as it reasons. Cache the profile for a day, the news for an hour, the financials until the next filing, and you collapse a dozen API calls into three, often the difference between a profitable per-outcome task and a losing one. This is also where retrieval caching lives: if your agent runs the same vector search repeatedly, cache the retrieved chunks rather than re-querying the index every loop.

The Economics: What Each Layer Saves

Rough, real-world ranges so you can prioritize:

• Prompt caching: 50-90% off the input portion that's cacheable. Since agents are input-heavy (big system prompts, tool defs, history), this commonly translates to a 30-60% cut in total token spend per task. Risk: essentially zero. Effort: low.
• Semantic caching: Up to 100% savings on every cache hit, but only on the hit rate you achieve. A 30% hit rate on repetitive workloads cuts roughly a third of inference cost. Risk: high if untuned. Effort: medium.
• Tool-result caching: Eliminates duplicate tool and API costs and their latency. Savings vary wildly by how tool-heavy the agent is, but for research and data-gathering agents it can rival prompt caching. Risk: low (exact-match keys). Effort: low-to-medium.

There's a latency dividend on top of the dollars: every cache hit is also a faster response, which compounds when an agent loops. McKinsey's analysis of the economic potential of generative AI frames the value of these workflows in trillions, but that value only materializes if the per-task cost structure works, and caching is how it works. The infrastructure layer turning these agents into a viable service, as a16z lays out in its writing on the emerging architectures for LLM applications, treats caching as a first-class component, not an afterthought.

When NOT to Cache

Caching is not free virtue. There are cases where it actively hurts:

• Genuinely novel, one-shot tasks. If every request is unique, bespoke analysis, creative generation with no repetition, semantic caching just adds an embedding lookup that always misses. You pay overhead for nothing.
• Anything where freshness is the product. Real-time monitoring agents, trading agents, anything where a five-minute-old answer is a wrong answer. Cache TTLs here approach zero, which means don't bother.
• Low-volume agents. Provider prompt caches expire in minutes. An agent that fires twice an hour never gets a warm cache, so the cache-write premium can cost more than it saves.
• Compliance-sensitive flows. Caching responses that contain regulated or personal data creates a data-retention surface you now have to govern. Sometimes the cleanest move is to not store it at all, a point the memory persistence and privacy discussion in this cluster develops.

The meta-point: caching's value is a function of repetition and tolerance for staleness. High repetition plus high staleness-tolerance equals cache aggressively. Low on either axis, and you should think twice.

A Practical Implementation Order

If you're standing up caching for a GaaS product, do it in this sequence, risk-ascending, so you bank the safe wins first:

1. Reorder prompts stable-first and enable provider prompt caching. Near-zero risk, immediate 30-60% input savings. Do this before anything else.
2. Add tool-result caching in the orchestration layer. Exact-match keys, per-tool TTLs, low risk, often surprising savings on tool-heavy agents.
3. Instrument hit rates and cost per task so the next two steps are measured, not guessed.
4. Introduce semantic caching carefully, scoped to non-personalized, repetitive query classes, with an empirically tuned threshold and a verification pass for anything that matters.
5. Set TTL and invalidation policy deliberately per data source, and monitor for staleness complaints as a leading indicator that a TTL is too long.

Run them in that order and you capture most of the savings in the first two steps, with the riskiest layer last and fully observable.

Insights Most People Overlook

Aggressive context-trimming and prompt caching are at war with each other. The instinct to summarize conversation history to save tokens can quietly destroy your KV-cache hit rate, because rewriting the prefix forces a full recompute. On a high-frequency agent, keeping the history append-only and letting the cache discount it is often cheaper than trimming it, the exact opposite of the conventional "shrink the context" advice. The only way to know is to measure recompute cost against trim savings for your specific traffic.

Semantic caching doesn't just save money, it silently homogenizes your agent's voice. Every cache hit returns a frozen past answer, so the more your cache works, the more your agent repeats itself verbatim across users. For a support agent that's fine. For anything where freshness or personalization is part of the perceived quality, a high hit rate is a stealth quality regression you won't see in your cost dashboard.

The cheapest cache is the one in your tool layer, and it's the one nobody builds. Teams obsess over token-level caching while their agent calls the same paid API four times per task. Tool-result caching has the best risk-to-savings ratio of any layer, exact-match keys, no embedding ambiguity, yet it's routinely skipped because it's not what "LLM caching" articles talk about.

Cache TTL is a product decision disguised as an infrastructure setting. Whoever picks the TTL is implicitly deciding how stale an answer your customer will tolerate. That's not a backend engineer's call to make alone, it belongs with whoever owns the outcome you're being paid for, especially under per-outcome pricing where a stale answer can mean a refunded task.

Prompt caching changes which model is actually cheapest. Model-routing math usually compares raw per-token prices. But once a large stable prefix is cached at a 90% discount, a "more expensive" model with strong caching can beat a "cheaper" model without it on real agent workloads. Re-run your routing decisions with caching turned on; the ranking can flip.

References

• Anthropic, Prompt Caching documentation
• OpenAI, Prompt Caching guide
• McKinsey, The economic potential of generative AI: The next productivity frontier
• Andreessen Horowitz (a16z), Emerging Architectures for LLM Applications