Modeling Worst-Case Spend: The Runaway-Agent Budget Scenario

Table of Contents

• Why Average Spend Is the Wrong Number to Watch
• Anatomy of a Runaway Agent
• The Multiplicative Cost Stack
• A Back-of-Envelope Worst-Case Formula
• Worked Example: A Research Agent Goes Feral
• The Caps That Actually Stop the Bleeding
• Who Eats the Cost When It Happens
• Insights Most People Overlook
• References

Why Average Spend Is the Wrong Number to Watch

Most teams selling or buying agents budget the way they budget cloud compute: take expected volume, multiply by average cost-per-task, add a fudge factor, call it done. That works right up until it doesn't. The trouble with agents is that their cost distribution has a long, ugly tail. Ninety-five percent of your tasks might cost eight cents and finish in four model calls. The other five percent are the ones that decide whether your gross margin survives the quarter.

Cloud taught a generation of operators to think in averages because a virtual machine costs roughly the same whether it's doing useful work or spinning idle. An agent is different. Its cost is a function of how many decisions it makes, and a misbehaving agent makes a near-unbounded number of them. That is a categorically different risk profile, and it deserves a categorically different budgeting discipline, one borrowed less from spreadsheets and more from how insurers model catastrophic loss.

The practical failure mode is this: you set a monthly budget based on the median, the agent fleet runs fine for six weeks, and then one prompt injection, one malformed tool response, or one ambiguous instruction sends a handful of agents into loops that each cost three hundred dollars before anyone notices. Your "average" was never wrong. It just never described the thing that hurt you. Worst-case modeling exists precisely to put a number on that tail before the tail puts a number on you.

Anatomy of a Runaway Agent

A runaway isn't usually a dramatic AI-gone-rogue story. It's mundane. The most common pattern is the retry loop: the agent calls a tool, the tool returns an error or an empty result, the agent reasons "let me try a slightly different approach," and tries again, forever, because nothing in its design tells it when to quit. Each iteration is a fresh, expensive model call, often with the entire growing conversation history attached.

The second pattern is fan-out without a budget. An orchestrator agent decides a task needs five sub-agents, each of those decides it needs three more, and now you have a tree of agents that grows geometrically. We cover the steady-state version of this in the fan-out economics discussion elsewhere in this cluster, but the runaway version is worse: a bug in the spawning logic means the tree never stops branching.

The third is context bloat feeding itself. Long-running agents accumulate history. The bigger the context, the more each call costs, and the slower the agent reasons, which makes it more likely to take wrong turns, which generates more history. It's a flywheel pointed in the wrong direction.

What ties all three together is the absence of a hard stop. Anthropic's own guidance on building agents stresses giving models explicit tools to signal completion and bounded loops rather than open-ended autonomy, precisely because models will happily keep working a problem long past the point of diminishing returns. You can read the reasoning in Anthropic's engineering guide to building effective agents. The runaway isn't a malfunction of the model. It's a malfunction of the harness around it.

The Multiplicative Cost Stack

Here's the part operators underestimate: runaway cost drivers don't add, they multiply. That's why worst-case spend is so far above the average, you're not summing a few bad factors, you're taking their product.

Walk the stack:

• Iterations (N): How many times around the loop before something stops it. In a well-bounded agent this is capped at, say, 10. In a runaway with no cap, it's bounded only by your wallet or a timeout.
• Tokens per iteration (T): Because history accumulates, T grows with N. Iteration fifty doesn't cost the same as iteration one, it might cost ten times as much because it's dragging fifty turns of context.
• Fan-out factor (F): Each agent may spawn sub-agents. A branching factor of three over four levels is 81 leaf agents, each running its own loop.
• Tool-call cost (C): Many tools aren't free. A search API, a data-enrichment vendor, a code-execution sandbox, each loop iteration may trigger one or several paid calls. These stack and compound across the workflow in ways worth their own treatment.
• Model tier (M): A runaway that escalates to a frontier reasoning model on every retry costs an order of magnitude more per token than one stuck on a cheap model.

Your worst case is roughly the product of these, not the sum. That's the whole insight. An agent that's 10x worse on iterations, 5x worse on tokens-per-call, and 3x worse on fan-out isn't 18x your expected cost, it's something closer to 150x. McKinsey's analysis of the economic potential of generative AI frames agentic systems as a step-change in automation value, but the flip side of that leverage is that the cost function inherits the same multiplicative shape. Leverage cuts both directions.

A Back-of-Envelope Worst-Case Formula

You don't need a simulation to get a usable ceiling. You need a pessimistic estimate for each factor and the discipline to multiply them. Here's a formula I'd actually use in a budget review:

Worst-case cost per triggered task ≈
 N_max × F_max × (T_avg × G) × price_per_token(M_max) + (N_max × F_max × C_tool)

Where: - N_max = the maximum iterations your harness permits (if there's no cap, this number is "infinity" and you should stop reading and go add a cap) - F_max = maximum concurrent sub-agents in the worst branching case - T_avg × G = average tokens per call multiplied by a growth factor G that accounts for context accumulation (use 3-5x if you don't trim history) - price_per_token(M_max) = the per-token price of the most expensive model the agent can escalate to - C_tool = total paid tool-call cost per iteration

Then multiply that per-task worst case by the number of tasks that could plausibly run away simultaneously. That last step is the one people skip, and it's the one that turns a $300 incident into a $30,000 incident. If a bad deploy or a poisoned input affects the whole fleet at once, your concurrency isn't one, it's your entire active agent count.

The output isn't a prediction. It's a fire-code occupancy limit: the number you architect against so that even the bad day stays survivable. If that number is larger than your runway, you have a design problem, not a forecasting problem.

Worked Example: A Research Agent Goes Feral

Make it concrete. Say you run a research agent that, on a normal task, does 6 iterations, averages 8,000 tokens per call on a mid-tier model at roughly $3 per million input tokens, spawns no sub-agents, and makes one paid search call (2 cents) per iteration. Normal cost: about 6 × 8,000 × $0.000003 + 6 × $0.02 ≈ $0.14 plus $0.12 = roughly $0.26 per task. Comfortable.

Now the bad day. A malformed source keeps returning partial data, so the agent retries. With no iteration cap it runs 200 times before a wall-clock timeout. Context has ballooned, so tokens-per-call have grown 4x to 32,000. A logic bug also has it spawning 3 sub-agents, each doing the same thing. And on each retry it escalated to a frontier model at $15 per million tokens.

Per leaf agent: 200 × 32,000 × $0.000015 + 200 × $0.02 ≈ $96 + $4 = $100. Times 4 agents (orchestrator plus three) ≈ $400 for one task that should have cost a quarter. Now imagine the malformed-source condition hits 50 active tasks before your alerting fires. You're looking at a $20,000 afternoon from a single bad input, a roughly 1,500x blowout over the expected per-task cost across that batch.

Nothing in that scenario required the model to do anything malicious or even unusual. Every individual decision was locally reasonable. The cost came entirely from the absence of bounds.

The Caps That Actually Stop the Bleeding

Modeling the worst case is only useful if it drives design. The good news is that the same multiplicative structure that makes runaways dangerous makes them cheap to defang, cap any one factor hard and you collapse the whole product. The caps worth building, roughly in order of return on effort:

• A hard iteration ceiling per task. Non-negotiable. Every agent loop gets a max-steps counter that terminates and escalates to a human or a failure state. This single cap converts "infinity" into a finite N and is the highest-leverage control you can ship.
• A per-task spend budget enforced at the harness level. Track cumulative cost mid-run and kill the task when it crosses a threshold. This is the financial equivalent of a circuit breaker, and it should live in code, not in a model instruction the agent can rationalize its way around.
• A fan-out limit. Cap total sub-agents per workflow and per depth level. A branching factor with no depth limit is a fork bomb wearing a trench coat.
• Context trimming. Bound how much history rides along on each call so T can't grow unbounded. Summarize-and-discard beats carry-everything for long-running work.
• Fleet-level anomaly alerting. Per-task caps stop a single runaway; they don't stop a thousand small runaways triggered by the same root cause. You need spend-rate monitoring across the fleet that pages a human when burn deviates from baseline, a topic worth a dedicated build in this cluster.

Worth naming a tension here: there's a quiet movement among agent startups to cap autonomy specifically to protect margin, and worst-case modeling is the honest accounting behind that decision. Every cap you add trades a sliver of capability for a chunk of cost predictability. That's not a compromise to be embarrassed about. It's the actual job.

Who Eats the Cost When It Happens

The budget model is incomplete until you answer the contractual question: when a runaway happens, whose money was it? This is where agent economics gets uncomfortable, because the answer is usually "the vendor, and they didn't price for it."

If you sell per-task or per-outcome, you quoted the customer a price based on the median. The runaway's cost lands on your COGS, not their invoice. One feral task can wipe out the margin on hundreds of healthy ones. That's the structural reason the margin trap of "we'll just pass through model costs" is a trap, pass-through sounds fair until a runaway makes the customer's bill incomprehensible and they churn, or until your fixed-price contract eats a loss you can't pass on.

The mature move is to price the tail in. Build the worst-case expected value, probability of a runaway times its cost, into your unit economics as a real line item, the way an insurer prices expected claims into a premium. a16z's writing on how AI is reshaping software business models and gross margins keeps circling the same point: agentic products carry variable COGS that behaves nothing like traditional SaaS, and pretending otherwise just defers the reckoning to a worse moment. If your pricing can't absorb a few runaways per thousand tasks without going underwater, your pricing is wrong, not your luck.

Insights Most People Overlook

The worst case is correlated, not independent. Naive models treat runaways as random isolated events, one bad task here, one there. In reality the trigger (a bad deploy, a poisoned input, an upstream API returning garbage) hits the whole fleet at once. Your worst-case concurrency isn't 1, it's N. Budget for the correlated blowout, because that's the one that actually shows up.

Falling token prices make runaways more likely, not less. Cheaper inference removes the natural economic friction that used to bound agent loops. When tokens were expensive, a verbose agent got noticed fast. As prices fall, the same loop can run far longer before the bill gets anyone's attention, so the runaway grows larger before detection. Cheaper unit cost, bigger blast radius.

A per-task spend cap is worth more than a faster model. Teams spend weeks optimizing average cost-per-task and skip the one-afternoon job of adding a hard budget breaker. The cap caps your downside tail, which is where the money actually lives. Median optimization is a rounding error next to tail control.

"Thinking" tokens are the silent multiplier. Reasoning models that deliberate at length are exactly the ones most prone to expensive loops, because their per-iteration cost is already high and they're inclined to keep deliberating. A runaway on a reasoning model isn't 2x worse than on a standard model, it's frequently 10x, because both N and the per-token price climb together.

Your timeout is your real spending limit, so price it like one. Most teams set a wall-clock timeout for latency reasons and never connect it to cost. But until you have a spend cap, the timeout is your budget ceiling: worst-case spend equals burn rate times timeout duration. A 30-minute timeout on a fast frontier model is a four-figure authorization you signed without reading.

References

• Building Effective Agents, Anthropic Engineering
• The Economic Potential of Generative AI: The Next Productivity Frontier, McKinsey
• The Economic Case for Generative AI and Foundation Models, Andreessen Horowitz (a16z)