Benchmarking Inference Spend Across the Top 10 Agent Platforms (And Why the Numbers Lie)

Table of Contents

• Why "inference spend" is the wrong number to compare
• The benchmark nobody runs correctly
• How the top 10 platforms differ structurally
• Frontier-model orchestrators
• Open-weight and self-hosted stacks
• Vertical and managed-agent vendors
• A reproducible benchmarking method
• Reading the results: cost per completed task, not per token
• Where the spend actually hides
• Insights Most People Overlook
• References

Why "inference spend" is the wrong number to compare

Walk into any procurement meeting for an agent platform and the first slide is almost always a price-per-million-tokens table. It feels rigorous. It is nearly useless.

Here's the problem. Two platforms can quote the identical per-token rate and still produce a 6x difference in your monthly bill, because the thing that determines your spend is not the price of a token but how many tokens the agent decides to spend on your behalf. An agent that reasons in long chains, re-reads its own context on every step, and retries a failed tool call three times will consume an order of magnitude more tokens than a tightly-scoped agent doing the same job. The token price is a rounding error next to the token volume, and token volume is a function of the platform's architecture, not its price sheet.

This is the central deceit of inference benchmarking. The number that's easy to measure (price per token) is the one that barely matters, and the number that matters (tokens consumed per successful outcome) is the one almost nobody publishes. If you're building or buying in this category, internalize that asymmetry before you do anything else.

I've watched teams sign a contract with the "cheapest" platform on a per-token basis and then spend the next quarter wondering why their inference line item tripled the forecast. The cheap tokens were real. The agent just used a lot more of them.

The benchmark nobody runs correctly

A proper inference-spend benchmark has to answer one question: for a fixed set of representative tasks, what does each platform cost to reach a correct outcome? Everything else is noise.

That means three things have to be held constant across platforms, and almost no public benchmark holds all three:

1. The task set. Identical inputs, identical success criteria. If platform A is graded on easy tickets and platform B on hard ones, the comparison is fiction.
2. The success bar. A task that "completes" but produces a wrong answer is not free, you pay for the tokens and you pay for the human who cleans it up. Spend has to be measured per correct completion, which ties directly to task completion rate versus genuine success rate.
3. The full call graph. You must count every model call the agent makes, including the ones it makes to other models, the retries, the planning steps, and the tool-call formatting tokens. Counting only the "main" call is how vendors make expensive agents look cheap.

When you enforce all three, the rankings scramble completely. Platforms that looked cheap on a price sheet move to the bottom. Platforms with higher token prices but disciplined architectures move up. McKinsey's work on the economic potential of generative AI repeatedly makes the point that value comes from workflow redesign rather than raw model access, and the same logic applies here, the efficiency of the surrounding workflow dominates the cost of the model itself.

How the top 10 platforms differ structurally

You can't benchmark ten platforms intelligently until you understand that they aren't ten versions of the same thing. They fall into three structural buckets, and the bucket largely predicts the spend profile.

Frontier-model orchestrators

These are the platforms built on top of frontier APIs, the ones wrapping models from the major labs and adding planning, memory, and tool use. Think of the agent frameworks and managed orchestration layers that call into the big providers' published rate cards. Their per-token cost is set by the upstream provider, so the differentiation is entirely in how efficiently they orchestrate.

The good ones aggressively cache context and prune what they re-send. The careless ones replay the entire conversation history on every reasoning step, which means token spend grows quadratically with task length. On a short task you'll never notice. On a 40-step research workflow, the careless orchestrator can cost five times the disciplined one at the same headline rate. Anthropic's own documentation on prompt caching lays out exactly why repeated context is the silent budget killer, and why platforms that ignore caching deserve to score badly.

Open-weight and self-hosted stacks

Platforms running open-weight models on rented or owned GPUs invert the cost structure. There's no per-token invoice; there's a GPU-hour bill. That changes the benchmark math entirely. Your effective cost per token is (GPU hourly rate ÷ tokens produced per hour), which means utilization is destiny. A self-hosted stack at 90% GPU utilization can undercut every frontier API on the planet. The same stack at 15% utilization, which is depressingly common for spiky agent workloads, is a money fire.

So benchmarking these platforms on "cost per token" is almost meaningless without also benchmarking their utilization under your traffic pattern. A vendor demoing on steady batch traffic will quote numbers you'll never reproduce with bursty, interactive agent demand.

Vertical and managed-agent vendors

The third bucket sells outcomes, not infrastructure, the coding agents, support agents, and SDR agents priced per task or per resolution. Here inference spend is invisible to you by design; it's the vendor's COGS, not your line item. Benchmarking these is a different exercise: you're not measuring their inference efficiency, you're measuring whether their per-outcome price leaves you with margin and leaves them solvent. A vendor pricing below their true inference cost to win the logo is a vendor whose pricing will change. Andreessen Horowitz has written extensively about how AI application gross margins compress when inference is the dominant cost of goods, and that pressure eventually reaches your contract.

A reproducible benchmarking method

If you want numbers you can trust, here's the method I'd run. It's not glamorous, but it survives scrutiny.

Build a golden task set of 50-100 real jobs. Pull them from your actual workload, not a synthetic suite. Include the messy ones, ambiguous inputs, tasks that require a tool call to an external system, tasks where the right answer is "I can't do this." The distribution of difficulty in your real traffic is what determines real spend.

Instrument the full call graph. Every platform worth using exposes token usage per call. Capture input tokens, output tokens, and reasoning/thinking tokens separately, because thinking tokens are billed and frequently dwarf the visible output. Sum across the entire agent run, including sub-agent calls and retries. If a platform won't expose this, that opacity is itself a benchmark result, score it down.

Run each task set three times per platform. Agent spend is non-deterministic. One run tells you nothing; you need a distribution. Report the median and the 90th percentile, because the tail is where runaway spend lives and the tail is what blows up your budget.

Normalize to cost per correct outcome. Grade every run against your success criteria, throw out the tokens spent on wrong answers (or rather, keep them as a cost but attribute zero value), and divide total spend by the count of correct completions. That single number, dollars per correct outcome, is the only honest cross-platform metric. Stanford HAI's AI Index has tracked the steep decline in raw inference prices, which makes it even more important to measure outcome cost rather than token cost; the token has gotten cheap, but the outcome has not gotten proportionally cheaper.

Reading the results: cost per completed task, not per token

When you finally have the table, resist the urge to rank by the average. Look at three columns.

The median cost per correct outcome tells you the typical case. The p90 cost tells you what happens on hard tasks, and the gap between median and p90 is a measure of how predictable your bill will be, a platform with a tight median-to-p90 spread is far easier to forecast and budget against than one with a long, fat tail. And the success rate at that cost keeps you honest, because a platform can win on cost simply by giving up early and producing cheap wrong answers.

The pattern I see again and again: the platform with the lowest median cost is rarely the platform with the lowest p90. The disciplined frontier orchestrators tend to have a slightly higher median (they spend a bit more being careful) but a dramatically tighter tail (they don't spiral). The cheap-token platforms win the median and then lose catastrophically on the 8% of tasks that trigger a retry storm. For most production deployments, the tighter tail is worth more than the lower median, because a predictable bill is a managed business and an unpredictable one is a quarterly fire drill.

Where the spend actually hides

Three places, consistently, and a benchmark that doesn't surface them is incomplete.

Reasoning tokens. Models that "think" before answering can spend more tokens on invisible reasoning than on the visible answer. On a hard task that ratio can run 5:1 or worse. If your benchmark only counts output tokens, you're undercounting the bill by a multiple, and you'll be blindsided on exactly the tasks that matter most.

Retries and the fan-out tax. When a tool call fails or an output fails validation, a well-built agent retries, and each retry is a full additional model call. When an agent spawns sub-agents, each sub-agent carries its own context and its own token bill. A single user-facing "task" can quietly become dozens of model calls. This is where the gap between the demo and the invoice opens up.

Idle and overhead tokens. Memory retrieval, context re-loading, system prompts re-sent on every turn, tool schemas re-serialized into the context window, none of these do visible work, all of them are billed. On long-running agents this overhead can exceed the cost of the actual reasoning. The platforms that win real benchmarks are usually the ones that have engineered this overhead down through caching and context discipline, not the ones with the lowest sticker price.

Insights Most People Overlook

The cheapest platform per token is usually the most expensive per outcome. This inverts so reliably that I'd treat a suspiciously low per-token quote as a warning sign rather than a selling point. Cheap tokens often signal a cheaper model that needs more attempts, more reasoning, and more human cleanup to reach the same result. You pay the difference somewhere, it just moves off the invoice and onto your team.

Token-price deflation is a trap for benchmark planning. Raw inference prices have fallen dramatically year over year, which tempts everyone to assume their agent bills will fall too. They mostly haven't, because agents immediately spend the savings on more reasoning, more retries, and more autonomy. Falling token prices fund rising token consumption. Benchmark on behavior under current pricing, not on the assumption that next year's price cut will rescue your margins.

Non-determinism makes single-run benchmarks actively misleading. Most published "platform X costs $Y per task" claims are based on one run, or a tiny sample. Agent spend has a fat right tail; one run can land anywhere in the distribution. A benchmark without a p90 and a variance figure isn't a benchmark, it's an anecdote with a dollar sign attached.

Opacity is a measurable cost. If a platform won't expose per-call, per-sub-agent token usage, you cannot attribute spend, cannot debug a cost spike, and cannot negotiate from a position of knowledge. Treat instrumentation transparency as a first-class benchmark dimension, not a nice-to-have. The platforms confident in their efficiency are the ones that show you everything.

The vendor's COGS becomes your problem eventually. When you buy per-outcome pricing, the vendor's inference efficiency is invisible to you, until it isn't. A vendor underwater on inference cost will eventually raise prices, cap autonomy to protect margin, or fail. Benchmarking a managed-agent vendor should include a rough estimate of their unit economics, because a price that's too good to be true is a price that won't survive the next funding environment.

References

• The economic potential of generative AI: The next productivity frontier, McKinsey & Company
• Prompt caching, Anthropic Documentation
• Navigating the High Cost of AI Compute, Andreessen Horowitz
• Stanford HAI AI Index Report