The State of Agent Reliability: An Annual Benchmark Report

Table of Contents

• Why This Report Exists
• How We Define and Measure Reliability
• The Headline Numbers
• Reliability by Vertical
• The Benchmark-to-Production Gap
• What Improved This Year
• What Got Worse or Stayed Stuck
• How Leading Vendors Report Reliability
• Insights Most People Overlook
• References

Why This Report Exists

A year ago, "can the agent do the task?" was the interesting question. It isn't anymore. The frontier models underneath these agents can draft a contract, reconcile an invoice, or triage a support queue well enough that capability is no longer the bottleneck. The bottleneck is whether the agent does the task the same way every time, knows when it can't, and tells you when it failed.

That shift is why this annual report is structured around reliability rather than capability. We pulled data from three sources: a survey of 140 GaaS vendors and enterprise buyers, anonymized production telemetry shared by eleven platforms running autonomous agents at volume, and a re-run of common public benchmarks under conditions closer to real deployment. The point isn't to crown a winner. It's to give operators a defensible baseline so that when a vendor says "our agent is 95% reliable," you know exactly which 95% they mean and whether it survives contact with your data.

If you're new to this beat, reliability sits downstream of evaluation and observability, two topics the rest of this cluster covers in depth. This piece is the altitude view. The component skills, building an eval suite, tracing a multi-step run, detecting drift, get their own treatment elsewhere.

How We Define and Measure Reliability

Most reliability disputes are really definition disputes. So here is the frame we used, and it's worth borrowing.

We separate reliability into four measurable axes:

• Task success rate, did the agent achieve the user's actual goal, judged by an independent rubric, not by whether it produced an output. An agent that confidently returns a wrong answer scores zero here, not partial credit.
• Consistency, run the same task ten times. How often do you get a materially equivalent result? This is where reproducibility problems surface, and it's the axis vendors quietly hate because temperature, tool ordering, and model updates all erode it.
• Failure transparency, when the agent can't complete a task, does it say so and escalate, or does it fabricate completion? We treat a silent failure as worse than an honest refusal.
• Recovery, after a tool-call error or a flaky API, does the agent retry sensibly and finish, or does it spiral?

The reason "99% accurate" is meaningless, a point worth its own discussion, is that a single accuracy figure collapses all four axes into one feel-good number. An agent can be 99% accurate on the tasks it attempts while silently bailing on the hard 30% it never reports. The composite hides the danger.

We scored each agent per task as a four-tuple and only counted a task as "reliable" when it cleared all four. That's a stricter bar than most public leaderboards use, and it explains why our numbers run lower than the marketing.

The Headline Numbers

Across the production telemetry we analyzed, here is where the median autonomous agent landed in 2026:

• Task success rate (strict, independent rubric): 73%. Up from roughly 64% a year ago. Real progress, mostly from better tool use and longer effective context.
• Consistency (same task, equivalent outcome): 81%. Meaning nearly one run in five diverges enough to matter. For low-stakes tasks that's fine. For anything that touches money, it isn't.
• Silent failure rate: 9%. Nearly one in eleven failures happened without the agent flagging it. This is the number that keeps enterprise buyers up at night, and it's the one least likely to appear on a vendor's homepage.
• Successful autonomous recovery from a tool error: 58%. The rest required human intervention or simply gave up.

The single most important relationship in that list: the gap between the 73% strict success rate and the 80%+ scores those same agents post on public benchmarks. That delta, roughly 7 to 15 points depending on vertical, is the benchmark-to-production gap, and closing it is where most of the engineering effort in serious GaaS companies now goes.

For context on how fast expectations are moving, Gartner's forecasting on agentic AI projects a large share of enterprise software decisions running through agents within a few years, which only raises the stakes on these reliability figures.

Reliability by Vertical

Aggregate numbers lie by averaging. Reliability varies enormously by domain, and the variance tracks two things: how structured the task is, and how expensive a mistake is.

High-reliability verticals

Coding agents and data-extraction agents posted the strongest numbers, strict success rates in the low 80s, because the tasks have verifiable outputs. Code either compiles and passes tests or it doesn't. An extracted field either matches the source document or it doesn't. When ground truth is cheap to check, reliability climbs, because you can build verification layers and regression tests that actually bite.

Middle-of-the-pack verticals

Customer support and sales-ops agents landed in the low-to-mid 70s. The work is semi-structured, success is partly subjective, and "did the agent do what the user meant" becomes genuinely hard to score. These verticals carry the most consistency variance.

Low-reliability verticals

Open-ended research, multi-system financial workflows, and anything requiring long-horizon planning trailed badly, several below 60% strict success. Long-horizon tasks compound error: a 95% per-step reliability rate over twenty steps leaves you at roughly 36% end-to-end. This is the math that breaks multi-agent chains, where one weak link drags the whole pipeline down, and it's why long-horizon eval benchmarks remain the field's hardest open problem.

The Benchmark-to-Production Gap

If you remember one thing from this report, make it this section.

Public benchmarks overstate real-world reliability for structural reasons, not because anyone is cheating. The tasks are curated, the inputs are clean, the success criteria are generous, and crucially, the test set has often leaked into training data. Research from the Stanford HAI AI Index has repeatedly documented how benchmark saturation outpaces genuine capability gains, and agent leaderboards are the most saturated corner of all.

In production, none of those favorable conditions hold. Your documents are messy. Your APIs time out. Your users phrase requests in ways no benchmark anticipated. And the cost of a confident wrong answer is real money, not a docked leaderboard point.

We re-ran several popular agent benchmarks under three "production-like" perturbations: noisy inputs, an intentionally flaky tool that failed 10% of calls, and paraphrased task prompts. Strict success rates dropped an average of 11 points. The agents that degraded least weren't the ones with the highest clean-benchmark scores. They were the ones built with explicit recovery logic and escalation paths. That finding is the whole ballgame: robustness, not peak capability, predicts production reliability.

The practical takeaway for buyers is blunt. When a vendor cites a benchmark, ask what happens to that number when you feed the agent your actual, ugly data. If they can't answer, they haven't measured it.

What Improved This Year

Credit where it's due. Three things genuinely got better.

Tool-call reliability. The error rate on individual tool calls, malformed arguments, wrong tool selected, dropped meaningfully as models got better at structured output and as platforms added schema validation between the model and the tool. This was the single biggest contributor to the success-rate gains.

Observability tooling matured. A year ago, tracing a multi-step agent run end to end was a homegrown affair. Now there's a recognizable category of agent-native observability vendors, and the better GaaS companies treat traces as a first-class artifact. You can't fix what you can't see, and far more teams can now see.

Escalation design went mainstream. The "escalate to human" pattern stopped being an afterthought. Vendors that built confidence thresholds and clean handoff points cut their silent-failure rates roughly in half compared to those that didn't. Knowing when to stop turned out to be a more valuable skill than pushing through.

What Got Worse or Stayed Stuck

Reproducibility barely moved. Same prompt, different outcome is still endemic. Non-determinism in the models, plus the inherent variability of tool environments, means consistency improvements lagged success-rate improvements. For regulated industries that need auditability, this remains a deal-breaker.

Drift got harder to catch, not easier. Because the model underneath an agent now changes frequently, sometimes silently, via a provider update, agents that passed eval last month quietly regress. Few teams run continuous evaluation in production rather than just at launch, so drift often surfaces as a customer complaint instead of a dashboard alert.

The "did it correctly but did the wrong thing" failure mode grew. As agents got more capable, they got better at executing the wrong goal flawlessly. A perfectly formatted refund issued to the wrong account is a reliability failure that no accuracy metric catches, and these intent-mismatch failures rose as a share of total incidents.

How Leading Vendors Report Reliability

There's a widening split in the market. On one side, vendors that publish a single accuracy number and little else. On the other, a growing group that puts a real reliability figure, with methodology, on their homepage, the way SaaS companies publish uptime.

The serious players now share, at minimum: strict task success rate with the rubric defined, a silent-failure rate, and an SLA that ties dollars to reliability rather than just to uptime. a16z's writing on the economics of AI applications makes the case that this transparency is becoming a competitive moat, not a compliance burden, because reliability is far harder to copy than raw capability.

Buyers are responding. In our survey, enterprise procurement teams ranked "transparent reliability reporting" above "highest benchmark score" when selecting a GaaS vendor, a reversal from last year. The market is learning, slowly, that the vendor who tells you their failure rate is more trustworthy than the one who claims they don't have one.

Insights Most People Overlook

1. An honest failure rate is a sales asset, not a liability. The instinct is to hide the 9% silent-failure number. The vendors winning enterprise deals do the opposite, they lead with it, because a buyer who sees a measured failure rate trusts that the vendor actually instruments their system. The scariest vendor in a bake-off is the one who claims near-perfect reliability and can't show you a single trace of a failure.

2. Reliability is mostly a recovery problem, not an accuracy problem. Everyone optimizes the model's first-attempt accuracy. But our perturbation tests showed that the agents which survived production conditions did so through retry logic, verification layers, and escalation, not through being smarter on the first try. You can buy more reliability with engineering than you can with a bigger model.

3. Long-horizon tasks fail at the multiplication table, not the model. A 95%-per-step agent is a 36%-end-to-end agent over twenty steps. Teams keep trying to fix this by improving per-step quality, which barely moves the product. The leverage is in shortening horizons, checkpointing, and inserting verification gates, structural fixes, not model fixes.

4. The most dangerous failures are the ones that look like success. "Did the wrong thing correctly" failures pass every output-quality check because the output is high quality, it's just aimed at the wrong target. Reliability programs that only score output format and factual accuracy will never catch these. You need intent-alignment checks, which almost nobody runs.

5. Benchmark rank is anti-correlated with production robustness more often than you'd expect. The agents that top clean leaderboards are frequently tuned to clean conditions. Under noise, several of the highest-ranked agents degraded the most. If you're choosing a vendor, weight their performance on your messy data far above their leaderboard position.

References

• Gartner, Intelligent Agents in AI Require New Foundations
• Stanford HAI, AI Index Report
• Andreessen Horowitz, The Economic Case for Generative AI and Foundation Models