Task Success Rate: How Leading Vendors Actually Measure It

Table of Contents

• Why "task success rate" is slipperier than it looks
• The unit problem: what counts as one task
• Who decides a task succeeded
• Programmatic checks
• LLM-as-judge
• Human graders
• Binary vs. partial credit
• The denominator games vendors play
• How the serious vendors actually run it
• Reading a vendor's number without getting fooled
• Insights Most People Overlook
• References

Why "task success rate" is slipperier than it looks

Every agentic AI-as-a-service company eventually puts a success rate on a slide. It's the metric buyers ask for first and the one vendors most want to control. And it's the one that breaks down fastest under scrutiny, because "did the agent succeed at the task" assumes three things that are rarely true: that everyone agrees what the task was, that there's an objective way to check the outcome, and that the result was either a clean win or a clean loss.

None of those hold in practice. A support agent that resolves a refund but leaves the customer mildly annoyed, pass or fail? A coding agent that writes correct code that doesn't match the house style? A research agent that produces a thorough report with one fabricated citation buried on page four? Each of these is a judgment call, and the vendor making the call has every incentive to call it a win.

This is why task success rate is less a measurement than a measurement protocol. The number is downstream of choices, and the choices are where the truth lives. If you're evaluating GaaS vendors, or building one, the question isn't "what's your success rate." It's "show me how you compute it." The rest of this article is a field guide to the choices that produce that number.

The unit problem: what counts as one task

Before you can measure success on a task, you have to define the task as a discrete unit. This is harder than it sounds, and vendors define it differently in ways that aren't comparable.

Consider an agent that handles inbound sales emails. Is the task "respond to one email"? "Resolve one lead end-to-end across a thread"? "Book one meeting"? These have wildly different success rates. Per-email is easy to ace, answering something is trivial. Per-meeting-booked is brutal, because it depends on the prospect, not just the agent. A vendor reporting per-email success and a vendor reporting per-outcome success can quote numbers 40 points apart while doing identical work.

This connects directly to how the agent is priced. Per-outcome pricing models, which much of the GaaS market is drifting toward, force a tighter task definition, because the vendor only gets paid when the outcome lands. That alignment is good for buyers: a vendor charging per resolved ticket can't hide behind a generous per-message success rate. But it also means task definitions are increasingly a commercial artifact, not just a measurement one. When you see a success rate, ask what unit the contract bills on, because that's almost always the unit being measured.

The unit also determines how you handle multi-step work. A task that requires twelve tool calls can "succeed" overall while three of the steps quietly failed and got retried. Whether your success rate counts the task or the steps changes the story entirely, a theme that runs through end-to-end tracing of multi-step agent runs and the silent-failure problem alike.

Who decides a task succeeded

Once you've defined the unit, something has to render a verdict on each instance. There are three mechanisms in production use, and the best vendors layer them rather than picking one.

Programmatic checks

The gold standard, when it's available. If the task is "extract the invoice total and write it to the ERP," you can check the ERP field against ground truth. Deterministic, cheap, reproducible. Coding agents lean on this heavily, did the code compile, did the tests pass, did the diff apply cleanly. OpenAI's and Anthropic's own agent evaluations make extensive use of programmatic verifiers exactly because they remove human and model subjectivity from the loop.

The catch: programmatic checks only cover tasks with a checkable artifact. A huge share of real agentic work, drafting, advising, summarizing, negotiating, produces output with no single correct answer. For those, you fall back to judges.

LLM-as-judge

The dominant method in 2025-2026 for open-ended tasks, because it scales. You hand a model the task, the agent's output, and a rubric, and ask it to grade. It's fast and cheap enough to run on every production interaction. The well-documented risk is that LLM judges are biased, they favor longer answers, they favor outputs from the same model family, and they can be gamed by confident-sounding wrong answers. Anthropic's guidance on building evaluations for AI systems is blunt that an LLM grader is only as good as the rubric and needs to be validated against human labels before you trust it.

Serious vendors calibrate the judge: they periodically sample its verdicts, have humans re-grade the same items, and measure agreement. A judge running unattended at scale will drift, and a 92% success rate from an uncalibrated judge is a number about the judge, not the agent.

Human graders

The fallback for ambiguity and the calibration anchor for everything else. Humans grade a sampled slice of production traffic, and that slice both produces a "true" success rate and validates the cheaper automated methods. The constraint is obvious, you can't human-grade millions of interactions, which is why human review workflows that scale with agent volume are their own discipline. The pattern that works: humans grade a statistically meaningful sample, the LLM judge grades everything, and you continuously measure the gap between the two.

Binary vs. partial credit

Here's a decision that swings the number more than people expect: is each task scored pass/fail, or on a gradient?

Binary scoring is honest and unforgiving. The task either fully succeeded or it didn't. It's the right default for outcome-priced work, a meeting was booked or it wasn't. It's also the number a buyer can actually reason about, because it maps to "how often does this thing do the whole job."

Partial credit feels fairer to the agent and is genuinely useful for development. An agent that completed 9 of 10 subtasks is better than one that completed 2, and a flat "fail" for both hides that signal. Eval suites used internally lean on partial credit precisely because it shows where the agent breaks. The danger is when partial credit leaks into the marketing number. "94% success" built from partial-credit averaging can mean almost no tasks fully completed, they just each got most of the way. A vendor that reports a high success rate should be asked, directly, whether that's pass/fail on whole tasks or an averaged partial score. The two are not the same product.

The cleanest vendors report both: a strict end-to-end completion rate for honesty, and a step-level or partial-credit breakdown for diagnosis. If you only get one number and nobody will tell you which kind it is, assume the more flattering interpretation.

The denominator games vendors play

The numerator, tasks that succeeded, gets all the attention. The denominator is where the real manipulation happens, because it's invisible. Every excluded task inflates the rate.

The common moves:

Excluding "out of scope" tasks. The agent declined to handle something, so it doesn't count against the success rate. Defensible if the scope is genuinely fixed and disclosed. A laundromat for failures if "out of scope" gets defined after the fact to exclude whatever the agent struggled with.

Excluding escalations. When the agent hands off to a human, is that a success (correct escalation), a failure (couldn't do the job), or excluded (not counted)? All three are defended somewhere. The honest framing treats a correct escalation as a success and a missed one as a failure, which ties directly into escalate-to-human design as a reliability feature, not an admission of defeat. Counting every escalation as "not a failure" is the tell of a soft number.

Cherry-picked traffic windows. Reporting the success rate from a curated benchmark or a clean cohort rather than the messy live distribution. This is why public benchmark numbers routinely overstate real-world reliability, the eval set is cleaner than production. McKinsey's work on getting real value from enterprise AI agents repeatedly lands on the same point: the gap between demo-condition performance and production performance is where most agent deployments stall.

Silent retries. If the agent fails, retries, and eventually succeeds, does the failed attempt count? For the user, the task succeeded. For reliability and cost, three attempts to do one job is a problem the success rate alone won't show. A rate that only counts eventual success hides latency and cost blowups underneath.

None of these are inherently dishonest. All of them can be. The rule: a success rate without a clearly stated denominator is not a measurement, it's a marketing asset.

How the serious vendors actually run it

Strip away the variance and a recognizable best-practice stack emerges among vendors who treat reliability as a moat rather than a slide.

They define the unit at the outcome level, aligned to billing, and disclose it. They maintain a golden dataset, a curated, versioned set of representative tasks with known-good outcomes, and run every agent change against it before shipping, which is the backbone of eval-driven development. They run a programmatic verifier wherever an artifact is checkable, an LLM judge for the rest, and a human-graded sample to calibrate the judge and catch drift. They report a strict binary completion rate as the headline and keep partial-credit breakdowns for internal diagnosis.

Critically, they measure in production continuously, not just pre-launch. Pre-launch evals tell you the agent passed your test set; continuous production evals tell you it's still working on real traffic that's drifting underneath you, especially when the underlying model gets updated by the provider. A vendor whose only success-rate evidence is a pre-launch benchmark has measured the easy half.

They also separate two things buyers conflate: capability (can the agent do this task in principle, on a clean example) and reliability (does it do this task on the live distribution, every time, without supervision). Capability numbers are always higher and always less useful. The vendors winning enterprise deals are the ones who lead with reliability, because reliability, not raw intelligence, is what decides whether the thing actually gets deployed.

Reading a vendor's number without getting fooled

If you're a buyer, here's the short interrogation that separates a real number from a decorated one. Ask what counts as one task, and whether that matches what they bill you for. Ask who renders the verdict, code, model, or human, and if it's a model judge, ask how it was validated against humans and how often. Ask whether scoring is binary or partial, and demand the strict end-to-end completion rate specifically. Ask what's in the denominator: are escalations, out-of-scope tasks, and retries counted, excluded, or hidden. And ask whether the number comes from a benchmark or from live production traffic over a stated window.

A vendor who can answer all five crisply has a real measurement practice, and the number probably means something. A vendor who gets vague on any of them is quoting you a number they can't fully defend, which tells you more about their reliability than the percentage ever could. The percentage is the answer; the methodology is the proof. In a market this young, only one of them is worth trusting.

Insights Most People Overlook

• A lower reported success rate is often the more trustworthy one. A vendor quoting 78% with a strict binary, full-traffic, human-calibrated methodology is more reliable than one quoting 95% on a partial-credit benchmark with escalations excluded. Counterintuitively, when comparing vendors, treat suspiciously clean high numbers as a red flag until the methodology is disclosed. The honest measurement costs you points.

• The success rate and the pricing model should match, and a mismatch is diagnostic. If a vendor charges per outcome but reports a per-step or per-message success rate, they've quietly decoupled what they measure from what they sell, usually because the outcome-level number is uncomfortable. Aligned vendors measure the exact unit they bill.

• "Eventual success" via retries is a hidden cost metric wearing a reliability costume. Two agents can both report 90% success while one does it in a single pass and the other averages 2.4 attempts. The success rates are identical; the unit economics are not. Always ask for first-attempt success alongside eventual success, the gap is the retry tax.

• The most dangerous failures never show up in the success rate at all. An agent that confidently does the wrong thing correctly, completes a task flawlessly that the user didn't actually want, scores as a success under almost every measurement scheme. Success rate measures "did it do the task as defined," not "did it do what the human meant." Those diverge exactly where it hurts most.

• Success rate is a lagging indicator; drift is the leading one. A flat success rate this week tells you nothing about next week if the underlying model is being updated beneath you. The vendors who get blindsided are the ones watching the headline number instead of watching the slope. A slowly declining rate is a fire alarm, and most dashboards aren't built to hear it.

References

• Anthropic, Develop your tests (building evaluations)
• McKinsey & Company, The state of AI and enterprise agent value