Agent Success Rate vs. Task Completion Rate: Why the Two Numbers Almost Never Match

Table of Contents

• The Two Numbers Are Not Synonyms
• Defining Each Metric Precisely
• Why the Gap Exists
• Silent Failures
• The Verification Problem
• Partial Completion
• The Economic Stakes of the Gap
• How to Actually Measure Both
• What "Good" Looks Like by Vertical
• Where Vendors Quietly Game the Numbers
• Insights Most People Overlook
• References

The Two Numbers Are Not Synonyms

Walk into any GaaS pitch meeting and you'll hear a percentage. "Our agent has a 94% success rate." Press on what that number actually counts, and the answer gets fuzzy fast. Half the time, what's being sold as success rate is really completion rate wearing a nicer suit.

The distinction matters because the entire pricing logic of agentic services rests on it. When you charge per completed task, or worse, per successful outcome, you've made a measurement decision that flows directly into revenue, COGS, and customer trust. A coding agent that opens a pull request has completed a task. Whether that PR passes review, compiles, and doesn't introduce a regression is a separate question entirely. One of those is easy to count. The other is the one that determines whether the customer renews.

I've watched teams conflate these for long enough to know it's rarely malicious. It's that completion is observable and success is judged. The agent emitting a final answer is an event your system can log. Whether that answer was right requires a verifier, a human, or a downstream signal that may not arrive for days. So teams measure what's easy and call it what sounds good.

Defining Each Metric Precisely

Let's nail down terms, because loose definitions are how the gap stays invisible.

Task completion rate is the share of initiated tasks where the agent reaches a terminal state without erroring out, timing out, or escalating. The agent ran, did not crash, and produced an output. Completion is about the process finishing. It says nothing about quality.

Agent success rate is the share of initiated tasks where the output is correct against some ground-truth or acceptance criterion. Success is about the result being right. It requires an evaluation step, automated checks, human review, or a downstream outcome signal like "the customer's problem stayed solved."

The relationship is almost always:

Success Rate ≤ Completion Rate

Completion is a precondition for success in most workflows, you generally can't be right if you never finished. But finishing doesn't make you right. So success is a subset, and the gap between the two is the population of tasks that completed but failed. That population is where the money leaks.

There's a third number worth naming because people fold it into the others: the escalation or intervention rate, tasks the agent handed to a human. A task that escalates is neither completed nor failed by the agent; it's deferred. How you account for escalations dramatically changes both metrics, which is exactly why human-intervention rate is emerging as the new churn signal for GaaS products. Count escalations as completions and your completion rate inflates. Count them as failures and you penalize an agent for knowing its limits, usually the behavior you want.

Why the Gap Exists

The gap isn't noise. It comes from three structural sources, and each one fails in its own way.

Silent Failures

The most dangerous tasks are the ones that complete confidently and wrongly. A support agent that answers a billing question with a plausible-but-incorrect policy has completed its task with a clean terminal state. No error, no timeout, no escalation. Your completion dashboard shows green. The customer, meanwhile, just got bad information they'll act on.

Large language models are unusually good at producing confident, well-formed, incorrect output, this is the practical face of what researchers describe as hallucination and calibration failure in language models. The output looks successful. Every surface-level signal says the task is done. Only a real verification step catches it, and most production systems don't have one tight enough.

The Verification Problem

Here's the uncomfortable truth: success is often as hard to measure as the task was to perform. For a coding agent, you can run the test suite, but the tests only catch what they cover, and "passes CI" is not the same as "correct." For a sales-development agent that wrote a personalized email, what's the ground truth? Reply rate? Booked meeting? Those signals are noisy, lagged, and confounded by a dozen other variables.

When the verifier is cheap and reliable (a calculator agent, a schema-validated data extraction), the success/completion gap is small and easy to track. When the verifier is expensive or subjective (anything involving judgment, persuasion, or open-ended generation), the gap widens and becomes harder to measure, a brutal combination. The verticals where agents create the most value tend to be exactly the ones where success is hardest to verify.

Partial Completion

Multi-step agentic workflows introduce a third failure mode: the agent gets most of the way there. A research agent asked to compile a market analysis across ten sources pulls eight, hits a paywall on the ninth, and writes its summary on incomplete data. Did it complete the task? Technically. Did it succeed? Not against the brief.

Partial completion is where binary metrics break down hardest. A pass/fail success rate throws away the information that the agent got 80% of the way there, which matters enormously for both customer experience and for deciding whether a retry is worth the compounding cost of re-running the workflow. Some operators move to graded scoring for exactly this reason, but graded scoring is harder to put on a pricing invoice.

The Economic Stakes of the Gap

Now the part that should keep GaaS founders up at night.

If you charge per completed task, you bill on completion but you're judged on success. Every silently-failed task is revenue you collected that erodes trust and seeds churn. The gap is a deferred liability, invisible this quarter, catastrophic when it compounds, which is the precise mechanism behind why churn stays invisible in GaaS until it's suddenly not.

If you charge per successful outcome, the holy grail of per-outcome pricing the category keeps chasing, the gap hits your P&L immediately and directly. You incur the full inference cost of every completed task, including the failed ones, but you only collect on the successful ones. Your effective cost-per-billable-outcome is:

Cost per success = (total inference cost) ÷ (successful tasks)

Not divided by completed tasks. Divided by successful ones. If completion is 95% and success is 70%, you're eating the compute for that 25-point gap with zero revenue against it. McKinsey's analysis of the economic potential of generative AI hinges on agents reliably producing value, not motion, and the gap between completion and success is precisely the difference between the two.

This is the metric that should anchor your unit economics, and it's why the category needs a real cost-per-completed-task standard that distinguishes "completed" from "billable." An agent that completes cheaply but succeeds rarely can look healthy on a completion-based dashboard and be deeply unprofitable on a per-outcome basis.

How to Actually Measure Both

Measuring completion is trivial, instrument terminal states and count. Measuring success is the real work. A few practices separate teams that know their true numbers from teams that are guessing:

Define acceptance criteria per task type, in advance. Success is meaningless without a spec. "Resolved the ticket" needs an operational definition: did the customer reply confirming resolution, did the ticket stay closed for N days, did a QA reviewer approve it? Write it down before you measure.

Use sampled human evaluation as ground truth. You can't human-review everything, but you can review a statistically meaningful sample and use it to calibrate cheaper automated checks. This is how you catch silent failures your automated verifiers miss.

Build LLM-as-judge verifiers, but trust them carefully. An LLM grading another LLM's output is cheap and scales, and it inherits its own biases and blind spots. Anchor it against human-labeled samples regularly and treat divergence as a signal the judge is drifting. Anthropic's guidance on building effective agents is blunt that evaluation and guardrails are not optional add-ons but core to whether an agentic system works at all.

Track the gap itself as a first-class metric. Completion minus success isn't just a derived number, it's a leading indicator. A widening gap means your agent is completing more while succeeding less, which usually signals a distribution shift, a prompt regression, or a tool that started failing silently. Most teams don't chart this delta. They should.

Segment by task type and cohort. Aggregate success rate hides everything. A 70% blended number might be 95% on routine tasks and 40% on the hard 30% of your volume, and those hard tasks are probably your most valuable customers' most important requests. Cohort analysis by use case is where the real story lives.

What "Good" Looks Like by Vertical

There's no universal target, and anyone quoting one is selling something. The acceptable gap depends entirely on the cost of a silent failure in that domain.

For a data-extraction agent with schema validation, completion and success should track closely, 90%+ completion, success within a few points, because the verifier is cheap and reliable. A wide gap here means something is broken.

For a customer-support agent, expect a meaningful gap. Completion might run 90%+ while genuine resolution sits lower, because "the agent answered" and "the customer's problem is actually solved" diverge constantly. The healthy move is a high, well-calibrated escalation rate that converts would-be silent failures into honest handoffs.

For a coding agent, completion (opened a PR) and success (PR merged without revert) can diverge enormously. A 50-point gap is not unusual on hard tasks, which is why honest coding-agent benchmarks report merge or resolution rates, not PR-open rates. Independent agentic benchmarks like SWE-bench for real-world software tasks exist precisely to measure verified success rather than activity.

The pattern across all three: the higher the cost of a confident wrong answer, the more you should weight success over completion in both your dashboards and your pricing.

Where Vendors Quietly Game the Numbers

Because completion is easy to inflate, the metric gets gamed, sometimes deliberately, often through motivated sloppiness. Worth knowing the moves so you can spot them in your own funnel or a competitor's pitch.

Counting escalations as completions. An agent that escalates everything hard can post a beautiful completion rate while doing almost nothing of value autonomously. Always ask for completion rate net of escalations.

Defining success at the lowest plausible bar. "Generated a response" is technically success for an absurdly low bar. The vendors with real numbers define success against customer outcomes; the ones with a story define it against output existence.

Reporting on cherry-picked task distributions. A 95% success rate on the easy 80% of tasks while quietly excluding the hard 20%. Always ask what's in the denominator.

Confusing benchmark numbers with production numbers. Benchmark success rates are measured on curated, stable distributions. Production traffic is messier, adversarial, and shifting. The gap between benchmark and production success is its own metric, and it's usually unflattering.

The honest GaaS operators are the ones who'll tell you their success rate is lower than their completion rate and can explain exactly why. That gap, openly reported, is a sign of measurement maturity, not weakness.

Insights Most People Overlook

A high completion rate can be a warning sign, not a good one. If completion approaches 100% but success lags, your agent has likely learned to always produce something rather than to know when it shouldn't. An agent that never escalates and never errors is often an agent that's confidently wrong on the hard cases. The healthiest systems have a deliberate, non-trivial escalation rate. Perfect completion is suspicious.

Success rate is non-stationary in a way completion isn't. Completion depends mostly on your plumbing, does the agent run end to end. That's stable. Success depends on the world: the data distribution, the difficulty mix, the model version, the tools' uptime. Your success rate can quietly collapse while completion holds perfectly steady, because the agent keeps finishing, it just stops being right. If you only watch completion, you'll never see the cliff.

The cheapest agent to operate is often the worst per-outcome deal. Teams optimize inference cost per completed task and declare victory. But if cutting cost dropped success from 80% to 65%, your cost per successful outcome went up even as cost per completion went down. The completion-cheap agent can be the success-expensive one. You have to do the division against the right denominator or you'll optimize yourself into negative margin.

Verification cost belongs in your unit economics, and almost nobody puts it there. If measuring success requires LLM-judge calls, human review, or downstream tracking, that's real COGS. A team proudly reporting a tight success/completion gap may be spending more on verifying outcomes than on producing them. The cost of knowing whether you succeeded is part of the cost of the service, and it scales with how hard success is to verify.

Per-outcome pricing only works where success is cheaply verifiable, which is a small slice of the market. The pricing model the whole category romanticizes requires that you and the customer can both agree a task succeeded without an expensive arbitration. That condition holds for narrow, checkable tasks and breaks for exactly the high-value judgment work where agents are most interesting. The completion/success gap is, in the end, the reason per-outcome pricing is harder to deploy than the marketing implies.

References

• Survey on Hallucination in Large Language Models (arXiv)
• McKinsey: The economic potential of generative AI
• Anthropic: Building effective agents
• SWE-bench: Software engineering benchmark for AI agents