Eval-Driven Development for Agent Teams: How to Ship Agents That Actually Work

Table of Contents

• What Eval-Driven Development Actually Means
• Why Agents Broke Test-After Development
• The Anatomy of an Eval Suite for Agent Teams
• Building the Workflow: EDD in Practice
• Scoring: The Hardest Part Nobody Warns You About
• Organizing the Team Around Evals
• Common Failure Modes in EDD Adoption
• Insights Most People Overlook
• References

What Eval-Driven Development Actually Means

Eval-driven development is test-driven development for systems whose output isn't deterministic. In classic TDD you write a failing test, write the minimum code to pass it, then refactor. The test is a binary oracle: green or red. EDD keeps the spirit, write the check before you write the behavior, but swaps the binary assertion for a graded evaluation, because an agent's "correct" answer lives on a spectrum and changes shape every time the underlying model updates.

In a GaaS context, an eval is a small, runnable experiment: a fixed input (or set of inputs), the agent under test, and a scorer that decides how well the run went. Stack a few hundred of those together and you have an eval suite, your living specification of what the agent is supposed to do. The defining feature of EDD is sequence and authority. Evals come first, and they hold veto power. No prompt tweak, no model swap, no new tool integration ships until the suite says it's at least as good as what's in production.

That sounds obvious written down. In practice most agent teams do the opposite. They build the agent, ship it because the three examples they tried by hand looked great, and then discover the failure modes through customer complaints. EDD is the discipline of paying that cost up front, on purpose, in a controlled environment.

Why Agents Broke Test-After Development

Traditional software is testable because it's deterministic. Same input, same output, forever. Agents detonate that assumption in several ways at once, which is why the reproducibility problem is the quiet villain behind so much agent flakiness.

First, the model itself is stochastic. Set temperature to zero and you still get drift from provider-side changes, nondeterministic floating-point on GPUs, and silent model updates. The same prompt can return different outcomes on Tuesday than it did on Monday. A test-after culture has no way to catch this, by the time it's noticed, it's a production incident.

Second, agents are multi-step and stateful. A single customer task might involve five tool calls, a memory lookup, and a branching decision tree. Failure can hide anywhere in that chain, and the silent failure problem, an agent that confidently produces a plausible-looking result that's actually useless, won't show up in a smoke test. You have to evaluate the trajectory, not just the final string.

Third, and this is the one that genuinely changes the economics: the ground keeps moving underneath you. When a provider ships a new model version, your agent's behavior shifts even if you changed nothing. Anthropic's own guidance on building reliable systems with their models leans hard on systematic evaluation precisely for this reason; their engineering writeups on agent design treat evals as the load-bearing wall, not a nice-to-have. Without an eval suite running as a gate, every upstream model update is a coin flip you're making blindly on behalf of paying customers.

This is why EDD isn't a process preference for GaaS teams. It's structural. You cannot sell per-outcome or per-task pricing, the heart of the agentic-as-a-service model, if you can't measure outcomes, and you can't measure outcomes without evals as a first-class artifact.

The Anatomy of an Eval Suite for Agent Teams

A real eval suite for a production agent has layers, and conflating them is a classic rookie mistake.

Unit-Level Evals

These check a single capability in isolation: can the agent correctly extract a date from a messy email? Does it pick the right tool given an ambiguous request? Unit evals are cheap, fast, and run on every commit. They're the equivalent of testing one function. You want hundreds of them and you want them to run in under a couple of minutes so they don't slow the dev loop.

Trajectory Evals

Here you grade the whole run, every tool call, every intermediate decision, the order of operations. This is where you catch the agent that reaches the right answer for the wrong reason, or the wrong thing done correctly failure where it executes a flawless workflow toward a goal the user never asked for. Trajectory evals are slower and harder to score, but they're where most of the real reliability signal lives.

Outcome Evals

The bottom line: did the customer get what they paid for? For a vertical agent, say, an agent that files insurance claims or reconciles invoices, this is measured against a golden dataset of known-correct results. Golden datasets for vertical agents are expensive to build and the single most valuable asset an eval team owns. Guard them like source code, version them, and never let test cases leak into the agent's training or prompt context.

Adversarial and Guardrail Evals

The set you hope never fires: prompt injection attempts, jailbreaks, out-of-scope requests, data-exfiltration probes. Red-teaming your own agent belongs in the suite, not in a separate annual security review. An agent that's 95% accurate and exfiltrates customer data 1% of the time is not a 94% product, it's an unshippable one.

The mix matters. A suite that's all unit evals gives false confidence; a suite that's all outcome evals is too slow and too coarse to drive daily development. Most mature teams I've seen run a pyramid: lots of fast unit evals at the base, a meaningful layer of trajectory evals, and a smaller, carefully curated set of outcome evals on real-world golden data.

Building the Workflow: EDD in Practice

The mechanics, in the order teams actually adopt them:

Start with the failures you already have. Don't theorize about what could go wrong, mine your logs and support tickets. Every real customer failure becomes a frozen eval case. This does two things: it grows your suite from reality instead of imagination, and it guarantees you never ship the same bug twice. This regression-on-real-incidents habit is the closest thing EDD has to a superpower.

Write the eval before the fix. When a bug comes in, the first commit isn't the fix, it's the failing eval that reproduces it. Then you fix until it's green. This is pure TDD muscle memory applied to agents, and it's the step teams skip most often because it feels slower. It isn't; it's the only thing that makes the fix durable.

Gate the CI pipeline on evals. No merge if the suite regresses against the production baseline. This is the move that converts EDD from a good intention into an actual constraint. It also forces an uncomfortable but healthy conversation: what's our pass threshold, and who's allowed to override it?

Run the suite on every model change, including ones you didn't make. This is the agent-specific twist. Regression testing when the model underneath changes means scheduling your eval suite to run against new model versions before you adopt them, so a provider's "minor" update doesn't become your major outage. The best GaaS teams treat model upgrades exactly like dependency upgrades: nothing ships to customers until the evals are green on the new version.

Close the loop with production. Pre-launch evals are necessary but never sufficient. Continuous evaluation, sampling real production traffic, scoring it, and feeding regressions back into the suite, is what keeps the agent honest after launch. Public benchmarks famously overstate real-world reliability; your production eval stream is the corrective.

McKinsey's analysis of why most enterprise AI pilots stall, they put the figure at a striking majority of GenAI initiatives failing to reach production scale, keeps circling back to the same root cause: teams can't reliably measure or trust outputs at scale. EDD is, fundamentally, the operational answer to that trust problem.

Scoring: The Hardest Part Nobody Warns You About

Here's the thing that turns smart teams into demoralized ones: writing the eval is easy, deciding whether the agent passed is brutally hard.

For unit evals with crisp answers, exact-match or regex scoring works fine. The trouble starts with anything open-ended. How do you score "summarize this contract" or "draft a polite rejection email"? Three approaches, each with teeth:

Deterministic checks, assertions on structure, presence of required fields, schema validity, tool-call correctness. Cheap and reliable, but only cover the mechanical parts. Use them everywhere you can; they catch a surprising amount.

LLM-as-judge, use a model to grade the output against a rubric. This scales, but it imports a second reliability problem: your judge can be wrong, biased toward verbosity, or inconsistent run to run. If you go this route, you must eval the judge itself, measure its agreement with human ratings on a calibration set, and recalibrate when you change judge models. A judge you haven't validated is just a vibe with an API key.

Human review, the gold standard and the bottleneck. The realistic answer is a tiered system: deterministic checks gate everything, LLM-judges handle the bulk of the graded cases, and humans review a sampled slice plus every disagreement between the other two. Human review workflows that scale with agent volume are their own engineering discipline, and underinvesting here is why so many eval suites quietly rot.

One opinionated point: chasing a single accuracy number is a trap. "99% accurate" is almost meaningless for agents because it hides the distribution of how it fails. A claims agent that's wrong 1% of the time by rounding a refund is fine; one that's wrong 1% of the time by approving fraudulent claims is a company-ending liability. Score by failure consequence, not just failure rate.

Organizing the Team Around Evals

EDD changes org structure, not just engineering practice. The emergence of the eval team as a distinct role inside GaaS companies is one of the clearer signals that the field is maturing. These aren't QA testers in the old sense, they're closer to a hybrid of domain expert, data engineer, and product owner, responsible for owning the golden datasets, maintaining scorer quality, and defining what "good enough to ship" means for each vertical.

The healthiest setup I've seen makes evals a shared artifact, not a gate one team imposes on another. Product defines the outcome that matters. Domain experts (the people who actually know how insurance claims or legal intake should work) author and bless the golden cases. Engineers wire the suite into CI and keep it fast. And everyone reads the same eval dashboard, because the eval suite is the one place where "is the agent good?" has a concrete, non-negotiable answer.

This is also where reliability becomes a competitive asset rather than an internal cost center. A vendor that can put a credible reliability number on its homepage, backed by a real eval suite and continuous production measurement, has a moat that's genuinely harder to copy than raw capability. Capability is a model away; reliability is a thousand frozen eval cases mined from two years of customer pain.

Common Failure Modes in EDD Adoption

Teams that bounce off EDD usually do it for predictable reasons.

They build the suite once and let it stagnate. An eval suite is a living spec; if it's not growing from new production failures every week, it's already lying to you about your real reliability.

They overfit to the suite. If developers start gaming evals, tweaking prompts to pass specific test cases rather than improve general behavior, the suite becomes a vanity metric. The defense is holdout sets and rotating real-world cases the dev team doesn't get to see.

They confuse coverage with confidence. A thousand evals that all probe the easy happy path tell you nothing about the long tail where agents actually break. Deliberately weight the suite toward edge cases, ambiguity, and adversarial input.

And they treat scoring as solved. It never is. Budget ongoing effort for judge calibration and human review, or the whole edifice slowly loses its grip on reality.

Insights Most People Overlook

Your eval suite is a more durable asset than your agent. Models will be replaced, possibly several times this year. Prompts will be rewritten. But a well-curated suite of golden cases and frozen real-world failures carries forward across every one of those changes. The teams treating evals as disposable test scaffolding have it exactly backwards: the suite is the crown jewel, and the current agent is the disposable part.

EDD is what makes per-outcome pricing legally and financially survivable. Everyone talks about the trust gap on the customer side. The under-discussed version is internal: your finance and legal teams cannot sign off on outcome-based pricing without a measurement system they believe. The eval suite is that system. It's the actuarial table for your pricing model, and without it, per-outcome GaaS is just gambling with extra steps.

The judge is part of your product surface, and it can regress silently. Teams obsess over the agent and forget that an LLM-as-judge is itself an agent with all the same drift problems. A judge that subtly recalibrates after a model update can make your reliability dashboard turn green while real quality drops, the most dangerous failure mode because it disables your own instruments. Version your judges, pin them, and run a human-agreement check on them as religiously as you eval the main agent.

Speed of the eval loop is a strategic variable, not an engineering detail. If your suite takes four hours to run, developers will stop running it, and EDD dies a quiet death. The teams that win optimize for a sub-five-minute inner loop on unit evals, even if it means caching, sampling, or sharding aggressively. The discipline only survives if it's faster than waiting for a customer to complain.

The most valuable evals come from incidents, not imagination. A brainstormed list of "things that could go wrong" produces a tidy suite that misses the weird, specific ways real users actually break agents. The single highest-leverage habit is the post-incident eval: every production failure becomes a permanent, named test case. Over time this gives you a suite shaped like your actual risk surface rather than your imagined one, and that's the difference between an eval suite that looks rigorous and one that is.

References

• Building Effective Agents, Anthropic Engineering
• The State of AI, McKinsey & Company (QuantumBlack)