Synthetic vs. Real-World Evals: Getting the Mix Right

Table of Contents

• Why this question decides whether your eval suite is worth anything
• What synthetic evals actually are (and aren't)
• What real-world evals buy you
• The gap that breaks teams: lab scores vs. production
• Getting the mix right: a maturity-based ratio
• How to manufacture good synthetic cases (most are bad)
• Harvesting real-world cases without leaking customer data
• The hybrid datasets the best benchmarks already use
• A practical operating rhythm
• Insights Most People Overlook
• References

Why this question decides whether your eval suite is worth anything

Every GaaS company that charges per task or per outcome is, whether they admit it or not, selling a reliability number. The customer doesn't care that your agent is "capable." They care whether it does the thing, correctly, this time and the next ten thousand times. The eval suite is how you find that number before a customer does, and the single biggest determinant of whether that number means anything is the data you feed it.

Here's the trap. Synthetic evals are seductive because you can generate ten thousand test cases over a weekend with another model and a prompt. The dashboard lights up green. Then you ship, and the agent face-plants on inputs no synthetic generator thought to produce: a customer who pastes a half-formatted invoice, an upstream API that returns a 200 with an empty body, a user who changes their mind mid-conversation. The eval said 97%. Production says 70%. That delta is where GaaS companies quietly die.

So the real question isn't "synthetic or real-world." It's: what ratio of each, at what stage, gives you an eval score you'd stake a contract SLA on? That's a strategy problem, and most teams never treat it as one.

What synthetic evals actually are (and aren't)

A synthetic eval is any test case you fabricate rather than observe. That covers a wide range: hand-written edge cases, templated permutations ("invoice in USD / EUR / JPY × net-30 / net-60 × with-tax / without-tax"), and, increasingly, cases generated by another LLM prompted to produce realistic-looking inputs and expected outputs.

What synthetic evals are genuinely good at:

• Coverage of the combinatorial surface. You can systematically enumerate currency × locale × document-type combinations that real traffic would take months to surface.
• Testing the rare-but-critical. If a wrong refund over $10,000 is catastrophic, you don't wait for a real one. You manufacture fifty.
• Speed and cost. Generation is near-free, runs are deterministic-ish, and you can regenerate the whole set when your task definition changes.
• Pre-launch testing, when you have no production traffic at all. Synthetic is often the only thing you've got.

What they are quietly bad at, and this is the part teams underweight, is distribution. Synthetic generators produce inputs that look like what a reasonable person imagines real inputs look like. Real inputs are weirder, messier, and more correlated in ways no generator anticipates. As one practitioner survey on evaluation bluntly put it, evaluating on clean test sets misses how users actually interact with the system, real inputs are vague, inconsistent, and inflate your scores until real traffic arrives and the numbers collapse.

There's also contamination. If you generate test cases with the same model family that powers your agent, you've baked the agent's blind spots into the eval. The agent and the examiner share a worldview, so the exam is rigged in the agent's favor.

What real-world evals buy you

A real-world eval is a test case drawn from actual usage: a logged production run, a real customer document, a genuine conversation transcript, with a verified-correct outcome attached. This is the gold standard for one reason, it's the only data drawn from the same distribution your agent will face in production.

The payoff is direct: real-world evals predict production behavior because they are production, replayed. They catch the failure modes synthetic data structurally cannot, the silent failures, the tool-call timeouts, the inputs that are technically valid but semantically deranged. They're also the only honest basis for the task success rate number you put on your homepage.

The costs are equally real. You need traffic before you have any. Labeling correct outcomes requires human judgment and often domain expertise, which is slow and expensive. And you inherit a privacy and compliance problem the moment you persist a customer's real input into a test fixture. None of these are reasons to avoid real-world evals, they're reasons to budget for them deliberately instead of pretending synthetic coverage is a substitute.

The gap that breaks teams: lab scores vs. production

The numbers here aren't subtle. Across enterprise agentic deployments, analysts have documented roughly a 37% gap between lab benchmark scores and real-world deployment performance, alongside cost swings of up to 50x for ostensibly similar accuracy. A controlled study comparing the two regimes directly found that even the strongest model dropped 11.3 points moving from synthetic to real-world inputs, while a weaker open model cratered 32 points, from 93.8% on synthetic data to 61.8% on the real thing.

Sit with that second figure. A model you'd have shipped on the strength of a 94% synthetic score is actually a coin-flip-and-a-half in production. If your eval mix was 100% synthetic, you never saw it coming. You found out from a churned customer.

The lesson is not "synthetic evals lie." It's that synthetic scores measure capability under idealized conditions, and real-world scores measure reliability under field conditions, and reliability, not capability, is what the GaaS market actually pays for. The gap between those two numbers is itself one of the most useful metrics you can track. A shrinking gap means your synthetic generator is getting more realistic and your agent is generalizing. A widening gap means your synthetic suite has drifted into fantasy.

Getting the mix right: a maturity-based ratio

There is no universal ratio, but there is a universal trajectory. The right mix shifts as your agent moves from pre-launch to scaled production.

Stage 0, Pre-launch (no traffic): You're at 90-100% synthetic by necessity. Use this phase to build the combinatorial coverage and the catastrophic-edge-case library. But seed even 5-10% real-world cases by hand: dogfood the agent yourself, pull a few real documents from a design partner under NDA. Those handful of real cases will catch more than the thousands of synthetic ones.

Stage 1, Early production (shadow mode / first customers): Now you have logs. Shift toward 60% synthetic / 40% real-world. The synthetic set still guards the rare catastrophic cases; the real-world set starts measuring the number you'll actually report. This is the stage where running the agent silently in shadow mode pays off, you accumulate real cases without exposing customers to failures.

Stage 2, Scaled production: Invert it. Aim for roughly 30% synthetic / 70% real-world. Real traffic is now your primary signal; synthetic survives only for the long-tail catastrophes that are too rare or too dangerous to wait for in the wild. Your regression suite should be dominated by real golden cases harvested from past failures.

Treat these as starting points, not gospel. The forcing function is the lab-vs-production gap: if it's wide, you need more real-world data in the loop. The mistake is freezing the ratio at whatever you started with and never revisiting it.

How to manufacture good synthetic cases (most are bad)

If you're going to lean on synthetic data at all, and at Stage 0 you have no choice, at least make it good. Most synthetic eval suites are bad in predictable ways. Three rules help.

Use a different model to generate than to answer. If your agent runs on one model family, generate adversarial test cases with another. This breaks the shared-blind-spot problem. The examiner shouldn't have the same intuitions as the student.

Generate from real seeds, not from imagination. The highest-value synthetic cases are perturbations of real ones: take a genuine production input and mutate it, corrupt a field, swap a currency, truncate the document, inject a contradiction. This keeps you anchored to the real distribution while still letting you scale coverage. Pure-imagination generation drifts; seeded perturbation stays grounded.

Programmatically verify, don't trust the generator's "expected output." When an LLM generates both the input and the supposed correct answer, the answer is frequently wrong, and you've now built a test that penalizes the agent for being right. Where possible, verify expected outputs with deterministic checks (a calculator, a schema validator, a database lookup) rather than another model's say-so.

Harvesting real-world cases without leaking customer data

The objection to real-world evals is almost always privacy. It's a solvable problem, not a blocker.

• Consent and contracts first. Bake eval-data usage into your terms or design-partner agreements. Many enterprise customers will agree to anonymized logging if you ask plainly.
• Redact at ingestion, not at use. Strip PII as logs enter your eval store, so the sensitive data never lands in a fixture in the first place.
• Prefer structure over content. Often you don't need the literal customer document, you need its shape: field count, format quirks, the specific malformation that broke the agent. Capture the structure, synthesize the surface.
• Promote failures into golden cases. Every production incident is a free, perfectly-distributed real-world eval. Your post-mortem process should end with a new regression case, redacted, added to the suite. This is how a post-mortem culture for agent failures compounds into reliability over time.

That last point is the highest-leverage habit on this list. Teams that convert every failure into a permanent regression test build a real-world eval set that no competitor can replicate, because it's made of their specific scars.

The hybrid datasets the best benchmarks already use

If you want a sanity check on this whole argument, look at what the strongest public benchmarks do. They don't pick a side. As a recent survey of agent evaluation and benchmarking notes, high-quality benchmarks rarely rely on purely human-curated or purely synthetic data, they use a hybrid. SWE-bench Verified starts from real GitHub issues, then adds human validation to filter the noise. AppWorld builds a fully synthetic "world" of fictitious users but validates every task with programmatic checks rather than trusting the generator.

That's the pattern worth stealing: synthetic for scale and coverage, real-world for grounding, programmatic verification to keep both honest. The benchmark builders converged on the mix because they were optimizing for the same thing you are, a score that survives contact with reality.

A practical operating rhythm

Pulling it together into something you can actually run:

1. Maintain two suites, scored separately. Never average synthetic and real-world into one headline number, you'll hide the gap that matters most. Report both, and report the delta.
2. Gate releases on the real-world suite. Synthetic regressions are a yellow light; real-world regressions are a red one.
3. Feed every production failure back as a real golden case. Non-negotiable. This is your moat.
4. Re-seed synthetic generation from fresh production logs monthly. Stops your synthetic distribution from drifting away from reality.
5. Watch the gap as a first-class metric. Widening lab-vs-production gap is an early warning that your evals are lying to you.

Get this rhythm running and the synthetic-vs-real-world debate dissolves. You stop asking which is better and start managing a portfolio, one tuned to give you a reliability number you'd actually sign a contract behind.

Insights Most People Overlook

• The gap between your synthetic and real-world scores is more valuable than either score alone. Most teams obsess over the absolute numbers. The difference is the leading indicator: it tells you whether your synthetic suite is still tethered to reality and whether your agent is generalizing or overfitting to imagined inputs. Track the delta as a primary metric, not an afterthought.

• LLM-generated "expected outputs" silently corrupt your eval set. When the same class of model generates both the question and the answer key, a meaningful fraction of those answer keys are wrong, and your agent gets penalized for being correct. Annotation error rates above 50% have been documented in some automated pipelines. If you can't programmatically verify a synthetic case's expected output, treat that case as suspect, not authoritative.

• Your competitors can copy your model, your prompts, and your architecture, they cannot copy your failure log. A real-world eval set built from your own production incidents is the most defensible asset in a GaaS company, because it encodes the exact distribution of your customers' weirdness. This is why reliability is a harder moat to copy than capability: capability is a model checkpoint; reliability is accumulated, redacted scar tissue.

• Pure synthetic coverage creates false confidence that's worse than no coverage. A team with no evals knows it's flying blind. A team with 100% synthetic evals at 97% green thinks it can see, and ships accordingly. The 11-to-32-point drops observed moving synthetic-to-real aren't edge cases; they're the norm. Confident wrongness is more dangerous than acknowledged ignorance.

• The right mix is a moving target, and freezing it is the actual mistake. Teams agonize over "60/40 or 70/30?" as if there's a correct constant. There isn't. The ratio should slide from synthetic-heavy at pre-launch toward real-world-heavy at scale, driven by how much real traffic you've accumulated and how wide your lab-vs-production gap runs. Revisit it quarterly; a ratio you set once and never touch is a ratio that's already stale.

References

• Evaluation and Benchmarking of LLM Agents: A Survey (arXiv)
• AI Benchmarks 2026: Top Evaluations and Their Limits (Kili Technology)
• A Survey on Evaluation of LLM-based Agents (arXiv)