Golden Datasets for Vertical Agent Evals: The Quiet Work That Decides Whether Your Agent Ships

Table of Contents

• Why golden datasets are the foundation of vertical agent reliability
• What a golden dataset actually is (and isn't)
• Sourcing cases: where the good examples come from
• Labeling ground truth in messy verticals
• Grading: outcome over transcript
• Sizing and stratifying the set
• Keeping the dataset alive
• The build sequence, start to finish
• Insights Most People Overlook
• References

Why golden datasets are the foundation of vertical agent reliability

Every GaaS company eventually hits the same wall. The demo dazzles. The pilot looks fine. Then a procurement team asks a deceptively simple question: "How do you know it works?" And the honest answer, for most teams, is a shrug dressed up in adjectives.

A golden dataset is how you stop shrugging. It's the difference between a vendor who says "our invoice-processing agent is highly accurate" and one who says "on our 600-case golden set, drawn from real customer invoices across 14 ERP formats, the agent reaches the correct posting decision 94.2% of the time, and here's the breakdown by case type." One of those vendors gets the enterprise signature. The other gets a follow-up meeting that never happens.

The reason this matters more for vertical agents than for general-purpose ones is concentration of risk. A general assistant that occasionally fumbles a trivia question annoys someone. A revenue-cycle agent that miscodes a medical claim, or a contract agent that misses an indemnification clause, creates real financial and legal exposure. The narrower the vertical, the higher the stakes per error, and the less forgiving the buyer. That's exactly why a golden set built from that vertical's real cases is non-negotiable. Generic benchmarks won't catch the failure modes that get you fired by a customer.

What a golden dataset actually is (and isn't)

Let's be precise, because the term gets thrown around loosely.

A golden dataset for a vertical agent is a curated collection of task instances, each one consisting of: the input the agent receives, the full context it would have in production, and a verified correct outcome that a domain expert has signed off on. It is the ground truth you grade against. The word "golden" carries weight: every entry is supposed to be trusted. If an entry's expected answer is wrong, it silently poisons every eval run forever, which is why labeling discipline matters so much.

Here's what it is not:

It is not your training data. Training data teaches the model; golden data judges the system. Mixing them is how you get the eval-engineering equivalent of grading students on the exact questions they studied. Keep a hard wall between the two.

It is not a benchmark leaderboard. Public leaderboards measure capability on shared tasks. Your golden set measures your agent on your customers' actual work. The two answer different questions, and as the cluster's pieces on why public benchmarks overstate real-world agent reliability and the benchmark wars explore, a high public score and a low golden-set score routinely coexist.

It is not static documentation. A golden set that hasn't changed in six months is either a perfect, finished product (it isn't) or a neglected one (it is).

The cleanest way to think about it: your golden dataset is the regression suite for a system whose behavior is probabilistic. Traditional software has unit tests with deterministic assertions. Agents have golden sets with statistical pass thresholds. The mindset shift from "did this exact line return true" to "did the agent reach the right outcome on enough of these representative cases" is the whole game.

Sourcing cases: where the good examples come from

The instinct of most teams is to sit in a room and write example cases from imagination. This produces a tidy, useless dataset. Synthetic cases cluster around what the team expects the agent to face, which is precisely the region where the agent already works. The cases that matter live in the long tail nobody imagined.

Better sources, roughly in order of value:

Production failures. When your agent screws up in the wild and a human catches it, that's gold, literally. Every escalation, every customer complaint, every "the agent did the wrong thing correctly" moment should be a candidate for the golden set. These are pre-validated as hard, because they already broke something. A disciplined team has a pipeline that routes flagged production cases straight into a labeling queue. This is where golden datasets and the broader observability stack connect: your tracing and human-review tooling is also your case-mining tool.

Historical human work. If your agent automates a job humans used to do, the archive of their past work is a treasure. Closed support tickets, processed claims, reviewed contracts, with their known-good resolutions, are ready-made input/outcome pairs. You usually need to clean and re-verify them, but the sourcing is half done.

Adversarial and red-team cases. Deliberately constructed edge cases, the ones that probe the seams of the vertical. What happens when the invoice is in a foreign currency? When the contract has a typo in a key clause? When two retrieved documents contradict each other? These won't show up often in random production sampling, so you have to manufacture them on purpose. The cluster piece on guardrail testing and red-teaming your own agents goes deeper here.

Stratified production sampling. Beyond just failures, periodically pull a random, representative sample of production traffic and label it. Failures over-represent the hard cases; random sampling keeps you honest about the easy-and-medium distribution so your headline number reflects reality.

A practical mix for a maturing vertical agent: maybe 40% mined from production failures and escalations, 30% from random stratified production sampling, 20% from historical human work, and 10% hand-built adversarial cases. Adjust to taste, but never let any single source dominate, especially not synthetic.

Labeling ground truth in messy verticals

This is where golden-dataset projects quietly die. Labeling sounds like data entry. In a real vertical, it's expert judgment under ambiguity, and it's expensive.

The core problem: in many verticals, there isn't one indisputable "correct" answer. Two experienced underwriters can reasonably reach different decisions on a borderline case. Two senior attorneys might flag different clauses as the most material risk. If your ground truth pretends to a certainty the domain doesn't have, your evals will punish the agent for being defensibly right.

A few hard-won practices:

Use domain experts, not annotators. The temptation to outsource labeling to cheap generalists is strong and almost always wrong for vertical agents. The whole point is that the vertical requires expertise; that's why customers pay for the agent. Cheap labels produce a golden set that's confidently wrong, which is worse than no set at all. Anthropic's own guidance on building evaluations stresses grading against criteria a knowledgeable human would actually apply, not a superficial string match.

Measure inter-annotator agreement. Have at least two experts label a subset independently. Where they disagree, you've found either an ambiguous case (flag it, possibly exclude it from the strict pass/fail set) or a place where your own rubric is underspecified. High disagreement on cases you thought were clear is one of the most useful signals a golden-dataset project produces, because it tells you the task definition itself is fuzzy.

Capture the rationale, not just the answer. "Correct outcome: deny the claim" is far less useful than "deny the claim because the policy lapsed on the date of service." The rationale lets you build better automated graders later and lets a new expert understand why a label is what it is when they revisit it in a year.

Accept ranges and rubrics where appropriate. For outputs that aren't binary, a rubric ("a passing summary must mention the termination date, the penalty amount, and the governing-law jurisdiction") often captures truth better than a single golden string. The golden "answer" becomes a checklist, which an LLM-as-judge or a deterministic checker can then evaluate.

Grading: outcome over transcript

Once you have inputs and verified expected outcomes, you have to decide how a run gets scored against them. The single most important principle here, and the one most often violated: grade the outcome, not the path.

Vertical agents are non-deterministic. Run the same input twice and the agent may take a different route, call tools in a different order, phrase things differently, the reproducibility problem that haunts the whole field. If your grader checks whether the transcript matches a reference transcript, every harmless variation registers as a failure and your eval becomes noise.

What you actually care about is whether the agent reached the right end state. Did the invoice get posted to the correct account? Did the right clause get flagged? Did the refund get issued for the right amount to the right customer? Define success at the level of the outcome the customer pays for, and let the agent take whatever reasonable path it wants to get there.

Practically, graders fall into three buckets:

Programmatic checks for anything verifiable. If the expected outcome is a structured value, a posting code, a dollar amount, a boolean decision, check it with code. These are cheap, fast, deterministic, and you should lean on them as heavily as the vertical allows.

LLM-as-judge for fuzzy outputs where a rubric exists but exact matching doesn't work, like assessing whether a generated summary covers the required points. Powerful, but treat the judge itself as a component that needs its own validation against human grades, or you've just moved the trust problem one layer down.

Human grading for the irreducible core where expertise is required and stakes are high. Expensive, so reserve it for the cases the other two methods can't handle confidently, and for periodic audits of the automated graders.

The output of all this is your reliability number, the kind that, as another cluster piece argues, every GaaS company should put on its homepage. But the number is only as trustworthy as the golden set beneath it. McKinsey's research on capturing real value from enterprise AI repeatedly lands on the same point: the organizations that get durable value are the ones that instrument and measure rigorously, rather than trusting impressive demos.

Sizing and stratifying the set

"How many cases do I need?" is the most common question and it has an annoying answer: enough to make your headline metric statistically meaningful, broken down by the slices you care about.

A 50-case set gives you a number with error bars wide enough to drive a truck through. A single percentage point of measured improvement is meaningless at that size. As a rough floor, a few hundred cases gets you a believable top-line number for a focused vertical; serious teams running fine-grained slice analysis push into the low thousands.

But raw count matters less than stratification. Your golden set should be deliberately divided into segments that mirror the real distribution and the segments you'd report on: by case difficulty, by customer type, by document format, by edge-case category. Why? Because an aggregate "93%" can hide that the agent is 99% on easy cases and 60% on a critical hard segment that happens to be rare in the data but expensive in the world. The per-vertical cost of a false positive versus a false negative varies enormously, and your stratification is how you make sure you're measuring the slices where errors actually hurt.

Stratification also protects you from a subtle trap: as you fix bugs, you tend to add cases for the failures you just found, which over-weights solved problems and lets your headline number drift upward without real improvement. Keeping a stable, stratified "core" set separate from a growing "frontier" set of new hard cases lets you track honest progress on a fixed baseline while still hunting new failures.

Keeping the dataset alive

A golden dataset is a living contract, and the maintenance is where most teams underinvest.

Three forces erode a golden set over time:

The model underneath changes. When you upgrade to a new model version, behavior shifts, sometimes improving overall while regressing on specific case types. Your golden set is exactly the instrument that catches this, which is the heart of regression testing agents when the model changes. But the set only does its job if it's run on every model change, automatically, as a gate, not an afterthought.

The world changes. Tax rules update. Contract norms shift. A customer migrates to a new ERP. Cases that were golden last year may now have different correct answers. Stale ground truth is insidious because the eval keeps running green while silently testing yesterday's reality. Schedule periodic expert re-review of the set, especially in fast-moving regulatory verticals.

The agent gets better and the set gets easier. As you fix failure modes, the cases that once stressed the agent become trivial. A set that's mostly solved cases stops being a useful signal. You need a steady inflow of fresh, hard cases, fed by the production-failure pipeline, to keep the set's difficulty calibrated to the agent's actual frontier.

The healthiest pattern I've seen: treat the golden dataset like source code. Version it. Review changes to it in pull requests, because changing an expected outcome is a consequential act that deserves scrutiny. Tie it into CI so evals run on every meaningful change. This is the practical core of eval-driven development for agent teams, and it's increasingly owned by a dedicated eval team inside mature GaaS companies rather than bolted onto whoever has spare time.

The build sequence, start to finish

Pulling it together, here's the order of operations for a team starting from zero:

1. Define the outcome. Write down, precisely, what "the agent did the right thing" means for this vertical. This is harder than it sounds and forces clarity you'll need everywhere else.
2. Seed from reality. Pull 50-100 real cases, ideally including known failures and a random production sample. Resist the urge to invent them.
3. Label with experts and capture rationale. Get domain sign-off on the correct outcome for each. Double-label a subset to measure agreement.
4. Build graders. Programmatic where you can, LLM-judge where you must, human for the hard core. Validate the automated graders against human grades.
5. Stratify and establish a baseline. Segment the set, run the current agent, record the number per slice. This is your line in the sand.
6. Wire it into CI. Make evals run automatically on model and prompt changes. A golden set nobody runs is decoration.
7. Feed the pipeline. Route production failures and escalations into the labeling queue continuously. The set grows where the agent hurts.

You don't need all seven on day one. But you need the trajectory, because reliability, more than raw capability, is what separates the GaaS companies that scale from the ones that stay stuck in pilot purgatory. The golden dataset is where that reliability gets built, measured, and defended.

Insights Most People Overlook

Your golden dataset is a competitive moat, and a strategic liability if leaked. Capability is increasingly commoditized; the next great model lifts everyone. What doesn't transfer is the accumulated, expert-labeled record of your vertical's hard cases. A rival can copy your prompt in an afternoon. They cannot copy three years of mined production failures with verified expert ground truth. This is why the reliability moat is harder to copy than capability. The flip side: that asset is so valuable it should be treated as sensitive IP, not a CSV in a shared drive.

The most dangerous golden-set entries are the confidently mislabeled ones. An obviously wrong label gets caught. A subtly wrong one, where a tired expert signed off on a defensible-but-incorrect outcome, becomes permanent. It silently penalizes correct agent behavior and rewards the wrong behavior, forever, on every run. Periodic re-auditing of high-confidence labels, not just ambiguous ones, catches these. Most teams only re-examine the cases they already flagged as uncertain, which is exactly backwards.

Inter-annotator disagreement is a product signal, not just a data-quality metric. When your own experts can't agree on the correct outcome for a case, you've discovered a boundary where the vertical is genuinely ambiguous. That's not a labeling problem to be smoothed over; it's information about where your agent should probably escalate to a human rather than decide. Some of the best autonomy-boundary decisions come straight out of golden-set disagreement analysis.

A shrinking pass rate can mean your dataset is improving, not your agent degrading. Teams panic when the golden-set score drops after they add new cases. But if you've been feeding in fresh production failures, a falling number means your set is getting more representative of the hard reality, not that the agent got worse. This is why you separate a stable core baseline from the growing frontier set: one tracks honest progress, the other tracks newly discovered pain. Conflate them and you'll misread your own metrics.

Grading the path instead of the outcome is the most common rookie mistake, and it scales badly. Early teams write graders that check tool-call sequences or match reference transcripts because it feels rigorous. It isn't, it's brittle. Every model upgrade, every harmless rephrasing, breaks these graders and floods you with false failures, until the team stops trusting the eval entirely. Outcome-based grading costs more to design up front and saves you from quietly abandoning your own reliability program six months in.

References

• Anthropic, Develop your tests and evaluations
• McKinsey & Company (QuantumBlack), Our insights on enterprise AI value