The Reproducibility Problem: Why the Same Prompt Gives You a Different Answer Every Time

Table of Contents

• What Reproducibility Actually Means for an Agent
• Where the Randomness Comes From
• Why Temperature Zero Doesn't Save You
• How Non-Determinism Compounds in Multi-Step Agents
• What This Breaks: Testing, Debugging, and Contracts
• How Serious GaaS Teams Handle It
• The Economics of Non-Determinism
• Insights Most People Overlook
• References

What Reproducibility Actually Means for an Agent

In conventional software, reproducibility is table stakes. Feed a function the same inputs and you get the same output, every time, forever. The whole discipline of testing rests on that assumption. When a test fails, it failed because the code changed, not because the dice came up differently.

Agentic systems break that assumption at the foundation. An LLM-backed agent is a stochastic system wearing the costume of a deterministic one. The same prompt, the same model, the same configuration, and you can still land in two different places. Sometimes the difference is cosmetic, a synonym here, a reordered sentence there. Sometimes it's the difference between booking the right flight and booking a flight to the wrong city.

It's worth being precise about what we mean, because people conflate two distinct things. Output reproducibility is whether the literal tokens come back the same. Outcome reproducibility is whether the agent accomplishes the same result, the refund gets processed, the ticket gets routed correctly, the code compiles. You almost never get the first one, and the second one is the only one that actually matters commercially. A customer paying per successfully resolved ticket doesn't care that the wording varied. They care that the resolution rate is stable. The reproducibility problem, properly framed, is a question about the distribution of outcomes an agent produces, not about byte-for-byte sameness.

Where the Randomness Comes From

Most explanations stop at "temperature," which is where most people's understanding goes wrong. There are at least four independent sources of variation, and temperature is only the most visible one.

The first and most obvious is sampling. When a language model generates the next token, it doesn't pick the single likeliest word, it samples from a probability distribution over the vocabulary. Temperature, top-p, and top-k all shape that distribution. Crank temperature up and the model gets more adventurous; the variance between runs widens. This is the source everyone knows about, and it's the one people assume they can turn off.

The second is floating-point non-associativity on parallel hardware. This is the one that surprises engineers. GPUs execute matrix multiplications across thousands of cores, and the order in which those partial sums get combined isn't guaranteed run to run. Because floating-point addition isn't perfectly associative, (a + b) + c can differ from a + (b + c) in the last bits, you get tiny numerical differences in the logits. Most of the time those differences vanish in the noise. But when two candidate tokens are nearly tied in probability, a rounding difference in the fifteenth decimal place can flip which token wins, and from that fork the whole generation diverges. Thinking Machines Lab published a widely-circulated technical breakdown of why LLM inference is non-deterministic that pins much of the blame not on floating point alone but on batch-dependent kernel behavior, the same request can be computed differently depending on what else is in the server's batch at that moment.

That batch effect is the third source, and it's the cruel one for a hosted service. When you call a model API, your request gets bundled with other users' requests into a batch for efficient GPU utilization. The batch composition changes constantly based on load. Different batch sizes can route through different kernels with different reduction orders. So your "identical" call at 2 a.m. on a quiet server and your identical call at peak traffic genuinely run through different computation paths. You don't control this. You usually can't even observe it.

The fourth source is the system around the model: provider-side model updates pushed silently, retrieval layers that return documents in a different order, tool calls that hit live APIs returning fresh data, and timestamps or random IDs injected into prompts. By the time you've wrapped a model in an agent with memory, tools, and retrieval, the model's own sampling randomness is often the smallest contributor to your variance.

Why Temperature Zero Doesn't Save You

The instinct, once people understand sampling, is to set temperature to 0 and declare the problem solved. Greedy decoding, always take the most probable token, should be deterministic, right?

In theory. In practice, temperature 0 reduces variance dramatically but does not eliminate it, for exactly the floating-point and batching reasons above. When two tokens are nearly tied, greedy decoding has to pick one, and the tie-break can come down to numerical noise that isn't stable across runs or across batch configurations. OpenAI's own API documentation has long carried a quiet caveat that even with a fixed seed and temperature 0, outputs are "mostly" deterministic but not guaranteed, and they expose a system_fingerprint field precisely so you can detect when the backend changed underneath you.

There's also a real cost to temperature 0 that teams underestimate. Greedy decoding can make agents worse at certain tasks, it tends to get stuck in repetitive loops, and for tasks that benefit from a little exploration (creative drafting, brainstorming alternatives, recovering from a dead end in a multi-step plan), zero temperature produces brittle, mode-collapsed behavior. So you're often trading away capability to chase a determinism you don't fully get anyway. That's usually a bad trade.

The honest conclusion: determinism is not a setting you flip. It's an asymptote you approach at increasing cost, and even at the limit, the hosted-inference reality means you can't promise it. Mature teams stop trying to make the system deterministic and start designing for the fact that it isn't.

How Non-Determinism Compounds in Multi-Step Agents

A single LLM call with mild variance is manageable. An agent is not a single call. It's a loop: reason, call a tool, observe the result, reason again, call another tool, and so on for five, ten, fifty steps.

Variance compounds across those steps, and it doesn't compound linearly, it branches. Suppose each step has a modest 95% chance of going the "right" way. String ten steps together and your end-to-end success probability is 0.95^10, which is about 60%. Twenty steps and you're near a coin flip. The arithmetic is unforgiving, and it explains a phenomenon that baffles people new to agents: each individual component looks reliable in isolation, yet the assembled agent is flaky. No single part is broken. The composition is.

Worse, the branches aren't independent of each other. A small divergence early, the agent retrieves a slightly different document, or phrases a sub-query differently, can route the entire downstream trajectory onto a completely different path. Two runs that started identically can be doing entirely unrelated things by step six. This is why "I ran it and it worked" tells you so little. You sampled one path out of a branching tree, and the tree has bad branches you never visited. This failure pattern connects directly to the silent failure problem, where agents confidently produce nothing useful, divergent trajectories are exactly how an agent ends up looking busy while accomplishing nothing.

The practical upshot is that for any agent doing real multi-step work, a single test run carries almost no information about reliability. You have to sample the distribution.

What This Breaks: Testing, Debugging, and Contracts

Three things break, and each one hurts a GaaS business in a specific way.

Regression testing breaks. The classic workflow, capture expected output, assert against it, alert on mismatch, assumes a stable output to assert against. With agents, a passing assertion can fail next run for no reason, and a real regression can hide behind a lucky run. Teams that bolt exact-match assertions onto agents end up with test suites that cry wolf so often everyone learns to ignore them, which is worse than no tests at all. Building evals that respect non-determinism is its own discipline, closely tied to eval-driven development for agent teams and the broader work of building an eval suite for an autonomous agent.

Debugging breaks. A user reports the agent did something wrong. You try to reproduce it. It works fine. Now what? Without the ability to replay the exact run, same retrieved context, same tool responses, same intermediate states, you're debugging a ghost. This is why the replay problem and end-to-end tracing have become first-class concerns in agent observability rather than nice-to-haves. You cannot fix what you cannot reproduce, and you cannot reproduce a stochastic run unless you captured everything around it.

Contracts break. This is the commercial sharp edge. When a GaaS vendor sells per-outcome pricing, "you pay when the invoice is correctly processed", they're implicitly making a reliability claim. But if the same invoice processes correctly 92% of the time and wrong 8% of the time, what exactly did you sell? Reproducibility failure turns crisp-sounding outcome contracts into statistical bets, and pricing, SLAs, and refund policies all have to account for a success rate, not a guarantee. A vendor who hasn't internalized this will underprice their risk and get burned by the tail.

How Serious GaaS Teams Handle It

The teams that have actually shipped reliable agents converged on a handful of practices, and none of them involve pretending the system is deterministic.

Measure distributions, not points. Run every eval case many times, five, ten, twenty, and report the rate of success along with its variance, not a single pass/fail. This is sometimes called pass@k or, more usefully for production, a stability metric: of N identical runs, how often did the agent reach an acceptable outcome? A case that passes 20/20 is genuinely solid. A case that passes 14/20 is a latent incident waiting for a customer to hit it.

Grade outcomes semantically, not literally. Since literal output varies, the assertion layer has to judge meaning. That usually means an LLM-as-judge or a rules engine that checks "did the refund get issued for the right amount" rather than "does the output string match." This shifts the brittleness from the agent to the grader, so the grader itself has to be evaluated for consistency, a subtlety many teams miss.

Capture everything for replay. Log the full trajectory: prompts, retrieved context, tool inputs and outputs, model fingerprints, seeds, timestamps. When something goes wrong, you replay the captured environment rather than re-running against a moving world. This is the backbone of modern agent observability and the reason the category is growing fast.

Pin what you can, monitor what you can't. Pin model versions explicitly rather than tracking a floating alias, so the provider can't silently swap the model underneath you. Then monitor for drift continuously, because even a pinned endpoint can shift behavior. Anthropic and other providers now publish guidance on building reliable, well-evaluated agents precisely because customers kept getting burned by the gap between a demo and production.

Add verification layers. For high-stakes outcomes, don't trust a single run. Run the agent, then have a separate check, another agent, a deterministic validator, or a human for the riskiest cases, confirm the work before it ships. Redundancy converts a flaky single sample into a reliable ensemble, at the cost of latency and compute.

The Economics of Non-Determinism

Every one of those practices costs money, and that's the part the reliability conversation usually skips. Running each eval case 20 times costs 20x the inference. LLM-judge grading adds another model call per check. Verification layers double or triple per-task compute. Replay logging balloons storage. Non-determinism isn't just an engineering headache, it's a line item.

This reframes a lot of GaaS strategy. The reason reliability is a moat that's harder to copy than raw capability is precisely that handling non-determinism well is expensive, unglamorous, and compounding. Any competitor can call the same frontier model and get the same average-case demo. Very few will invest in the distributional testing, replay infrastructure, and verification layers needed to make per-outcome pricing actually safe at scale. The vendor who absorbs that cost and prices it correctly wins the enterprise deals; the vendor who waves it away wins the demo and loses the renewal.

There's a temperature-on-the-margin decision here too. Higher determinism buys you tighter SLAs and lower verification cost, but at the price of capability and, sometimes, customer-visible quality. Lower determinism buys you better task performance but a fatter tail of failures you have to catch. Where you set that dial is a business decision dressed up as a technical one, and it should be made by someone who understands both the failure costs and the margins, not defaulted to whatever the SDK ships with.

Insights Most People Overlook

Determinism would not actually solve reliability, and might hurt it. People treat "make it deterministic" as the goal, but a perfectly deterministic agent that's deterministically wrong on 8% of cases hasn't fixed anything. You'd have a stable, reproducible failure. Reproducibility and correctness are orthogonal axes. The real target is a tight, high-mean distribution of good outcomes, and a little stochasticity that lets the agent escape dead ends can raise the mean even as it lowers reproducibility. Chasing determinism for its own sake optimizes the wrong variable.

The model's sampling is rarely your biggest source of variance. Engineers obsess over temperature because it's the knob they can see. But in a deployed agent, retrieval ordering, live tool data, batch effects, and silent provider updates typically swamp the contribution of the model's own token sampling. If you spend all your variance-reduction effort on temperature and ignore the system around the model, you're polishing the smallest term in the equation.

"It works on my machine" is now a statistical claim, and a dishonest one. In deterministic software, a successful local run is real evidence. With agents, a single successful run is one sample from a distribution you haven't characterized. When someone demos an agent live and it works, that's genuinely weaker evidence than the same demo would be for normal software, yet people instinctively read it as stronger because it's a live agent doing something impressive. Buyers and builders both need to recalibrate: one good run is closer to anecdote than proof.

Non-determinism quietly sets a floor on how clean your SLAs can be. A vendor can promise a deterministic API a 99.99% uptime number with a straight face. An agent vendor selling outcome correctness fundamentally cannot promise a hard guarantee on any single task, only a rate, because the same input can fail through no fault of the code. This is why 99% accurate is close to meaningless for agents without specifying the measurement, the distribution, and the tail. The honest GaaS contract is probabilistic by nature, and vendors who pretend otherwise are writing checks the math can't cash.

The grader inherits the disease. Teams move to LLM-as-judge to handle output variance, then forget that the judge is itself non-deterministic. The same correct output can be graded pass on one run and fail on the next. If you don't measure the consistency of your evaluator the same way you measure your agent, you've just moved the reproducibility problem one layer up and stopped looking at it.

References

• Defeating Nondeterminism in LLM Inference, Thinking Machines Lab
• Reproducible Outputs and the seed Parameter, OpenAI Platform Documentation
• Building Effective Agents, Anthropic Engineering