Pilot Purgatory: Why GaaS Projects Get Stuck Between "Promising Demo" and "Production"

Table of Contents

• What Pilot Purgatory Actually Is
• Why GaaS Pilots Are Especially Prone to It
• The Five Forces That Keep an Agent Stuck
• The Trust Threshold Nobody Defines Up Front
• How to Design a Pilot That Can Actually Graduate
• Reading the Warning Signs Early
• When Killing the Pilot Is the Right Call
• Insights Most People Overlook
• References

What Pilot Purgatory Actually Is

Every company running agents has at least one of these: a vendor-sold agent that did something genuinely impressive in a six-week trial, got a round of applause in a steering-committee deck, and then sat there. It didn't fail. Nobody pulled the plug. It just never expanded past the original three users, the original one workflow, or the original sandbox account. Six months later it's still "in pilot."

That's pilot purgatory. The agent is neither dead nor alive. It consumes a little budget, a little attention, and a recurring line on someone's roadmap, but it isn't producing the outcome anybody bought it for. And because it's technically "still going," nobody has to admit it stalled.

This matters more in the GaaS world than in traditional software because of how these deals are sold. A SaaS license you've already paid for tends to get used eventually, sunk cost drags it into production. An agent sold per-task or per-outcome is different. If it never gets the task volume, the vendor isn't getting paid, the buyer isn't getting value, and the relationship slowly goes cold without anyone declaring a death. The pricing model that makes GaaS attractive, you only pay when it works, also makes it unusually easy to leave an agent half-deployed forever.

The distinction worth holding onto: a failed pilot produces a clear no. Pilot purgatory produces a permanent maybe. The permanent maybe is more expensive, because it ties up organizational capacity you could have spent on something that would have shipped.

Why GaaS Pilots Are Especially Prone to It

Agentic systems carry a few properties that make the trap deeper than it was for the previous waves of enterprise tech.

The first is nondeterminism. A traditional integration either works or throws an error you can ticket. An agent mostly works, occasionally does something weird, and the "occasionally" is exactly the part that scares people out of expanding. A 92% success rate looks fantastic on a slide and feels terrifying to the ops lead who has to own the other 8%. Pilots are small enough to babysit. Production isn't. So the very thing that makes the pilot feel safe, a human watching every run, is the thing that can't scale, and nobody has budgeted for what replaces it.

The second is that agents touch real systems and take real actions. A read-only analytics pilot graduates easily. An agent that issues refunds, modifies records, or sends customer emails hits a wall the moment someone asks, "what happens when it's wrong at 50x the volume?" That question is legitimate, and most pilots are structured to dodge it rather than answer it.

The third is organizational. Agents straddle the line between IT and the business, which means ownership is genuinely ambiguous, a problem deep enough that it gets its own treatment elsewhere in this cluster. The team running the pilot is often an innovation group with no mandate to operate anything in production. They can prove the agent works. They cannot make payroll, security, and the line-of-business owner all say yes at once. So the pilot succeeds and then has nowhere to go.

McKinsey's research on enterprise AI has repeatedly found that the gap between experimentation and scaled value capture is where most of the money evaporates, companies pile into pilots and a small minority convert them into enterprise-wide impact. Agents have inherited that gap and arguably widened it.

The Five Forces That Keep an Agent Stuck

When I look at agents that stalled, the same handful of forces show up in almost every case. They compound.

No owned outcome. The pilot was framed as "let's see what this can do," not "this agent will own ticket triage for tier-1 support and we will measure deflection rate." Without a named owner accountable for a specific number, there is no one whose job it is to push the agent into production. It's everybody's interesting experiment and nobody's deliverable.

The trust threshold was never defined. Nobody wrote down, before the pilot started, what success rate, error profile, and oversight model would justify expansion. So when results come in, the conversation becomes a vibe check. "Is 92% good enough?" has no answer if you never decided what good enough was.

The integration cliff. The pilot ran on exported data or a sandbox. Real production means wiring the agent into the CRM, the ticketing system, the identity provider, and three legacy databases, and that integration burden is frequently larger than building the agent in the first place. The pilot conveniently skipped it, so the hard 80% of the work is still ahead and now looks like a brand-new project.

No operating model. Who watches the agent in production? Who gets paged when it misbehaves? Who retrains it? In a pilot, the answer is "the smart person who set it up." In production, that person has moved on, and the AgentOps function that should own it doesn't exist yet. Without an operating model, scaling means scaling chaos, and sensible people decline to do that.

Procurement and security never engaged. Innovation teams love to run pilots on a corporate card and a handshake. Then production triggers a full security review, a data-processing assessment, and vendor risk management, and the agent that breezed through the demo now faces three months of procurement scrutiny it was never prepared for. Many agents die in this queue, quietly, as "pending review."

Each force alone is survivable. Stacked, they produce an agent that works perfectly and ships never.

The Trust Threshold Nobody Defines Up Front

I want to dwell on the trust threshold, because it's the most fixable and the most ignored.

Humans don't delegate to a new hire because the hire is perfect. They delegate because they've calibrated how often the hire is wrong, how bad the wrong looks, and how easily they can catch it. That calibration is a number, fuzzy, but real. Agents need the same thing, and almost no pilot establishes it deliberately.

Here's the concrete move: before the pilot starts, write down the production bar. Something like, "We will expand this agent to the full queue when it sustains a 95% task-completion rate over 1,000 runs, with zero unrecoverable errors, and a human can review any flagged decision in under 30 seconds." Now the pilot has a finish line. The conversation stops being "does this feel safe?" and becomes "are we at 95% or not?" One of those questions can be answered; the other festers forever.

This also reframes what a pilot is for. A pilot is not a demo of the best case. It's an instrument for measuring the worst case at small scale so you can predict it at large scale. If your pilot never logged a single failure, that's not a triumph, it means the sample was too small or too curated to tell you anything about production. The most useful pilots deliberately surface the agent's failure modes, because the trust-building curve that gets a team to delegate runs through seeing the failures and finding them survivable, not through never seeing them. Anthropic's own guidance on deploying agentic systems leans hard on this point: you build reliability by bounding what the agent can do and instrumenting how it fails, not by hoping it won't.

How to Design a Pilot That Can Actually Graduate

The fixes are mostly structural and mostly cheap, and they belong in the pilot's design before it launches, not in a rescue effort six months later.

Pick a workflow with a real owner and a real number. "Reduce average handle time on billing tickets by 20%" beats "explore agentic support" every time, because the first one has someone who wins if it ships.

Scope to a thin slice that includes the ugly parts. Counterintuitively, you want the pilot to touch at least one real integration and one real edge case, even if it makes the pilot harder. A pilot that skips the integration is testing the easy half and lying to you about the timeline.

Define the trust threshold and the oversight model in the same document that authorizes the pilot. Decide who owns the agent in production before you start, even if that person is just on standby. An agent with a named production owner from day one converts at a dramatically higher rate than one searching for a home after the demo lands.

Bring security and procurement in at the start, not the finish. A 30-minute conversation up front about data flows and vendor risk saves the three-month review ambush later. This is unglamorous and it's the difference between an agent that ships and one that dies in a queue.

Budget for the operate phase explicitly. The cost of running an agent in production, monitoring, human oversight, retraining, vendor fees at volume, is the real total cost of ownership, and pilots that ignore it set up a sticker-shock moment that kills expansion. If the per-outcome economics only work in the demo, you'd rather learn that in week three than month nine.

Reading the Warning Signs Early

You can usually smell pilot purgatory before it sets in. A few reliable tells:

The pilot keeps getting extended without a changed scope. "Let's run it another month" with the same three users is the sound of a project avoiding a decision.

Nobody can state the graduation criteria when asked. If you walk up to the team and say "what specifically has to be true for this to go to production?" and you get a meandering answer, you're already in purgatory; you just haven't dated it.

The results are reported as anecdotes, not rates. "It wrote this amazing summary" is a demo story. "It hits 94% on a sample of 800" is a production conversation. Anecdote-driven status updates mean the instrumentation to make a real decision was never built.

The agent's sponsor is excited but its eventual operator has never been in the room. Enthusiasm without an operator is a strong predictor of a stall, because the people who would have to live with the thing haven't bought in, and getting them to delegate is the whole game.

When Killing the Pilot Is the Right Call

Not every stalled pilot deserves a rescue. Sometimes the honest answer is that the agent solved a problem nobody important enough cared about, or that the production economics don't clear, or that the workflow wasn't ready to be handed off. Killing it cleanly is a legitimate outcome and a far better one than the permanent maybe.

The discipline worth building is the willingness to say no out loud. A portfolio of agent pilots should have a kill rate, if every pilot you've ever run is technically "still going," you don't have a portfolio, you have a graveyard with the lights left on. Declaring a pilot dead frees the budget, the attention, and the roadmap slot for the next one. Companies that scale agents well aren't the ones that never fail; they're the ones that fail fast, declare it, and recycle the capacity. Pilot purgatory is what happens when an organization can't bring itself to either ship or quit, and the cure is the same either way: force the decision.

Insights Most People Overlook

Per-outcome pricing hides the stall. With a SaaS seat license you've already paid, so finance nags people to use it and sunk cost pushes it toward production. GaaS priced per task means an unused agent costs almost nothing, which removes the financial pressure that would otherwise force a ship-or-kill decision. The pricing model everyone loves is quietly an incentive to leave agents in limbo.

A pilot with zero failures is a failed pilot. Most teams treat a clean trial as a win. It's the opposite. If you never saw the agent break, your sample was too small or too curated to predict production behavior, and you've learned nothing about the 8% that will actually decide whether you can scale. Demand failures from your pilot; a pilot's job is to find the edges cheaply.

The integration work, not the agent, is the real project. Vendors demo the intelligence because that's the impressive part. But the reason agents stall is almost always the unsexy plumbing, auth, data access, legacy system connections. If your pilot ran on exported CSVs, you have validated roughly 20% of the actual work and badly underestimated the timeline. Treat any "it works on sample data" pilot as ground not yet broken.

Naming the production owner on day one matters more than the agent's accuracy. An 88%-accurate agent with a committed operator ships. A 96%-accurate agent that's nobody's job stalls. Ownership is a stronger predictor of graduation than model quality, which is uncomfortable for vendors selling on benchmark numbers and exactly why the org question outranks the tech one.

The "permanent maybe" is more expensive than a clean no. A killed pilot returns its budget and attention. A pilot stuck in purgatory keeps drawing both, indefinitely, while producing nothing, and worse, it occupies the organizational slot a shippable agent could have used. The most underrated skill in an agent program is the willingness to declare death.

References

• The State of AI, McKinsey & Company
• Building Effective Agents, Anthropic