Pilot Purgatory: When the Demo Works and Nothing Ships
Most AI pilots do not fail. Failing would be useful, because a failure produces a decision. Instead they enter a stable state where the demo still works, the steering committee still meets quarterly, the slide still says promising results, and there is no production date, no named owner and no budget line. We call this pilot purgatory, and once you can name it you start seeing it everywhere. It is not a technology problem and it is rarely a talent problem. It is a specific set of gaps between what a pilot proves and what a production system has to prove, and every one of those gaps is predictable.
The state is stable because nobody is punished for it. The pilot is cheap enough to keep alive and impressive enough that cancelling it looks like a lack of ambition. The vendor has been paid, the sponsor has something to present, and the operational team that would have to run it has not been asked to commit to anything. Meanwhile the model providers ship new versions, the corpus grows, the people who built the demo move on to other work, and the thing quietly becomes harder to ship every month it is not shipped. Purgatory has a real cost, but that cost never appears on a line anybody owns.
The underlying gap is that a pilot proves feasibility on a friendly slice of reality, while production requires proving reliability against hostile reality. Feasibility takes a curated corpus, one user, no permission rules and no deadline. Reliability takes a random sample, thirty concurrent users, enforced entitlements, a latency budget, a cost ceiling, an on-call rota and a rollback path. In our experience the hardening effort between those two states is 2.5 to 4 times the engineering that produced the demo, which is why a project scoped as if the demo were 80 percent complete always stalls at what feels like the finish line.
The good news is that these gaps are enumerable, and each one is far cheaper to close in week two than in week twenty. What follows is the list we work through, in the order the failures usually arrive, followed by the exit criteria we think a pilot should have to satisfy before anyone releases production budget. Treat it as a checklist rather than an essay, because its value is in being applied early. If you also need the money side of this, we set out the full spend structure in what an AI project actually costs, including the run-rate that a pilot never reveals.
What Gartner Actually Said, and What It Did Not
The number people quote at each other comes from Gartner's June 2025 press release, which stated that over 40 percent of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls, based on a poll of over 3,400 respondents. It is worth reading the three cited reasons carefully, because none of them is the model being insufficiently capable. All three are things a buyer and a supplier control jointly: how the spend is managed, whether the value was ever defined, and whether anyone designed the controls before the pilot started.
It is equally worth noting what the finding does not say. Canceled is not the same as technically failed, and a prediction about agentic projects specifically is not a verdict on document assistants, retrieval systems or classifiers. Gartner has also projected that 40 percent of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5 percent in 2025. Both statements can hold at once: broad adoption of narrow, well-scoped agents alongside a high cancellation rate for ambitious, ill-defined ones. The difference between those two populations is mostly scoping discipline rather than technical sophistication.
One more data point is worth carrying because it points at the same place. An NVIDIA 2026 survey reported 48 percent of respondents citing data-related issues, 38 percent a shortage of AI expertise and 30 percent a lack of ROI clarity as blockers. Data, people and measurement, in that order. Nothing in that list is solved by choosing a different model, and nothing in it is solved after the pilot has finished either. The rest of this article is an attempt to be specific about what those three categories actually look like on a real project on a real Tuesday afternoon.
Failure Mode One: Demo-Grade Data Versus Production Data
The pilot ran on a corpus that somebody chose. Typically 200 to 2,000 documents, hand-picked because they were clean, current, representative of the interesting cases and conveniently available. Production runs on 50,000 to two million documents that nobody chose, including the scans made on a fax-era device in 2011, the four near-identical versions of the same policy, the documents in a second language, the ones that were superseded but never marked, and the folder that three departments each believe belongs to someone else. These are different problems wearing the same interface.
What changes between them is measurable and specific. The distribution of scan quality widens, so extraction accuracy stops being a single number. Duplicates and superseded versions start competing for the same retrieval slots, which is worse than missing documents because the system answers confidently from a document that was correct two years ago. Document types nobody mentioned in the workshop appear at 3 to 8 percent of volume. And permission boundaries, which did not exist in the pilot because there was one user with access to everything, suddenly determine what may legally be retrieved for a given person.
The effect is large enough to derail a project's credibility. In our experience, retrieval quality measured at around 90 percent on a curated demo slice frequently lands 20 to 35 points lower the first time it meets a random production sample. Nobody has done anything wrong; the pilot simply measured a different thing from the thing that matters. But the meeting where that number is presented is the meeting where a promising pilot becomes a controversial one, and it usually happens after the budget conversation rather than before it, which is the worst possible sequence.
The fix is procedural and costs nothing at the start. Draw the pilot corpus by random sample, not by curation, and let the ugly documents in. Report the pilot's accuracy against that random sample and against the curated slice separately, so everyone can see the gap and plan for it. Then treat closing the gap as data work with its own budget, which is the subject of our note on preparing data for an AI project. A pilot that starts with representative data is slower to impress and far faster to ship.
Failure Mode Two: Nobody Agreed the Acceptance Criteria Before the Demo
Ask the six people in a demo review what would make the system good enough and you will get six answers, none of them written down. The head of operations is thinking about whether it saves time. The compliance lead is thinking about the worst possible output. The engineer is thinking about retrieval precision. The sponsor is thinking about the board. Because no threshold was agreed in advance, the demo is judged on impression, and impression is not a state you can engineer towards. Every subsequent iteration is therefore aimed at a target that moves whenever the audience changes.
An acceptance criterion is a sentence with numbers in it that a non-engineer can check. For example: on the agreed golden set of 240 questions, at least 85 percent of answers must cite the correct clause; no more than 2 percent may be confidently wrong in a way a reviewer would not catch; p95 response time must stay under 6 seconds at 30 concurrent users; cost per answered question must stay under an agreed ceiling; and every answer must carry a citation a user can open. Five sentences, agreed in week one, signed by the people who will approve the budget.
The political function of these sentences matters as much as the technical one. Without them, not accurate enough is unfalsifiable and therefore unfixable; with them, the conversation becomes which of these five numbers is failing and what would move it. It also protects the pilot from the opposite failure, where a system that comfortably passes every stated criterion is still refused because someone found one bad answer during a demo. Both outcomes waste the same budget, both damage the same credibility, and both are avoided by the same one-page document written at the start.
Write the criteria before the first demo, not after it, and write them jointly. A supplier who proposes the thresholds alone is guessing at your risk appetite; a buyer who sets them alone usually asks for accuracy that costs three times what the use case is worth. The negotiation itself is valuable, because it forces someone to say out loud what happens when the system is wrong and who carries that. We treat this as part of scoping rather than delivery, and the mechanics are in how to scope an AI project.
Failure Mode Three: No Owner After Handover, and the Integration Debt
A pilot has a sponsor. Production needs an owner, and they are different roles. The sponsor secures funding and attention for a period; the owner accepts a permanent operational burden: an on-call rota, a backlog, a monthly cost line, responsibility when the system says something wrong to a customer, and the annual argument about whether to keep paying for it. In pilot purgatory, the sponsor exists and the owner does not, so the project cannot move forward even when every technical box is ticked. Nobody is refusing; there is simply no one whose job it becomes.
The other half of this failure is integration debt. The pilot read a CSV export that someone dropped into a folder each Friday. Production needs a live connector with authentication, incremental sync, permission propagation, retries, and a defined behaviour when the upstream system is down for maintenance. As a planning figure, that is 3 to 6 person-weeks per system, and it is normally discovered around week nine, after the demo has already set expectations about the timeline. Multiply by the three or four systems that turned out to be involved and the remaining work exceeds everything done so far.
Both are cheap to prevent. Name the production owner in the pilot charter, before the pilot starts, and make the budget line theirs rather than the sponsor's. Then build exactly one real connector during the pilot rather than after it, chosen as the most awkward of the systems involved rather than the easiest. The pilot will look less polished at the demo and will carry a realistic estimate for everything that remains. That trade is almost always worth making, because a slightly less impressive demo with a named owner ships, and a spectacular one without an owner does not.
Failure Mode Four: No Evaluation Harness, So Nobody Can Prove a Change Is Safe
Sooner or later something changes that you did not choose. A provider deprecates the model version you built on, a library changes its default, your corpus grows by a third, or someone edits a prompt to fix one complaint. Without an evaluation harness, the only available answer to did it get worse is a person's recollection of a handful of queries. That is not a basis on which any risk committee will approve a production release, and quite reasonably so. The absence of a harness is what converts a working pilot into a permanent pilot.
Concretely, a harness is five things. A golden set of 150 to 300 real items with agreed correct answers. Retrieval metrics scored separately from answer quality, so you can tell whether the system failed to find the document or found it and reasoned badly. An automated regression run that any team member can trigger and that finishes fast enough to be run before every change. Assertions on cost and latency alongside the quality scores. And a scoring rubric that a human agreed to before anyone saw the numbers. Initial construction is typically 4 to 8 person-weeks on a mid-size system.
The organisational effect is larger than the engineering one. A harness converts an argument into a number, and a number can be signed off. It lets you tell a compliance function exactly how the system behaves on the hard cases, and it lets you accept a model upgrade on Tuesday instead of scheduling a two-week evaluation project around it. It also changes the vendor relationship, because both sides are now looking at the same scoreboard rather than negotiating over anecdotes collected by whoever complained loudest last week.
If a pilot has no harness, that is the single most valuable thing to add before asking for production budget, ahead of any accuracy improvement anyone is proposing. It is also the first artefact to ask for when you inherit a system somebody else built, because its absence tells you roughly how much unmeasured risk you have just taken on. We go through the metric choices, the golden set construction and the trap of scoring with the same model you are testing in our note on evaluating AI systems.
Failure Mode Five: Latency and Cost That Only Appear at Real Volume
A pilot is one enthusiastic user running perhaps 50 queries a day, usually in the afternoon, usually with short questions. Production is 5,000 queries a day arriving in bursts, with thirty people logging in within the same ten minutes on Monday morning, asking longer questions against a larger index, several of them uploading a document at the same time. Nothing about the pilot measured that, so nothing about the pilot predicts it. This is the failure mode that arrives last and hurts most, because it appears after go-live rather than before it.
Latency degrades in ways that are invisible at one user. A demo that answers in 2 to 4 seconds at p95 can land 3 to 5 times higher under real concurrency if embedding calls are not batched, if there is no cache in front of repeated queries and if requests are not queued with a sensible backpressure policy. Reranking a top-50 candidate set adds roughly 100 to 400 milliseconds on its own, which is a good trade for quality and a bad surprise if nobody budgeted for it. Users abandon a tool that takes fifteen seconds, whatever its accuracy.
Cost moves by roughly the same multiple as volume, but the per-unit figure also moves, because production prompts are longer than pilot prompts. More retrieved context, longer conversation history, more system instructions accumulated from edge cases. We plan cost per completed task rather than per token, measure it in the pilot at realistic prompt lengths, then multiply by honest volume and add a margin for the growth in context that always happens. A pilot costing a rounding error can become a monthly line that requires an approval nobody has requested.
The fix is to load test during the pilot at three times expected peak concurrency, to record p95 rather than average, and to ship a monthly cost ceiling with an alert before go-live rather than after the first invoice. Put both numbers in the acceptance criteria so they are gates rather than observations. And carry them into the business case, because a system that is worth building at one cost per task may not be at four times that; the method for keeping that honest is in measuring AI return on investment.
Failure Mode Six: The Security and Permission Review That Arrives at the End
The pilot ran with a service account that could read everything, over a copy of a folder, on a laptop or in a sandbox tenant. Nobody asked about permissions, because there was one user and that user was allowed to see all of it. Then the project approaches production, information security is invited for the first time, and the questions begin. They are all reasonable questions. They are also, collectively, several weeks of work that was never in the plan, arriving at the moment when the plan has the least slack left in it.
The review will ask whose permissions apply at query time and how the system proves it, what is logged and retained and for how long, where data physically goes and which sub-processors touch it, what happens when a user pastes confidential text into a prompt, how the system behaves when a retrieved document contains instructions aimed at the model, and who can see the logs. Under Law No. 6698 the duty to inform in article 10 and the data security obligations in article 12 both bear directly on those answers, and they are much cheaper to design for than to retrofit.
In our experience a security review that lands in week nine adds 3 to 6 person-weeks of engineering plus two to four weeks of calendar for approvals, and it occasionally invalidates an architectural choice outright, such as an index built without per-document access control. The fix is to hold a one-hour review in week two with a one-page architecture diagram, a list of data flows and a named reviewer. It costs almost nothing and it moves the expensive discovery to the point where changing the design is still cheap, a pattern we describe further in AI and data privacy.
The Exit Criteria a Pilot Must Satisfy to Earn Production Budget
Here is the gate we would apply, and it is deliberately unromantic. One, an evaluation harness exists, with a golden set of 150 to 300 items and thresholds agreed in writing by the people who approve the budget. Two, accuracy is reported against a random sample of the production corpus, not a curated slice, with both numbers shown side by side. Three, a named production owner exists, with a budget line and an on-call arrangement. Four, at least one real integration is live rather than simulated by a file export. Five, p95 latency has been measured at three times expected peak concurrency.
Six, cost per completed task has been measured at production prompt lengths, with a monthly ceiling and an alert configured. Seven, the permission model has been tested with a least-privileged account and a deliberately hostile query set that tries to reach documents the user cannot open. Eight, a rollback path exists: the previous prompt, index and model version can be restored in under thirty minutes by someone who is not the original author. Nine, an incident runbook names an on-call owner and documents a kill switch. Ten, logging and retention have been reviewed against the relevant KVKK duties.
The point of writing this as a gate is that a pilot must be allowed to fail it. If a pilot cannot pass, the honest outcomes are to fund the specific gap or to stop, and both are better than another quarter of promising results. We would add one more habit: run a three to four week shadow period where the system answers alongside the humans without being acted on, and compare the two. It is the cheapest production-grade evidence available anywhere. If you are also evaluating who should build the production version, the questions to ask are in how to choose an AI development partner.
How We Run Pilots So They Can Graduate
Our pilots are designed to be boring in the first two weeks and credible in the last two. We draw the corpus by random sample and show the accuracy gap against a curated slice rather than hiding it. We build the golden set with your subject experts rather than for them, because a set written by engineers measures what engineers assumed. We write the acceptance criteria before the first demo and put the numbers on the same page as the demo itself. And we build one real connector during the pilot, choosing the most awkward system rather than the friendliest.
We also try to be explicit about the branch where the answer is not to build at all. If the capability is a commodity, if the volume is low, and if a product on the market already does it acceptably, a pilot is an expensive way to reach that conclusion and a two-week product evaluation is a cheap one. We set out that decision properly in build versus buy for AI. A pilot should be the answer to a question that a purchase cannot settle, not a default first step taken because a pilot is what everyone else did.
Across caseon.ai, DiligenceAI, SYDhub and VinçTakip, the consistent pattern has been that the work between demo and production was larger than the work before the demo, and that the projects which planned for that ratio from the start were the ones that shipped. If your pilot has been promising for two quarters, the useful next step is usually the exit-criteria review rather than another model experiment. Our custom software development team runs that review as a fixed-scope engagement, and the wider context for the decision sits in our overview of AI for business.