Most AI pilots are designed to succeed as demos and then fail as workflows
In July 2025 a group at MIT's NANDA initiative circulated a report titled "The GenAI Divide: State of AI in Business 2025," and one number from it traveled much faster than the rest of the document: 95% of organizations, it said, were getting no measurable return from their generative AI efforts. The figure was repeated in headlines within weeks. Read in the report's own terms, it rests on a systematic review of more than 300 publicly disclosed AI initiatives, structured interviews with 52 organizations and survey responses from 153 senior leaders, collected between January and June 2025, and the authors label the work preliminary findings. Company data in it is anonymized, which makes the sample hard to check, and, as one critic observes, the initiative that produced the report promotes agent infrastructure, which is the direction its conclusions point. None of that makes the finding false. It means 95% is a headline and not a measurement, and that anyone planning around it should look for corroboration.
Corroboration exists, with different definitions and different numbers. McKinsey's survey "The state of AI in 2025" found that nearly two-thirds of respondents had not yet begun scaling AI across their enterprise, and only 39% reported any enterprise-level effect on earnings. S&P Global Market Intelligence surveyed 1,006 IT and business professionals in North America and Europe and, according to secondary coverage, found that 42% of companies had abandoned most of their AI initiatives in 2025, up from 17% a year earlier, with the average organization scrapping 46% of its proofs of concept before they reached production. Gartner had predicted in 2024 that at least 30% of generative AI projects would be abandoned after the proof-of-concept stage by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value.
The statistics differ in quality, and I would not rely on any one of them. What they share is a shape. Trying AI is nearly universal, and turning trials into scaled, measurable value is not. This essay asks why, and argues that the cause lies mostly in how pilots are designed and owned, with the technology a smaller part of the story than the vocabulary of the debate suggests.
A counterexample that shows what working looks like
Before the diagnosis, a case where the value was real and measured. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the rollout of a generative AI assistant to 5,179 customer support agents, introduced to different agents at different times. The staggered timing is what makes the result credible, since agents who had not yet received the tool served as a comparison group. Access to the assistant raised productivity, measured as issues resolved per hour, by about 14% on average. For novice and low-skilled workers the gain was 34%, and for experienced, highly skilled workers it was minimal. The authors also found improved customer sentiment and higher employee retention.
Notice what the deployment had. The assistant sat inside the agents' working screen, on the task they already performed. The outcome was a number the company already tracked. The rollout was designed so that its effect could be separated from everything else going on. Few pilots have all three properties, and I think their absence explains more stalled projects than any model limitation.
Five ways a pilot is the wrong shape
The first is that the pilot answers the wrong question. A typical pilot asks whether the model can do the task, and the answer is nearly always yes, since demos are easy. The question that decides whether the project survives is whether the workflow will change: who will do what differently on Monday, using which screen, with what authority, and who loses a task or gains one. McKinsey's analysis found that intentionally redesigning workflows was one of the strongest contributors to business impact of all the factors it tested, and that the roughly 6% of respondents it classed as high performers, attributing at least 5% of earnings to AI, were nearly three times as likely as others to have redesigned workflows. A pilot that bolts a chat window onto an unchanged process proves the technology and leaves the process alone.
The second is that there is no baseline and no counterfactual. Without a record of how the work was done before, and a comparison group during the pilot, the claim of improvement is an impression. Enthusiastic users will report that the tool is helpful, which is a different thing from a measured change in time, error or revenue. The same problem arises whenever a treatment is evaluated: you cannot tell what it did without something to compare with. When the CFO asks for the return, the answer then consists of anecdotes, and a project that cannot state its return is the first to be cut in a budget review.
The third is ownership. Pilots are usually owned by the technology function, because that is where the skills sit, while the benefit lands in operations, sales or service. The person who would have to accept the change in the workflow, and whose results would improve or not, has no stake in the pilot and no obligation to adopt its outcome. A project owned by someone whose quarterly goals do not depend on it will survive only as long as the sponsor's attention does.
The fourth, which the MIT report emphasizes, is that tools do not fit the way work is actually done. The report's authors describe a learning gap: generic tools are flexible but do not retain feedback or adapt to local context, which makes them fine for individual use and poor for workflows with their own vocabulary, exceptions and approvals. I would put this as a testable hypothesis and not a finding, since the evidence is interview-based. It matches a common experience, though. A model that is excellent on a benchmark of public documents and unaware of the company's product codes, discount rules and escalation paths will produce answers that are plausible and slightly wrong, and slightly wrong at scale is expensive to review.
The fifth is that pilots are scoped to avoid the hard parts. To get approval quickly, a pilot runs on a sandbox with a clean extract of data, a small set of willing users and no connection to production systems. That avoids security review, permission models, integration with the system of record, data quality problems and the cost of operating at volume, which are precisely the things that decide whether the project can be run in production. When the pilot ends and the real work begins, the team finds that most of the effort and risk lie ahead and the budget was sized for the part that is finished.
Where the effort should go
Boston Consulting Group's October 2024 survey of 1,000 senior executives across 59 countries offers a way to think about allocation. It found that 74% of companies had yet to show tangible value from AI. Only 4% had built advanced capabilities across functions and generated significant value consistently, and another 22% were beginning to see substantial gains. A secondary summary of the report describes the leaders as putting roughly 70% of their effort into people and processes, 20% into technology and data and 10% into the algorithms themselves, and as concentrating on a few high-priority projects instead of spreading across dozens. I would check the split against the full report before quoting it, but the direction matches the argument here. Most of the work in an AI deployment is outside the model, and a company that budgets as if the model were most of it will run out of money and patience at the point where the pilot ends.
The concentration point has a practical corollary. A portfolio of thirty small pilots divides the scarce resource, which is the attention of business owners who can change a workflow. Five projects with owners, baselines and a decision date will teach more than thirty without them.
Measure adoption and value separately
A pilot report that leads with usage statistics is describing curiosity. Logins, prompts and messages sent show that people tried the tool, and say nothing about whether work improved. Track the two families apart. Adoption measures show whether the tool is part of the routine: the share of eligible tasks handled with it, repeat use after the first week, the rate at which users override or discard its output. Value measures show what changed: time per unit of work, error or rework rate, customer satisfaction, revenue per interaction, with cost per unit set against them.
The relationship between the two is informative. High adoption with no change in value means the tool is pleasant and changes nothing, or that users are spending the saved time on rework. Low adoption with high value among those who use it points to a training or workflow problem that is worth solving. Neither pattern appears if only one family is measured.
Abandonment can be rational
A caution is in order. High abandonment is not necessarily bad. If trying an idea is cheap and informative, then a company that tests many and kills most is doing what an experimental portfolio should do. S&P's figure that organizations scrap 46% of proofs of concept before production could describe a healthy filter or a dysfunctional process, and the number cannot tell you which.
The test is how the decision to stop is made. A healthy kill happens on a stated date, against stated criteria, with a written lesson. A dysfunctional one happens by drift: the sponsor moves on, the budget lapses, the team is reassigned, and no one can say what was learned or whether the idea failed or was merely neglected. The first produces knowledge. The second produces a graveyard and a sour memory that makes the next proposal harder.
Design the pilot as the first slice of production
The practical remedy is to change what a pilot is. Treat it as the first slice of the real system, built with the real users, the real data, the real permissions and a real measurement plan, only smaller in scope. A one-page charter forces the discussion.
| Field | What it must say |
|---|---|
| Business owner | A named person who runs the process and whose targets depend on it |
| Workflow change | What people will do differently, step by step |
| Value metric | One measure in operational or financial terms, already tracked |
| Baseline and comparison | How the current performance was measured, and which group is the control |
| Success and kill criteria | The numbers that mean scale, and the numbers that mean stop |
| Decision date | When the owner decides, regardless of mood |
| Run cost | The expected cost per unit at full volume, including review labor |
| Risk and compliance | Who approved the data use, permissions and failure handling |
The measurement plan deserves a little arithmetic, because pilots are often too small to prove anything. Suppose the goal is to cut average handling time by 10%, and handling times vary with a standard deviation of about 40% of the mean, which is typical of service work. Detecting a 10% reduction with 80% power at the 5% level takes about 250 tickets in each group. A pilot that observes 40 tickets will produce a number, and the number will be mostly noise. Run the calculation before the pilot and set its length to match, with the staggered or randomized rollout that gave the support-agent study its credibility.
Run cost needs the same discipline. A pilot with fifty users and a mid-tier model may cost little, and the same workflow at fifty thousand users, with a human reviewing a share of outputs, may not pay back. Put the per-unit cost at full volume in the charter, next to the value per unit, so that the economics are visible before the architecture is.
What to do with the stalled ones
A company with a shelf of stalled pilots can triage them with three questions. Does a business owner exist who still wants the result? Is there a measured baseline against which the pilot can be judged? Is the workflow change specified? Pilots with three yeses deserve a second chance with a proper charter. Pilots with none should be closed, with the lessons written down and the data and prompts archived so the work is not wasted. The ones in between usually need an owner more than they need more engineering.
The point to take from the statistics, whichever of them one believes, is that the technology has crossed the line of being good enough for many tasks, and the companies that benefit are the ones that treat deployment as an organizational design problem. A pilot is the cheapest place to learn that, and the lessons are worth more when the pilot was built to teach them.
Sources
- Challapally, A., Pease, C., Raskar, R., Chari, P. (July 2025). The GenAI Divide: State of AI in Business 2025. MIT NANDA, preliminary findings.
- McKinsey & Company (2025). The state of AI in 2025: Agents, innovation, and transformation. mckinsey.com.
- S&P Global Market Intelligence (2025). Survey of 1,006 IT and business professionals on AI initiatives, as reported by CIO Dive (2025).
- Gartner (2024). Gartner predicts 30% of generative AI projects will be abandoned after proof of concept by end of 2025. Press release.
- Brynjolfsson, E., Li, D., Raymond, L. R. (2023, revised). Generative AI at Work. NBER Working Paper 31161; published in the Quarterly Journal of Economics (2025).
- Boston Consulting Group (24 October 2024). Where's the Value in AI? Survey of 1,000 senior executives, press release and report summary.