Hours saved are not money earned, and AI business cases confuse the two

In the first half of 2025 a research group called METR ran a randomized trial with sixteen experienced open-source developers. Each worked on mature projects they knew well, with about five years of prior experience on average, and each task, 246 in total, was randomly assigned either to allow or to forbid the AI tools of the time, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet. Before starting, the developers predicted that AI would cut their completion time by 24%. The measured result was the opposite: with AI allowed, tasks took 19% longer. After the study, when asked, the developers estimated that AI had reduced their time by 20%. Economics and machine-learning experts surveyed beforehand had predicted speedups near 40%.

The study was small, with a particular population of highly skilled people working in codebases they knew intimately, and its authors say plainly that it should not be extended to all developers or all tasks. It is nonetheless a useful warning about the evidence most business cases rest on. The people using the tool were wrong about its effect on them, by about forty percentage points and in the wrong direction. A business case built on surveys of how much time people think they saved is built on a measurement that, in at least one careful test, did not track the truth.

This essay follows an hour from the moment someone says it was saved to the moment it appears as money, and argues that most AI returns that are reported are potential and not realized. The distinction matters, because the cost side of the ledger is real and the benefit side is often a forecast.

Four numbers that look like return

The first is usage: how many people logged in, how many prompts were sent. It shows interest. The second is self-reported time saved, usually from a survey. It shows belief, and the METR result shows how far belief can be from measurement. The third is measured time per task for tasks done with and without the tool, which is genuine evidence of a change in efficiency, though only for those tasks. The fourth is financial value: a cost that fell, a revenue line that rose, a headcount that was avoided. Only the fourth is a return, and it needs a path from the third.

That path has four conversions, and each loses something.

The first conversion is from reported time to real time. Self-reports overstate, as the trial showed, and tasks differ in how much the tool helps. A haircut is appropriate until measurement replaces it.

The second is from time saved on a task to time freed in a week. Five minutes saved on each of twelve tasks a day is an hour only if the tasks can be compressed and the person is not waiting for something else. Time saved in fragments tends to be absorbed, spent on slower coffee, more careful emails or work expanding to fill the day.

The third is from freed time to redeployed capacity. The hour must go to something valuable: more customers served, a project finished sooner, a backlog cleared. If the extra capacity has nothing to absorb it, the value is zero, however large the number of hours looks.

The fourth is from redeployed capacity to cash. Capacity becomes money only when it changes revenue or cost, through additional sales, avoided hires, reduced overtime or fewer contractors. Until it does, it is a possibility.

A worked example, and how fragile it is

Take a company that deploys an assistant to 1,000 employees who report saving three hours a week, working 46 weeks a year. Gross reported time saved is 138,000 hours. Suppose the loaded cost of an hour of their time is $40.

Now apply the conversions. Half of the reported time is real, so 69,000 hours. Suppose 40% of that is redeployed to valuable work, which leaves 27,600 hours. At $40 each, the benefit is about $1.1 million a year. The costs: licenses at $30 per user per month come to $360,000, and integration, training, evaluation, review labor and support add, say, $300,000, for a total of $660,000. The net is about +$444,000, a return of 67% on cost.

The result is sensitive. If only 20% of the real time is redeployed, the benefit falls to $552,000, and the net is −$108,000. The break-even realization, with half of reported time real, is about 24%. A company that reported the first figure had a good business case. A company whose employees redeploy their time less effectively is paying more than it earns. Both would show the same usage dashboards and the same enthusiastic surveys. Skipping the haircut, and treating all 138,000 reported hours as value, gives a benefit of $5.5 million and a return of more than eight times cost, which is the type of number that circulates in decks.

Why the hours do not show up

The economist Robert Solow wrote in 1987 that the computer age could be seen everywhere but in the productivity statistics. The remark has been revived for AI because it keeps being true in a particular way. Studies find real task-level gains, and aggregate numbers move slowly.

Erik Brynjolfsson, Daniel Rock and Chad Syverson offered a mechanism in their work on the productivity J-curve. A general-purpose technology needs complementary investments in processes, skills and organization, which are intangible and unrecorded in the usual statistics. While those investments are being made, measured productivity can stall or fall, and the gains arrive later, once the complements are in place. For a company, the practical meaning is that a first-year measurement understates the long-run effect when the organization has not yet changed, and overstates it when the numbers come from the early enthusiasts.

The macro estimates are modest. A study from the Federal Reserve Bank of St. Louis surveyed Americans on generative AI use and found that respondents reported time savings equivalent to 1.4% of total work hours, with a range of 1% to 5% of work hours currently assisted by the technology. Daron Acemoglu's simple macroeconomic model, built from the share of tasks affected and the cost savings on those tasks, put the total factor productivity gain at no more than 0.66% over ten years, and argued the number might be lower because early evidence comes from easy-to-learn tasks. These are economy-wide figures, and an individual firm can do much better or much worse. They do suggest that claims of immediate, large aggregate gains deserve skepticism.

Where the evidence is strong

Some settings let the conversion happen. The customer-support study by Brynjolfsson, Li and Raymond measured issues resolved per hour for 5,179 agents, with staggered access that made a comparison possible, and found a 14% increase on average and 34% for novices. In that setting the output unit (resolved issues) was countable, the work was a stream of similar items so freed time could be filled by more items, and the company could convert capacity into money through fewer hires, faster response or better service. The properties that made the evidence strong are a standard unit of work, a queue of waiting items and a comparison group.

Where those hold, you can report cost per unit of work. Where they do not, as with knowledge work that is varied and has no queue, the honest report is a range with stated assumptions, labeled as potential.

The same study carries a second lesson about where to look. The gain was 34% for novices and minimal for experienced agents, and the developer trial found a slowdown among people with deep expertise in their own codebases. Benefits tend to concentrate among less experienced workers, for whom a tool supplies knowledge they lack, and to shrink or reverse for people who already do the task fluently and have to check or fight the tool's suggestions. A rollout that treats every employee alike will average these effects and hide both. Segment the measurement by experience, and the deployment plan by who is likely to gain, with onboarding speed and retention counted as benefits where they apply, since the support study also found higher employee retention.

A measurement design

A credible plan decides several things before the tool is turned on.

Choose the unit of work to count: a ticket resolved, a claim processed, an invoice reconciled, a pull request merged. Measure the baseline for a few weeks before deployment. Introduce the tool in stages across teams so that some are comparison groups, or randomize access within a team. Track quality alongside speed, with error rates, rework, customer satisfaction or defects, because faster and worse is a cost. Count the review time: when a person must check and correct the tool's output, that time belongs on the cost side. Compute cost per completed unit, including license, inference, support and review, and compare it with the baseline.

For varied knowledge work with no queue of identical items, the METR design offers a template that a company can copy. Randomize at the level of the task inside the same person: for each incoming task, a coin flip decides whether the tool may be used. Over a few hundred tasks, each person acts as their own control, and the comparison is not contaminated by differences in skill between people. Add a blind quality rating, in which a manager who does not know which condition produced the output scores it. A month of this costs little next to a license bill of several hundred thousand dollars a year, and it replaces a survey of beliefs with a measurement.

Quality belongs in the ledger as money. Suppose a tool cuts the time for a back-office transaction from six minutes to 4.8, and the company processes 100,000 a year. The saving is 120,000 minutes, or 2,000 hours, worth $80,000 at $40 an hour. If the tool also raises the error rate by one percentage point, there are 1,000 more errors a year, and at $200 to find and fix each, the cost is $200,000. The net is a loss of $120,000, hidden behind a time saving that every user would report with satisfaction. Guardrail metrics for errors, rework and customer complaints belong in the same dashboard as speed.

Then decide in advance what happens to the capacity. If the plan is to handle more volume with the same team, say what volume is waiting. If the plan is to avoid hiring, name the roles. If neither is true, the forecast benefit should be marked as a possibility.

The cost side deserves the same scrutiny

Costs are easier to see and easy to underestimate. License fees are visible. Inference charges are variable and depend on design: Anthropic's account of its research system reports that an agent uses roughly four times the tokens of a chat and a multi-agent system roughly fifteen times, so a usage-based bill can grow faster than the seat count. The larger costs are often internal: engineering time to integrate, the evaluation and monitoring work that keeps quality up, legal and security review, training and change management, and the managers' time spent redesigning workflows. A business case that lists only licenses understates the investment by a large factor.

What to tell the board

Boards and finance committees ask for a single return figure, and a single figure invites false precision. A better answer has three parts. State the realized value to date, with its source in the accounts. State the potential value that is being converted, with the plan and the date by which it should appear. State the range of outcomes under pessimistic, expected and optimistic assumptions about the conversions, so the committee sees the same table of sensitivities that the team used. A committee that has seen the break-even realization, such as the roughly one quarter in the example above, can judge whether the program is likely to clear it, and can ask the right question about each initiative: which of the four conversions is still unproven?

Two habits guard against self-deception. Have the measurement owned by someone outside the team that built the deployment, because the builders are the people most inclined to read an ambiguous number favorably. And publish the failed estimates: a program that reports only its wins teaches the organization nothing about which conversions fail, and the same errors will come back with the next tool.

Report realized value, and label potential

A practical convention separates two ledgers. Realized value covers changes that appear in a budget or a revenue line and can be traced to the deployment. Potential value covers freed capacity that has not yet been turned into either. Present both, with the conversion assumptions and the sensitivity, so the reader can see how much of the case depends on what has not yet happened. A program that can show realized value has a defensible case. A program that can show only potential has a hypothesis, and the remedy is a plan to convert it.

Sources

1. Becker, J., Rush, N., Barnes, E., Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR. arXiv:2507.09089.
2. Solow, R. M. (12 July 1987). We'd better watch out. New York Times Book Review.
3. Brynjolfsson, E., Rock, D., Syverson, C. (2021). The Productivity J-Curve: How Intangibles Complement General Purpose Technologies. American Economic Journal: Macroeconomics 13(1).
4. Bick, A., Blandin, A., Deming, D. J. (2024, revised 2025). The Rapid Adoption of Generative AI. Federal Reserve Bank of St. Louis Working Paper 2024-027; NBER Working Paper 32966.
5. Acemoglu, D. (2024). The Simple Macroeconomics of AI. NBER Working Paper 32487.
6. Brynjolfsson, E., Li, D., Raymond, L. R. (2023, revised). Generative AI at Work. NBER Working Paper 31161.
7. Anthropic (June 2025). How we built our multi-agent research system. anthropic.com/engineering/multi-agent-research-system.
8. Illustrative business case: 1,000 users, three reported hours saved per week for 46 weeks, $40 loaded hourly cost, $30 per user-month licenses and $300,000 of other costs, assumptions stated in the text.