How to measure AI impact and ROI: cycle time, quality and cost per task

How do you measure the ROI of AI?

Measure what AI changes in the work, one workflow at a time. First, record how the workflow runs today. Track the same measures once AI is part of it, and compare. The return is the difference, priced in hours and dollars, minus what the new way costs to run.

Measure at the level of the workflow, because that is where AI enters the work. Company-wide numbers, such as revenue per employee, move for many reasons at once. A workflow has a clear start and end, a volume and a check for a correct result, so a change shows up in its numbers. Start with the workflows that take the most hours or carry the most risk; see how to find where AI belongs.

Five measures cover most workflows:

Each measure guards against a way AI can look better on paper than in the work. Speed without quality only makes mistakes faster. A cheaper model saves little if people redo its work. Freed hours are worth money only once they go to other work. And a tool that half the team drops after a month won’t deliver what the pilot promised.

Why don’t usage numbers and hours-saved surveys prove AI impact?

Because they measure activity and opinion. The UK Department for Business and Trade’s evaluation of its Microsoft 365 Copilot pilot shows the gap. During the three-month pilot, with 1,000 licenses, 72% of respondents were satisfied, and in an average week 64% of license holders used the tool. Yet the evaluators found no robust evidence that the time saved led to higher productivity, while noting that they collected little data on that question.

Self-reported time savings also shrink under follow-up questions. In the same pilot, people logged their tasks in a diary and estimated the time each one saved. For data analysis, they reported an average of two hours per task. Once outputs they never used, and tasks they only did because they had the tool, were taken out, the saving was 0.6 hours. In a cross-government Copilot trial with 20,000 UK government employees, survey respondents reported saving 26 minutes a day on average. The Government Digital Service, which ran it, notes that the savings are self-reported and that it could not tell how the saved time was spent.

Even experienced people misjudge how much AI speeds them up. In a randomized trial by METR, a research group that evaluates AI models, 16 experienced open-source developers worked on 246 real tasks in their own projects. Each task was randomly assigned to allow or forbid AI tools.

Read it as a snapshot of one setting. METR now calls the results out of date, and believes developers were likely sped up more by early 2026. Its finding on method still holds: anecdotal reports and estimates of speed-up, METR concluded, “can be very inaccurate.” Time the work instead of asking about it.

How do you set a baseline before AI goes live?

Measure the workflow as it runs today, with the measures and checks you will use afterwards, before anyone changes how they work. Do it early, because the old way gets harder to measure once people rely on AI. When METR began a second trial in August 2025, more developers declined to take part because they didn’t want to work without AI, and 30% to 50% of those surveyed said they had held back tasks for the same reason.

A good baseline takes five steps:

1. Define the unit and what “done correctly” means. One ticket routed, one invoice matched, one deal approved, each judged by the team’s own check.
2. Pull the history. Timestamps and volumes in the ticketing system, ERP or CRM give you months of cycle time and throughput.
3. Time a sample of the hands-on work. Systems show when work moved, not how long people spent on it. Time a few dozen real cases rather than asking for estimates.
4. Score a sample on the team’s checks. Record rework and serious errors separately, and price each case: people’s time at a loaded hourly cost, plus tools.
5. Cover a full cycle. Include a month-end or a seasonal peak, and the exceptions at their real rate. Then write the baseline down and don’t change it.

Lock it in before the announcement. The UK government’s guidance on evaluating AI notes that outcomes can shift before a rollout, in anticipation of it. The past cases you collect also make a test set for choosing between setups; see how to benchmark AI on your own work.

Which metrics show the impact of AI on a workflow?

The five measures from Figure 1, defined so they mean the same thing before and after:

• Speed. Cycle time runs from the moment a case arrives to the moment it is done correctly, waits included. Throughput is how many cases you finish a week, in total or per person-hour; a study of 5,172 customer-support agents used issues resolved per hour as its main measure. A faster step rarely shortens cycle time if the case still waits for an approval.
• Quality. Accuracy on your own work: the share of cases that pass the check you set in the baseline. Count serious errors on their own, because a good average can hide one expensive mistake.
• Cost per task done correctly. Everything a case costs (people’s time, model usage, review and rework), divided by the cases done right. It puts people, agents and agents with a reviewer on one scale. See cost per task, or try the calculator.
• People’s time, and where it went. Freed hours turn into savings only when they go to other work, fewer hires or less overtime. In the Department for Business and Trade’s pilot, some people said they moved saved time to more strategic work; others used it to “attend training sessions or take a lunchtime walk.”
• Adoption that lasts. The share of eligible cases handled the new way, a month in and three months in. In the customer-support study, agents followed the AI’s suggestions more closely over time, and those who followed them more closely gained more.

Here are the measures for ticket triage at the illustrative company from Nolli’s product demo, where an agent now routes every ticket.

Two things make this comparison fair. The volume is the same before and after, so the drop in hours isn’t a quiet month. And people and the agent are scored on the same check: routed correctly. The 192 hours freed are worth about $152,000 a year at $66 an hour, once they go to other work.

How do you know the gain came from AI?

Compare against what would have happened anyway. Volumes swing, seasons change, people gain experience and other projects land in the same quarter. A plain before-and-after comparison credits AI with all of it.

The customer-support study shows how much this matters. Agents with the AI assistant resolved 2.5 issues an hour, against 1.7 for agents who never got it. But the agents who got it had already been faster, at 2.0 an hour before the rollout. Once the researchers accounted for who got the tool and when, and for how long each agent had been on the job, the gain was 15%.

Most companies can run a staggered rollout. Training, licenses and budgets force a rollout in waves anyway, so set the order in advance and keep measuring the teams still waiting. In the support study, the waves came from limits on training capacity and budget. Microsoft, Accenture and a Fortune 100 company went further. As part of ordinary business, each gave a random subset of its developers an AI coding assistant. Each experiment on its own was noisy. Pooled across 4,867 developers, completed tasks rose 26% among those who used the tool. Two of the study’s six authors work at Microsoft.

If you redesign the workflow at the same time, report that too. Part of the gain may come from deleted steps and plain code, and that part doesn’t depend on the model.

When should you measure AI impact: pilot or steady state?

Measure from day one, but judge at steady state. Pilot numbers can mislead in both directions. Volunteers, hand-picked cases and extra support flatter them; about 70% of the Department for Business and Trade’s pilot licenses went to volunteers. Learning curves hide gains. METR notes that learning effects might only show after several hundred hours of use, while its developers had typically used the main tool, Cursor, for a few dozen. In the support study, agents leaned on the AI more as time went on.

• Pilot, the first weeks. Does it work on real cases, and what does each case cost? Run a new agent in shadow mode first, so its mistakes don’t reach customers.
• Steady state, after a full cycle. Once people have settled in and the work has been through a peak, such as a month-end. Report this number.
• Every quarter after that. Measure against the same baseline, and again whenever the model, a prompt, a price or the process changes. The UK government’s guidance recommends quick checks during rollout and regular evaluations after it.

How do you report AI impact to the board?

Give each workflow one page, in the same format every quarter, so the board can compare workflows and follow trends. Put the decision you need at the top and the evidence under it.

Four rules keep the pages honest:

• Show the baseline next to every number. Triage taking 0.3 minutes a ticket means little without the 3.4 it replaced.
• Separate measured from estimated. Label anything self-reported, and state the hourly cost you used for people’s time. Our examples use $66 an hour.
• Say how you know. Name the design behind each result, from Figure 4. A plain before-and-after comparison deserves less weight than a staggered rollout.
• Report what you stopped. List the workflows you kept human or dropped, and why. Choosing to forego AI is a result too.

Put a summary page on top: total hours freed and where they went, cost per task by workflow, and the full cost to run. Add workflows together only after each has its own baseline.

What are the most common mistakes in measuring AI ROI?

Watch for these six:

• Counting activity as impact. Licenses, logins and prompts show use. Report them as context, never as return.
• Trusting estimates of time saved. Time a sample of the work, and ask where the time went.
• Skipping the baseline. Rebuilding one after rollout is guesswork, especially for people’s time and quality.
• Tracking speed without quality. In the support study, the most skilled agents got slightly faster with AI and slightly worse on quality.
• Leaving out review and rework. A model that is cheap per call but often wrong can cost more per correct task than a pricier one.
• Calling the pilot the result. Report steady-state numbers, and measure again when anything underneath changes.

Sources

1. Digital Data and Technology Monitoring and Evaluation team. “The Evaluation of the M365 Copilot Pilot in the Department for Business and Trade.” UK Department for Business and Trade, August 28, 2025.
2. “Microsoft 365 Copilot Experiment: Cross-Government Findings Report.” Government Digital Service, GOV.UK, June 2, 2025.
3. Joel Becker, Nate Rush, Beth Barnes and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR, July 10, 2025.
4. “We are Changing our Developer Productivity Experiment Design.” METR, February 24, 2026.
5. Frontier Economics and the Department for Transport. “Guidance on the Impact Evaluation of AI Interventions.” UK Evaluation Task Force, an annex to HM Treasury’s Magenta Book, August 2024.
6. Erik Brynjolfsson, Danielle Li and Lindsey Raymond. “Generative AI at Work.” The Quarterly Journal of Economics, May 2025.
7. Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, February 27, 2026.