The AI productivity illusion: what people feel versus what studies measure

How we read the studies

Four numbers, and what each was measured on

METR’s trial, its follow-up, and a small replication

METR recruited 16 experienced developers from large open-source repositories (averaging more than 22,000 stars and over a million lines of code), had them list 246 real issues from their own projects, and randomly assigned each issue to allow or forbid AI tools. Issues averaged two hours, and developers were paid $150 an hour. With AI allowed, issues took 19% longer. Before starting, developers had expected AI to cut their time by 24%. After finishing, they still believed it had cut their time by 20%. All of this is from METR’s own page, which now labels the results out of date.

METR lists its own limits. There were 16 developers, all experienced and all working in repositories they know. It could not rule out learning effects beyond 50 hours of using Cursor. It also could not rule out that developers who expected large gains declined to join, because they did not want to be forced to work without AI on half their tasks. The interval around the 19% runs from 2% to 39% longer.

In February 2026, METR published what happened next, and the answer is awkward. It had started a second experiment in August 2025 with 57 developers (10 from the first study and 47 new), 800+ tasks, and pay cut from $150 to $50 an hour. Raw results leaned the other way: for the returning developers, 18% less time with AI (interval 38% less to 9% more); for the new ones, 4% less time (15% less to 9% more). METR then judged the new data an unreliable signal. Developers increasingly declined to take part because they did not want to work without AI, and when surveyed, 30% to 50% said they were holding back tasks they would not want to do without AI. METR believes developers were likely sped up more in early 2026 than in early 2025, and calls its new data “very weak evidence” for the size of that change.

Read the chart for its width, not its middle. Only the first interval sits wholly above 100. The other three cross it, so they cannot rule out no effect, and the Evolver one spans 40% faster to 26% slower. A measured result with an interval that wide is a weak instrument too. That is a reason to measure more tasks, not a reason to trust the feeling instead.

The Evolver study repeated METR’s design in a software consultancy of under 30 people, as a master’s thesis at Aalto University, full text here. Seven developers finished 28 tasks over two weeks, 15 with AI allowed and 13 without, mostly with GitHub Copilot. Tasks took 13% less time with AI, with an interval from 40% less to 26% more (p = 0.449). The developers’ own estimate after the study was 14.3% less time, and before each task they had estimated about 17% less. Here belief and clock agree. With 28 tasks, we cannot say whether that agreement is information or luck. The thesis also records that in no task did a developer expect AI to slow them down.

Short tasks: a 58% forecast and an 8% result

Sunny Yu, Myra Cheng, Robert Hawkins and colleagues ran three preregistered studies with 2,691 US adults on Prolific, on 24 tasks that each take under five minutes: naming an Olympic medalist, adding 27 and 56, finding spelling errors in a sentence. In the second study, 600 people predicted how long tasks would take alone and with a chatbot, and 1,001 others did the tasks, alone or with GPT-4o. Predictors expected AI to cut time by 58%, from 97.1 to 41.4 seconds. The people doing the tasks took 93.7 seconds alone and 86.2 with AI. That is 7.5 seconds, about 8% (our calculation), and the overall difference was not statistically significant (p = 0.07). On the easy versions, AI was slower: 70.2 seconds against 60.2.

One detail matters for anyone running a survey. People predicted their own unaided time well, at 99 seconds against 93.7 actual. The error sat in the AI half: they predicted 43.3 seconds and took 86.2. The authors’ reading is that people can sense how long their own work takes and cannot sense how long working with a chatbot takes, with prompting and reading included. Two more results from the same paper point at behaviour. In a separate study of 498 people, AI was used on 47% of tasks against 33% predicted. And people who used AI on two tasks went on to use it on 44.5% of later tasks, against 27.7% for people who did the first two alone.

A shorter companion paper from the same team, with 1,237 participants, reports the same pattern and one result that cuts the other way: on the harder versions of the tasks AI made people 26.14 seconds faster. We count the two papers as one line of evidence, because the papers do not say whether their samples overlap. The caveat is the authors’ own: the tasks are simple, the sample is a general one, and the chatbot was GPT-4o. Do not carry 7.5 seconds into a finance close.

Okta: fewer hours reported, output unchanged

Fei Liu and Todd McKinnon followed engineers through a GitHub Copilot rollout at Okta, and published in Communications of the ACM on September 17, 2026. They invited 261 people who received licences between July and September 2024. Ninety-seven answered, and 74 had complete before-and-after data. The authors joined each survey to the company’s engineering systems and compared April to June 2024 with September to November 2024.

The engineers reported weekly work hours falling from 44.77 to 40.83, about 9% (our calculation), with 1.06 fewer hours on coding and 1.56 fewer on testing. From the systems, monthly pull request count, pull request cycle time and lines of code showed no significant change. Only the number of reviews performed rose significantly. The authors call the distance the efficiency-throughput gap. Same output in fewer hours would, taken at face value, be a gain per hour, and the authors say as much: an individual efficiency gain, not an organizational throughput gain. The weak link is that the hours are self-reported and the output is a count of pull requests, which the authors call a proxy. One more figure cuts against a simple “people are wrong” reading: the engineers’ self-reported Copilot use correlated with logged use at r = 0.48, a medium-to-large correlation.

The authors list their limits: one company, a moderate sample (so a non-significant result may be low power and not a true zero), the early rollout phase, and no randomization. They also note that survey respondents used Copilot more than all licence holders did, so the picture may be rosier than the whole population’s. They report that output metrics still showed no improvement when they kept watching from December 2024 to May 2025. The authors write about Okta as their own company, so this is a company measuring its own rollout.

The other direction: gains people did not credit

Garg and colleagues at The Hague University of Applied Sciences gave 159 staff, across teaching, research and support work, access to an institutional GenAI platform for four weeks. Inside the pilot, a micro-randomized trial had a work assignment done with or without the platform. Staff with the platform finished faster, and automated evaluators rated their output higher. Their own quality ratings were no higher than the others’. And their rating of how useful the platform was fell from 5.30 to 4.20 on a 7-point scale (p < .001). Researchers gained the most speed; support staff lost the most enthusiasm. We read the SSRN abstract page only, so we cannot give the size of the speed gain, how the tasks were timed, or how many staff were in each arm. The authors say hands-on use made staff more aware of the tool’s limits, and that automated and human raters may weigh quality differently. It is a preprint from one institution, in a pilot.

Cui and colleagues have no feelings to set against their measure, which is why they are here as a counterweight and not as a contrast. A random subset of developers at Microsoft, Accenture and a Fortune 100 company were given an AI coding assistant that suggests code completions, in experiments the companies ran in the ordinary course of business. Each experiment alone was noisy and results varied between them. Combined, across 4,867 developers, completed tasks rose 26.08% (standard error 10.3%). Less experienced developers adopted more and gained more. Two of the six authors work at Microsoft Research, and the funding statement names the MIT GenAI Initiative for two authors. It is the strongest design in our set, a published randomized result at three companies, and it is a software result.

The UK Department for Business and Trade evaluated its own three-month Microsoft 365 Copilot pilot, with 1,000 licences, about 70% of them to volunteers. People logged their tasks in diaries, recording hours taken and the hours they expected the task to take without Copilot. For data analysis (47 entries), the mean reported saving was 2.0 hours a task, and 0.6 hours once outputs people never used and tasks they only did because they had the tool were taken out. Then the department watched a few people work. Six Copilot users and five non-users in all did short tasks, so each task had only a few people. The Excel analysis took users 25:01 and non-users 20:33, with accuracy scored 1.5 against 2.7 on a five-point scale. Summarising reports took users 12:37 against 41:34, with accuracy 4 against 2.5. PowerPoint was faster, 10:47 against 18:30, and scored worse: accuracy 1.5 against 5. The department says plainly that these observed samples are small and supplementary. It also says it found no evidence that the saved time raised productivity, with little data collected on that question.

What a survey-only number looks like

METR also ran a survey from February to April 2026 of 349 technical workers (including 87 software engineers, 71 researchers, 129 academics and PhD students, and 48 founders and managers). The median respondent reported a 3x change in speed and a 1.4x to 2x change in the value of their work. Retrospectively they put their March 2025 gain at 1.3x. METR’s own cautions are the useful part: survey results are not necessarily grounded in reality; response was about 2% among people it emailed; and METR staff, who know its 2025 trial, gave the lowest answers of any group. METR also notes that public survey estimates have tended to run higher than field experiments, and that it is hard to say by how much. A dashboard that reports hours saved from a survey is this kind of number.

Why the studies may disagree

We cannot prove any of these. Each has something in the papers behind it.

What the data cannot say

• Whether the work got cheaper. None of the eight measured cost per task with review and rework included, so none says whether faster work was also cheaper work.
• Whether any of it applies to your workflow. Four of the eight studied software developers. The survey covered technical workers. Short online tasks, university staff and civil servants fill most of the rest. Finance, support and operations work is barely there.
• What happens over a year. The longest follow-up in the set is Okta’s, with output tracked into May 2025. METR suggests learning effects may only show after several hundred hours of use, and its developers had typically used Cursor for a few dozen.
• How agents compare. Every study here measures a person using a tool. None measures an agent running a workflow, so none tells you how people alone, people with AI and agents compare.
• That a null result means no effect. The Evolver interval runs from 40% faster to 26% slower. The Okta authors say their non-significant results could come from low power. An interval that wide cannot tell no gain from a modest one.
• That a feeling is wrong. A person who says the work got easier may be right about the part of it that matters to them. In the Yu companion paper, reported mental effort fell by 0.61 points on a 7-point scale with AI, while completion time was significantly faster on only three of 24 tasks. Time and effort are different things, and a survey that asks about one may be measuring the other.

What to measure instead

The studies suggest an order of preference. Asking people is cheap and the least able to separate AI from everything else that changed. Timing the same task before and after is a step up. A comparison group is another, and randomizing tasks is the strongest and the hardest: METR found people increasingly unwilling to work without AI. Below is our reading, with positions that are judgments and not measurements.

1. 01

   Baseline

   Take it before the change, from system history and a timed sample. METR’s experience in 2026 is the warning: once a tool is part of the work, the old way is hard to observe.

   Do this todayTime a sample of real cases by hand. Our guide’s example uses 40.

2. 02

   Same task, before and after

   Compare like with like. The Yu papers and the UK pilot both found that AI helped on some task types and hurt on others, so break results out by task type.

   Do this todayTag each case with a task type before you time it.

3. 03

   A timed sample, not a recollection

   Okta’s engineers reported fewer hours while counted output stayed flat. Only the output was counted by a system, so the hours could not be checked.

   Do this todayAsk people for their estimate afterward, then show them the clock. The size of the gap tells you where to look.

4. 04

   A quality check

   In the UK pilot, the faster PowerPoint was the lower-scoring one. In Garg and colleagues, self-rated quality did not move while automated ratings did. Faster only counts if the result is as good.

   Do this todayUse your team’s own check and let someone other than the author apply it.

Sources

1. Joel Becker, Nate Rush, Beth Barnes and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR, July 10, 2025.
2. Joel Becker, Nate Rush, Tom Cunningham, David Rein and Khalid Mahamud. “We are Changing our Developer Productivity Experiment Design.” METR, February 24, 2026.
3. Joel Becker. “Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity.” METR, May 11, 2026.
4. Lauri Nieminen. “For a few tokens more: The productivity effect of AI tools on software development tasks at Evolver Business Solutions Oy (master’s thesis).” Aalto University, April 25, 2026.
5. Sunny Yu, Myra Cheng, Ahmad Jabbar, Ilia Sucholutsky, Katherine M. Collins, Dan Jurafsky and Robert D. Hawkins. “The efficiency-gain illusion: People underestimate the rate of AI use and overestimate its benefits on simple tasks.” arXiv (2605.22687), May 2026.
6. Sunny Yu, Myra Cheng, Ahmad Jabbar, Ilia Sucholutsky, Katherine M. Collins, Dan Jurafsky and Robert D. Hawkins. “Cognitive offloading and the speedup illusion in human-AI interaction.” arXiv (2605.23177), May 22, 2026.
7. Fei Liu and Todd McKinnon. “Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot.” Communications of the ACM, September 17, 2026.
8. Manika Garg, Samuel Kernan Freire, Sunčica Bruck, Aslam Tanjung and Theo Bakker. “Generative AI in Higher Education Staff Work: Perceived versus Measured Effects (abstract page).” SSRN (7570368), October 6, 2026.
9. Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, February 27, 2026.
10. Digital Data and Technology Monitoring and Evaluation team. “The Evaluation of the M365 Copilot Pilot in the Department for Business and Trade.” UK Department for Business and Trade, August 2025.
11. “Connect to the Microsoft Copilot Dashboard for Microsoft 365 customers.” Microsoft Learn, accessed October 16, 2026.
12. “The system for AI enablement teams.” Nolli, accessed October 16, 2026.

Plain-text copy: /blog/ai-productivity-illusion-perceived-vs-measured.md