How to benchmark AI on your own work: people, people with AI, and agents

Why don’t public AI benchmarks predict results on your work?

Because they test a model on someone else’s tasks, graded by someone else’s idea of correct, and usually without the review your people would do. Public benchmarks are good for tracking which models are getting stronger. They can’t tell you which setup will do your work best, or what it will cost.

Take GDPval, one of the most realistic public tests of knowledge work. OpenAI built it from 1,320 tasks in 44 occupations, written by professionals who average 14 years of experience. A task takes an expert about seven hours on average. In the GDPval paper, the best model tested, Claude Opus 4.1, matched or beat the expert’s work 47.6% of the time on the tasks the authors published. Scores have climbed fast since.

All four numbers are real, and none of them is your answer. Each benchmark answers the question its builders asked. GDPval’s tasks are precisely specified and one-shot, and its authors leave out work that needs tacit knowledge, proprietary software or talking to colleagues. Mercor notes that its simpler accounting tasks tested exactly the skills models are best at, with no coworkers to ask and no accumulated context. Your work has all of these: unclear requests, unwritten rules, colleagues to ask and years of context.

Review is the other gap. OpenAI’s headline that models finish GDPval tasks roughly 100 times faster and cheaper than experts counts model time and API cost only. The paper’s authors then added an expert who reviews each attempt and does the task when the model keeps falling short. GPT-5 went from 90 times faster than the expert to 1.39 times, and from 474 times cheaper to 1.63 times. GPT-4o, an older model, ended up slower and more expensive than the expert working alone.

Which setups should you compare?

A model is only one part of how work gets done. For each workflow, run the same cases through up to four setups:

People alone is the baseline every other setup has to beat, so measure it with the same care: same cases, same rubric, timed. People with AI means your team using an assistant or shared skills. An agent alone uses the least people’s time and sends every mistake straight into the work. An agent with a reviewer puts a person in the loop to check each result before it goes out.

Here is the comparison for two workflows at the example company in Nolli’s product demo. It tested three of the four setups. In the revenue review, the agent’s 15 minutes of people’s time is a person reviewing its deck.

Speed alone would have picked the agent both times. In the revenue review, it caught all three conflicts and cut people’s time from two and a half hours to 15 minutes. In pricing exceptions, it decided in 4 minutes instead of 25, but a reviewer reversed 5 of its 12 decisions, and customers accepted only half of its explanations.

Do people and AI do better together?

Sometimes. The research says it depends on the task, and that you often can’t tell in advance which tasks AI will help with.

In a preregistered experiment run with Boston Consulting Group and published in Organization Science in 2026, Fabrizio Dell’Acqua and colleagues gave 758 BCG consultants realistic tasks, with or without GPT-4. On 18 product-development tasks inside what they call the “jagged technological frontier,” consultants with AI completed 12.2% more tasks, worked 25.1% faster and produced work rated 30% to 34% higher in quality. On a business case built to fall outside it, where the right answer depended on details in interview notes, consultants with AI were more often wrong.

The frontier is jagged because tasks that look equally hard to people can fall on either side of it, even within one workflow. The study used GPT-4, the leading model in mid-2023, and only one task outside the frontier, so treat its numbers as a snapshot. The unevenness hasn’t gone away: on GDPval, model win rates vary by occupation and fall as tasks get longer.

A 2024 meta-analysis in Nature Human Behaviour by Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone pooled 370 results from 106 experiments published between 2020 and mid-2023. On average, people working with AI beat people working alone, but did worse than the better of the person or the AI on its own. Three findings matter for a benchmark:

• The type of task. Combinations did worse on decisions, such as choosing between set options, and tended to do better on creating content, such as drafting.
• Who is better alone. When people outperformed the AI, combining them helped. When the AI outperformed people, adding a person made results worse.
• How the work is split. In more than 95% of the setups, a person made the final call after seeing the AI’s suggestion. Only 3 experiments gave people and AI separate parts of the task up front.

The authors flag possible publication bias and differences between study designs, and the studies predate today’s agents. The practical lesson is simple: a person checking an AI, or an AI helping a person, is not automatically better than either one alone. The BCG study’s authors suggest that, for each task in a workflow, companies should

“evaluate the value of using different configurations of humans and AI”

— Dell’Acqua et al., Organization Science, 2026

How do you build a personalized benchmark?

Start with two or three workflows, a few dozen real cases for each, and the people who already do the work. A personalized benchmark then takes five steps:

1. Pick workflows that matter. Choose work that is frequent, expensive or risky, where you can say what a good result looks like. Give each workflow an owner who will act on the result.
2. Collect real cases, exceptions included. Pull 30 to 50 recent cases from the work as it actually ran: the emails, files and records people used, and what they decided. Keep the awkward ones, such as late renewals, missing purchase orders and conflicting figures, at their real rate.
3. Agree on what correct means, and write a rubric. Before any setup runs, have two experienced people settle the right answer for each case. Then turn their judgment into yes-or-no checks: the right total, the right policy applied, the conflict flagged, nothing sent that shouldn’t be. Specific checks are easier to agree on and to automate. Ramp and Mercor’s accounting benchmark uses about 14 expert-written checks per task, and Mercor reports that its AI grader agrees with expert graders 97% of the time. Holistic judgments are harder to pin down: GDPval’s expert graders, comparing whole deliverables, agreed with each other 71% of the time.
4. Run every setup on the same cases. Time everything, including your people. A 2025 review of 115 human baselines in AI evaluations found substantial shortcomings in how rigorously people’s performance was measured. Run agents several times per case, because a model can solve a case once and miss it the next time. Ramp and Mercor ran every task eight times.
5. Compare, then re-run. Rank the setups by cost per correct result, and keep the cases as your evals. When a model, a price or the process changes, re-run them and, in Ramp’s words, “see whether the outcome improved—or the failure just moved.”

What should you measure?

Four things, for every case in every setup: time, quality, cost and review effort. Then combine them into one number, the cost per correct result.

Cost per correct result is everything a setup cost (people’s time, model usage, review and fixes) divided by the number of results that were right. It is the same idea as cost per task, applied to people as well as agents. A setup that is cheap per attempt but often wrong can lose to one that costs more and is nearly always right.

Never judge on speed alone. On the BCG task outside the frontier, consultants with AI spent 18% to 30% less time, and more of them got it wrong. Set a minimum quality bar first, especially where a mistake is costly or hard to undo, and compare cost only among the setups that clear it.

What are the most common benchmarking mistakes?

Most weak benchmarks fail in one of six ways:

• Testing the documented process. Cases built from the SOP leave out much of the work people do. In our revenue review example, the process document lists five steps; four real Mondays show 20. Build cases from records of the real work.
• Too few cases. With 20 cases, each one moves accuracy by five percentage points, so a gap of one or two cases between setups is noise. Add cases until the difference is clear.
• Scoring only the final answer. Mercor found that about seven in ten failures by the top models on its accounting benchmark came from flawed reasoning, such as spotting a discrepancy early and then dropping it from the final journal entry. Check the steps and the evidence as well as the total.
• Ignoring review time. If a person checks every output, the check is part of the cost. On GDPval, assessing a model’s deliverable took experts 109 minutes on average, against about 404 minutes to do the task.
• Asking instead of timing. In METR’s early-2025 trial, experienced developers took 19% longer with AI but believed it had sped them up by 20%. METR now says those results are out of date. The gap between feeling faster and being faster is the reason to time the work.
• Benchmarking once. Models, prices and your process all change. Last quarter’s result is this quarter’s hypothesis.

How do benchmark results turn into decisions?

Read the results one workflow at a time. Pick the setup with the lowest cost per correct result that clears your quality bar and your risk limits, then act on it:

• People alone wins: keep the work with people and re-test when models improve. That is the forego option in build, buy or forego.
• People with AI wins: give people the assistant or skill that worked, write down how they use it, and share it with every team that does the same work.
• An agent wins: decide whether to build it or buy it, and keep a reviewer until your results show that checking a sample is enough.

Often the answer splits by step, as the four buckets predict. At the example company, an agent prepares the revenue review and a person makes the call; in pricing exceptions, AI drafts the explanation and a person decides. Before a winning agent takes on real work, run it in shadow mode on live cases. Then keep re-running the benchmark, so the decision changes when the evidence does.

Sources

1. Tejal Patwardhan et al.. “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.” OpenAI, in the ICLR 2026 proceedings, April 2026.
2. “Introducing GPT-5.5.” OpenAI, April 23, 2026.
3. “Measuring the performance of our models on real-world tasks.” OpenAI, September 25, 2025.
4. “APEX–Accounting: Testing AI on the work accountants actually do.” Ramp Labs and Mercor, 2026.
5. Aden Barton. “Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants.” Mercor, October 1, 2026.
6. Jasmin Kern, Austin Bennett, Ryan Stevens, Julien Benchek and Bertie Vidgen. “Introducing the AI Productivity Index for Accounting.” Mercor, July 31, 2026.
7. Fabrizio Dell’Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani. “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality.” Organization Science, March–April 2026.
8. Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone. “When combinations of humans and AI are useful: A systematic review and meta-analysis.” Nature Human Behaviour, October 28, 2024.
9. Kevin L. Wei et al.. “Recommendations and Reporting Checklist for Rigorous & Transparent Human Baselines in Model Evaluations.” arXiv, June 2025.
10. Joel Becker, Nate Rush, Beth Barnes and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR, July 10, 2025.