Guide · Decide
How to choose an AI model by cost per task
Price per token is misleading. How to measure cost per task, when small and open-weight models do the same job for less, and when a frontier model is worth paying for.
The short answer
Choose a model for each task by its cost per task: the full cost of getting one task done correctly, including every attempt, the time people spend reviewing, and the cost of fixing mistakes. Test the cheapest models that could plausibly work on your own past cases. For many well-defined tasks, a small or open-weight model matches a frontier model for far less.
Key takeaways
- Price per token is misleading. Cost per task includes retries, review and fixing mistakes.
- Public leaderboards don’t predict how a model does on your work. Test it on your own cases, more than once.
- Route by difficulty: small or open-weight models for routine cases, frontier models for hard judgment.
What is cost per task?
Most teams compare AI models by their price per million tokens. That number is easy to find and almost useless on its own. What you actually pay for is a task done correctly, so measure that:
Two parts are easy to miss. Attempts: agents often call a model several times per task, and failed attempts are billed in full. Mistakes: every wrong answer costs someone time to notice and fix. For cheap, high-volume tasks, fixing mistakes can cost far more than the model itself.
Frontier, mid-size, small and open-weight models
Models come in roughly four tiers, and prices differ widely even within one provider. On Anthropic’s price list, its largest model costs ten times as much per token as its smallest.
| Tier | Best for | Examples | Price per million tokens, in / out |
|---|---|---|---|
| Frontier | Hard reasoning, ambiguous judgment, long multi-step work | Claude Fable 5.1, Claude Opus 5 | $10 / $50 and $5 / $25 |
| Mid-size | Most everyday drafting and analysis | Claude Sonnet 5 | $2 / $10 |
| Small | Classifying, extracting and routing at volume | Claude Haiku 4.5 | $1 / $5 |
| Open-weight | High volume, data that must stay in your network, custom tuning | Families such as Qwen, GLM, Kimi and Llama | Low per token, plus the cost of hosting and running it |
Other providers’ lineups follow a similar pattern. Open-weight models can be the cheapest per token, but you pay for hosting and for the people who run them.
Why public benchmarks are not enough
Public leaderboards tell you which model is strongest in general. They don’t tell you how a model will do on your work, or how consistently. Two accounting studies from 2026 show why.
On 160 realistic month-end close tasks, the best model met 56.4% of the experts’ grading criteria, and no model completed more than 2.6% of tasks correctly on all eight attempts.
On four simpler, well-defined close tasks, frontier models scored perfectly. Twelve licensed CPAs averaged about 37%, and cost more than ten times as much per task.
Both results are true. The difference is the task: short, well-defined, detail-oriented work versus long, messy work that has to be right every time. Mercor itself notes that its simpler tasks tested the skills models are best at, not the whole job. The only way to know which kind of work you have is to test models on it, case by case and run after run.
How to test models on your own work
You don’t need a research lab. A focused test takes days, not months:
- Pick one workflowFrequent, bounded, with a clear right answer.
- Collect 20 to 50 past casesInclude the awkward exceptions.
- Run each model several timesConsistency matters as much as accuracy.
- Check the output and the evidenceRight, and right for the right reasons.
- Price it per taskModel cost, review time and fixes.
Compare three setups, not one: your people doing the work today, an agent alone, and an agent with a reviewer. That is a personalized benchmark, and it often shows that the best option isn’t the cheapest model or the most autonomous agent, but the combination with the lowest cost per correct result.
Before giving an agent real responsibility, run it in shadow mode: it works alongside the real process, sends nothing, and is scored against what people actually did.
Worked examples: when the cheaper model wins, and when it doesn’t
Here is the math for three tasks, comparing a frontier model (Claude Fable 5.1, at list price) with a small one (Claude Haiku 4.5). Token counts, accuracy and fix times are illustrative, and people’s time is costed at $66 an hour.
97.4% right$0.183
97.1% right$0.164
98.3% right$0.216
97.9% right$0.085
98.2% right$1.64
88.4% right$5.82
- Model cost
- Fixing mistakes
Show the data
| Task · model | Right | Model cost | Fixing mistakes | Total per task |
|---|---|---|---|---|
| Route a support ticket · Frontier | 97.4% | $0.040 | $0.143 | $0.183 |
| Route a support ticket · Small | 97.1% | $0.004 | $0.160 | $0.164 |
| Summarize a one-hour meeting · Frontier | 98.3% | $0.160 | $0.056 | $0.216 |
| Summarize a one-hour meeting · Small | 97.9% | $0.016 | $0.069 | $0.085 |
| Reconcile conflicting revenue figures · Frontier | 98.2% | $0.750 | $0.891 | $1.64 |
| Reconcile conflicting revenue figures · Small | 88.4% | $0.075 | $5.74 | $5.82 |
The results point three different ways:
- Routing a ticket: the small model costs a tenth as much per call, but fixing its extra mistakes eats most of the saving. It still wins, by about 10%.
- Summarizing a meeting: long inputs make the model cost dominate. The small model is about 60% cheaper per summary, with almost no loss in quality.
- Reconciling figures: mistakes are expensive, and the small model makes many more of them. The frontier model is more than three times cheaper per correct result.
When is a frontier model worth it?
Use a frontier model where judgment is hard and mistakes are costly, and route everything else.
Routing is often the biggest single saving: a small model handles the routine cases and passes the hard ones, such as multi-currency deals or unusual contract clauses, to a frontier model. Varick Agents says its production systems use “mostly small fast cheap models,” saving a frontier model for the steps that genuinely need one.
Open-weight or closed: price, control and policy
Open-weight models publish their weights, so you can run them yourself or through a cloud provider. That brings three trade-offs:
- Price: lower per token, but self-hosting adds GPUs, monitoring and people. At low volume, a hosted API is usually cheaper.
- Control: data can stay inside your network, and you decide when the model changes.
- Policy: some enterprises restrict which models they may use, for example by country of origin, whatever the benchmark scores say.
Whatever you choose, keep your test cases. Models and prices change every few months, and a good set of evals lets you re-run the comparison in minutes. As Vasuman Moza, Varick Agents’ CEO, put it on The Startup Ideas Podcast, benchmark every workflow against every model before deciding which one it gets.
Questions people ask
What is a token, and how is AI usage priced?
A token is the unit a model reads and writes, roughly three-quarters of an English word. Providers charge per million input tokens and per million output tokens, and output usually costs several times more than input.
Are open-weight models really cheaper?
Per token, usually. But running them yourself adds servers, GPUs and people to operate them. At low volume a hosted API is often cheaper; at high volume, or when data must stay in your network, open-weight models can win.
How often should we re-test models?
Whenever a new model ships, a price changes or your process changes, and on a regular schedule, such as monthly. Keep your test cases so that each re-test takes minutes, not weeks.
Can different steps of one workflow use different models?
Yes, and they often should. A small model can classify and route, a frontier model can handle the few hard cases, and plain code can do the rest.
Does a cheaper model put quality at risk?
Only if you switch without testing. Compare quality on your own cases first, set a minimum bar, and keep a frontier model wherever the gap is real.
Sources
- “APEX–Accounting: Testing AI on the work accountants actually do.” Ramp Labs and Mercor, 2026.
- Jasmin Kern, Austin Bennett, Ryan Stevens, Julien Benchek and Bertie Vidgen. “Introducing the AI Productivity Index for Accounting.” Mercor, July 31, 2026.
- Aden Barton. “Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants.” Mercor, October 1, 2026.
- “Pricing.” Anthropic, accessed October 2, 2026.
- “AI Forward Deployed Engineering: The blueprint to breaking into FDE in 30 days.” Varick Agents.
- Daniel Kornum. “If AI is so great, why isn’t it working?” Varick Agents.
- Greg Isenberg, with Vasuman Moza of Varick Agents. “Become a $1M/yr FDE (Full Course).” The Startup Ideas Podcast, October 1, 2026.