What is shadow mode? Testing AI agents on real work before they go live

What is shadow mode?

Shadow mode means running an AI agent alongside the real work without letting it send or change anything. The agent gets the same inputs as your team, such as the same deal requests, invoices or support tickets, and does the same task. Its output is kept for scoring and compared with what your people actually did.

It works like a trial period for a new hire on a critical desk. They work every case alongside the team, but nothing goes out under their name until someone has compared their work with the team’s.

Four things make it shadow mode:

• Real inputs. Live work as it arrives, not cases picked for a demo.
• The full task. The agent reads, decides and drafts as it would in production.
• No effects. It can read the systems your people use, but it writes only to a log. Nothing is sent, posted, paid or changed.
• A score. Every output is compared with what your team did, using the team’s own checks.

Where does shadow mode come from?

Software and machine-learning teams have tested new systems this way for years, under names such as dark launching, shadow testing and traffic mirroring. The pattern is the same: send a copy of real requests to the new version, and keep its answers away from users.

• Facebook, 2008. Facebook expected its new chat feature to go from zero to 70 million users almost overnight. Before switching it on, the team ran a “dark launch”: Facebook pages quietly connected to the chat servers and simulated sending messages, with nothing on screen, so bugs could be fixed first.
• Parallel running. Martin Fowler describes running a rewritten feature alongside the old one, comparing their results, and returning only one answer to users.
• Regulated decisions. Because of regulatory requirements, PayPal’s fraud-detection models must be tested in a shadow platform before release, its engineers wrote in 2021. The shadow copy gets the same data as production, but its scores are only logged and play no part in decisions.
• Built into the cloud. Amazon SageMaker’s shadow tests send a copy of live requests to a new model running in shadow mode, and return only the current model’s answers.

Agents raise the stakes. A fraud model produces a score; an agent takes actions. It sends emails, updates records and approves deals. So shadow mode for agents adds one rule: every action the agent would take is captured as a proposal and never carried out. In practice, that rule lives in the agent’s harness, which gives it read-only tools and sends its would-be actions to a log.

Why test an AI agent on your own work?

Because neither a public benchmark nor a demo can tell you how an agent will handle your work. Two 2026 accounting studies show how far apart results can be, even within one field. Both use tasks from APEX–Accounting, a benchmark built by Ramp and Mercor.

Both results are true. The difference is the work. Mercor notes that its simpler tasks tested the skills models are best at, such as following detailed instructions and searching through files. Your work has its own mix of routine cases, traps and missing context, and no leaderboard has seen it.

How an agent fails matters as much as how often. In Mercor’s failure analysis, roughly seven in ten failures by the top three models came from flawed reasoning, not from failing to find information. A model might spot a discrepancy early, then leave it out of its final journal entry or contradict it there. You only catch that by checking the work behind the answer.

Even your own offline tests have limits. Anthropic’s engineers point out that automated evals can create false confidence when they don’t match real usage, while monitoring a live agent finds problems only after they reach users. Shadow mode sits between the two: real work, real exceptions, and no consequences.

How do you run an AI agent in shadow mode?

Pick one workflow, agree on what “correct” means, and let the agent work real cases alongside the team until you have enough evidence to decide. Seven steps:

1. Pick a bounded workflow. Choose work that is frequent, with clear inputs and a result you can check, such as deal approvals, invoice matching or ticket routing. The best candidates need judgment and have past cases to check against; see the four buckets.
2. Agree on what “correct” means, with the people who do the work. Write the checks down before the first case: which figures must match, which policy applies, when a case must go to a person. Anthropic’s engineers suggest a test: two experts should reach the same pass-or-fail verdict on their own.
3. Collect real cases, awkward ones included. Let live work flow through for a full cycle, such as a month-end, and add past cases that went wrong: the vendor that always sends the wrong purchase order, the multi-currency deal, the renewal that arrives late.
4. Run it more than once. An agent can answer the same case differently on different runs. One that is right 75% of the time passes three runs in a row only about 42% of the time, as Anthropic’s guide to evals shows. What counts in production is being right every time.
5. Score the answer and the records behind it. Check the final output and the evidence it used: the documents it read, the figures it pulled, the policy it applied. A right answer for the wrong reason will fail on the next case.
6. Measure corrections and review time. Have a reviewer mark each output as accepted, edited or rejected, and time the review. That shows what an agent with a person checking it would really cost.
7. Compare with how the team does it today. Score your people on the same cases and the same checks. People make mistakes too, so the bar is the team’s real performance, not perfection.

What should you measure in shadow mode?

Six measures cover most workflows: agreement with your people, accuracy on the team’s own checks, how serious each error is, how often the agent hands a case to a person, time, and cost per task. Set the bar for each before the first case, so the goalposts can’t move.

A scorecard like this teaches three things:

• Agreement and accuracy are different measures. When the agent and a person disagree, someone who owns the work decides who was right. Sometimes it is the agent.
• Serious errors count on their own. Two expensive mistakes matter more than a strong average. AWS architects make the same point: in their design, safety is a floor that strong scores elsewhere can’t average away.
• Hand-offs cut both ways. An agent that escalates everything looks accurate but saves little. One that never escalates is a risk. Test both: cases it should hand over and cases it should keep.

Count the people around the agent, too: the reviewer’s minutes and the time spent fixing mistakes. Our guide to cost per task shows the math.

When is an agent ready to go live?

When it has cleared every bar you agreed, on enough real cases, and only for the kinds of case where it cleared them. Even then, it moves up one step at a time:

AWS architects describe a similar ladder and call it graduated autonomy: four tiers, from read-only “probation” to full access with after-the-fact audits. Every agent starts at the bottom, whatever its test scores, and moves up only through sustained performance. Their rule:

“Promote slowly, demote immediately”

— AWS Architecture Blog, August 2026

• Move an agent up when it has met every bar across a full cycle of real work, with no serious errors and a low, steady rate of corrections. Promote by type of case: routine deals can move up while multi-currency deals stay with people.
• Move it down the moment a serious error occurs or quality drops below the bar. Re-run your shadow cases whenever something changes underneath it, such as the model, a prompt, a policy or a vendor’s format. As the AWS team notes, an agent that is accurate on Monday can fail on Tuesday after a prompt change or a model update.

Keep each approval meaningful. A person who clicks “approve” on everything isn’t a control. Show reviewers what the agent checked, what it is unsure of, and which decision is theirs to make. See human-in-the-loop.

At the last step, people stop checking individual cases and watch the record instead: observability on how the agent performs, and an audit trail of every action it takes.

How do shadow-mode cases become your evals?

Every shadow case, scored against an answer your team agreed on, is a ready-made test. Keep them, and you have an eval set built from your own work, exceptions included.

Re-run the set whenever something changes: a new model, a new prompt, a price cut, a policy update. Ramp and Mercor describe the payoff of fixed tasks: change one thing, rerun the same tasks, and “see whether the outcome improved—or the failure just moved.”

The set keeps growing after go-live, because every correction a reviewer makes is a new case with a known answer. It also speeds up change. Anthropic’s engineers write that teams with evals can test a new model and upgrade in days, while teams without them face weeks of testing.

That is why your evals are a company asset. Models, prompts and vendors will change. A record of what “correct” means at your company, built from real cases, stays yours. It is also the basis of a personalized benchmark that compares people, agents and both together. See how to benchmark AI on your own work.

Sources

1. Eugene Letuchy. “Facebook Chat.” Engineering at Meta, May 13, 2008.
2. Martin Fowler. “Dark Launching.” martinfowler.com, April 29, 2020.
3. Quinn Zuo, Sudhi Murthy and Nitin Sharma. “Deploying Large-scale Fraud Detection Machine Learning Models at PayPal.” The PayPal Technology Blog, October 26, 2021.
4. “Shadow tests.” Amazon SageMaker AI Developer Guide, Amazon Web Services, accessed October 3, 2026.
5. “APEX–Accounting: Testing AI on the work accountants actually do.” Ramp Labs and Mercor, 2026.
6. Aden Barton. “Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants.” Mercor, October 1, 2026.
7. Jasmin Kern, Austin Bennett, Ryan Stevens, Julien Benchek and Bertie Vidgen. “Introducing the AI Productivity Index for Accounting.” Mercor, July 31, 2026.
8. Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe. “Demystifying evals for AI agents.” Anthropic, January 9, 2026.
9. Dev Arora, Meera Kezhukoot and Sathish Kumar Prabakaran. “Closing the AI agent trust gap with graduated autonomy.” AWS Architecture Blog, August 26, 2026.