AI agent observability and governance: how to monitor agents after go-live

What is AI agent observability?

AI agent observability means being able to see what each of your agents did, step by step, and why. It rests on one record: a trace of every run. A trace shows what came in, what the agent saw, each model call and tool call it made, what it produced, what it cost, how long it took, what went wrong and where a person stepped in.

Agent frameworks are built to keep this record. OpenAI’s Agents SDK, for example, turns tracing on by default and records each model call, tool call, hand-off between agents and guardrail check in a run. Anthropic’s engineers call the same record a transcript, trace or trajectory.

Monitoring and observability work together. Monitoring watches a few numbers, such as quality, cost and errors, and raises an alert when one crosses a line. Observability lets you answer the next question, which is why, by opening the traces behind the number.

Take a deal desk agent that checks discount requests and drafts a summary for an approver. Its trace shows the deal it read, the pricing policy it applied, the CRM records it pulled, the summary it wrote and what the approver changed. When an approver rewrites a summary, the trace shows which step went wrong.

Why does observability matter more for agents than for chatbots?

Because agents act. A chatbot writes an answer, and a person decides what to do with it. An agent updates records, sends messages and approves requests, often over many steps, so a mistake can take effect before anyone reads it.

Agents are also harder to judge from the outside:

• Many steps. One run can include dozens of model and tool calls, and a good final message can hide a bad step. Anthropic’s engineers separate what an agent says from what it did: a booking agent may report that the flight is booked, but what counts is whether the reservation exists.
• Different paths. Model outputs vary between runs, so the same request can go differently tomorrow. That is why Anthropic’s team runs each test more than once.
• Changes underneath. Models, prompts, tools and data change after launch. AWS architects warn that the same agent can be accurate on Monday and wrong on Tuesday after a prompt change or a model update.

Oversight also shifts as trust grows. People stop approving each step and watch the record instead, stepping in when something looks wrong. In an Anthropic study of millions of human-agent interactions, experienced Claude Code users approved less and interrupted more:

The study’s authors conclude that overseeing agents well will take new kinds of monitoring after deployment. When people approve less, the record is how they stay in control.

What should you monitor after an AI agent goes live?

Watch five things, continuously and after every change. Use the same checks and bars you used to prove the agent before go-live, in shadow mode and in your benchmarks, so the numbers stay comparable.

• Quality on your own checks. Score a sample of finished work against what “correct” means for your team, and check the result in the system, not just the agent’s report. Anthropic’s engineers suggest reading a sample of transcripts every week.
• Drift. Re-run your evals whenever the model, a prompt, a tool, a policy or a data source changes. Anthropic’s engineers suggest turning mature evals into a regression suite that runs continuously to catch drift.
• Overrides and escalations. Track how often people edit, reject or reverse the agent’s work, and how often the agent escalates a case to a person. NIST’s AI RMF suggests collecting data on “the frequency and rationale with which humans overrule AI system output.” Rising overrides point to a quality problem. Overrides near zero can mean reviewers have stopped looking, a tendency the EU AI Act calls automation bias.
• Cost per task. Add up model calls, retries and people’s review time for each finished task, and set an alert on tokens per task: a sudden jump can mean the agent is retrying or stuck in a loop. See cost per task.
• Incidents. Decide in advance what counts as one, such as a wrong payment, a message to the wrong customer or data sent to the wrong place, and who is called when it happens.

Read a panel like this by exception. Every measure here is within its bar, but the triage agent just moved to a smaller model, so it gets a closer look for a few weeks. The approver’s edits on the deal desk are useful too: each one is a new test case with a known answer.

Changes you didn’t make count as well. In April 2025, OpenAI rolled out an update to GPT-4o in ChatGPT that made it too eager to please users, and began rolling it back three days later. Its offline evaluations and A/B tests had looked good. When a provider updates the model under one of your agents, re-run your own checks.

Is there an open standard for tracing AI agents?

Yes, and it is still settling. OpenTelemetry, an open-source project for collecting traces, metrics and logs from software, publishes semantic conventions for generative AI: shared names for what to record about model calls, agents and tools. When every product records a run the same way, you can follow one run across products and keep the record in your own monitoring systems.

The conventions define agent operations such as creating an agent, invoking it, planning and running a tool. They also cover metrics, events, the Model Context Protocol and conventions for specific providers, including Anthropic, AWS Bedrock, Azure AI Inference and OpenAI. Two cautions:

• They are not final. Their status is Development, so names and fields can still change. In May 2026, OpenTelemetry moved them into a repository of their own.
• Content is sensitive. Recording the text of prompts and responses is opt-in, with a warning that it is likely to contain sensitive information, including personal data. Decide what you capture, who can read it and how long you keep it.

For buyers, the practical question is whether an agent product exports its traces in an open format, so your record of what agents did isn’t locked inside one vendor’s dashboard. Some already do. Claude Code, for example, exports usage, cost and tool activity through OpenTelemetry, and labels its model calls with names from the GenAI conventions.

What governance controls does every AI agent need?

Governance decides what an agent may do and checks that it did only that. It works as a set of controls before, during and after every run. Most of them belong in the agent’s harness, the software around the model, so they apply whichever model you use.

Start with least privilege. OWASP’s Top 10 for LLM applications includes excessive agency: an agent with more functions, permissions or autonomy than its task needs. Its advice is to cut all three. Offer only the tools the task needs, give each tool the narrowest permissions, and have a person approve high-impact actions.

Design so that the worst mistakes can’t happen. In July 2025, a Replit coding agent deleted data from a user’s production database. Replit’s CEO called it “unacceptable and should never be possible,” Business Insider reported. The company changed what its agent could reach: it began separating development and production databases automatically, so the agent works in a sandbox and any change to production needs explicit approval.

Make each approval specific. The approver should see what the agent checked, what it is unsure of and which decision is theirs, or approving becomes a reflex. Then cover every agent you run: an AI policy that names the models and tools agents may use, an owner for each agent, and one inventory of them all, as NIST’s framework asks for AI systems in general. Agents that nobody approved are part of shadow AI, and they leave no record you can review.

Finally, keep an audit trail: what each agent did, with which inputs, under whose approval, and what changed. Record the state before each action, too. In the design AWS architects describe, that is what makes it possible to recover from an incorrect action.

What should you do when an agent gets something wrong?

Handle it like any production incident, with one extra step: turn the case into a test. A short playbook:

1. Contain it. Pause the agent, or move it down a step, for example from acting alone to drafting for approval. The AWS team’s rule for graduated autonomy is to “promote slowly, demote immediately.”
2. Trace it. Open the run and find the step that failed: the input, the context, a model call or a tool call.
3. Repair the effect. Reverse what you can, using the state recorded before the action, and tell anyone affected.
4. Fix the cause. Change the permission, prompt, policy or tool, then re-run your evals before you restore the agent’s autonomy.
5. Learn from it. Add the case to your eval set and hold a short after-action review, as NIST’s Generative AI Profile suggests.
6. Report it where required. Once the EU AI Act’s high-risk rules apply, deployers of high-risk systems must suspend a system that presents a risk and report serious incidents to its provider and the authorities.

Plan for retirement, too. Processes change, better tools arrive and owners move on. NIST’s AI RMF asks for processes to decommission AI systems safely. When an agent retires, revoke its access, keep its audit trail and hand its evals to whatever replaces it.

What do the NIST AI RMF, ISO/IEC 42001 and the EU AI Act ask for?

Three frameworks are worth knowing, and they differ in kind. NIST’s AI RMF is voluntary guidance, ISO/IEC 42001 is a standard a company can be certified against, and the EU AI Act is law.

NIST’s framework organizes AI risk work into four functions: Govern, Map, Measure and Manage. Two of its outcomes read like an operator’s checklist: monitor the system’s behavior in production, and have post-deployment monitoring plans that cover override, decommissioning, incident response, recovery and change management. NIST’s page says version 1.0 is being revised under the White House AI Action Plan, and in February 2026 NIST launched an AI Agent Standards Initiative.

ISO/IEC 42001 specifies how to set up and run an AI management system: the policies, objectives and processes a company uses to develop, provide or use AI responsibly, improved in a plan-do-check-act cycle. It covers how you manage AI across the company, not the details of each application. Accredited bodies can certify companies against it, under rules set by ISO/IEC 42006.

The EU AI Act sets rules by level of risk. Its duties fall mainly on high-risk uses, such as AI that sorts CVs or scores credit, and on transparency, such as telling people they are talking to a chatbot. The European Commission says the vast majority of AI systems used in the EU are minimal or no risk, with no new rules. The Act has applied in general since August 2, 2026. The Digital Omnibus on AI, in force since July 27, 2026, moved the high-risk rules to December 2, 2027, for uses such as hiring and credit scoring, and to August 2, 2028, for AI built into regulated products. From then, deployers of high-risk systems must assign trained people to oversee them, monitor them, keep their logs for at least six months and report serious incidents.

Whether a given agent is in scope depends on what it is used for and on your role as its provider or deployer. Treat this as a summary, not legal advice, and check your own uses with counsel.

Sources

1. “Tracing.” OpenAI Agents SDK docs, accessed October 4, 2026.
2. Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe. “Demystifying evals for AI agents.” Anthropic, January 9, 2026.
3. Dev Arora, Meera Kezhukoot and Sathish Kumar Prabakaran. “Closing the AI agent trust gap with graduated autonomy.” AWS Architecture Blog, August 26, 2026.
4. Miles McCain, Thomas Millar, Saffron Huang and 17 others. “Measuring AI agent autonomy in practice.” Anthropic, February 18, 2026.
5. “Artificial Intelligence Risk Management Framework (AI RMF 1.0).” National Institute of Standards and Technology, NIST AI 100-1, January 26, 2023.
6. “Article 14: Human oversight.” AI Act Service Desk, European Commission, accessed October 4, 2026.
7. “Expanding on what we missed with sycophancy.” OpenAI, May 2, 2025.
8. “Semantic conventions for generative AI systems.” OpenTelemetry, GitHub, accessed October 4, 2026.
9. “Semantic Conventions for GenAI agent and framework spans.” OpenTelemetry, GitHub, accessed October 4, 2026.
10. “Move GenAI semantic conventions to its own dedicated repository (pull request #3696).” OpenTelemetry, GitHub, May 5, 2026.
11. “Semantic conventions for generative client AI spans.” OpenTelemetry, GitHub, accessed October 4, 2026.
12. “Monitoring.” Anthropic, Claude Code docs, accessed October 3, 2026.
13. “LLM06:2025 Excessive Agency.” OWASP Gen AI Security Project, accessed October 4, 2026.
14. Lee Chong Ming. “Replit’s CEO apologizes after its AI agent wiped a company’s code base in a test run and lied about it.” Business Insider, July 22, 2025.
15. Amjad Masad. “Replit is moving to separate dev vs production databases for developers automatically.” LinkedIn, July 24, 2025.
16. Chloe Autio, Reva Schwartz, Jesse Dunietz and 5 others. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.” National Institute of Standards and Technology, NIST AI 600-1, July 26, 2024.
17. “Article 26: Obligations of deployers of high-risk AI systems.” AI Act Service Desk, European Commission, accessed October 4, 2026.
18. “AI Risk Management Framework.” National Institute of Standards and Technology, accessed October 4, 2026.
19. “Announcing the ‘AI Agent Standards Initiative’ for Interoperable and Secure Innovation.” National Institute of Standards and Technology, February 17, 2026.
20. “ISO/IEC 42001:2023, AI management systems.” International Organization for Standardization, December 2023.
21. “ISO/IEC 42006:2025, Requirements for bodies providing audit and certification of artificial intelligence management systems.” International Organization for Standardization, July 2025.
22. “AI Act.” European Commission, Shaping Europe’s digital future, accessed October 4, 2026.
23. “Regulation (EU) 2026/1744 amending Regulation (EU) 2024/1689 and other acts (Digital Omnibus on AI).” Official Journal of the European Union, July 24, 2026.