What is an on-screen AI assistant? AI that sees your screen and helps in any app

Why do people need help inside their apps?

Companies buy software to speed up work, then use a fraction of it. People move between many apps a day, each with features they never find. When they get stuck, the help sits somewhere else: a training course, a help center or a colleague who knows the trick.

In a Gartner survey of 4,861 employees, run in late 2022, the average desk worker used 11 applications, up from six in 2019, and 47% struggled to find the information they needed to do their jobs. Two other studies show the cost:

The studies measure different things but point the same way: people move between apps all day and use a small part of each. When they need an answer, they leave the app to find it, which costs one more switch.

AI chat helps, but a chatbot in another tab can’t see the work. People paste a screenshot or retype the problem, then carry the answer back by hand. On-screen assistants try to remove that round trip.

What is an on-screen AI assistant?

An on-screen AI assistant is an AI assistant that can see what is on a person’s screen, with their permission, and help in the app they are using. Instead of asking the person to describe their work, it looks at the same window they do. It can help in four ways, each with more reach than the last:

• Explain. Say what a field means, why a number looks wrong or what the policy says about this case.
• Guide. Show where to click next, or walk through a task one step at a time.
• Fill in. Draft the text, complete the fields or prepare a request for the person to check and send.
• Act. Click, type and move between apps to finish the task, in products built to do that.

Explaining and guiding leave every click to the person. Filling in and acting hand part of the work to the assistant, so they need tighter limits and a person on the decisions that matter, the core of human-in-the-loop design.

Assistants also see in different ways. Vision in Microsoft 365 Copilot processes what you share as a series of images and reads the text, charts and interfaces in them. ChatGPT’s desktop app for macOS reads most supported apps through the macOS Accessibility API. The method shapes what an assistant can read and what it could expose, so ask every vendor how theirs works.

What can on-screen AI assistants do today?

On-screen AI is a spectrum rather than one product. At one end, a chatbot sees only what you paste into it. At the other, an agent operates your apps for you.

Many business apps now have an assistant of their own. Copilot in Excel builds and edits workbooks, and Gemini in Gmail summarizes threads and drafts replies. These assistants know their own app well, but they stop at its edge, and most work crosses several apps.

Assistants that see the screen close that gap. As of October 2026, each major platform offers one:

• Microsoft. Copilot Vision on Windows lets you share two apps at a time, and its Highlights feature shows you where to click. It became available in the U.S. on June 12, 2025, and Microsoft says it won’t click, type or scroll for you. For businesses, vision in Microsoft 365 Copilot sees a shared screen during voice chats.
• Google. Gemini Live’s screen sharing became free on Android and iOS on May 20, 2025.
• OpenAI. ChatGPT’s desktop app for macOS can work with apps: it reads content from supported coding and text apps, and can edit code in some of them.
• Apple. Siri AI began rolling out in beta, in English, on September 14, 2026. Apple says its onscreen awareness lets Siri answer questions or take actions related to what is on screen.

Agents that act on screen go furthest. Anthropic introduced computer use in October 2024, describing Claude using computers “by looking at a screen, moving a cursor, clicking buttons, and typing text.” Today, ChatGPT’s desktop app offers Computer Use on macOS and Windows, in supported regions. It asks permission before it uses each app, and OpenAI advises keeping tasks narrow.

Some products straddle two steps: Siri, for one, can act on what it sees. Ask each vendor where its product sits today, because the answer changes every few months.

All of these work with a person, in the moment. That makes them sidekicks, unlike background agents, which run on their own when a schedule or an event triggers them. Our guide to sidekick and background agents explains when to use each.

How is AI changing digital adoption platforms?

Before assistants could see screens, enterprises bought digital adoption platforms (DAPs) to put help inside their software. A DAP adds a layer of in-app guidance on top of apps such as a CRM or an HR system: tooltips, step-by-step walkthroughs, checklists and announcements, plus analytics that show where people get stuck. Vendors include WalkMe, which SAP bought in 2024, Pendo and Whatfix.

Classic in-app guidance is written by hand: someone builds a walkthrough for each task, and rules decide when it appears. WalkMe itself says rule-based display logic can’t account for every situation on screen. Hand-written guides cover the common paths but miss the long tail of rare tasks and exceptions.

Vendors describe four changes:

• Writing the guides. Whatfix’s Authoring Agent builds walkthroughs and pop-ups from a plain-language prompt, and Pendo can draft guides with AI for a person to edit.
• Reading the screen. Whatfix says its ScreenSense engine is built on a computer use model. WalkMe says its AI Launchers “read the screen and your policies” to suggest the next step.
• Answering in place. Whatfix’s Guidance Agent answers questions inside the app from internal documents, such as return policy exceptions in an order management system.
• Working with agents. Pendo’s Agent Toolkit, in beta, lets an AI agent offer a walkthrough when someone asks how to do a task, and still respects who may see which guide.

DAPs and on-screen assistants are converging. DAPs bring authoring, targeting by role, analytics and the admin controls enterprises expect; general assistants bring flexibility. Whichever you choose, ask how it learns your workflows and whether it measures if people finish the task, not just whether they saw a tip.

Where does on-screen help pay off?

On-screen help pays off at moments of need: the points where someone has to stop and find out how. Learning experts Conrad Gottfredson and Bob Mosher name five: learning something new, learning more, applying it, solving a problem and adapting to change. Gottfredson notes that traditional training focuses on the first two, which leaves people on their own for the other three. Those three happen mid-task, inside the app, where on-screen help can reach them.

The payoff is largest where the work is frequent but the situation is rare. Take a deal desk that reviews about 30 deals a week, most of which pass the same three checks. Then a three-year renewal arrives with a 22% discount and Net 60 payment terms, a combination the analyst hasn’t seen in months. The pricing policy is a PDF, the approver is busy and the answer is a Slack thread away. Or take an accounts payable clerk whose invoice doesn’t match its purchase order, when the rule for that exception lives in one colleague’s head. That is key-person risk, felt one invoice at a time.

A new tool is the other big moment. It changes every screen at once, and the training usually comes before people meet the cases they need help with.

Two cautions. If people need the same help every day, fix the process instead: the step may need deleting, plain code or an agent rather than a tip, so sort it with the four buckets first. And help that interrupts has a cost. Prefer assistants that wait to be asked, or that speak up only when something is wrong, such as a discount outside policy.

Why does company context matter more than pixels?

An assistant that sees only pixels knows what is on the screen. It doesn’t know what your company does about it: your approval limits, who approves what, which exceptions are allowed, or which system holds the right number. That is the gap between generic help and useful help.

Vendors are moving the same way. Vision in Microsoft 365 Copilot grounds its answers in both the shared screen and your work data, such as documents, emails and meetings. WalkMe and Whatfix pair the screen with your policies and business knowledge.

The context that matters most is how your company actually works: who owns what, how each workflow runs, where the data lives, the rules and their exceptions, and the methods your best people use. Written down where software can use it, that becomes a digital org. The same map that helps AI agents work inside your company, which is what good agent experience means, makes on-screen help specific. And the skills your experts write for agents can guide people at the same moments.

Context also goes stale. If the deal desk raises its limit to 18% and the assistant still cites 15%, it gives everyone the same wrong answer with confidence. Ask how the assistant’s knowledge stays current, and who owns it.

What are the privacy risks, and which controls matter?

A screen shows everything a person can see: customer records, salaries, health details, private messages, sometimes passwords. An assistant that sees the screen sees all of it, including data about customers and colleagues who never agreed to be seen. Treat privacy as the first test of any on-screen assistant.

Microsoft learned this in public. On May 20, 2024, it introduced Recall, a Windows feature that saves snapshots of the screen so people can search them later. Within weeks, after what it called a clear signal from customers, it made Recall opt-in, required Windows Hello sign-in and added more encryption. Today Recall is off and removed by default on managed business devices: IT has to allow it, and each person still has to turn it on. Recall is a memory feature, not an assistant, but the lesson carries over.

Look for five controls, and check each in the product, not the brochure:

• Opt-in, every time. The assistant looks only when a person starts it. Microsoft says Copilot Vision is active only during a session you start.
• A visible sign. People should always know when the assistant is looking. Copilot Vision draws a glowing outline around the apps you share, and ChatGPT’s desktop app shows a banner naming the apps it is working with.
• The smallest view. Share one window, not the whole screen. Copilot Vision on Windows lets you pick which apps it sees, while Gemini Live on Android shares your full screen.
• Per-app permissions. ChatGPT’s Computer Use asks before it uses each app, and people can revoke apps later. Recall lets people filter out apps and websites, and by default tries to keep passwords, national ID numbers and credit card numbers out of what it saves.
• Admin controls. IT should decide which assistants run on company devices, and where. Vision in Microsoft 365 Copilot is on by default, and admins can turn it off. ChatGPT Enterprise admins can switch off Work with Apps, and workspace admins can limit which apps Computer Use may reach.

Then ask what is kept, and for how long. The answers vary, even within one vendor. Microsoft says consumer Copilot Vision deletes images, audio and context when a session ends, but stores the model’s responses and a transcript. Vision in Microsoft 365 Copilot keeps audio and video for 48 hours so users can give feedback. Content that ChatGPT reads through Work with Apps becomes part of the chat history. Get each answer in writing, including whether any of it is used to train models.

Agents that act raise the stakes. They can click the wrong button, and they can be steered by instructions hidden in a web page or a document, an attack called prompt injection. OpenAI warns that web pages can contain malicious or misleading content, and that sites may treat the agent’s clicks as coming from your account. Keep a person on approvals, payments and anything sent outside the company, and keep an audit trail of every action.

Last, tell people what the assistant sees, why, and what is kept, and involve employee representatives where they have a say. If the company offers no approved option, people may paste screenshots of their work into personal chatbots: shadow AI, with sensitive data attached.

How do you evaluate an on-screen AI assistant?

Judge an on-screen assistant on three things: what it sees and keeps, what it knows about your company, and whether it helps on your own work. Put these questions to every vendor, including the ones you already pay for:

Then test it where it matters. Pick three or four moments of need from real workflows, such as a non-standard discount or an invoice exception. Give the assistant to a small group and compare with how they handle the same cases today: time to finish, errors, how often they had to ask someone, and whether they still use it a month later. Our guide to benchmarking AI on your own work shows how to set up the cases.

Sources

1. “Gartner Survey Reveals 47% of Digital Workers Struggle to Find the Information Needed to Effectively Perform Their Jobs.” Gartner, May 10, 2023.
2. Rohan Narayana Murty, Sandeep Dadlani and Rajath B. Das. “How Much Time and Energy Do We Waste Toggling Between Applications?” Harvard Business Review, August 29, 2022.
3. “The 2019 Feature Adoption Report.” Pendo, February 2019.
4. “How vision in Microsoft 365 Copilot works.” Microsoft Support, accessed October 4, 2026.
5. “Work with Apps on macOS.” OpenAI Help Center, accessed October 4, 2026.
6. “Get started with Copilot in Excel.” Microsoft Support, accessed October 4, 2026.
7. “Collaborate with Gemini in Gmail.” Gmail Help, Google, accessed October 4, 2026.
8. The Copilot Team. “Copilot Vision on Windows with Highlights is now available in the U.S.” Microsoft Copilot Blog, June 12, 2025.
9. “Using Copilot Vision with Microsoft Copilot.” Microsoft Support, accessed October 4, 2026.
10. Josh Woodward. “Gemini App: 7 updates from Google I/O 2025.” Google, May 20, 2025.
11. “Siri AI, a profoundly more capable and personal assistant powered by the next generation of Apple Intelligence, is here.” Apple Newsroom, September 14, 2026.
12. “Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku.” Anthropic, October 22, 2024.
13. “Computer Use.” ChatGPT Learn, OpenAI, accessed October 4, 2026.
14. “SAP Completes Acquisition of WalkMe.” SAP News Center, September 12, 2024.
15. “Proactive AI: Always-On Action Bar Assistance.” WalkMe, accessed October 4, 2026.
16. “Whatfix Launches AI Agents to Accelerate Business Outcomes for Enterprises.” Whatfix, September 9, 2025.
17. “Create guides using AI.” Pendo Help Center, accessed October 4, 2026.
18. “Whatfix Unveils ScreenSense: An AI Technology to Shape the Next Frontier of Digital Adoption.” Whatfix, February 27, 2025.
19. “Deliver guides through your AI agent (beta).” Pendo Help Center, updated September 22, 2026.
20. Conrad Gottfredson. “Extending Learning into the Workflow.” The 5 Moments of Need Blog, February 2023.
21. Pavan Davuluri. “Update on the Recall preview feature for Copilot+ PCs.” Windows Experience Blog, Microsoft, June 7, 2024, updated October 31, 2024.
22. “Manage Recall for Windows clients.” Microsoft Learn, accessed October 4, 2026.
23. “Talk naturally with Gemini Live (Android).” Gemini Apps Help, Google, accessed October 4, 2026.
24. “Update on Recall security and privacy architecture.” Windows Experience Blog, Microsoft, September 27, 2024.
25. “Manage Microsoft Copilot Settings.” Microsoft Learn, accessed October 4, 2026.