Computer use
A model operating a computer as a person would, looking at the screen and moving the mouse and keyboard.
Also: browser agent, computer-using agent
Computer-use models receive screenshots, decide where to click or what to type, and see the result in the next screenshot. Anthropic released a computer-use feature for Claude in October 2024, and OpenAI's Operator, Google's Project Mariner and others followed. The approach works with any software, even without an API, but is slower and less reliable than direct tool calls. Because the agent reads whatever is on screen, prompt injection through web pages and documents is a particular risk.