How to let AI agents control desktop apps via accessibility trees
Instead of having an AI agent look at the screen and move the mouse, try a flow where it reads information from the operating system's accessibility tree to control apps. The agent runs a loop that receives the app's structure as JSON, performs clicks or inputs, and verifies the results, rather than capturing the screen or matching pixels. This tool does not include a model itself; it provides a CLI and a C library to help the agent execute this loop.
Manipulation via the accessibility tree
agent-desktop does not use screenshots or pixel matching; instead, it reads the desktop app's accessibility tree and returns it in JSON format. Each element is assigned a reference (ref) that the agent can use to perform clicks or text inputs. After manipulation, the process of reading the UI again to verify the result is repeated, and this loop runs in the user's agent code. The tool does not have an embedded AI model, so it must be used in combination with an external agent.
Platform requirements and installation
It supports macOS 13 and later, Windows 10 1809 and later, and Ubuntu 24.04 and later. Each platform uses its own accessibility API (AX API, UI Automation, AT-SPI), and on macOS, accessibility permissions must be granted. You can install it with the npm install -g agent-desktop command and check the current platform and permission status with agent-desktop status. On Windows, no permissions are required for apps corresponding to the terminal's integrity level, while on Linux, it must be run with at-spi2-core in the user's graphical session.
Agent integration and command structure
The CLI has 60 command names, grouped into observation, manipulation, keyboard/mouse, app/window management, notifications, clipboard, waiting, sessions, and system management. Use the snapshot command to receive the current UI state as JSON and check reference IDs, then manipulate those references with click or type commands. If the UI changes, previous references return a STALE_REF error, so you must take a new snapshot to update the references. You can configure the agent to use this tool correctly by adding skills to the agent harness or including instructions in AGENTS.md.
Security and data handling considerations
Snapshots, clipboard reads, screenshots, command outputs, and debug artifacts may all contain personal desktop content. Be careful that this data is not included in public logs or uploads. Operations are performed headless by default, so no pointer movement or focus changes occur; use the --headed option only when physical input is required. In PowerShell, references must be enclosed in quotes; otherwise, they may be interpreted as splatting operators, causing errors.
Pre-implementation checks
This tool is not an AI model but provides an interface for agents to manipulate the desktop. Therefore, you must implement the agent logic directly or integrate it into an existing agent framework. There may be limitations in specific platforms' accessibility APIs or features not supported in headless mode, so you should check the support scope in the documentation for the platform you intend to use. It is distributed under the Apache-2.0 license, and you can also call it directly within a process via the C library.
