Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): fix stale references after recent tool consolidation (#1417)
Francesco Bonacci4ae26cfde325
diff --git a/docs/content/docs/cua-driver/guide/getting-started/quickstart.mdx b/docs/content/docs/cua-driver/guide/getting-started/quickstart.mdxindex 912cbcda03..f3915cc994 100644--- a/docs/content/docs/cua-driver/guide/getting-started/quickstart.mdx+++ b/docs/content/docs/cua-driver/guide/getting-started/quickstart.mdx@@ -103,7 +103,7 @@ Pixel clicks are for surfaces the AX tree doesn't reach: canvases, video players```bash# Write the screenshot to disk. Works in every capture mode.-cua-driver get_window_state '{"pid":844,"window_id":10725}' --image-out /tmp/shot.png+cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file /tmp/shot.png# Look at /tmp/shot.png, pick a target pixel, then:cua-driver click '{"pid":844,"window_id":10725,"x":120,"y":240}'diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdxindex c5af31770c..da22b2e458 100644--- a/docs/content/docs/cua-driver/reference/cli-reference.mdx+++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx@@ -56,7 +56,7 @@ cua-driver <tool-name> '<JSON-args>'**Flags:**- `--raw` — Print the raw `CallTool.Result` JSON (content + structuredContent + isError) instead of unwrapping `structuredContent`.-- `--image-out <path>` — Write the first image content block from the response to `path` (PNG bytes). The default text formatter would drop image content otherwise.+- `--screenshot-out-file <path>` — Write the first image content block from the response to `path` (PNG bytes). The default text formatter would drop image content otherwise.- `--compact` — Emit minified JSON instead of pretty-printed.- `--no-daemon` — Skip the running daemon and run the tool in-process. Element-indexed workflows fail without a daemon because the per-pid cache dies between CLI invocations.- `--socket <path>` — Override the daemon Unix socket path.@@ -67,7 +67,7 @@ cua-driver <tool-name> '<JSON-args>'cua-driver call list_appscua-driver call launch_app '{"bundle_id":"com.apple.finder"}'echo '{"pid":844,"window_id":1234}' | cua-driver call get_window_state-cua-driver get_window_state '{"pid":844,"window_id":1234}' --image-out /tmp/shot.png+cua-driver get_window_state '{"pid":844,"window_id":1234}' --screenshot-out-file /tmp/shot.png```### cua-driver list-tools@@ -380,6 +380,7 @@ The following MCP tools are callable via `cua-driver <tool>`. For input schemas- `set_agent_cursor_enabled` — toggle the visual overlay.- `set_agent_cursor_motion` — tune the Bezier-arc + spring motion knobs.+- `set_agent_cursor_style` — customize the cursor's visual appearance (gradient colors, bloom color, image path).- `get_agent_cursor_state` — read the current overlay configuration.**Config**diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdxindex 2b110dfceb..e00f452577 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx@@ -7,7 +7,7 @@ import { Callout } from 'fumadocs-ui/components/callout';`cua-driver` exposes 28 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.-Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/cua-driver/reference/cli-reference) for CLI-specific flags like `--raw` and `--image-out`.+Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/cua-driver/reference/cli-reference) for CLI-specific flags like `--raw` and `--screenshot-out-file`.<Callout type="info">Tool names here match the CLI form exactly. `cua-driver list_apps` and the MCP `list_apps` tool run the same code path.@@ -81,6 +81,7 @@ Snapshot a single window: AX element tree plus a screenshot. Populates the per-p- `pid` (integer, required): Process ID from `list_apps`.- `window_id` (integer, required): CGWindowID of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array.- `query` (string, optional): Case-insensitive substring. When set, `tree_markdown` only contains matching lines plus their ancestor chain; element indices and `element_count` are unchanged.+- `screenshot_out_file` (string, optional): Absolute path to write the screenshot to. When set, the screenshot bytes are written to this file, the MCP image content block is omitted, and `screenshot_file_path` is returned instead. Useful for agents that cannot consume inline base64 without saturating their context window.```json{"pid": 844, "window_id": 10725, "query": "save"}diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.mdindex e442825136..5bec2c32e2 100644--- a/libs/cua-driver/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md@@ -167,11 +167,11 @@ failure mode and it steals focus every time.When a cua-driver call surprises you, diagnose cua-driver first:- **Tiny screenshot / empty `tree_markdown`?** Check- `cua-driver get_config` → `capture_mode`. Default `"vision"` omits- the AX tree (PNG only), `"ax"` omits the PNG, `"som"` returns- both. If a snapshot lacks a tree, `capture_mode` is almost- certainly `"vision"` — either reason purely from the PNG or flip- to `"som"` / `"ax"` via `set_config`.+ `cua-driver get_config` → `capture_mode`. Default `"som"` returns+ both the AX tree and screenshot. `"vision"` omits the AX tree+ (PNG only), `"ax"` omits the PNG. If a snapshot lacks a tree,+ `capture_mode` is almost certainly `"vision"` — either reason+ purely from the PNG or flip to `"som"` / `"ax"` via `set_config`.- **`has_screenshot: false`?** The window capture failed (transientrace against a close, or the window has no backing store yet).Re-snapshot; if persistent, pick a different `window_id` via@@ -414,17 +414,14 @@ single-window case you can skip `list_windows` entirely and read theCall `get_window_state({pid, window_id})` with the `window_id` from`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if-you're interacting with a long-lived process). In the default-`vision` capture_mode the response carries **only the screenshot**-— no AX tree — so the canonical loop is `list_windows →-get_window_state → reason over PNG → pixel click`. When you need-`element_index` dispatch (AX-addressable elements, backgrounded-clicks), flip to `som` first: `cua-driver set_config '{"key":-"capture_mode", "value": "som"}'`. The rest of this section walks through `som` mode, which-is what you want once you've decided element-indexed addressing is-required.--In `som` mode the response carries:+you're interacting with a long-lived process). The default `som`+capture_mode returns **both the AX tree and screenshot**, so the+canonical loop works immediately without any config change. The rest+of this section walks through `som` mode. If you're in `vision` mode+(PNG only, no AX tree), flip back: `cua-driver set_config '{"key":+"capture_mode", "value": "som"}'`.++In `som` mode (the default) the response carries:- `tree_markdown` — every actionable element tagged `[N]`. That `N`is the `element_index`. The tree can be very large (Finder isdiff --git a/libs/cua-driver/Skills/cua-driver/TESTS.md b/libs/cua-driver/Skills/cua-driver/TESTS.mdindex fdc835b329..98b949c25e 100644--- a/libs/cua-driver/Skills/cua-driver/TESTS.md+++ b/libs/cua-driver/Skills/cua-driver/TESTS.md@@ -59,7 +59,7 @@ backgrounded throughout, and Claude uses the canonical### 4. Notes — native text entry**Prompt:** `Create a new note in Notes titled 'cua-driver test' with the body 'native text entry verification'.`-**Exercises:** `type_text_in` via `kAXSelectedText` against native AppKit text fields.+**Exercises:** `type_text` via `kAXSelectedText` against native AppKit text fields.**Success:**- Re-snapshot of Notes shows a note row in the sidebar whose title contains `cua-driver test`.@@ -72,7 +72,7 @@ backgrounded throughout, and Claude uses the canonical### 5. Numbers — cell-addressable UI**Prompt:** `Open ~/Documents/test.numbers and put 42 in cell B2.`-**Exercises:** unusual AX shape (spreadsheet cells), click + `set_value` or `type_text_in` + Tab/Return.+**Exercises:** unusual AX shape (spreadsheet cells), click + `set_value` or `type_text` + Tab/Return.**Success:**- Re-snapshot shows cell B2's AXValue is `42`.