Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

perf(cua-driver): front-load efficient workflow, trim hot tool descriptions (CUA-1195, CUA-1198) (#4742)

Francesco Bonaccic0762938ba76
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
index b59f881f06..fcf24286a5 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
@@ -15,7 +15,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
| [`list_apps`](#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |
-| [`list_windows`](#list_windows) | List all layer-0 top-level windows currently known to WindowServer. | macOS, Linux, Windows |
+| [`list_windows`](#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |
| [`launch_app`](#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows |
| [`kill_app`](#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows |
| [`bring_to_front`](#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |
@@ -94,16 +94,16 @@ Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-List all layer-0 top-level windows currently known to WindowServer. Includes off-screen windows (minimized, on another Space, hidden-launched). Use this to find a window_id before calling get_window_state. AppKit-internal helper windows (off screen, untitled, in no Space, and absent from the app's AXWindows) are omitted: no tool can read, move, or focus them.
+List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `window_id` for get_window_state. AppKit-internal helper windows are omitted.
-Per-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one), is_on_screen, space_ids, current_space_id (the active Space on that window's display), and on_current_space. The top-level current_space_id is WindowServer's main/global active Space and can differ from a record's current_space_id when displays use independent Spaces. To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order.
+Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="list_windows--macos--on_screen_only"></span>`on_screen_only` | `boolean` | | When true, drop windows not on the current Space. Default false. |
-| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Optional pid filter. When set, only this pid's windows are returned. |
+| <span id="list_windows--macos--on_screen_only"></span>`on_screen_only` | `boolean` | | Drop windows not on the current Space. |
+| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Only this pid's windows. |
</Tab>
<Tab value="Linux">
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
index 70f5dcc83f..2acc0c96d6 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
@@ -21,34 +21,30 @@ Effect: destructive. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Click against a target pid. **Prefer `element_token` over pixel coordinates**: the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree.
+Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.
-Two addressing modes:
-
-- element_token (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window: re-snapshot every turn before clicking.
-
-- x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion.
+- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.
+- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.
-button: "left" (default), "right", or "middle". Defaults to left so the field is fully back-compat: omit it and you get the legacy left-click behaviour. Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. AX path: "right" maps to AXShowMenu (same surface as the dedicated `right_click` tool); "middle" has no AX equivalent and falls back to a pixel middle-click at the element's center.
-action: press (default), show_menu, pick, confirm, cancel, open.
-from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordinates to full-window space.
+`button` defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="click--macos--action"></span>`action` | `string` | | AX action: press, show_menu, pick, confirm, cancel, open. |
-| <span id="click--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button. Default: "left"; omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle". |
-| <span id="click--macos--capture_id"></span>`capture_id` | `string` | | Optional immutable source capture ID returned by get_window_state or get_desktop_state. With x,y, Driver atomically admits and consumes that exact capture before dispatch; stale, mismatched, or out-of-bounds captures are refused without fallback. |
+| <span id="click--macos--action"></span>`action` | `string` | | AX action. |
+| <span id="click--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button. Default left. |
+| <span id="click--macos--capture_id"></span>`capture_id` | `string` | | Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused. |
| <span id="click--macos--count"></span>`count` | `integer` | | Click count (pixel path only). Default 1. |
-| <span id="click--macos--debug_image_out"></span>`debug_image_out` | `string` | | Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom. |
-| <span id="click--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require "foreground" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_token) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground". |
-| <span id="click--macos--from_zoom"></span>`from_zoom` | `boolean` | | When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates. |
+| <span id="click--macos--debug_image_out"></span>`debug_image_out` | `string` | | Write a screenshot with a crosshair at (x, y) to this path. Needs window_id; not with from_zoom. |
+| <span id="click--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default "background" (no fronting). "foreground" briefly fronts the window, acts, then restores the prior app; needs window_id and is required for modified clicks. Escalate in order: AX element_token, background pixel, foreground, checking state between rungs; a click has no read-back except list-row selection. |
+| <span id="click--macos--from_zoom"></span>`from_zoom` | `boolean` | | x and y are in the last zoom image for this pid. |
| <span id="click--macos--modifier"></span>`modifier` | `string[]` | | Modifier keys: cmd, shift, option/alt, ctrl. |
| <span id="click--macos--pid"></span>`pid` | `integer` | | Target process ID. |
-| <span id="click--macos--scope"></span>`scope` | `"window" \| "desktop"` | | Coordinate frame for a windowless screen-absolute click (default "window"). Pass "desktop" when sending x,y with NO pid/window_id: the coordinates are then true screen pixels (read from get_desktop_state with scope="desktop"). Per-call; not a setting. |
-| <span id="click--macos--window_id"></span>`window_id` | `integer` | | Target window ID. Omit when element_token is supplied (the token carries it). |
-| <span id="click--macos--x"></span>`x` | `number` | | X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale. |
+| <span id="click--macos--scope"></span>`scope` | `"window" \| "desktop"` | | Default "window". "desktop": x,y with no pid/window_id are true screen pixels from get_desktop_state. |
+| <span id="click--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
+| <span id="click--macos--window_id"></span>`window_id` | `integer` | | Target window ID. Omit with element_token. |
+| <span id="click--macos--x"></span>`x` | `number` | | X in screenshot pixels (get_window_state PNG, or get_desktop_state PNG for a desktop target). |
| <span id="click--macos--y"></span>`y` | `number` | | Y in screenshot pixels from the image selected by target. |
</Tab>
@@ -78,6 +74,7 @@ button: "left" (default), "right", or "middle". Defaults to left so the field is
| <span id="click--linux--modifier"></span>`modifier` | `string[]` | | Modifier keys held during the action: cmd, shift, option/alt, ctrl. |
| <span id="click--linux--pid"></span>`pid` | `integer` | | Target process ID. Required unless scope is "desktop". |
| <span id="click--linux--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Coordinate frame (default "window"). Pass "desktop" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates. |
+| <span id="click--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="click--linux--window_id"></span>`window_id` | `integer` | | Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it). |
| <span id="click--linux--x"></span>`x` | `number` | | Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop"). |
| <span id="click--linux--y"></span>`y` | `number` | | Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x. |
@@ -110,6 +107,7 @@ Windows-only convenience: `button: "left"|"right"|"middle"` switches the mouse b
| <span id="click--windows--modifier"></span>`modifier` | `string[]` | | Modifier keys held during the action: cmd, shift, option/alt, ctrl. |
| <span id="click--windows--pid"></span>`pid` | `integer` | | Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state. |
| <span id="click--windows--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Coordinate frame (default "window"). Pass "desktop" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates. |
+| <span id="click--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="click--windows--window_id"></span>`window_id` | `integer` | | HWND of the target window. Omit when element_token is supplied (the token carries it). |
| <span id="click--windows--x"></span>`x` | `number` | | X in window-local screenshot pixels: same space as the PNG get_window_state returns. Must be provided together with y. |
| <span id="click--windows--y"></span>`y` | `number` | | Y in window-local screenshot pixels. Must be provided together with x. |
@@ -122,5 +120,4 @@ Windows-only convenience: `button: "left"|"right"|"middle"` switches the mouse b
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="click--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
-| <span id="click--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="click--target"></span>`target` | `window target \| desktop target` | | Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
index ae4fbb1927..b5d019a317 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
@@ -55,7 +55,7 @@ List, launch, quit, front and arrange apps and windows. [Reference](/cua-driver/
| Tool | Description | Platforms |
| --- | --- | --- |
| <span id="list_apps"></span>[`list_apps`](/cua-driver/reference/mcp-tools/apps-and-windows#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |
-| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List all layer-0 top-level windows currently known to WindowServer. | macOS, Linux, Windows |
+| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |
| <span id="launch_app"></span>[`launch_app`](/cua-driver/reference/mcp-tools/apps-and-windows#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows |
| <span id="kill_app"></span>[`kill_app`](/cua-driver/reference/mcp-tools/apps-and-windows#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows |
| <span id="bring_to_front"></span>[`bring_to_front`](/cua-driver/reference/mcp-tools/apps-and-windows#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |
@@ -69,7 +69,7 @@ Screenshots, accessibility trees, verification and visual regions of a window. [
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). | macOS, Linux, Windows |
+| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |
| <span id="get_accessibility_tree"></span>[`get_accessibility_tree`](/cua-driver/reference/mcp-tools/window-state#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
| <span id="verify_state"></span>[`verify_state`](/cua-driver/reference/mcp-tools/window-state#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
| <span id="parse_visual_regions"></span>[`parse_visual_regions`](/cua-driver/reference/mcp-tools/window-state#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
@@ -91,7 +91,7 @@ Click by element or pixel. [Reference](/cua-driver/reference/mcp-tools/click).
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="click"></span>[`click`](/cua-driver/reference/mcp-tools/click#click) | Click against a target pid. | macOS, Linux, Windows |
+| <span id="click"></span>[`click`](/cua-driver/reference/mcp-tools/click#click) | Click a target pid. | macOS, Linux, Windows |
## Double-click and right-click tools
@@ -122,7 +122,7 @@ Type text into a field or window. [Reference](/cua-driver/reference/mcp-tools/ke
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="type_text"></span>[`type_text`](/cua-driver/reference/mcp-tools/keyboard#type_text) | Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. | macOS, Linux, Windows |
+| <span id="type_text"></span>[`type_text`](/cua-driver/reference/mcp-tools/keyboard#type_text) | Insert text via `AXSetAttribute(kAXSelectedText)` into the element given by `element_token`, or the pid's focused element. | macOS, Linux, Windows |
## Key and shortcut tools
@@ -130,7 +130,7 @@ Press keys and keyboard shortcuts. [Reference](/cua-driver/reference/mcp-tools/k
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="press_key"></span>[`press_key`](/cua-driver/reference/mcp-tools/keys-and-shortcuts#press_key) | Press and release a single key. | macOS, Linux, Windows |
+| <span id="press_key"></span>[`press_key`](/cua-driver/reference/mcp-tools/keys-and-shortcuts#press_key) | Press and release one key. | macOS, Linux, Windows |
| <span id="hotkey"></span>[`hotkey`](/cua-driver/reference/mcp-tools/keys-and-shortcuts#hotkey) | Press a key combination: e.g. | macOS, Linux, Windows |
## Value and clipboard tools
@@ -139,7 +139,7 @@ Set element values and read or write the clipboard. [Reference](/cua-driver/refe
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="set_value"></span>[`set_value`](/cua-driver/reference/mcp-tools/values-and-clipboard#set_value) | Set a value on a UI element. | macOS, Linux, Windows |
+| <span id="set_value"></span>[`set_value`](/cua-driver/reference/mcp-tools/values-and-clipboard#set_value) | Set an element's value by `element_token`. | macOS, Linux, Windows |
| <span id="clipboard_read"></span>[`clipboard_read`](/cua-driver/reference/mcp-tools/values-and-clipboard#clipboard_read) | List available system clipboard types and optionally return privacy-sensitive plain text. | macOS, Linux, Windows |
| <span id="clipboard_write"></span>[`clipboard_write`](/cua-driver/reference/mcp-tools/values-and-clipboard#clipboard_write) | Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL from an absolute local path. | macOS, Linux, Windows |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
index 86a56ddbc7..43d5896e23 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
@@ -21,25 +21,24 @@ Effect: destructive. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa text fields and text views. No keystrokes are synthesized: special keys (Return / Escape / arrows) go through `press_key` / `hotkey`. For Chromium / Electron inputs that don't implement `kAXSelectedText`, the tool falls back to CGEvent character synthesis automatically when the estimated route stays within the daemon transport budget. Longer synthesized routes are refused before character events and return a safe chunk size; one-call AX insertion remains uncapped.
+Insert text via `AXSetAttribute(kAXSelectedText)` into the element given by `element_token`, or the pid's focused element. Does not press keys: use `press_key` / `hotkey` for Return, Escape, arrows. If AX insertion is unsupported (Chromium/Electron), it falls back to CGEvent typing when the route fits the transport budget; longer routes are refused with a safe chunk size.
-Optional `element_token` (from the last `get_window_state` snapshot) directs the write to a specific field. Without `element_token`, the write goes to the pid's currently focused element.
-
-WEB CONTENT (Chromium/WebKit/Electron: browser tabs, Slack, VS Code, X's compose box): AXValue is not independent proof that the renderer/DOM observed an AX write or synthesized keystrokes. The driver detects this at the element level (an AXWebArea ancestor) and refuses to trust AXValue-only read-back there. Electron AX targets that are web content or cannot be proven native refuse background delivery before mutation because the AX route cannot establish exact renderer focus; use the px form or explicit foreground delivery. Other web-content paths return effect:"unverifiable" + escalation, never a false "confirmed" (a browser's own native address bar/toolbar stays trusted). For a browser TAB the reliable path is the `page` tool (drives the DOM via CDP); for an embedded web view use this tool's px form: pass x,y (no element_token) to pixel-click the field then type, in one call. NOTE: a px focus-click won't reliably open+focus a CLOSED control; AX-press to open/activate it first (works in the background), then px-type. Always confirm via the screenshot; if px-background still drops, escalate to delivery_mode:"foreground".
+Web content (browser tabs, Slack, VS Code): AXValue is not proof the DOM saw the text, so the result is `effect:"unverifiable"` and Electron web content refuses background delivery. For a browser tab use the `page` tool. For an embedded web view pass `x, y` (no element_token) to pixel-click the field then type in one call; a pixel click will not reliably open a closed control, so AX-press it open first. Confirm from the screenshot, and use `delivery_mode:"foreground"` only if background still drops the text.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="type_text--macos--delay_ms"></span>`delay_ms` | `integer` | | Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds. Range: 0 to 200. |
-| <span id="type_text--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed; no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:"unverifiable". "foreground": briefly front the window, type, restore the prior frontmost; the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear. |
+| <span id="type_text--macos--delay_ms"></span>`delay_ms` | `integer` | | Ms between characters in the CGEvent fallback. Default 30. Range: 0 to 200. |
+| <span id="type_text--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default "background": AX insert, then CGEvent keys, no focus steal. "foreground": front the window, type, restore; last resort for surfaces (WhatsApp/Catalyst) where background keys do not land. |
| <span id="type_text--macos--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="type_text--macos--pid"></span>`pid` | `integer` | | Target process ID. |
-| <span id="type_text--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use desktop with no pid/window_id to type into the frontmost application. |
-| <span id="type_text--macos--text"></span>`text` | `string` | required | Text to insert at the target's cursor. |
-| <span id="type_text--macos--window_id"></span>`window_id` | `integer` | | CGWindowID. Omit when element_token is supplied (the token carries it). |
-| <span id="type_text--macos--x"></span>`x` | `number` | | Screenshot-pixel X of the field to type into: the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click. |
-| <span id="type_text--macos--y"></span>`y` | `number` | | Screenshot-pixel Y of the field (see x). |
+| <span id="type_text--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "desktop" with no pid/window_id types into the frontmost app. |
+| <span id="type_text--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
+| <span id="type_text--macos--text"></span>`text` | `string` | required | Text to insert. |
+| <span id="type_text--macos--window_id"></span>`window_id` | `integer` | | Window ID. Omit with element_token. |
+| <span id="type_text--macos--x"></span>`x` | `number` | | Screenshot-pixel X of the field (get_window_state PNG, as in click). With y and no element_token, pixel-clicks to focus, then types. |
+| <span id="type_text--macos--y"></span>`y` | `number` | | Screenshot-pixel Y (see x). |
</Tab>
<Tab value="Linux">
@@ -55,6 +54,7 @@ Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.
| <span id="type_text--linux--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="type_text--linux--pid"></span>`pid` | `integer` | | Target process ID. Omit with scope "desktop" to type into the focused application. |
| <span id="type_text--linux--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use "desktop" with no pid to type into the focused application. Default "window". |
+| <span id="type_text--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="type_text--linux--text"></span>`text` | `string` | required | Text to type. |
| <span id="type_text--linux--window_id"></span>`window_id` | `integer` | | Window id from list_windows. Required with x/y; optional with element_token (the token carries it). |
| <span id="type_text--linux--x"></span>`x` | `number` | | Pixel X of the field to type into: the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop"): a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget. |
@@ -80,6 +80,7 @@ Special keys (Return, Escape, arrows, Tab) go through `press_key` / `hotkey`: th
| <span id="type_text--windows--element_token"></span>`element_token` | `string` | | Opaque element handle from get_window_state. When supplied, type_text writes through UIA ValuePattern and reads that exact element back by handle. The shared ActionResult is confirmed only when the complete expected value is synchronously visible. If SetValue succeeds but read-back is stale or unavailable, the result is unverifiable with no escalation; take a fresh snapshot before retrying because deferred providers may publish only after this call returns. |
| <span id="type_text--windows--pid"></span>`pid` | `integer` | | Target process ID. |
| <span id="type_text--windows--scope"></span>`scope` | `"window" \| "desktop"` | | Use desktop with no pid/window_id to type into the current foreground application. |
+| <span id="type_text--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="type_text--windows--text"></span>`text` | `string` | required | Text to insert at the focused element's cursor. |
| <span id="type_text--windows--window_id"></span>`window_id` | `integer` | | HWND of the target window. Omit when element_token is supplied (the token carries it). |
| <span id="type_text--windows--x"></span>`x` | `number` | | Window-local screenshot-pixel X of the field to type into: the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the UIA/WM_CHAR path can't reach. Read straight off the get_window_state PNG, same convention as click. |
@@ -92,7 +93,6 @@ Special keys (Return, Escape, arrows, Tab) go through `press_key` / `hotkey`: th
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="type_text--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="type_text--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
index d412b16b6b..a5ea771eba 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
@@ -14,7 +14,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
-| [`press_key`](#press_key) | Press and release a single key. | macOS, Linux, Windows |
+| [`press_key`](#press_key) | Press and release one key. | macOS, Linux, Windows |
| [`hotkey`](#hotkey) | Press a key combination: e.g. | macOS, Linux, Windows |
Served by `cua-driver mcp`; see [MCP tools](/cua-driver/reference/mcp-tools) for every tool.
@@ -26,23 +26,24 @@ Effect: destructive. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text: it does NOT raise the window by default:
-• `background` (default): post to the pid WITHOUT fronting/raising; the auth-message path (Chromium-safe). With element_token it focuses that AX element first. `window_id` only targets; it does not raise.
-• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id.
+Press and release one key. `delivery_mode` as in click/type_text.
+- `background` (default): posts to the pid without raising the window; with `element_token` it focuses that element first.
+- `foreground`: fronts the exact window (needs `window_id`), sends a real HID key transition (Chromium content, inline editors, menu equivalents), then restores the prior app.
-A key press is confirmed only when a bounded native AX value/selection read-back changes on the same control. Otherwise a successfully attempted post remains effect:"unverifiable" without implying delivery failure or recommending foreground. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.
+Confirmed only when a native AX value/selection read-back changes; otherwise `effect:"unverifiable"` (not a delivery failure). Keys: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, any letter or digit.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="press_key--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target; no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost; the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. |
-| <span id="press_key--macos--key"></span>`key` | `string` | required | Key name: return, tab, escape, up, down, etc. |
-| <span id="press_key--macos--modifiers"></span>`modifiers` | `string[]` | | Modifier keys: cmd, shift, option/alt, ctrl, fn. |
+| <span id="press_key--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default "background": no fronting or focus steal. "foreground": briefly front the window, act, restore the prior app; last resort when background did not land. |
+| <span id="press_key--macos--key"></span>`key` | `string` | required | Key name. |
+| <span id="press_key--macos--modifiers"></span>`modifiers` | `string[]` | | cmd, shift, option/alt, ctrl, fn. |
| <span id="press_key--macos--pid"></span>`pid` | `integer` | | Target process ID. |
-| <span id="press_key--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use desktop with no pid/window_id to send the key to the frontmost application. |
-| <span id="press_key--macos--window_id"></span>`window_id` | `integer` | | Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window: raising is gated on delivery_mode. |
-| <span id="press_key--macos--x"></span>`x` | `number` | | Screenshot-pixel X: the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token. |
+| <span id="press_key--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "desktop" with no pid/window_id sends to the frontmost app. |
+| <span id="press_key--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
+| <span id="press_key--macos--window_id"></span>`window_id` | `integer` | | Target window; required for foreground. Does not raise by itself. |
+| <span id="press_key--macos--x"></span>`x` | `number` | | Screenshot-pixel X: pixel-click to focus, then send the key. With y, no element_token. |
| <span id="press_key--macos--y"></span>`y` | `number` | | Screenshot-pixel Y (see x). |
</Tab>
@@ -60,6 +61,7 @@ Press a key via XSendEvent to a window. No focus steal.
| <span id="press_key--linux--modifiers"></span>`modifiers` | `string[]` | | Modifier keys held while the key is pressed: ctrl, shift, alt, super. |
| <span id="press_key--linux--pid"></span>`pid` | `integer` | | Target process ID. Omit with scope "desktop" to send the key to the focused application. |
| <span id="press_key--linux--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use "desktop" with no pid to send the key to the focused application. Default "window". |
+| <span id="press_key--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="press_key--linux--window_id"></span>`window_id` | `integer` | | Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it). |
| <span id="press_key--linux--x"></span>`x` | `number` | | Pixel X: the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop"). |
| <span id="press_key--linux--y"></span>`y` | `number` | | Pixel Y (see x). |
@@ -84,6 +86,7 @@ Key vocabulary: return, tab, escape, up/down/left/right, space, delete, home, en
| <span id="press_key--windows--modifiers"></span>`modifiers` | `string[]` | | Optional modifier names held while the key is pressed (ctrl/shift/alt/win). |
| <span id="press_key--windows--pid"></span>`pid` | `integer` | | Target process ID. |
| <span id="press_key--windows--scope"></span>`scope` | `"window" \| "desktop"` | | Use desktop with no pid/window_id to send the key to the current foreground application. |
+| <span id="press_key--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="press_key--windows--window_id"></span>`window_id` | `integer` | | HWND for the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window. |
| <span id="press_key--windows--x"></span>`x` | `number` | | Window-local screenshot-pixel X: the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the UIA path can't focus. Pass with y, no element_token. |
| <span id="press_key--windows--y"></span>`y` | `number` | | Window-local screenshot-pixel Y (see x). |
@@ -96,7 +99,6 @@ Key vocabulary: return, tab, escape, up/down/left/right, space, delete, home, en
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="press_key--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
-| <span id="press_key--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="press_key--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index e731498fa7..e94ffa8885 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -132,25 +132,24 @@ Effect: mutating. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Scroll the target pid. Two paths, picked by how you address the scroll:
+Scroll the target pid.
+- Targeted wheel: pass `element_token` (preferred) or window-local `x, y`. Sends a real wheel event at that point, so it scrolls whatever is under it. The only way to scroll nested `overflow:auto` regions in web views.
+- Keystroke: no target, just pid + direction. PageDown/PageUp (by='page') or arrows (by='line') on the focused scroller.
-• **Targeted wheel path**: when you pass a target, either `element_token` (preferred) or window-local `x, y` pixels: the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, at that screen point. The renderer hit-tests the wheel at the cursor, so the scroll lands on whatever element is under the point: exactly like physically rolling the wheel over it. This is the ONLY way to scroll a nested `overflow:auto` region (e.g. a scrollable &lt;div&gt; with no tabindex): such regions never take keyboard focus, so the keystroke path below no-ops on them. Use this for inner/nested scrollers in web views.
-
-• **Keystroke path (focused region)**: when you pass NO target (just pid + direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows (by='line'); horizontal uses Left/Right arrows. Drives the focused / page scroller only.
-
-Mapping: by='page' → larger step; by='line' → smaller step; amount = number of wheel notches (targeted path) or keystroke repetitions (keystroke path).
+`amount` is wheel notches (targeted) or key repetitions (keystroke).
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="scroll--macos--amount"></span>`amount` | `integer` | | Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3. Range: 1 to 50. |
-| <span id="scroll--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target; no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost; the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. |
+| <span id="scroll--macos--amount"></span>`amount` | `integer` | | Wheel notches or key repetitions, clamped to the maximum. Default 3. Range: 1 to 50. |
+| <span id="scroll--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default "background": no fronting or focus steal. "foreground": briefly front the window, act, restore the prior app; last resort when background did not land. |
| <span id="scroll--macos--pid"></span>`pid` | `integer` | | Target process ID. Required unless scope is "desktop". |
-| <span id="scroll--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates. |
-| <span id="scroll--macos--window_id"></span>`window_id` | `integer` | | CGWindowID of the target window. Required with x/y; optional with element_token (the token carries it). |
-| <span id="scroll--macos--x"></span>`x` | `number` | | Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point: use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion. |
-| <span id="scroll--macos--y"></span>`y` | `number` | | Window-local screenshot Y. See `x`. |
+| <span id="scroll--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "desktop" with x,y and no pid/window_id uses get_desktop_state coordinates. |
+| <span id="scroll--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
+| <span id="scroll--macos--window_id"></span>`window_id` | `integer` | | Window ID. Required with x/y; optional with element_token. |
+| <span id="scroll--macos--x"></span>`x` | `number` | | Window-local screenshot X (get_window_state PNG). With y, wheel-scrolls at that point; needs window_id. |
+| <span id="scroll--macos--y"></span>`y` | `number` | | Window-local screenshot Y. |
</Tab>
<Tab value="Linux">
@@ -167,6 +166,7 @@ Scroll the target pid's focused region via XSendEvent Button4/5. direction requi
| <span id="scroll--linux--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | `"background"` | Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus: Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window; a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. |
| <span id="scroll--linux--pid"></span>`pid` | `integer` | | Target process ID. Required unless scope is "desktop". |
| <span id="scroll--linux--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use "desktop" with x,y and no pid/window_id for get_desktop_state screen coordinates. Default "window". |
+| <span id="scroll--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="scroll--linux--window_id"></span>`window_id` | `integer` | | Window id from list_windows. Required with x/y; optional with element_token (the token carries it). |
| <span id="scroll--linux--x"></span>`x` | `number` | | Window-local screenshot-pixel X of the scroll target. Pass with y and without element_token. |
| <span id="scroll--linux--y"></span>`y` | `number` | | Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_token. |
@@ -190,6 +190,7 @@ Note: `element_token` is accepted for cross-platform parity but currently no-op
| <span id="scroll--windows--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Input delivery mode. 'background' (default) never swaps foreground: it routes through UIA Invoke / PostMessage and the window is never raised. For targets whose input stack silently drops posted events (Chromium/Electron content, GTK buttons, VCL/LibreOffice accelerators) the tool returns a structured background_unavailable error rather than fronting. 'foreground' is the explicit escalation: a brief SetForegroundWindow swap + SendInput, restoring the prior foreground afterward. IMPORTANT: 'background' is not a hint to weigh; it is the mandatory first attempt. Do NOT pass 'foreground' preemptively because a target 'looks like' GTK/Chromium/Electron; the DRIVER decides when background is impossible and tells you so via a background_unavailable error (or a verified no-op). Only THEN re-issue the same action with 'foreground'. The lists above are the driver's detectors, not a checklist for you to front on a guess: fronting up-front needlessly steals the user's focus and is a bug, not a shortcut. Matches the macOS delivery_mode surface. |
| <span id="scroll--windows--pid"></span>`pid` | `integer` | | Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state. |
| <span id="scroll--windows--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use "desktop" with x,y and no pid/window_id for screen-absolute coordinates from get_desktop_state. Default "window". |
+| <span id="scroll--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="scroll--windows--window_id"></span>`window_id` | `integer` | | HWND of the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window. |
| <span id="scroll--windows--x"></span>`x` | `number` | | With pid/window_id: window-local screenshot X used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute X for desktop scope. Must be paired with y. |
| <span id="scroll--windows--y"></span>`y` | `number` | | With pid/window_id: window-local screenshot Y used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute Y for desktop scope. Must be paired with x. |
@@ -204,7 +205,6 @@ Note: `element_token` is accepted for cross-platform parity but currently no-op
| <span id="scroll--by"></span>`by` | `"line" \| "page"` | | Scroll granularity. Default: line. |
| <span id="scroll--direction"></span>`direction` | `"up" \| "down" \| "left" \| "right"` | required | Scroll direction. |
| <span id="scroll--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
-| <span id="scroll--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="scroll--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
index 90327d3604..84e942e4e1 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
@@ -14,7 +14,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
-| [`set_value`](#set_value) | Set a value on a UI element. | macOS, Linux, Windows |
+| [`set_value`](#set_value) | Set an element's value by `element_token`. | macOS, Linux, Windows |
| [`clipboard_read`](#clipboard_read) | List available system clipboard types and optionally return privacy-sensitive plain text. | macOS, Linux, Windows |
| [`clipboard_write`](#clipboard_write) | Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL from an absolute local path. | macOS, Linux, Windows |
@@ -27,22 +27,17 @@ Effect: destructive, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Set a value on a UI element. Two modes depending on element role:
+Set an element's value by `element_token`. A popup button / select dropdown: presses the child option whose title or value matches `value` (case-insensitive) without opening the menu or stealing focus. Any other element: writes AXValue (sliders, steppers, date pickers, native text fields).
-- **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly; the native macOS popup menu is never opened, so focus is never stolen. Use this for HTML &lt;select&gt; elements in Safari or any native NSPopUpButton.
-
-- **All other elements**: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose settable AXValue).
-
-- **A file's name as Finder lists it** (a text field carrying AXFilename and a file URL, not being edited) or **as Finder's Get Info window shows it** (Finder's text field with AXIdentifier `Name`, not being edited): refused with `file_name_needs_rename`, because the write changes only what Finder shows, never the file. The refusal names the keyboard route that renames it.
-
-For free-form text entry into web inputs, prefer `type_text_chars` which synthesises key events: AXValue writes are ignored by WebKit.
+A Finder file name (list row or Get Info `Name` field, not being edited) is refused with `file_name_needs_rename`; the refusal names the keyboard route. For free-form text in web inputs use `type_text_chars`: WebKit ignores AXValue writes.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="set_value--macos--value"></span>`value` | `string` | required | New value. AX will coerce to the element's native type. |
-| <span id="set_value--macos--window_id"></span>`window_id` | `integer` | | CGWindowID. Omit when element_token is supplied (the token carries it). |
+| <span id="set_value--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
+| <span id="set_value--macos--value"></span>`value` | `string` | required | New value, coerced to the element's type. |
+| <span id="set_value--macos--window_id"></span>`window_id` | `integer` | | Window ID. Omit with element_token. |
</Tab>
<Tab value="Linux">
@@ -54,6 +49,7 @@ Set value of an AT-SPI element via SetValue action.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="set_value--linux--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | `"background"` | Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus: Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window; a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. |
+| <span id="set_value--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="set_value--linux--value"></span>`value` | `string \| number` | required | New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back. |
| <span id="set_value--linux--window_id"></span>`window_id` | `integer` | | Omit when element_token is supplied (the token carries it). |
@@ -73,6 +69,7 @@ For free-form text entry that the target accepts via keystrokes only, prefer `ty
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="set_value--windows--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Input delivery mode. 'background' (default) never swaps foreground: it routes through UIA Invoke / PostMessage and the window is never raised. For targets whose input stack silently drops posted events (Chromium/Electron content, GTK buttons, VCL/LibreOffice accelerators) the tool returns a structured background_unavailable error rather than fronting. 'foreground' is the explicit escalation: a brief SetForegroundWindow swap + SendInput, restoring the prior foreground afterward. IMPORTANT: 'background' is not a hint to weigh; it is the mandatory first attempt. Do NOT pass 'foreground' preemptively because a target 'looks like' GTK/Chromium/Electron; the DRIVER decides when background is impossible and tells you so via a background_unavailable error (or a verified no-op). Only THEN re-issue the same action with 'foreground'. The lists above are the driver's detectors, not a checklist for you to front on a guess: fronting up-front needlessly steals the user's focus and is a bug, not a shortcut. Matches the macOS delivery_mode surface. |
+| <span id="set_value--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="set_value--windows--value"></span>`value` | `string` | required | New value. UIA will coerce to the element's native type. |
| <span id="set_value--windows--window_id"></span>`window_id` | `integer` | | HWND of the window. Omit when element_token is supplied (the token carries it). |
@@ -85,7 +82,6 @@ For free-form text entry that the target accepts via keystrokes only, prefer `ty
| --- | --- | --- | --- |
| <span id="set_value--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="set_value--pid"></span>`pid` | `integer` | required | Target process ID. |
-| <span id="set_value--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
index c22bbe101f..6e25873936 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
@@ -14,7 +14,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
-| [`get_window_state`](#get_window_state) | Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). | macOS, Linux, Windows |
+| [`get_window_state`](#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |
| [`get_accessibility_tree`](#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
| [`verify_state`](#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
| [`parse_visual_regions`](#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
@@ -28,35 +28,28 @@ Effect: read-only. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to click, type_text, press_key, etc.
+Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.
-INVARIANT: call get_window_state once per turn per (pid, window_id) before any element action. The next snapshot of the window replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.
+START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.
-PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value` (the element's text/AXValue when present: use it to verify what a field holds), `actions` (names of AX actions exposed by the element, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side.
+A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
-Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only: the cheap path when you're just re-indexing before an element ax action.
-
-The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata, `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title`, the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.
-
-The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with `delivery_mode:"foreground"`. When that pid is an app still launching (its window exists before it answers accessibility), the walk first waits up to `timeout_ms` for it; if it never answers, the tree comes back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and `truncation_reason: app_lookup_timeout`. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.
-
-Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.
-
-Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).
+Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="get_window_state--macos--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title): the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |
-| <span id="get_window_state--macos--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. |
-| <span id="get_window_state--macos--max_depth"></span>`max_depth` | `integer` | | Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. Minimum: 1. |
+| <span id="get_window_state--macos--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error. |
+| <span id="get_window_state--macos--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture. |
+| <span id="get_window_state--macos--max_depth"></span>`max_depth` | `integer` | | Cap on walk depth. Default 25. Minimum: 1. |
| <span id="get_window_state--macos--max_dimension"></span>`max_dimension` | `integer` | | Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved): the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. Minimum: 1. |
-| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. Minimum: 1. |
+| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000. Minimum: 1. |
| <span id="get_window_state--macos--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior. Minimum: 0. |
| <span id="get_window_state--macos--pid"></span>`pid` | `integer` | required | Target process ID. |
-| <span id="get_window_state--macos--query"></span>`query` | `string` | | Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values. |
+| <span id="get_window_state--macos--query"></span>`query` | `string` | | Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read. |
| <span id="get_window_state--macos--screenshot_out_file"></span>`screenshot_out_file` | `string` | | When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. |
+| <span id="get_window_state--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
| <span id="get_window_state--macos--window_id"></span>`window_id` | `integer` | required | Target window ID from list_windows. |
</Tab>
@@ -91,6 +84,7 @@ POPUP MENUS: a context menu / popover / combo list is an override-redirect windo
| <span id="get_window_state--linux--pid"></span>`pid` | `integer` | required | Process ID that owns the window. |
| <span id="get_window_state--linux--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |
| <span id="get_window_state--linux--screenshot_out_file"></span>`screenshot_out_file` | `string` | | When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead. |
+| <span id="get_window_state--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="get_window_state--linux--window_id"></span>`window_id` | `integer` | | Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window. |
</Tab>
@@ -133,6 +127,7 @@ Windows requires no special permissions.
| <span id="get_window_state--windows--pid"></span>`pid` | `integer` | required | Process ID from `list_apps`. |
| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |
| <span id="get_window_state--windows--screenshot_out_file"></span>`screenshot_out_file` | `string` | | When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead. |
+| <span id="get_window_state--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="get_window_state--windows--window_id"></span>`window_id` | `integer` | required | HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array. |
</Tab>
@@ -143,7 +138,6 @@ Windows requires no special permissions.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |
-| <span id="get_window_state--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |
**Example arguments**
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 9b468e718e..05cb1225a0 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -31,7 +31,16 @@ metadata:
# cua-driver
-Operate one exact target, observe its state, act once, and verify the user's postcondition.
+Operate one exact target, observe it narrowly, act by element, and verify the user's postcondition at checkpoints.
+
+## Efficient loop
+
+The MCP server instructions carry the same rules, so they apply even when this file is not opened.
+
+1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`, and pass `include_screenshot:false` when the tree is enough. Widen only if the target is missing.
+2. Act by `element_token`. A narrowed read keeps the full snapshot actionable, so you can act on a match without re-reading. Use pixels only for surfaces missing from the tree.
+3. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.
+4. Batch known steps in one call once the batch tool exists (CUA-1194, not yet available).
## Act
@@ -39,7 +48,7 @@ Operate one exact target, observe its state, act once, and verify the user's pos
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Check installation and capabilities | `cua-driver --version`, `status`, `doctor`, `describe <tool>`; MCP `tools/list` | [Runtime](RUNTIME.md) |
| Find or open the requested app | `list_apps`, `list_windows`, `launch_app` | Current platform guide below |
-| Observe one window | `get_window_state({pid, window_id})` | [Workflow](WORKFLOW.md) |
+| Observe one window | `get_window_state({pid, window_id, query})` | [Workflow](WORKFLOW.md) |
| Act on a control | `click` / `type_text` with a fresh `element_token` and exact window target | [Workflow](WORKFLOW.md) |
| Use pixels when semantics cannot reach it | Fresh target screenshot, then `x,y` on the same target | [Workflow](WORKFLOW.md) |
| Verify the outcome | `verify_state({pid, window_id, expect})` or a fresh snapshot read by the agent | [Workflow](WORKFLOW.md) |
@@ -57,7 +66,7 @@ Check the installed version and advertised schema before using unfamiliar parame
## Rules
1. Select the exact target on each action. A session is lifecycle metadata, not capture scope or permission authority.
-2. Observe before input and verify after it. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.
+2. Observe before input and verify the outcome at checkpoints, not after every action. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.
3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`.
4. Keep background window actions non-interfering. Foreground delivery and desktop input require authorization for visible control; an unavailable route is not permission to escalate.
5. Never infer pixels from a missing image, a different window, or an unaccounted-for resized preview. Capture failure and an empty accessibility tree are different failures.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
index dc6e0509f8..80ea3c4536 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
@@ -35,7 +35,7 @@ These objects are argument fragments for tools advertising `target`, not standal
## Observe
-`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.
+`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Start with `query` (or `max_elements`) instead of a full read, and verify at checkpoints rather than after every action. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.
Prefer `structuredContent.elements` in MCP (the CLI prints structured fields directly) over parsing `tree_markdown`. Rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
index d7e7da82a0..6cc446d128 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
@@ -378,22 +378,22 @@ fn agent_instructions() -> String {
)
};
+ // TODO(CUA-1194): when the batch tool lands, add a loop step: send several
+ // known steps in one batch call instead of one call per action.
format!(
- r#"cua-driver: cross-platform background computer-use automation.
+ r#"cua-driver: background GUI automation. No shell: for non-GUI outcomes use an app API/SDK, CLI or filesystem and read the result back there.
-For non-GUI outcomes, prefer a client-provided app API/SDK, headless/background interface, CLI, or filesystem operation and read the result back in that semantic domain. This server has no shell.
+Efficient loop:
+1. `list_windows` or `launch_app` for `pid` and `window_id`.
+2. Read narrowly: `get_window_state` with `query:"Save"` (or `max_elements`) returns ~2K chars vs ~42K for a full snapshot. Widen only if the target is missing; `include_screenshot:false` when the tree suffices.
+3. Act by `element_token` ({tree_kind}) from the latest snapshot; pixel `x,y` only for surfaces missing from the tree. A new snapshot stales older tokens.
+4. Verify at checkpoints, not after every action: `verify_state(pid, window_id, expect)` or one targeted read. `unknown` and `effect:"unverifiable"` are not success.
-On continuation/recent-work, when available, call `history_status`; if ready, make one bounded initial `history_query` before broad discovery; otherwise continue.
+Use the narrowest semantic route first: `set_window_frame` plus `list_windows` for geometry, typed browser tools for pages, clipboard tools for the clipboard. Stay in the background; `delivery_mode:"foreground"` only if refused or unverifiable.
-For app/window outcomes, use the narrowest semantic Cua route first: `set_window_frame` plus `list_windows` readback for geometry, typed browser tools for supported page content, and clipboard tools for clipboard state. Then climb through background `element_token` ({tree_kind}), background pixels, foreground delivery, and desktop fallback. Never advance on transport success alone.
+`start_session` is optional; repeat a short `session` label on every call that accepts it. On continuation/recent-work, call `history_status`; if ready, make one bounded initial `history_query` before broad discovery.
-Workflow per turn:
-0. `start_session` is optional. For multi-call work, prefer a short `session` label and repeat it on every call that accepts it. Unnamed calls use the transport's implicit session. Only `start_session` revives an ended name; `end_session` explicitly cleans up.
-1. `launch_app`, then `get_window_state(pid, window_id)` to refresh element indices.
-2. Act with the fresh index.
-3. `verify_state(pid, window_id, expect)` checks bounded postconditions. `unknown` is not success; `include_screenshot:true` lets the multimodal agent judge visual evidence.
-
-Read `skill://cua-driver/SKILL.md` via `skills/get` or `resources/read`. Hosts control activation/consent. When activated, follow SKILL.md and {platform_skill_pointer}."#
+Details: `skill://cua-driver/SKILL.md` and {platform_skill_pointer}."#
)
}
@@ -485,19 +485,21 @@ mod agent_instruction_tests {
fn instructions_route_structured_and_visual_verification_to_the_right_owner() {
let instructions = agent_instructions();
assert!(instructions.contains("verify_state"));
- assert!(instructions.contains("`unknown` is not success"));
- assert!(instructions.contains("multimodal agent"));
- assert!(instructions.contains("client-provided app API/SDK"));
- assert!(instructions.contains("headless/background interface"));
- assert!(instructions.contains("read the result back in that semantic domain"));
- assert!(instructions.contains("narrowest semantic Cua route first"));
- assert!(instructions.contains("`set_window_frame` plus `list_windows` readback"));
- assert!(instructions.contains("typed browser tools for supported page content"));
- assert!(instructions.contains("has no shell"));
+ assert!(instructions.contains("`unknown` and `effect:\"unverifiable\"` are not success"));
+ assert!(instructions.contains("app API/SDK"));
+ assert!(instructions.contains("read the result back there"));
+ assert!(instructions.contains("narrowest semantic route first"));
+ assert!(instructions.contains("`set_window_frame` plus `list_windows`"));
+ assert!(instructions.contains("typed browser tools"));
+ assert!(instructions.contains("No shell"));
+ // The efficient workflow must be front-loaded: targeted read, act by
+ // token, verify at checkpoints.
+ assert!(instructions.contains("`query:\"Save\"`"));
+ assert!(instructions.contains("Act by `element_token`"));
+ assert!(instructions.contains("Verify at checkpoints, not after every action"));
assert!(
- instructions.find("client-provided app API/SDK")
- < instructions.find("background `element_token`"),
- "semantic/headless operations must precede native UI dispatch"
+ instructions.find("Read narrowly") < instructions.find("Act by `element_token`"),
+ "the targeted read must precede acting"
);
assert!(
instructions.split_whitespace().count() <= 200,
@@ -513,11 +515,9 @@ mod agent_instruction_tests {
.expect("initialize result should carry agent instructions");
assert!(instructions.contains("`start_session` is optional"));
- assert!(instructions.contains("prefer a short `session` label"));
- assert!(instructions.contains("repeat it on every call that accepts it"));
- assert!(instructions.contains("transport's implicit session"));
- assert!(instructions.contains("Only `start_session` revives an ended name"));
- assert!(instructions.contains("`end_session` explicitly cleans up"));
+ assert!(
+ instructions.contains("repeat a short `session` label on every call that accepts it")
+ );
assert!(
!instructions.contains("`start_session(session)` once"),
"initialize instructions must not require explicit session setup"
@@ -540,7 +540,6 @@ mod agent_instruction_tests {
assert!(status < bounded_query);
assert!(bounded_query < discovery);
assert!(instructions.contains("if ready"));
- assert!(instructions.contains("otherwise continue"));
assert!(instructions.split_whitespace().count() <= 200);
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index 70c094627c..daefb4b914 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -225,32 +225,12 @@ fn background_pixel_restore_pid(
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "click".into(),
- description:
- "Click against a target pid. **Prefer `element_token` over pixel \
- coordinates** — the token works on backgrounded / minimized / hidden / \
- off-Space windows, identifies one exact snapshot element, and tells \
- you what you're clicking via the cached element's role + label. Reach for \
- `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface \
- that doesn't appear in the AX tree.\n\n\
- Two addressing modes:\n\n\
- - element_token (from get_window_state): AX action path. \
- Works on backgrounded/hidden windows. No cursor move, no focus steal. \
- The snapshot cache is scoped per (pid, window_id) and is replaced by the \
- next snapshot of the same window — re-snapshot every turn before clicking.\n\n\
- - x, y (window-local screenshot pixels, top-left origin of the PNG returned \
- by get_window_state): CGEvent path. Synthesizes mouse events and posts to \
- pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen \
- window to anchor the conversion.\n\n\
- button: \"left\" (default), \"right\", or \"middle\". Defaults to left so the \
- field is fully back-compat — omit it and you get the legacy left-click behaviour. \
- Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. \
- AX path: \"right\" maps to AXShowMenu (same surface as the dedicated `right_click` \
- tool); \"middle\" has no AX equivalent and falls back to a pixel middle-click at the \
- element's center.\n\
- action: press (default), show_menu, pick, confirm, cancel, open.\n\
- from_zoom: set true after a zoom call to auto-translate zoom-image pixel \
- coordinates to full-window space."
- .into(),
+ description: "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\
+ \n\
+ - element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n\
+ - x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\
+ \n\
+ `button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call.".into(),
input_schema: serde_json::json!({
"type": "object",
// `pid` is conditionally required — needed for window/element clicks
@@ -260,18 +240,18 @@ fn def() -> &'static ToolDef {
// cua_driver_core::tool_schema.)
"required": [],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID." },
- "window_id": { "type": "integer", "description": "Target window ID. Omit when element_token is supplied (the token carries it)." },
+ "window_id": { "type": "integer", "description": "Target window ID. Omit with element_token." },
"element_token": cua_driver_core::tool_schema::element_token_schema(),
- "capture_id": { "type": "string", "description": "Optional immutable source capture ID returned by get_window_state or get_desktop_state. With x,y, Driver atomically admits and consumes that exact capture before dispatch; stale, mismatched, or out-of-bounds captures are refused without fallback." },
- "x": { "type": "number", "description": "X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale." },
+ "capture_id": { "type": "string", "description": "Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused." },
+ "x": { "type": "number", "description": "X in screenshot pixels (get_window_state PNG, or get_desktop_state PNG for a desktop target)." },
"y": { "type": "number", "description": "Y in screenshot pixels from the image selected by target." },
- "action": { "type": "string", "description": "AX action: press, show_menu, pick, confirm, cancel, open." },
+ "action": { "type": "string", "description": "AX action." },
"button": {
"type": "string",
"enum": ["left", "right", "middle"],
- "description": "Mouse button. Default: \"left\" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps \"right\" to AXShowMenu and falls back to a pixel middle-click at the element's center for \"middle\"."
+ "description": "Mouse button. Default left."
},
"count": { "type": "integer", "description": "Click count (pixel path only). Default 1." },
"modifier": {
@@ -281,21 +261,21 @@ fn def() -> &'static ToolDef {
},
"from_zoom": {
"type": "boolean",
- "description": "When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates."
+ "description": "x and y are in the last zoom image for this pid."
},
"debug_image_out": {
"type": "string",
- "description": "Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom."
+ "description": "Write a screenshot with a crosshair at (x, y) to this path. Needs window_id; not with from_zoom."
},
"delivery_mode": {
"type": "string",
"enum": ["background", "foreground"],
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require \"foreground\" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_token) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."
+ "description": "Default \"background\" (no fronting). \"foreground\" briefly fronts the window, acts, then restores the prior app; needs window_id and is required for modified clicks. Escalate in order: AX element_token, background pixel, foreground, checking state between rungs; a click has no read-back except list-row selection."
},
"scope": {
"type": "string",
"enum": ["window", "desktop"],
- "description": "Coordinate frame for a windowless screen-absolute click (default \"window\"). Pass \"desktop\" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope=\"desktop\"). Per-call; not a setting."
+ "description": "Default \"window\". \"desktop\": x,y with no pid/window_id are true screen pixels from get_desktop_state."
}
},
"additionalProperties": false
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index e582f27472..2d1f2e4687 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -27,88 +27,29 @@ const AX_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_se
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "get_window_state".into(),
- description: "Walk a running app's AX tree and return BOTH a structured \
- `elements` array (preferred) AND a Markdown rendering of the same tree \
- (back-compat). Every actionable element is tagged with [element_index N] \
- in the markdown and as `element_index` in the structured array; pass \
- each element's `element_token` to click, type_text, press_key, etc.\n\n\
- INVARIANT: call get_window_state once per turn per (pid, window_id) before any \
- element action. The next snapshot of the window replaces this one, stales its \
- element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\n\
- PREFERRED CONSUMERS read `structuredContent.elements` (one entry per \
- indexed row with `element_index`, `role`, `label`, `value` (the \
- element's text/AXValue when present — use it to verify what a field \
- holds), `actions` (names of AX actions exposed by the element, \
- omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \
- `tree_markdown` stays available \
- and unchanged in shape for existing text-parsing callers — but new \
- fields will only be added to the structured side.\n\n\
- Always returns BOTH the element tree AND a screenshot — ground on \
- both and cross-check (the tree lies on some surfaces: Electron \
- echo-confirms, Catalyst null values, virtualized off-viewport rows \
- with `h:1` frames). You choose the modality at ACTION time, not here: \
- an element ax action (pass `element_token` → the \
- accessibility rung) or an element px action (pass `x`,`y` → the pixel \
- rung, read straight off this screenshot). `capture_mode` is deprecated \
- and ignored. Pass `include_screenshot:false` to skip the grab and get \
- the tree only — the cheap path when you're just re-indexing before an \
- element ax action.\n\n\
- The mirror image: pass `include_accessibility_tree:false` to SKIP the \
- AX walk entirely (the expensive part, bounded by timeout_ms) and return just the \
- screenshot plus window metadata — `window_bounds`, `screenshot_scale`, \
- `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` \
- — the capture-only path for rendering a live window preview / \
- picture-in-picture without paying for perception. Setting BOTH \
- `include_accessibility_tree:false` and `include_screenshot:false` is an \
- error (nothing to return). Optional `max_image_dimension` overrides the \
- configured screenshot long-edge limit for this call; use 0 for native \
- resolution. The legacy `max_dimension` remains a tighter cap for \
- compatibility.\n\n\
- The snapshot is SCOPED to `window_id`: a window_id that no longer exists is \
- refused with `window_id_not_found`, and one owned by another process is \
- refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry \
- with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its \
- window belongs to the panel service, not the app). If the window is live under \
- this pid but its accessibility surface can't be resolved, the tree comes back \
- EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the \
- requested window; background input is refused until it resolves, so \
- re-snapshot or act with `delivery_mode:\"foreground\"`. When that pid is an \
- app still launching (its window exists before it answers accessibility), the \
- walk first waits up to `timeout_ms` for it; if it never answers, the tree comes \
- back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and \
- `truncation_reason: app_lookup_timeout`. A window on another \
- Space still resolves by its exact CGWindowID. This tool never returns another \
- surface's elements under your window_id. Before exposing a screenshot, \
- its raw dimensions are validated as a coherent 1x/2x representation of \
- the requested WindowServer bounds. `px_frame_mismatch` or \
- `px_capture_unavailable` omits an unprovable screenshot/pixel frame \
- instead of guessing a transform; the truthful AX payload remains available.\n\n\
- Optional `query` projects both tree_markdown and structured `elements` to \
- matching lines plus their ancestor chain (case-insensitive substring). The \
- element_index values are unchanged, the complete snapshot remains actionable, \
- and `element_count` continues to report its total size; \
- `filtered_element_count` reports the projected response size.\n\n\
- Optional `max_elements` / `max_depth` bound the AX walk to mitigate \
- context-window blow-up on Electron / Obsidian / large web apps that \
- produce 10k+ element trees. When applied, BOTH the markdown \
- and the structured elements are truncated identically. Omit both for \
- current default behaviour (≤2 000 elements, depth ≤25).".into(),
+ description: "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\
+ \n\
+ START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\
+ \n\
+ A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
+ \n\
+ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["pid", "window_id"],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID." },
"window_id": { "type": "integer", "description": "Target window ID from list_windows." },
- "query": { "type": "string", "description": "Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values." },
+ "query": { "type": "string", "description": "Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read." },
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
"include_accessibility_tree": {
"type": "boolean",
- "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."
+ "description": "Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error."
},
"include_screenshot": {
"type": "boolean",
- "description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk."
+ "description": "Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture."
},
"screenshot_out_file": {
"type": "string",
@@ -117,12 +58,12 @@ fn def() -> &'static ToolDef {
"max_elements": {
"type": "integer",
"minimum": 1,
- "description": "Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows."
+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000."
},
"max_depth": {
"type": "integer",
"minimum": 1,
- "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees."
+ "description": "Cap on walk depth. Default 25."
},
"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
"max_dimension": {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
index 2c346149ab..bf8bb1e7d7 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
@@ -12,30 +12,19 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "list_windows".into(),
- description: "List all layer-0 top-level windows currently known to WindowServer. \
- Includes off-screen windows (minimized, on another Space, hidden-launched). \
- Use this to find a window_id before calling get_window_state. \
- AppKit-internal helper windows (off screen, untitled, in no Space, and absent \
- from the app's AXWindows) are omitted: no tool can read, move, or focus them.\n\n\
- Per-record fields: window_id, pid, app_name, title, bounds \
- (x/y/width/height, top-left origin), z_index (integer or null; higher values are \
- closer to the front; null means stacking order is unavailable and callers must not \
- infer one), is_on_screen, space_ids, current_space_id (the active Space on that \
- window's display), and on_current_space. The top-level current_space_id is \
- WindowServer's main/global active Space and can differ from a record's \
- current_space_id when displays use independent Spaces. To select a frontmost candidate, take the \
- maximum integer z_index; if every value is null, use an explicit fallback instead of \
- relying on array order.".into(),
+ description: "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `window_id` for get_window_state. AppKit-internal helper windows are omitted.\n\
+ \n\
+ Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.".into(),
input_schema: serde_json::json!({
"type": "object",
"properties": {
"pid": {
"type": "integer",
- "description": "Optional pid filter. When set, only this pid's windows are returned."
+ "description": "Only this pid's windows."
},
"on_screen_only": {
"type": "boolean",
- "description": "When true, drop windows not on the current Space. Default false."
+ "description": "Drop windows not on the current Space."
}
},
"additionalProperties": false
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index 516f049eae..74a2206549 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -181,39 +181,29 @@ fn delivery_failed(error: impl std::fmt::Display) -> ToolResult {
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "press_key".into(),
- description: "Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text \
- — it does NOT raise the window by default:\n\
- • `background` (default): post to the pid WITHOUT fronting/raising — the \
- auth-message path (Chromium-safe). With element_token it focuses that AX \
- element first. `window_id` only targets; it does not raise.\n\
- • `foreground`: guard and briefly front the exact window, focus an addressed AX \
- element when supplied, send a genuine HID key transition so Chromium content, \
- inline editors, and native menu equivalents receive it, then restore prior \
- frontmost. Requires window_id.\n\n\
- A key press is confirmed only when a bounded native AX value/selection read-back \
- changes on the same control. Otherwise a successfully attempted post remains \
- effect:\"unverifiable\" without implying delivery failure or recommending foreground. \
- Key names: return, tab, escape, up/down/left/right, space, delete, \
- home, end, pageup, pagedown, f1-f12, plus any letter or digit. \
- Modifiers array: cmd, shift, option/alt, ctrl, fn.".into(),
+ description: "Press and release one key. `delivery_mode` as in click/type_text.\n\
+ - `background` (default): posts to the pid without raising the window; with `element_token` it focuses that element first.\n\
+ - `foreground`: fronts the exact window (needs `window_id`), sends a real HID key transition (Chromium content, inline editors, menu equivalents), then restores the prior app.\n\
+ \n\
+ Confirmed only when a native AX value/selection read-back changes; otherwise `effect:\"unverifiable\"` (not a delivery failure). Keys: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, any letter or digit.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["key"],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID." },
- "key": { "type": "string", "description": "Key name: return, tab, escape, up, down, etc." },
+ "key": { "type": "string", "description": "Key name." },
"modifiers": {
"type": "array",
"items": { "type": "string" },
- "description": "Modifier keys: cmd, shift, option/alt, ctrl, fn."
+ "description": "cmd, shift, option/alt, ctrl, fn."
},
- "window_id": { "type": "integer", "description": "Target window. Required for delivery_mode:\"foreground\". Does NOT itself raise the window — raising is gated on delivery_mode." },
+ "window_id": { "type": "integer", "description": "Target window; required for foreground. Does not raise by itself." },
"element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x": { "type": "number", "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token." },
+ "x": { "type": "number", "description": "Screenshot-pixel X: pixel-click to focus, then send the key. With y, no element_token." },
"y": { "type": "number", "description": "Screenshot-pixel Y (see x)." },
- "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the key to the frontmost application." },
- "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "\"desktop\" with no pid/window_id sends to the frontmost app." },
+ "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema_with("Default \"background\": no fronting or focus steal. \"foreground\": briefly front the window, act, restore the prior app; last resort when background did not land.")
},
"additionalProperties": false
}),
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
index b31ca0bf2d..7b8d94ed28 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
@@ -61,29 +61,18 @@ fn clamp_amount(requested: u64) -> usize {
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "scroll".into(),
- description: "Scroll the target pid. Two paths, picked by how you address the scroll:\n\n\
- • **Targeted wheel path** — when you pass a target, either \
- `element_token` (preferred) or window-local `x, y` pixels: \
- the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, \
- at that screen point. The renderer hit-tests the wheel at the \
- cursor, so the scroll lands on whatever element is under the point — exactly \
- like physically rolling the wheel over it. This is the ONLY way to scroll a \
- nested `overflow:auto` region (e.g. a scrollable <div> with no tabindex): such \
- regions never take keyboard focus, so the keystroke path below no-ops on them. \
- Use this for inner/nested scrollers in web views.\n\n\
- • **Keystroke path (focused region)** — when you pass NO target (just pid + \
- direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows \
- (by='line'); horizontal uses Left/Right arrows. Drives the focused / page \
- scroller only.\n\n\
- Mapping: by='page' → larger step; by='line' → smaller step; amount = number of \
- wheel notches (targeted path) or keystroke repetitions (keystroke path).".into(),
+ description: "Scroll the target pid.\n\
+ - Targeted wheel: pass `element_token` (preferred) or window-local `x, y`. Sends a real wheel event at that point, so it scrolls whatever is under it. The only way to scroll nested `overflow:auto` regions in web views.\n\
+ - Keystroke: no target, just pid + direction. PageDown/PageUp (by='page') or arrows (by='line') on the focused scroller.\n\
+ \n\
+ `amount` is wheel notches (targeted) or key repetitions (keystroke).".into(),
input_schema: serde_json::json!({
"type": "object",
// `pid` conditionally required (validated in code), not pinned in the
// schema — keeps the contract consistent across platforms.
"required": ["direction"],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID. Required unless scope is \"desktop\"." },
"direction": {
"type": "string",
@@ -99,14 +88,14 @@ fn def() -> &'static ToolDef {
"type": "integer",
"minimum": AMOUNT_MIN,
"maximum": AMOUNT_MAX,
- "description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3."
+ "description": "Wheel notches or key repetitions, clamped to the maximum. Default 3."
},
- "window_id": { "type": "integer", "description": "CGWindowID of the target window. Required with x/y; optional with element_token (the token carries it)." },
+ "window_id": { "type": "integer", "description": "Window ID. Required with x/y; optional with element_token." },
"element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x": { "type": "number", "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion." },
- "y": { "type": "number", "description": "Window-local screenshot Y. See `x`." },
- "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates." },
- "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
+ "x": { "type": "number", "description": "Window-local screenshot X (get_window_state PNG). With y, wheel-scrolls at that point; needs window_id." },
+ "y": { "type": "number", "description": "Window-local screenshot Y." },
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "\"desktop\" with x,y and no pid/window_id uses get_desktop_state coordinates." },
+ "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema_with("Default \"background\": no fronting or focus steal. \"foreground\": briefly front the window, act, restore the prior app; last resort when background did not land.")
},
"additionalProperties": false
}),
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
index 2db6493821..ded77100d1 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
@@ -45,42 +45,23 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "set_value".into(),
- description:
- "Set a value on a UI element. Two modes depending on element role:\n\
- \n\
- - **AXPopUpButton / select dropdown**: finds the child option whose \
- title or value matches `value` (case-insensitive) and AXPresses it \
- directly — the native macOS popup menu is never opened, so focus \
- is never stolen. Use this for HTML <select> elements in Safari or \
- any native NSPopUpButton.\n\
- \n\
- - **All other elements**: writes AXValue directly (sliders, steppers, \
- date pickers, native text fields that expose settable AXValue).\n\
- \n\
- - **A file's name as Finder lists it** (a text field carrying \
- AXFilename and a file URL, not being edited) or **as Finder's Get \
- Info window shows it** (Finder's text field with AXIdentifier \
- `Name`, not being edited): refused with `file_name_needs_rename`, \
- because the write changes only what Finder shows, never the file. \
- The refusal names the keyboard route that renames it.\n\
- \n\
- For free-form text entry into web inputs, prefer `type_text_chars` \
- which synthesises key events — AXValue writes are ignored by WebKit."
- .into(),
+ description: "Set an element's value by `element_token`. A popup button / select dropdown: presses the child option whose title or value matches `value` (case-insensitive) without opening the menu or stealing focus. Any other element: writes AXValue (sliders, steppers, date pickers, native text fields).\n\
+ \n\
+ A Finder file name (list row or Get Info `Name` field, not being edited) is refused with `file_name_needs_rename`; the refusal names the keyboard route. For free-form text in web inputs use `type_text_chars`: WebKit ignores AXValue writes.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["pid", "value"],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID." },
"window_id": {
"type": "integer",
- "description": "CGWindowID. Omit when element_token is supplied (the token carries it)."
+ "description": "Window ID. Omit with element_token."
},
"element_token": cua_driver_core::tool_schema::element_token_schema(),
"value": {
"type": "string",
- "description": "New value. AX will coerce to the element's native type."
+ "description": "New value, coerced to the element's type."
}
},
"additionalProperties": false
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 7551db4cbe..d977e2dad0 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -57,64 +57,34 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "type_text".into(),
- description:
- "Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. \
- Works for standard Cocoa text fields and text views. No keystrokes are \
- synthesized — special keys (Return / Escape / arrows) go through \
- `press_key` / `hotkey`. For Chromium / Electron inputs that don't \
- implement `kAXSelectedText`, the tool falls back to CGEvent \
- character synthesis automatically when the estimated route stays \
- within the daemon transport budget. Longer synthesized routes are \
- refused before character events and return a safe chunk size; \
- one-call AX insertion remains uncapped.\n\n\
- Optional `element_token` (from the last \
- `get_window_state` snapshot) directs the write to a specific field. \
- Without `element_token`, the write goes to the pid's currently \
- focused element.\n\n\
- WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, \
- X's compose box): AXValue is not independent proof that the \
- renderer/DOM observed an AX write or synthesized keystrokes. The \
- driver detects this at the element level (an AXWebArea ancestor) and \
- refuses to trust AXValue-only read-back there. Electron AX targets that \
- are web content or cannot be proven native refuse background delivery \
- before mutation because the AX route cannot establish exact renderer \
- focus; use the px form or explicit foreground delivery. Other web-content \
- paths return effect:\"unverifiable\" + \
- escalation, never a false \"confirmed\" (a \
- browser's own native address bar/toolbar stays trusted). For a browser \
- TAB the reliable path is the `page` tool (drives the DOM via CDP); for \
- an embedded web view use this tool's px form: pass x,y (no \
- element_token) to pixel-click the field then type, in one call. NOTE: \
- a px focus-click won't reliably open+focus a CLOSED control; AX-press \
- to open/activate it first (works in the background), then px-type. \
- Always confirm via the screenshot; if px-background still drops, \
- escalate to delivery_mode:\"foreground\"."
- .into(),
+ description: "Insert text via `AXSetAttribute(kAXSelectedText)` into the element given by `element_token`, or the pid's focused element. Does not press keys: use `press_key` / `hotkey` for Return, Escape, arrows. If AX insertion is unsupported (Chromium/Electron), it falls back to CGEvent typing when the route fits the transport budget; longer routes are refused with a safe chunk size.\n\
+ \n\
+ Web content (browser tabs, Slack, VS Code): AXValue is not proof the DOM saw the text, so the result is `effect:\"unverifiable\"` and Electron web content refuses background delivery. For a browser tab use the `page` tool. For an embedded web view pass `x, y` (no element_token) to pixel-click the field then type in one call; a pixel click will not reliably open a closed control, so AX-press it open first. Confirm from the screenshot, and use `delivery_mode:\"foreground\"` only if background still drops the text.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["text"],
"properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it." },
"pid": { "type": "integer", "description": "Target process ID." },
- "text": { "type": "string", "description": "Text to insert at the target's cursor." },
+ "text": { "type": "string", "description": "Text to insert." },
"window_id": {
"type": "integer",
- "description": "CGWindowID. Omit when element_token is supplied (the token carries it)."
+ "description": "Window ID. Omit with element_token."
},
"element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x": { "type": "number", "description": "Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click." },
- "y": { "type": "number", "description": "Screenshot-pixel Y of the field (see x)." },
+ "x": { "type": "number", "description": "Screenshot-pixel X of the field (get_window_state PNG, as in click). With y and no element_token, pixel-clicks to focus, then types." },
+ "y": { "type": "number", "description": "Screenshot-pixel Y (see x)." },
"delay_ms": {
"type": "integer",
"minimum": 0,
"maximum": 200,
- "description": "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds."
+ "description": "Ms between characters in the CGEvent fallback. Default 30."
},
- "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to type into the frontmost application." },
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "\"desktop\" with no pid/window_id types into the frontmost app." },
"delivery_mode": {
"type": "string",
"enum": ["background", "foreground"],
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:\"unverifiable\". \"foreground\": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with \"foreground\" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear."
+ "description": "Default \"background\": AX insert, then CGEvent keys, no focus steal. \"foreground\": front the window, type, restore; last resort for surfaces (WhatsApp/Catalyst) where background keys do not land."
}
},
"additionalProperties": false
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 2a84f7a268..4b7b270507 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -18,18 +18,18 @@
"read_only": true
},
{
- "description": "List all layer-0 top-level windows currently known to WindowServer. Includes off-screen windows (minimized, on another Space, hidden-launched). Use this to find a window_id before calling get_window_state. AppKit-internal helper windows (off screen, untitled, in no Space, and absent from the app's AXWindows) are omitted: no tool can read, move, or focus them.\n\nPer-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one), is_on_screen, space_ids, current_space_id (the active Space on that window's display), and on_current_space. The top-level current_space_id is WindowServer's main/global active Space and can differ from a record's current_space_id when displays use independent Spaces. To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order.",
+ "description": "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `window_id` for get_window_state. AppKit-internal helper windows are omitted.\n\nPer record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.",
"destructive": false,
"idempotent": true,
"input_schema": {
"additionalProperties": false,
"properties": {
"on_screen_only": {
- "description": "When true, drop windows not on the current Space. Default false.",
+ "description": "Drop windows not on the current Space.",
"type": "boolean"
},
"pid": {
- "description": "Optional pid filter. When set, only this pid's windows are returned.",
+ "description": "Only this pid's windows.",
"type": "integer"
},
"session": {
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to click, type_text, press_key, etc.\n\nINVARIANT: call get_window_state once per turn per (pid, window_id) before any element action. The next snapshot of the window replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value` (the element's text/AXValue when present — use it to verify what a field holds), `actions` (names of AX actions exposed by the element, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.\n\nThe snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with `delivery_mode:\"foreground\"`. When that pid is an app still launching (its window exists before it answers accessibility), the walk first waits up to `timeout_ms` for it; if it never answers, the tree comes back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and `truncation_reason: app_lookup_timeout`. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.\n\nOptional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.\n\nOptional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).",
+ "description": "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\nSTART NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -58,15 +58,15 @@
"type": "string"
},
"include_accessibility_tree": {
- "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
+ "description": "Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error.",
"type": "boolean"
},
"include_screenshot": {
- "description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk.",
+ "description": "Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture.",
"type": "boolean"
},
"max_depth": {
- "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees.",
+ "description": "Cap on walk depth. Default 25.",
"minimum": 1,
"type": "integer"
},
@@ -76,7 +76,7 @@
"type": "integer"
},
"max_elements": {
- "description": "Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows.",
+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000.",
"minimum": 1,
"type": "integer"
},
@@ -90,7 +90,7 @@
"type": "integer"
},
"query": {
- "description": "Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values.",
+ "description": "Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read.",
"type": "string"
},
"screenshot_out_file": {
@@ -98,7 +98,7 @@
"type": "string"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"timeout_ms": {
@@ -480,18 +480,18 @@
"read_only": false
},
{
- "description": "Click against a target pid. **Prefer `element_token` over pixel coordinates** — the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree.\n\nTwo addressing modes:\n\n- element_token (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking.\n\n- x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion.\n\nbutton: \"left\" (default), \"right\", or \"middle\". Defaults to left so the field is fully back-compat — omit it and you get the legacy left-click behaviour. Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. AX path: \"right\" maps to AXShowMenu (same surface as the dedicated `right_click` tool); \"middle\" has no AX equivalent and falls back to a pixel middle-click at the element's center.\naction: press (default), show_menu, pick, confirm, cancel, open.\nfrom_zoom: set true after a zoom call to auto-translate zoom-image pixel coordinates to full-window space.",
+ "description": "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\n- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\n`button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call.",
"destructive": true,
"idempotent": false,
"input_schema": {
"additionalProperties": false,
"properties": {
"action": {
- "description": "AX action: press, show_menu, pick, confirm, cancel, open.",
+ "description": "AX action.",
"type": "string"
},
"button": {
- "description": "Mouse button. Default: \"left\" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps \"right\" to AXShowMenu and falls back to a pixel middle-click at the element's center for \"middle\".",
+ "description": "Mouse button. Default left.",
"enum": [
"left",
"right",
@@ -500,7 +500,7 @@
"type": "string"
},
"capture_id": {
- "description": "Optional immutable source capture ID returned by get_window_state or get_desktop_state. With x,y, Driver atomically admits and consumes that exact capture before dispatch; stale, mismatched, or out-of-bounds captures are refused without fallback.",
+ "description": "Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused.",
"type": "string"
},
"count": {
@@ -508,11 +508,11 @@
"type": "integer"
},
"debug_image_out": {
- "description": "Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.",
+ "description": "Write a screenshot with a crosshair at (x, y) to this path. Needs window_id; not with from_zoom.",
"type": "string"
},
"delivery_mode": {
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require \"foreground\" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_token) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\".",
+ "description": "Default \"background\" (no fronting). \"foreground\" briefly fronts the window, acts, then restores the prior app; needs window_id and is required for modified clicks. Escalate in order: AX element_token, background pixel, foreground, checking state between rungs; a click has no read-back except list-row selection.",
"enum": [
"background",
"foreground"
@@ -525,7 +525,7 @@
"type": "string"
},
"from_zoom": {
- "description": "When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.",
+ "description": "x and y are in the last zoom image for this pid.",
"type": "boolean"
},
"modifier": {
@@ -540,7 +540,7 @@
"type": "integer"
},
"scope": {
- "description": "Coordinate frame for a windowless screen-absolute click (default \"window\"). Pass \"desktop\" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope=\"desktop\"). Per-call; not a setting.",
+ "description": "Default \"window\". \"desktop\": x,y with no pid/window_id are true screen pixels from get_desktop_state.",
"enum": [
"window",
"desktop"
@@ -548,7 +548,7 @@
"type": "string"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"target": {
@@ -597,11 +597,11 @@
]
},
"window_id": {
- "description": "Target window ID. Omit when element_token is supplied (the token carries it).",
+ "description": "Target window ID. Omit with element_token.",
"type": "integer"
},
"x": {
- "description": "X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale.",
+ "description": "X in screenshot pixels (get_window_state PNG, or get_desktop_state PNG for a desktop target).",
"type": "number"
},
"y": {
@@ -870,20 +870,20 @@
"read_only": false
},
{
- "description": "Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa text fields and text views. No keystrokes are synthesized — special keys (Return / Escape / arrows) go through `press_key` / `hotkey`. For Chromium / Electron inputs that don't implement `kAXSelectedText`, the tool falls back to CGEvent character synthesis automatically when the estimated route stays within the daemon transport budget. Longer synthesized routes are refused before character events and return a safe chunk size; one-call AX insertion remains uncapped.\n\nOptional `element_token` (from the last `get_window_state` snapshot) directs the write to a specific field. Without `element_token`, the write goes to the pid's currently focused element.\n\nWEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's compose box): AXValue is not independent proof that the renderer/DOM observed an AX write or synthesized keystrokes. The driver detects this at the element level (an AXWebArea ancestor) and refuses to trust AXValue-only read-back there. Electron AX targets that are web content or cannot be proven native refuse background delivery before mutation because the AX route cannot establish exact renderer focus; use the px form or explicit foreground delivery. Other web-content paths return effect:\"unverifiable\" + escalation, never a false \"confirmed\" (a browser's own native address bar/toolbar stays trusted). For a browser TAB the reliable path is the `page` tool (drives the DOM via CDP); for an embedded web view use this tool's px form: pass x,y (no element_token) to pixel-click the field then type, in one call. NOTE: a px focus-click won't reliably open+focus a CLOSED control; AX-press to open/activate it first (works in the background), then px-type. Always confirm via the screenshot; if px-background still drops, escalate to delivery_mode:\"foreground\".",
+ "description": "Insert text via `AXSetAttribute(kAXSelectedText)` into the element given by `element_token`, or the pid's focused element. Does not press keys: use `press_key` / `hotkey` for Return, Escape, arrows. If AX insertion is unsupported (Chromium/Electron), it falls back to CGEvent typing when the route fits the transport budget; longer routes are refused with a safe chunk size.\n\nWeb content (browser tabs, Slack, VS Code): AXValue is not proof the DOM saw the text, so the result is `effect:\"unverifiable\"` and Electron web content refuses background delivery. For a browser tab use the `page` tool. For an embedded web view pass `x, y` (no element_token) to pixel-click the field then type in one call; a pixel click will not reliably open a closed control, so AX-press it open first. Confirm from the screenshot, and use `delivery_mode:\"foreground\"` only if background still drops the text.",
"destructive": true,
"idempotent": false,
"input_schema": {
"additionalProperties": false,
"properties": {
"delay_ms": {
- "description": "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds.",
+ "description": "Ms between characters in the CGEvent fallback. Default 30.",
"maximum": 200,
"minimum": 0,
"type": "integer"
},
"delivery_mode": {
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:\"unverifiable\". \"foreground\": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with \"foreground\" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear.",
+ "description": "Default \"background\": AX insert, then CGEvent keys, no focus steal. \"foreground\": front the window, type, restore; last resort for surfaces (WhatsApp/Catalyst) where background keys do not land.",
"enum": [
"background",
"foreground"
@@ -901,7 +901,7 @@
},
"scope": {
"default": "window",
- "description": "Use desktop with no pid/window_id to type into the frontmost application.",
+ "description": "\"desktop\" with no pid/window_id types into the frontmost app.",
"enum": [
"window",
"desktop"
@@ -909,7 +909,7 @@
"type": "string"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"target": {
@@ -966,19 +966,19 @@
"description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
},
"text": {
- "description": "Text to insert at the target's cursor.",
+ "description": "Text to insert.",
"type": "string"
},
"window_id": {
- "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "description": "Window ID. Omit with element_token.",
"type": "integer"
},
"x": {
- "description": "Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click.",
+ "description": "Screenshot-pixel X of the field (get_window_state PNG, as in click). With y and no element_token, pixel-clicks to focus, then types.",
"type": "number"
},
"y": {
- "description": "Screenshot-pixel Y of the field (see x).",
+ "description": "Screenshot-pixel Y (see x).",
"type": "number"
}
},
@@ -991,14 +991,14 @@
"read_only": false
},
{
- "description": "Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:\n• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_token it focuses that AX element first. `window_id` only targets; it does not raise.\n• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id.\n\nA key press is confirmed only when a bounded native AX value/selection read-back changes on the same control. Otherwise a successfully attempted post remains effect:\"unverifiable\" without implying delivery failure or recommending foreground. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.",
+ "description": "Press and release one key. `delivery_mode` as in click/type_text.\n- `background` (default): posts to the pid without raising the window; with `element_token` it focuses that element first.\n- `foreground`: fronts the exact window (needs `window_id`), sends a real HID key transition (Chromium content, inline editors, menu equivalents), then restores the prior app.\n\nConfirmed only when a native AX value/selection read-back changes; otherwise `effect:\"unverifiable\"` (not a delivery failure). Keys: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, any letter or digit.",
"destructive": true,
"idempotent": false,
"input_schema": {
"additionalProperties": false,
"properties": {
"delivery_mode": {
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "description": "Default \"background\": no fronting or focus steal. \"foreground\": briefly front the window, act, restore the prior app; last resort when background did not land.",
"enum": [
"background",
"foreground"
@@ -1011,11 +1011,11 @@
"type": "string"
},
"key": {
- "description": "Key name: return, tab, escape, up, down, etc.",
+ "description": "Key name.",
"type": "string"
},
"modifiers": {
- "description": "Modifier keys: cmd, shift, option/alt, ctrl, fn.",
+ "description": "cmd, shift, option/alt, ctrl, fn.",
"items": {
"type": "string"
},
@@ -1027,7 +1027,7 @@
},
"scope": {
"default": "window",
- "description": "Use desktop with no pid/window_id to send the key to the frontmost application.",
+ "description": "\"desktop\" with no pid/window_id sends to the frontmost app.",
"enum": [
"window",
"desktop"
@@ -1035,7 +1035,7 @@
"type": "string"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"target": {
@@ -1092,11 +1092,11 @@
"description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
},
"window_id": {
- "description": "Target window. Required for delivery_mode:\"foreground\". Does NOT itself raise the window — raising is gated on delivery_mode.",
+ "description": "Target window; required for foreground. Does not raise by itself.",
"type": "integer"
},
"x": {
- "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token.",
+ "description": "Screenshot-pixel X: pixel-click to focus, then send the key. With y, no element_token.",
"type": "number"
},
"y": {
@@ -1232,7 +1232,7 @@
"read_only": false
},
{
- "description": "Set a value on a UI element. Two modes depending on element role:\n\n- **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly — the native macOS popup menu is never opened, so focus is never stolen. Use this for HTML <select> elements in Safari or any native NSPopUpButton.\n\n- **All other elements**: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose settable AXValue).\n\n- **A file's name as Finder lists it** (a text field carrying AXFilename and a file URL, not being edited) or **as Finder's Get Info window shows it** (Finder's text field with AXIdentifier `Name`, not being edited): refused with `file_name_needs_rename`, because the write changes only what Finder shows, never the file. The refusal names the keyboard route that renames it.\n\nFor free-form text entry into web inputs, prefer `type_text_chars` which synthesises key events — AXValue writes are ignored by WebKit.",
+ "description": "Set an element's value by `element_token`. A popup button / select dropdown: presses the child option whose title or value matches `value` (case-insensitive) without opening the menu or stealing focus. Any other element: writes AXValue (sliders, steppers, date pickers, native text fields).\n\nA Finder file name (list row or Get Info `Name` field, not being edited) is refused with `file_name_needs_rename`; the refusal names the keyboard route. For free-form text in web inputs use `type_text_chars`: WebKit ignores AXValue writes.",
"destructive": true,
"idempotent": true,
"input_schema": {
@@ -1248,15 +1248,15 @@
"type": "integer"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"value": {
- "description": "New value. AX will coerce to the element's native type.",
+ "description": "New value, coerced to the element's type.",
"type": "string"
},
"window_id": {
- "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "description": "Window ID. Omit with element_token.",
"type": "integer"
}
},
@@ -1270,14 +1270,14 @@
"read_only": false
},
{
- "description": "Scroll the target pid. Two paths, picked by how you address the scroll:\n\n• **Targeted wheel path** — when you pass a target, either `element_token` (preferred) or window-local `x, y` pixels: the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, at that screen point. The renderer hit-tests the wheel at the cursor, so the scroll lands on whatever element is under the point — exactly like physically rolling the wheel over it. This is the ONLY way to scroll a nested `overflow:auto` region (e.g. a scrollable <div> with no tabindex): such regions never take keyboard focus, so the keystroke path below no-ops on them. Use this for inner/nested scrollers in web views.\n\n• **Keystroke path (focused region)** — when you pass NO target (just pid + direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows (by='line'); horizontal uses Left/Right arrows. Drives the focused / page scroller only.\n\nMapping: by='page' → larger step; by='line' → smaller step; amount = number of wheel notches (targeted path) or keystroke repetitions (keystroke path).",
+ "description": "Scroll the target pid.\n- Targeted wheel: pass `element_token` (preferred) or window-local `x, y`. Sends a real wheel event at that point, so it scrolls whatever is under it. The only way to scroll nested `overflow:auto` regions in web views.\n- Keystroke: no target, just pid + direction. PageDown/PageUp (by='page') or arrows (by='line') on the focused scroller.\n\n`amount` is wheel notches (targeted) or key repetitions (keystroke).",
"destructive": false,
"idempotent": false,
"input_schema": {
"additionalProperties": false,
"properties": {
"amount": {
- "description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3.",
+ "description": "Wheel notches or key repetitions, clamped to the maximum. Default 3.",
"maximum": 50,
"minimum": 1,
"type": "integer"
@@ -1291,7 +1291,7 @@
"type": "string"
},
"delivery_mode": {
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "description": "Default \"background\": no fronting or focus steal. \"foreground\": briefly front the window, act, restore the prior app; last resort when background did not land.",
"enum": [
"background",
"foreground"
@@ -1319,7 +1319,7 @@
},
"scope": {
"default": "window",
- "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates.",
+ "description": "\"desktop\" with x,y and no pid/window_id uses get_desktop_state coordinates.",
"enum": [
"window",
"desktop"
@@ -1327,7 +1327,7 @@
"type": "string"
},
"session": {
- "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
"target": {
@@ -1384,15 +1384,15 @@
"description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
},
"window_id": {
- "description": "CGWindowID of the target window. Required with x/y; optional with element_token (the token carries it).",
+ "description": "Window ID. Required with x/y; optional with element_token.",
"type": "integer"
},
"x": {
- "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion.",
+ "description": "Window-local screenshot X (get_window_state PNG). With y, wheel-scrolls at that point; needs window_id.",
"type": "number"
},
"y": {
- "description": "Window-local screenshot Y. See `x`.",
+ "description": "Window-local screenshot Y.",
"type": "number"
}
},