Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver)!: make native desktop actions snapshot-safe (#2783)

Francesco Bonaccid8ae6df643df
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 410cc5f8bb..5bc8134ecc 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposes
import { Callout } from 'fumadocs-ui/components/callout';
-`cua-driver` exposes 53 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
+`cua-driver` exposes 54 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`.
@@ -67,7 +67,7 @@ Always returns BOTH the element tree AND a screenshot — ground on both and cro
The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.
-Optional `query` filters the tree_markdown to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged — filtering only trims the rendered Markdown.
+Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.
Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).
@@ -78,7 +78,7 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind
- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded
- `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. range: 1–unbounded
- `pid` (integer, required): Target process ID.
-- `query` (string, optional): Case-insensitive filter for tree_markdown.
+- `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values.
- `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `window_id` (integer, required): Target window ID from list_windows.
@@ -220,11 +220,11 @@ Set one exact top-level window's frame in the desktop-coordinate space reported
### `click`
-Click against a target pid. **Prefer `element_index` over pixel coordinates** — element_index works on backgrounded / minimized / hidden / off-Space windows, surfaces a stable handle that survives rebuilds, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree.
+Click against a target pid. **Prefer `element_token` over pixel coordinates** — the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree.
Two addressing modes:
-- element_index + window_id (from last get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. element_index cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking.
+- element_token, or element_index + snapshot_id (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking.
- x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion.
@@ -239,13 +239,14 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin
- `count` (integer, optional): Click count (pixel path only). Default 1.
- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".
-- `element_index` (integer, optional): Element index from last get_window_state. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token` of the last get_window_state. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded — re-snapshot in that case.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.
- `modifier` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl.
- `pid` (integer, optional): Target process ID.
- `scope` (string, optional): Coordinate frame for a windowless screen-absolute click (default "window"). Pass "desktop" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope="desktop"). Per-call; not a setting.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `window_id` (integer, optional): Target window ID. Required for element_index. Optional when element_token is supplied (the token carries it).
- `x` (number, optional): X in screenshot pixels, read straight off the image you were handed — no scaling math needed. With pid+window_id (capture_scope=window): window-local pixels from the get_window_state PNG (top-left origin). Windowless (no pid/window_id, capture_scope=desktop): pixels from the get_desktop_state PNG (the native full-display image). Either way, the pixel you read IS the pixel that gets clicked; the driver undoes the Retina backing scale + any downscale internally.
- `y` (number, optional): Y in screenshot pixels (see x). Window-local from get_window_state, or full-display from get_desktop_state under capture_scope=desktop.
@@ -261,10 +262,11 @@ Pixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordina
**Arguments:**
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
-- `element_index` (integer, optional): Element index from last get_window_state. Uses AX path. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `pid` (integer, required)
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it).
- `x` (number, optional): Screen X coordinate (pixel path).
- `y` (number, optional): Screen Y coordinate (pixel path).
@@ -286,11 +288,12 @@ Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` always r
**Arguments:**
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
-- `element_index` (integer, optional): Element index from last get_window_state. Routes through AXShowMenu. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `modifier` (array of string, optional): Modifier keys held during the right-click: cmd/shift/option/ctrl. Pixel path only.
- `pid` (integer, required): Target process ID.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it).
- `x` (number, optional): X in window-local screenshot pixels. Must be provided together with y.
- `y` (number, optional): Y in window-local screenshot pixels. Must be provided together with x.
@@ -344,11 +347,12 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp
- `delay_ms` (integer, optional): Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds. range: 0–200
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:"unverifiable". "foreground": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear.
-- `element_index` (integer, optional): Element index from last get_window_state. Directs the write to a specific field. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `pid` (integer, optional): Target process ID.
- `scope` (string, optional): Use desktop with no pid/window_id to type into the frontmost application. default: `"window"`
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `text` (string, required): Text to insert at the target's cursor.
- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it).
- `x` (number, optional): Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click.
@@ -360,22 +364,23 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp
### `press_key`
-Press and release a single key, delivered to the target pid via CGEventPostToPid. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:
+Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:
• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_index it focuses that AX element first. `window_id` only targets; it does not raise.
-• `foreground`: briefly front the window (NSMenu path, &lt; 1 ms) so native menu key-equivalents dispatch, then restore prior frontmost — the explicit escalation for menu shortcuts an app drops in the background. Requires window_id (and no element_index).
+• `foreground`: guard and briefly front the exact window, send a genuine HID key transition so Chromium content and native menu equivalents both receive it, then restore prior frontmost. Requires window_id (and no element_index).
A key press is never driver-verifiable → effect:"unverifiable"; confirm via screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.
**Arguments:**
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
-- `element_index` (integer, optional)
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `key` (string, required): Key name: return, tab, escape, up, down, etc.
- `modifiers` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl, fn.
- `pid` (integer, optional)
- `scope` (string, optional): Use desktop with no pid/window_id to send the key to the frontmost application. default: `"window"`
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window — raising is gated on delivery_mode.
- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index.
- `y` (number, optional): Screenshot-pixel Y (see x).
@@ -421,10 +426,11 @@ For free-form text entry into web inputs, prefer `type_text_chars` which synthes
**Arguments:**
-- `element_index` (integer, optional): Element index from last get_window_state. Must be supplied unless element_token is provided. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `pid` (integer, required)
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `value` (string, required): New value. AX will coerce to the element's native type.
- `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used; optional when element_token is supplied (the token carries it).
@@ -448,11 +454,12 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number
- `by` (string, optional): Scroll granularity. Default: line.
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
- `direction` (string, required): Scroll direction.
-- `element_index` (integer, optional): Element from last get_window_state. Routes through the pixel-wheel path AT this element's center — use it to scroll a nested overflow region you located in the AX tree.
-- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `pid` (integer, optional)
- `scope` (string, optional): Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates. default: `"window"`
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `window_id` (integer, optional)
- `x` (number, optional): Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion.
- `y` (number, optional): Window-local screenshot Y. See `x`.
@@ -564,8 +571,8 @@ Start trajectory recording. Every subsequent action-tool invocation (click, righ
- `evidence.json` — capture status and a stable classification when an expected artifact could not be captured.
- `app_state.json` — post-action AX/UIA snapshot for the target pid.
- `screenshot.png` — compatibility alias of `after.png`.
-- `action.json` — tool name, full input arguments, result summary, pid, click point (when applicable), ISO-8601 timestamp.
-- `click.png` — for click-family actions only, `before.png` with a red marker at the click point.
+- `action.json` — tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp.
+- `click.png` — for dispatched click-family actions only, `before.png` with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead.
Turn folders are named `turn-00001/`, `turn-00002/`, etc. Turn numbering restarts at 1 each time recording is (re-)started.
@@ -788,6 +795,21 @@ caller. The driver does not interpret that image.
{"expect":[{"window":{"exists":true}}],"pid":844,"window_id":10725}
```
+### `invoke_menu`
+
+Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.
+
+**Arguments:**
+
+- `path` (array of string, required): items: 1–16
+- `pid` (integer, required): range: 1–unbounded
+- `session` (string, optional): Optional session id.
+- `window_id` (integer, required): range: 1–unbounded
+
+```json
+{"path":["example"],"pid":844,"window_id":10725}
+```
+
### `set_agent_cursor_theme`
Select an already-installed cursor theme for a session.
diff --git a/libs/cua-driver/contract/README.md b/libs/cua-driver/contract/README.md
index ac72f33057..06715af7ed 100644
--- a/libs/cua-driver/contract/README.md
+++ b/libs/cua-driver/contract/README.md
@@ -29,6 +29,7 @@ It also covers the portable whole-desktop loop:
- `get_cursor_position`
- `move_cursor` with the required `scope="desktop"`
- `set_window_frame` for exact, read-back-verified top-level window geometry
+- `invoke_menu` for an exact native application-menu path resolved live at each hop
- `click` with the required `scope="desktop"`
- `drag` and `scroll` in native desktop coordinates
- `type_text`, `press_key`, and `hotkey` against the foreground application
@@ -88,7 +89,7 @@ Compatibility is tracked separately at each boundary:
| Field | Current | Meaning |
| --- | --- | --- |
-| `contract_version` | `0.5.0` | Generated manifest and typed SDK shape |
+| `contract_version` | `0.6.0` | Generated manifest and typed SDK shape |
| `tools_list_schema_version` | `1` | cua-driver `tools/list` extension shape |
| `capability_version` | `1` | Additive capability-token vocabulary |
| `mcp_protocol_version` | `2025-06-18` | MCP initialization protocol served to agent runtimes |
diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.json
index ccf7747dd5..0860a225f3 100644
--- a/libs/cua-driver/contract/manifest.json
+++ b/libs/cua-driver/contract/manifest.json
@@ -1,7 +1,7 @@
{
"generated_notice": "Generated by cua-contract-gen; do not edit by hand.",
"experimental": true,
- "contract_version": "0.5.0",
+ "contract_version": "0.6.0",
"tools_list_schema_version": "1",
"capability_version": "1",
"mcp_protocol_version": "2025-06-18",
@@ -1365,6 +1365,177 @@
"type": "object"
}
},
+ {
+ "name": "invoke_menu",
+ "description": "Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.",
+ "platforms": [
+ "macos",
+ "windows",
+ "linux"
+ ],
+ "capabilities": [
+ "menu.path.invoke",
+ "accessibility.menu.native"
+ ],
+ "annotations": {
+ "read_only": false,
+ "destructive": true,
+ "idempotent": false,
+ "open_world": true
+ },
+ "schema_mode": "portable_subset",
+ "cursor_semantics": {
+ "action": "app"
+ },
+ "input_schema": {
+ "additionalProperties": false,
+ "description": "Exact, immediate-child application menu path to resolve and invoke through\nthe operating system's accessibility API. Path labels are matched after\ntrimming surrounding whitespace and otherwise remain case-sensitive.",
+ "properties": {
+ "path": {
+ "items": {
+ "maxLength": 200,
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 16,
+ "minItems": 1,
+ "type": "array"
+ },
+ "pid": {
+ "minimum": 1,
+ "type": "integer"
+ },
+ "session": {
+ "description": "Optional session id.",
+ "type": "string"
+ },
+ "window_id": {
+ "minimum": 1,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "pid",
+ "window_id",
+ "path"
+ ],
+ "type": "object"
+ },
+ "success_output_schema": {
+ "additionalProperties": false,
+ "properties": {
+ "delivery": {
+ "additionalProperties": false,
+ "properties": {
+ "delivered_count": {
+ "format": "uint32",
+ "minimum": 0,
+ "type": [
+ "integer",
+ "null"
+ ]
+ },
+ "mode": {
+ "enum": [
+ "background",
+ "foreground",
+ "not_applicable",
+ "unknown"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "mode"
+ ],
+ "type": [
+ "object",
+ "null"
+ ]
+ },
+ "effect": {
+ "enum": [
+ "confirmed",
+ "partial",
+ "unverifiable",
+ "suspected_noop",
+ "refused"
+ ],
+ "type": "string"
+ },
+ "escalation": {
+ "additionalProperties": false,
+ "properties": {
+ "reason": {
+ "enum": [
+ "route_unavailable",
+ "delivery_failed",
+ "effect_unconfirmed",
+ "suspected_noop",
+ "permission_required"
+ ],
+ "type": "string"
+ },
+ "target": {
+ "enum": [
+ "pixel",
+ "foreground",
+ "page",
+ "session"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "target",
+ "reason"
+ ],
+ "type": [
+ "object",
+ "null"
+ ]
+ },
+ "evidence": {
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "kind": {
+ "enum": [
+ "value_readback",
+ "window_change"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind"
+ ],
+ "type": "object"
+ },
+ "type": [
+ "array",
+ "null"
+ ]
+ },
+ "route": {
+ "enum": [
+ "accessibility",
+ "synthetic_events",
+ "global_input",
+ "system_api",
+ "dom",
+ "trusted_input"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "effect",
+ "route"
+ ],
+ "type": "object"
+ }
+ },
{
"name": "move_cursor",
"description": "Move the real OS pointer in get_desktop_state coordinates.",
diff --git a/libs/cua-driver/docs/e2e-ci-reporting.md b/libs/cua-driver/docs/e2e-ci-reporting.md
index 40a1657d37..31b9921ff0 100644
--- a/libs/cua-driver/docs/e2e-ci-reporting.md
+++ b/libs/cua-driver/docs/e2e-ci-reporting.md
@@ -65,7 +65,7 @@ artifacts/cua-driver/<platform>/
|-- after.png
|-- app_state.json
|-- screenshot.png
- `-- click.png (click-family turns)
+ `-- click.png (dispatched click-family turns)
```
`cases.jsonl` is the executed catalog. `results.jsonl` contains one result for
@@ -85,6 +85,13 @@ accessibility data was captured, and the required trajectory and finalized MP4
remain present. Any
other missing image still fails closed.
+A click-family call refused before its target can be resolved is the other
+explicit non-capture case: `action.json` records `result_error: true` without a
+`click_point`, and `evidence.json` records the click phase as
+`not_applicable/action_refused_before_target_resolution`. No input was aimed or
+dispatched, so that turn must not invent a `click.png`; successful and
+target-resolved click-family turns still require it.
+
The testkit prepares each cell's artifact directory before fixture setup but
does not start `recording.mp4` yet. On Windows GitHub-hosted runners it obtains
the test process's inherited top-level console with `GetConsoleWindow`, verifies
diff --git a/libs/cua-driver/python/src/cua_driver/__init__.py b/libs/cua-driver/python/src/cua_driver/__init__.py
index 865cbd6fcc..a699f5362f 100644
--- a/libs/cua-driver/python/src/cua_driver/__init__.py
+++ b/libs/cua-driver/python/src/cua_driver/__init__.py
@@ -80,6 +80,7 @@ from ._native_contract import (
GetScreenSizeInput,
GetSessionStateInput,
HotkeyInput,
+ InvokeMenuInput,
MoveCursorInput,
Platform,
PredicateOutcome,
@@ -227,6 +228,7 @@ __all__ = [
"GetScreenSizeInput",
"GetSessionStateInput",
"HotkeyInput",
+ "InvokeMenuInput",
"ImageContent",
"MacOsPermissionStatus",
"MoveCursorInput",
diff --git a/libs/cua-driver/python/src/cua_driver/_native.py b/libs/cua-driver/python/src/cua_driver/_native.py
index c30444c3c8..54f92889ef 100644
--- a/libs/cua-driver/python/src/cua_driver/_native.py
+++ b/libs/cua-driver/python/src/cua_driver/_native.py
@@ -544,6 +544,8 @@ def _uniffi_check_api_checksums(lib):
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey() != 53333:
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
+ if lib.uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu() != 13118:
+ raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available() != 42961:
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriver_list_tools_json() != 33039:
@@ -604,6 +606,8 @@ def _uniffi_check_api_checksums(lib):
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey() != 12820:
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
+ if lib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu() != 2742:
+ raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor() != 35768:
raise InternalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project")
if lib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_press_key() != 36768:
@@ -1171,6 +1175,11 @@ _UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_hotkey.argtypes = (
cua_driver._native_contract._UniffiRustBuffer,
)
_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_hotkey.restype = ctypes.c_uint64
+_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu.argtypes = (
+ ctypes.c_uint64,
+ cua_driver._native_contract._UniffiRustBuffer,
+)
+_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu.restype = ctypes.c_uint64
_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_is_available.argtypes = (
ctypes.c_uint64,
ctypes.POINTER(_UniffiRustCallStatus),
@@ -1319,6 +1328,11 @@ _UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_hotkey.argtypes = (
cua_driver._native_contract._UniffiRustBuffer,
)
_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_hotkey.restype = ctypes.c_uint64
+_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu.argtypes = (
+ ctypes.c_uint64,
+ cua_driver._native_contract._UniffiRustBuffer,
+)
+_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu.restype = ctypes.c_uint64
_UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_move_cursor.argtypes = (
ctypes.c_uint64,
cua_driver._native_contract._UniffiRustBuffer,
@@ -1517,6 +1531,9 @@ _UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_get_session_state.res
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey.argtypes = (
)
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey.restype = ctypes.c_uint16
+_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu.argtypes = (
+)
+_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu.restype = ctypes.c_uint16
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available.argtypes = (
)
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available.restype = ctypes.c_uint16
@@ -1607,6 +1624,9 @@ _UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_get_session_st
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey.argtypes = (
)
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey.restype = ctypes.c_uint16
+_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu.argtypes = (
+)
+_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu.restype = ctypes.c_uint16
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor.argtypes = (
)
_UniffiLib.uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor.restype = ctypes.c_uint16
@@ -4136,6 +4156,8 @@ class _UniffiFfiConverterTypeSdkClientKind(_UniffiConverterRustBuffer):
+
+
@@ -4174,6 +4196,8 @@ class CuaDriverProtocol(typing.Protocol):
raise NotImplementedError
async def hotkey(self, input: cua_driver._native_contract.HotkeyInput) -> ToolResult:
raise NotImplementedError
+ async def invoke_menu(self, input: cua_driver._native_contract.InvokeMenuInput) -> ToolResult:
+ raise NotImplementedError
def is_available(self, ) -> bool:
raise NotImplementedError
async def list_tools_json(self, ) -> str:
@@ -4798,6 +4822,23 @@ class CuaDriver(CuaDriverProtocol):
_uniffi_lift_return,
_uniffi_error_converter,
)
+ async def invoke_menu(self, input: cua_driver._native_contract.InvokeMenuInput) -> ToolResult:
+
+ cua_driver._native_contract._UniffiFfiConverterTypeInvokeMenuInput.check_lower(input)
+ _uniffi_lowered_args = (
+ self._uniffi_clone_handle(),
+ cua_driver._native_contract._UniffiFfiConverterTypeInvokeMenuInput.lower(input),
+ )
+ _uniffi_lift_return = _UniffiFfiConverterTypeToolResult.lift
+ _uniffi_error_converter = _UniffiFfiConverterTypeDriverError
+ return await _uniffi_rust_call_async(
+ _UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu(*_uniffi_lowered_args),
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_poll_rust_buffer,
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_complete_rust_buffer,
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_free_rust_buffer,
+ _uniffi_lift_return,
+ _uniffi_error_converter,
+ )
def is_available(self, ) -> bool:
_uniffi_lowered_args = (
self._uniffi_clone_handle(),
@@ -5124,6 +5165,8 @@ class CuaDriverSessionProtocol(typing.Protocol):
raise NotImplementedError
async def hotkey(self, input: cua_driver._native_contract.HotkeyInput) -> ToolResult:
raise NotImplementedError
+ async def invoke_menu(self, input: cua_driver._native_contract.InvokeMenuInput) -> ToolResult:
+ raise NotImplementedError
async def move_cursor(self, input: cua_driver._native_contract.MoveCursorInput) -> ToolResult:
raise NotImplementedError
async def press_key(self, input: cua_driver._native_contract.PressKeyInput) -> ToolResult:
@@ -5411,6 +5454,23 @@ class CuaDriverSession(CuaDriverSessionProtocol):
_uniffi_lift_return,
_uniffi_error_converter,
)
+ async def invoke_menu(self, input: cua_driver._native_contract.InvokeMenuInput) -> ToolResult:
+
+ cua_driver._native_contract._UniffiFfiConverterTypeInvokeMenuInput.check_lower(input)
+ _uniffi_lowered_args = (
+ self._uniffi_clone_handle(),
+ cua_driver._native_contract._UniffiFfiConverterTypeInvokeMenuInput.lower(input),
+ )
+ _uniffi_lift_return = _UniffiFfiConverterTypeToolResult.lift
+ _uniffi_error_converter = _UniffiFfiConverterTypeDriverError
+ return await _uniffi_rust_call_async(
+ _UniffiLib.uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu(*_uniffi_lowered_args),
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_poll_rust_buffer,
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_complete_rust_buffer,
+ _UniffiLib.ffi_cua_driver_sdk_rust_future_free_rust_buffer,
+ _uniffi_lift_return,
+ _uniffi_error_converter,
+ )
async def move_cursor(self, input: cua_driver._native_contract.MoveCursorInput) -> ToolResult:
cua_driver._native_contract._UniffiFfiConverterTypeMoveCursorInput.check_lower(input)
diff --git a/libs/cua-driver/python/src/cua_driver/_native_contract.py b/libs/cua-driver/python/src/cua_driver/_native_contract.py
index d9b92836fa..2a2c4c3d0f 100644
--- a/libs/cua-driver/python/src/cua_driver/_native_contract.py
+++ b/libs/cua-driver/python/src/cua_driver/_native_contract.py
@@ -3096,6 +3096,59 @@ class _UniffiFfiConverterTypeHotkeyInput(_UniffiConverterRustBuffer):
_UniffiFfiConverterTypeDesktopScope.write(value.scope, buf)
_UniffiFfiConverterOptionalString.write(value.session, buf)
+@dataclass
+class InvokeMenuInput:
+ """
+ Exact, immediate-child application menu path to resolve and invoke through
+ the operating system's accessibility API. Path labels are matched after
+ trimming surrounding whitespace and otherwise remain case-sensitive.
+"""
+ def __init__(self, *, pid:int, window_id:int, path:typing.List[str], session:typing.Optional[str]):
+ self.pid = pid
+ self.window_id = window_id
+ self.path = path
+ self.session = session
+
+
+
+
+ def __str__(self):
+ return "InvokeMenuInput(pid={}, window_id={}, path={}, session={})".format(self.pid, self.window_id, self.path, self.session)
+ def __eq__(self, other):
+ if self.pid != other.pid:
+ return False
+ if self.window_id != other.window_id:
+ return False
+ if self.path != other.path:
+ return False
+ if self.session != other.session:
+ return False
+ return True
+
+class _UniffiFfiConverterTypeInvokeMenuInput(_UniffiConverterRustBuffer):
+ @staticmethod
+ def read(buf):
+ return InvokeMenuInput(
+ pid=_UniffiFfiConverterUInt32.read(buf),
+ window_id=_UniffiFfiConverterUInt64.read(buf),
+ path=_UniffiFfiConverterSequenceString.read(buf),
+ session=_UniffiFfiConverterOptionalString.read(buf),
+ )
+
+ @staticmethod
+ def check_lower(value):
+ _UniffiFfiConverterUInt32.check_lower(value.pid)
+ _UniffiFfiConverterUInt64.check_lower(value.window_id)
+ _UniffiFfiConverterSequenceString.check_lower(value.path)
+ _UniffiFfiConverterOptionalString.check_lower(value.session)
+
+ @staticmethod
+ def write(value, buf):
+ _UniffiFfiConverterUInt32.write(value.pid, buf)
+ _UniffiFfiConverterUInt64.write(value.window_id, buf)
+ _UniffiFfiConverterSequenceString.write(value.path, buf)
+ _UniffiFfiConverterOptionalString.write(value.session, buf)
+
@dataclass
class MoveCursorInput:
def __init__(self, *, x:float, y:float, scope:DesktopScope, session:typing.Optional[str]):
@@ -4706,6 +4759,7 @@ __all__ = [
"GetScreenSizeInput",
"GetSessionStateInput",
"HotkeyInput",
+ "InvokeMenuInput",
"MoveCursorInput",
"PredicateOutcome",
"PressKeyInput",
diff --git a/libs/cua-driver/python/tests/test_uniffi_loader.py b/libs/cua-driver/python/tests/test_uniffi_loader.py
index 3addb1ba4b..7bb925be58 100644
--- a/libs/cua-driver/python/tests/test_uniffi_loader.py
+++ b/libs/cua-driver/python/tests/test_uniffi_loader.py
@@ -58,7 +58,7 @@ while True:
if request["method"] == "metadata":
result = {
"driver_version": "0.10.0",
- "contract_version": "0.5.0",
+ "contract_version": "0.6.0",
"tools_list_schema_version": "1",
"capability_version": "1",
"mcp_protocol_version": "2025-06-18",
@@ -142,7 +142,7 @@ except FileNotFoundError:
if request["method"] == "metadata":
result = {
"driver_version": "0.12.6",
- "contract_version": "0.5.0",
+ "contract_version": "0.6.0",
"tools_list_schema_version": "1",
"capability_version": "1",
"mcp_protocol_version": "2025-06-18",
diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
index 92d6a5594f..8a9c9d4808 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
@@ -23,7 +23,7 @@ AT-SPI is talked to natively over D-Bus (the `atspi`/zbus crate) — no
target window. No raise, no activate, no real-pointer warp. (It does
**not** use `XTestFakeButtonEvent`, which would route through the
focused window.)
-- **Element click** (`element_index`) — AT-SPI `do_action` on the
+- **Element click** (`element_token`, or `element_index` + `snapshot_id`) — AT-SPI `do_action` on the
accessible. Toolkit-native, focus-free.
- **type_text** — AT-SPI EditableText first (focus-free; lands in an
*unfocused* window's editable for Qt6 / GTK4). When a **non-editable**
@@ -103,7 +103,8 @@ is no `ax`/`vision`/`som` capture choice anymore; drop that vocabulary.
Modality is chosen at **action time**, by how you address the target:
-- **element ax action** — `element_index` / `element_token` → AT-SPI
+- **element ax action** — `element_token` (preferred), or the matching
+ `element_index` + `snapshot_id` pair → AT-SPI
`do_action`. Backgroundable, driver-verifiable.
- **element px action** — `x,y` → pixel rung, read straight off the screenshot
already in the `get_window_state` response. Best-effort; caller-confirmed.
@@ -119,7 +120,7 @@ experimental per-surface routes.
The capture/dispatch/addressing params are a shared cross-platform
contract (see `SKILL.md` → *Cross-platform parameter contract*) — the
same `session`, `delivery_mode`, `capture_mode`, `scope`, `modifier`,
-`element_index`/`element_token` *shapes* as macOS and Windows, gated in
+`element_index`/`snapshot_id`/`element_token` *shapes* as macOS and Windows, gated in
CI so the three surfaces can't drift. Linux-relevant notes:
- **`session` is now accepted on every action/cursor tool.** Earlier
@@ -132,6 +133,16 @@ CI so the three surfaces can't drift. Linux-relevant notes:
an explicitly escalated `auto` session). Strict window sessions receive
`desktop_scope_disabled`; no persistent config is read or written.
+## Native application menus
+
+Use `invoke_menu({pid, window_id, path:[...]})` for a known GTK/Qt application
+menu command. It activates the exact target only for the duration of the
+operation, resolves each labelled AT-SPI menu descendant again after the prior
+menu expands, and refuses missing, duplicate, disabled, or non-actionable
+segments. It works through AT-SPI on both X11 and Wayland and never falls back
+to coordinates. Verify the command's semantic effect from fresh state; the
+native `do_action` acknowledgement alone is not task completion.
+
## AT-SPI needs the session bus (headless / containers / `runuser`)
AT-SPI — the accessibility tree behind `get_window_state`, element-indexed
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index f9fb9cc1bb..b72279cd73 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -78,19 +78,14 @@ Reading frontmost state is fine (`osascript -e 'tell application
"System Events" to get name of first application process whose
frontmost is true'`). Mutating it is not.
-**Corollary — the AXMenuBar rule.** `AXMenuBarItem` + AXPick
-dispatches at the AX layer regardless of which app is frontmost,
-but macOS's on-screen menu bar always belongs to the frontmost
-app. If you drive a _backgrounded_ app's menu bar, the AX call
-succeeds but the viewer sees the dispatch rendered over the
-_frontmost_ app's menu bar — confusing in any observed session and
-routinely a silent no-op too, because action menu items go
-`DISABLED` when their owning app isn't the key window. **So: only
-use menu-bar navigation when the target is already frontmost.** For
-backgrounded targets, read state via in-window AX (window title,
-toolbar `AXStaticText`) and dispatch via in-window `element_index`
-or pixel clicks — both paths are frontmost-insensitive. Full
-rationale in "Navigating native menu bars" below.
+**Corollary — the AXMenuBar rule.** Do not manually drive a background
+application's `AXMenuBarItem`: the visible macOS menu bar belongs to the
+frontmost app, and command items may be disabled otherwise. Use `invoke_menu`
+for a known application-menu path. It owns the necessary temporary activation,
+resolves every AX level live, and restores the prior app on a best-effort basis. Prefer an in-window
+element action when the same command has an ordinary control, and prefer
+`set_window_frame` for exact geometry. Full rationale is in “Navigating native
+menu bars” below.
**"Open \<app\>" in user speech means launch, not activate.**
`cua-driver launch_app` is the one correct path for process
@@ -141,12 +136,10 @@ When a cua-driver call surprises you, diagnose cua-driver first:
race against a close, or the window has no backing store yet).
Re-snapshot; if persistent, pick a different `window_id` via
`list_windows`.
-- **`Invalid element_index` / `No cached AX state`?** You either
- skipped `get_window_state` this turn or passed a different
- `window_id` than the one the snapshot cached against. The cache
- is keyed on `(pid, window_id)` — indices don't carry across
- windows of the same app. Re-snapshot with the same window_id
- you're about to click in.
+- **`snapshot_id_required` / `stale_element_token` / no cached AX state?**
+ Re-snapshot the exact window and use the new `element_token`, or pair its
+ `snapshot_id` with the matching `element_index`. A new snapshot of that
+ window invalidates older targets immediately.
- **Sparse Chromium AX tree?** Retry `get_window_state` once — the
tree populates on second call.
@@ -416,95 +409,31 @@ move is the px form of `type_text` — focus and type in one call.
## Navigating native menu bars (AXMenuBar)
-**Only drive the menu bar when the target app is frontmost.** This
-is the single most-misused cua-driver capability. If the target is
-backgrounded, don't reach for `AXMenuBarItem` + AXPick — use
-in-window `element_index` or pixel clicks instead. Two reasons, one
-functional and one perceptual:
-
-- **Functional:** menu items that touch document/playback/editor
- state go `DISABLED` when their owning app isn't the key window
- (Preview rotate, IINA speed change, most editor commands).
- AXPick + AXPress will dispatch successfully from the driver's side but
- no-op at the target — you get a silent false-pass.
-- **Perceptual (matters for demos, screen recordings, and anything
- the user watches live):** macOS's screen-rendered menu bar
- always belongs to the _frontmost_ app. AXPick on a backgrounded
- app's `AXMenuBarItem` dispatches to that app's per-process menu at
- the AX layer, but any visible menu render happens over the
- frontmost app's menu bar — the viewer sees an IINA submenu
- flashing on top of Chrome's menus, which reads as "the agent
- clicked the wrong app." The AX call was correct; the frame the
- user sees is not. For recorded or observed sessions, this is an
- integrity bug even though it's not a correctness bug.
-
-**Good decision rule:** if the target is not already frontmost, do
-not use `AXMenuBarItem` at all. For _reading_ in-window state,
-snapshot the window AX tree — most apps expose the same state via
-an in-window `AXStaticText`, title bar, or toolbar. For _dispatching_
-actions, use in-window `element_index` (buttons, toolbar items) or
-pixel clicks on in-window controls — both dispatch via AppKit's
-window-under-pointer hit-test and are **not** frontmost-gated.
-
-When the target IS frontmost, the menu-bar flow below is fine and
-the canonical path for menus.
-
-### The two-snapshot pattern (target frontmost only)
-
-Menu contents are a two-snapshot flow. Closed AXMenu subtrees are
-deliberately skipped during snapshot — otherwise every app's File /
-Edit / View hierarchy plus every Recent Items macOS has ever seen
-would inflate the tree 10-100x. But once a menu is _open_, its
-AXMenuItem children do receive `element_index` values so you can
-click them normally.
-
-1. Find the `[N] AXMenuBarItem "<Menu Name>"` in the tree.
-2. `click({pid, element_index: N, action: "pick"})` — menu bar items
- implement `AXPick` ("open my submenu"), not `AXPress`. Using the
- default action on an AXMenuBarItem is a no-op.
-3. Re-snapshot. The expanded menu's items now appear under the bar
- item as `[M] AXMenuItem "<Item Name>"`.
-4. Click the target item — most items respond to `AXPress` (default
- action). Submenus nest under the item and are walked the same way.
-5. Re-snapshot and verify.
-
-If you ever need to back out without selecting, `press_key({pid, key:
-"escape"})` closes the open menu. Leaving a menu expanded between
-turns poisons subsequent snapshots for that pid.
-
-### Commands gated on the target being frontmost
-
-Some menu items and global shortcuts (Preview's Tools → Rotate
-Right, ⌘R; anything in the View menu that manipulates the current
-document; most editor commands) are **disabled unless the target
-app is the key / frontmost window**. You'll see it in the AX tree
-as `DISABLED` on the menu item even though the user's intent is
-obviously valid.
-
-Before activating, confirm you're in this narrow case — the menu
-item still reads `DISABLED` after a fresh snapshot AND the action
-the user requested genuinely requires frontmost (Preview rotate,
-View menu document manipulation, editor commands). If either
-check fails, don't activate.
-
-When both checks pass, the driver has no `activate` tool
-(deliberately — the whole point is backgroundable control), so
-this is the one legitimate `osascript` fallback:
+Use `invoke_menu({pid, window_id, path:[...]})` when the desired command has a
+known native application-menu path. The tool temporarily activates the exact
+target window because the on-screen macOS menu bar belongs to the frontmost
+application, resolves each immediate child from live AX state, uses only
+`AXPress`/`AXPick`-class actions, and restores the prior application afterward
+on a best-effort basis.
+It refuses missing, duplicate, disabled, or non-actionable segments and never
+falls back to pixels.
```bash
-osascript -e 'tell application "<App Name>" to activate'
+cua-driver invoke_menu \
+ '{"pid":844,"window_id":10725,"path":["Window","Move & Resize","Left"]}'
```
-Then re-snapshot — the menu item loses its `DISABLED` tag — and
-`click({action: "pick"})` the item. Alternatively, a `hotkey`
-call delivered to the now-frontmost app works for the shortcut
-form (`⌘R`, `⌘+`, etc.).
-
-**Always name the focus steal in your response** so the user isn't
-surprised — "Briefly activating Preview to enable Tools → Rotate
-Right" or similar. Don't silently steal focus. You don't need to
-restore the previous frontmost afterwards unless the user asks —
-they can cmd-tab back.
+The native action acknowledgement does not prove the application's semantic
+postcondition, so re-snapshot or use `list_windows`/`verify_state` afterward.
+For geometry commands, prefer `set_window_frame` because its independent frame
+readback is stronger and avoids menu state entirely.
+
+Do not manually reproduce the old open → snapshot → reuse-index sequence.
+Menu expansion mutates the accessibility tree, and closed AppKit submenus may
+remain present in `AXChildren`; disabled descendants are now non-addressable,
+but their presence is not a visibility guarantee. `invoke_menu` re-resolves the
+live path after every expansion, so it never carries a snapshot index across
+that mutation.
## Browsers on macOS
diff --git a/libs/cua-driver/rust/Skills/cua-driver/RECORDING.md b/libs/cua-driver/rust/Skills/cua-driver/RECORDING.md
index 9fcd1362a4..e141f7460c 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/RECORDING.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/RECORDING.md
@@ -78,15 +78,18 @@ Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
- `app_state.json` and `screenshot.png` — compatibility aliases for
`after_state.json` and `after.png`.
- `action.json` — the tool name, full input arguments, result
- summary, pid, click point (when applicable), ISO-8601 timestamp.
+ summary, result-error flag, pid, click point (when applicable), ISO-8601
+ timestamp.
- `click.png` — for click-family actions (`click`, `double_click`,
`right_click`): a copy of `before.png` with a red marker drawn at
the click point. **Both addressing modes are covered:** explicit
`x, y` clicks use the supplied coordinates directly, and
`element_index`-addressed clicks resolve to the element's center
via the live AX/UIA cache, then convert to window-local screenshot
- pixels. Absent for non-click tools and classified as unavailable when
- a click point was expected but could not be resolved or rendered.
+ pixels. Absent for non-click tools. It is also absent, and explicitly
+ classified as not applicable, when the driver refuses a click before target
+ resolution; no input was aimed in that case. A dispatched click whose marker
+ cannot be resolved or rendered remains an evidence failure.
## When to use it
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 828aa35375..b3a392bd91 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -1,6 +1,6 @@
---
name: cua-driver
-description: Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, click/type/scroll by element_index or pixel coordinates, and verify via re-snapshot without bringing the target to the foreground. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.
+description: Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.
version: 0.16.0 # x-release-please-version
metadata:
openclaw:
@@ -78,8 +78,9 @@ before stopping or advancing:
conversion, and process inspection. Read the resulting semantic state back;
a zero exit status alone is not proof.
1. **Typed Cua operation for an application or window outcome.** Use
- `set_window_frame` for exact geometry, typed browser tools for supported page
- content, and clipboard tools for clipboard state. Verify with
+ `set_window_frame` for exact geometry, `invoke_menu` for a known native
+ application-menu path, typed browser tools for supported page content, and
+ clipboard tools for clipboard state. Verify with
`list_windows`, `get_browser_state`, or `clipboard_read`, respectively.
2. **Background accessibility action.** Use a fresh AX/UIA/AT-SPI target.
3. **Background pixel action.** Use the pixels from the same state snapshot.
@@ -178,7 +179,8 @@ cua-driver serve
cua-driver launch_app '{"bundle_id":"..."}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
cua-driver get_window_state '{"pid":844,"window_id":10725}'
-cua-driver click '{"pid":844,"window_id":10725,"element_index":14}'
+# Use the returned structuredContent.elements[].element_token:
+cua-driver click '{"pid":844,"element_token":"s0000002a:14"}'
cua-driver verify_state '{"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}'
cua-driver stop
```
@@ -396,7 +398,8 @@ capture, no mode flip.
You don't pick a capture mode; you pick **how you address the target**
on the action call, and that one choice selects the rung:
-- **element ax action** — pass `element_index` / `element_token`.
+- **element ax action** — pass `element_token` (preferred), or the exact
+ `element_index` + `snapshot_id` pair from the same response.
Dispatches through the **accessibility rung**: AXPress (macOS) / UIA
Invoke (Windows) / AT-SPI `doAction` (Linux). Backgroundable,
z-order-independent, and the only **driver-verifiable** rung.
@@ -410,7 +413,7 @@ for the _dispatch_ path — it conflated perception with dispatch.
Perception is always both; dispatch is `ax` or `px`.
**The keyboard family has both forms too.** `type_text`, `press_key`,
-and `hotkey` take `element_index` (ax) **or** `x,y` (px) — mutually
+and `hotkey` take a snapshot-bound element target (ax) **or** `x,y` (px) — mutually
exclusive, same as the pointer tools. The px form **pixel-clicks at
`(x,y)` to establish real renderer focus, then delivers the
keystroke(s)** to the now-focused element (it reuses `click`'s
@@ -420,7 +423,7 @@ for Chromium/Electron inputs the AX path can't reach, and
`hotkey({pid, x, y, keys:["cmd","v"]})` to paste into a specific field.
**Typing default (the ladder).** Call `type_text` directly with
-`element_index` (ax) — it targets the field, no pre-click. On
+`element_token` (ax) — it targets the field, no pre-click. On
Electron/Catalyst the AX layer echoes the write without rendering it,
so the driver returns `effect:"unverifiable"` with
`escalation.target:"pixel"` there (never a false `effect:"confirmed"`) —
@@ -474,12 +477,13 @@ _dispatch rung_ on a real signal. Walk the rungs:
# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first
# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.
# For exact window geometry: set_window_frame(...), then list_windows(...) readback.
+# For a known native menu command: invoke_menu(pid, window_id, path), then verify its effect.
# For supported page content: get_browser_state(...), typed browser action, refresh refs.
# Continue below only when the postcondition actually requires native UI interaction.
# Route 2 — element AX/UIA/AT-SPI action, backgrounded
get_window_state(pid, window_id) # tree + screenshot, both, always
-resp = click(pid, window_id, element_index) # or type_text / set_value / press_key
+resp = click(pid, element_token) # or type_text / set_value / press_key
check = verify_state( # bounded structured read-back
pid, window_id,
expect=[...],
@@ -614,21 +618,21 @@ app record doesn't carry window state on purpose. In the common
single-window case you can skip `list_windows` entirely and read the
`windows` array that `launch_app` already returned.
-### Snapshot and act by element_index
+### Snapshot and act with a snapshot-bound target
Call `get_window_state({pid, window_id})` with the `window_id` from
`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if
you're interacting with a long-lived process). It returns **the tree
and the screenshot together** by default, so you can both dispatch by
-`element_index` and ground on pixels from one call — no config change,
+`element_token` and ground on pixels from one call — no config change,
no mode flip. When you're just re-indexing before an element ax action
and don't need fresh pixels, pass `include_screenshot:false` to skip
the grab (a perf knob, not a modality choice).
The response carries:
-- `tree_markdown` — every actionable element tagged `[N]`. That `N`
- is the `element_index`. The tree can be very large (Finder is
+- `tree_markdown` — every actionable element tagged `[N]`; the structured row
+ with the same `element_index` carries its opaque `element_token`. The tree can be very large (Finder is
~1600 elements, ~190 KB); when it exceeds token limits the MCP
harness saves it to a file and returns the path. Use `Bash` +
`jq -r '.tree_markdown'` + `grep` to pull the section you need.
@@ -665,7 +669,7 @@ they come from the _same_ call.** Each half carries signal the other
can't, which is exactly why you cross-check them:
- The **tree** tells you _what's clickable_ — roles, labels,
- `element_index` handles, advertised actions, parent-child
+ snapshot-bound element handles, advertised actions, parent-child
structure. This is the ground truth for an **element ax action**.
- The **screenshot** tells you _which one_ — the tree often has many
buttons with similar or empty labels ("Delete", "OK", anonymous
@@ -676,7 +680,7 @@ can't, which is exactly why you cross-check them:
catch the tree _lying_ (an `h:1`/off-viewport row, a Catalyst null
value).
-Default to dispatching by `element_index` (the **element ax action**) —
+Default to dispatching by `element_token` (the **element ax action**) —
it's the verifiable, backgroundable rung. Do an **element px action**
(`x,y` off the same screenshot) when the tree can't disambiguate
(repeated/empty labels), when it's empty (`degraded` — non-AX
@@ -691,30 +695,31 @@ video / WebGL / custom-drawn surface that isn't in the tree
The `actions=[...]` list on each element is **advisory**, not
authoritative. cua-driver does not pre-flight check against it —
-`click({pid, element_index})` always attempts the default action (or
+`click({pid, element_token})` always attempts the default action (or
the action you pass) and surfaces whatever the target returns. **Try
the click first** — pivot only on the returned error code.
### Tool dispatch table
-Every row assumes a `(pid, window_id)` pair from the last
-`get_window_state`; `window_id` is required alongside `element_index`,
-ignored on pixel-only forms unless you want to anchor the conversion
-against a specific window.
+Every row assumes a fresh `get_window_state`. Prefer its opaque
+`element_token`. If a client uses the visible integer instead, it must send
+the response's `snapshot_id` with `element_index`; bare indices fail closed in
+0.17. Pixel-only forms remain independent of snapshot handles.
| Intent | Tool | Notes |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| List an app's windows | `list_windows({pid})` | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids |
| Set an exact window frame | `set_window_frame({pid, window_id, x, y, width, height})` | uses the platform window manager and returns `confirmed` only after geometry readback; inspect `list_windows` again before continuing when the result is not confirmed |
+| Invoke a native application menu | `invoke_menu({pid, window_id, path:["Window","Arrange","Left"]})` | resolves exact immediate-child labels from live native state at every hop; refuses missing, ambiguous, or disabled segments and never falls back to pixels; verify the command's semantic effect afterward |
| Snapshot a window | `get_window_state({pid, window_id})` | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache |
| Verify a postcondition | `verify_state({pid, window_id, expect, include_screenshot?})` | polls bounded structured predicates; returns `satisfied`, `unsatisfied`, or `unknown`. Optional final image is interpreted by the agent harness, never by the driver |
-| Left click | `click({pid, window_id, element_index})` | default `action: "press"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: ["cmd"\|"ctrl"]` |
-| Double-click / open | `double_click({pid, window_id, element_index})` | Default action when the element advertises one (Open on Finder items / openable rows), else stamped pixel double-click at the element's center |
-| Right click / context menu | `right_click({pid, window_id, element_index})` or `click({pid, window_id, element_index, action: "show_menu"})` | Browser page content should use the typed route where available; see `BROWSER.md` |
-| Type at cursor | `type_text({pid, text, window_id, element_index})` (ax) or `type_text({pid, text, window_id, x, y})` (px) | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium/Electron inputs the AX path can't reach |
-| Set whole non-text control value | `set_value({pid, window_id, element_index, value})` | **AX-only by design** — dropdown/`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`/`drag` |
-| Scroll | `scroll({pid, direction, amount, by, window_id, element_index})` | synthesizes per-pid PageUp/PageDown/arrows |
-| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` (ax) or `press_key({pid, key, x, y})` (px) | ax `element_index` sets focus then posts the key; **px** pixel-clicks `(x,y)` to focus, then sends the key |
+| Left click | `click({pid, element_token})` or `click({pid, window_id, element_index, snapshot_id})` | default `action: "press"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: ["cmd"\|"ctrl"]` |
+| Double-click / open | `double_click({pid, element_token})` | Default action when the element advertises one (Open on Finder items / openable rows), else stamped pixel double-click at the element's center |
+| Right click / context menu | `right_click({pid, element_token})` or `click({pid, element_token, action:"show_menu"})` | Browser page content should use the typed route where available; see `BROWSER.md` |
+| Type at cursor | `type_text({pid, text, element_token})` (ax) or `type_text({pid, text, window_id, x, y})` (px) | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium/Electron inputs the AX path can't reach |
+| Set whole non-text control value | `set_value({pid, element_token, value})` | **AX-only by design** — dropdown/`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`/`drag` |
+| Scroll | `scroll({pid, direction, amount, by, element_token})` | synthesizes per-pid PageUp/PageDown/arrows |
+| Focus + send key | `press_key({pid, key, element_token, modifiers})` (ax) or `press_key({pid, key, x, y})` (px) | ax targets the element before posting the key; **px** pixel-clicks `(x,y)` to focus, then sends the key |
| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus |
| Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `["cmd","v"]` to paste into it |
@@ -735,9 +740,10 @@ target's per-pid event-post path. Only a strict/effective desktop session may
omit `pid`, and it intentionally routes keyboard input to the current
foreground application.
-**Why `element_index` is the primary path:** works on hidden /
-occluded / off-desktop windows, no focus steal, stable across
-rebuilds, labels tell you what you're clicking. Reach for pixel
+**Why the snapshot-bound element target is the primary path:** works on hidden /
+occluded / off-desktop windows, avoids focus steal, and fails closed after a
+tree rebuild instead of silently retargeting a reused index. Labels tell you
+what you're clicking. Reach for pixel
coordinates only when the accessibility tree can't.
## Cross-platform parameter contract
@@ -745,7 +751,7 @@ coordinates only when the accessibility tree can't.
The capture, dispatch, and addressing params — `session`,
`delivery_mode`, `capture_mode` (deprecated/ignored — see the behavior
matrix; still in the schema only so old callers don't error), `scope`,
-`modifier`, `button`, `element_index`, `element_token` — are a **shared
+`modifier`, `button`, `element_index`, `snapshot_id`, `element_token` — are a **shared
schema contract**: identical _shape_ (`type`/`enum`/`items`) on macOS,
Windows, and Linux.
They compose from canonical fragments in
@@ -929,7 +935,7 @@ doesn't-survive-across-sessions caveat.
| Error text | Meaning | Fix |
| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `No cached AX state for pid X window_id W` | You either skipped `get_window_state` this turn, or passed a different `window_id` to the click than the one the snapshot cached against | Call `get_window_state({pid: X, window_id: W})` first — the same window_id you intend to click in |
-| `Invalid element_index N for pid X window_id W` | Index is stale or out of range | Re-run `get_window_state` with the same window_id, pick a fresh index from the new tree |
+| `snapshot_id_required` / `stale_element_token` | A bare index was supplied, or a newer snapshot superseded this target | Re-run `get_window_state`; use the new `element_token`, or send its `snapshot_id` with the matching integer |
| `window_id W belongs to pid P, not …` | Passed a window_id that's owned by a different process | Use `list_windows({pid: X})` to enumerate this pid's own windows |
| `ambiguous_window_target` | A PID-only window action matched multiple eligible top-level windows | Use the returned candidates or `list_windows({pid: X})`, select the intended sibling, and retry with its explicit `window_id` |
| `AX action … failed with code …` / `UIA invoke failed` | Element doesn't support the default action | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center |
@@ -941,7 +947,9 @@ respective companion files.
## Things to avoid
-- **Never** reuse an `element_index` across a re-snapshot of the same pid.
+- **Never** reuse an element target across a re-snapshot of the same window.
+ A new snapshot invalidates older tokens immediately. Bare `element_index`
+ input is rejected; use `element_token` or `element_index` + `snapshot_id`.
- **Don't conflate the two addressing modes.** The tree gives you
`element_index` handles; the screenshot (same call) gives you the
pixel frame. An **element ax action** addresses by index, an
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index 3c9e80b1d6..64f8157e3b 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -534,10 +534,11 @@ Canonical multi-step workflow:
# Snapshot the UIA tree.
'{"pid":6004,"window_id":459672}' | & cua-driver call get_window_state
-# Returns: tree_markdown with [N] element indices, screenshot, dimensions.
+# Returns: tree_markdown with [N] indices plus structured element_token values,
+# snapshot_id, screenshot, and dimensions.
-# Click element [22] (the "Equals" button per the tree).
-'{"pid":6004,"window_id":459672,"element_index":22}' | & cua-driver call click
+# Click the "Equals" row with its opaque token from that response.
+'{"pid":6004,"element_token":"s0000002a:22"}' | & cua-driver call click
# → "✅ Performed UIA Invoke on [22] ..."
# Re-snapshot to verify the action landed.
@@ -548,13 +549,14 @@ Canonical multi-step workflow:
**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:
-- **Before** — the pre-action snapshot resolves the `element_index`
- you're about to use. Indices from previous turns are stale; the
+- **Before** — the pre-action snapshot resolves the `element_token`
+ you're about to use. A bare integer is rejected in 0.17; clients that keep
+ integers must send the same response's `snapshot_id`. Targets from previous turns are stale; the
server replaces the element index map on every snapshot, keyed
on `(pid, window_id)`. Indices from turn N don't resolve in turn
- N+1, and indices from window A don't resolve against window B of
+ N+1, and targets from window A don't resolve against window B of
the same app. Skip this and element-indexed actions fail with
- `Invalid element_index`.
+ `stale_element_token` or `snapshot_id_required`.
- **After** — the post-action snapshot verifies the action actually
landed. Without it you can't tell a silent no-op from a real
effect. The UIA tree change (new value, new window, disappeared
@@ -569,13 +571,13 @@ Canonical multi-step workflow:
Two click addressing modes, both gated by `pid`:
-### `element_index` mode (preferred)
+### Snapshot-bound element mode (preferred)
```json
-{ "pid": 6004, "window_id": 459672, "element_index": 22 }
+{ "pid": 6004, "element_token": "s0000002a:22" }
```
-Looks up the cached UIA element from the last `get_window_state`,
+Looks up the exact cached UIA element from the named snapshot,
fires `IUIAutomationInvokePattern::Invoke()` on it directly.
Properties:
@@ -599,8 +601,17 @@ Properties:
`accessibility` means UIA/MSAA and `synthetic_events` means the
targeted event fallback. Do not parse the human-readable text.
-This is the right path for **any** "click button N" / "click menu
-item X" / "click checkbox Y" intent.
+This is the right path for **any** "click button N" or "click checkbox Y"
+intent. For a known application-menu hierarchy, prefer `invoke_menu`:
+
+```json
+{ "pid": 6004, "window_id": 459672, "path": ["Window", "Arrange", "Left"] }
+```
+
+It uses `ExpandCollapsePattern` at intermediate hops and `InvokePattern` or
+`SelectionItemPattern` at the leaf, resolving the live UIA hierarchy again
+after every expansion. It refuses ambiguous, missing, or disabled segments and
+never falls back to pixels. Verify the command's semantic effect afterward.
### `(x, y)` mode (element px action / pixel)
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
index de9e98de9d..87bc75ea75 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
@@ -234,8 +234,8 @@ pub fn classify_cursor_semantics(name: &str, args: &Value) -> Option<CursorSeman
CursorAction::Navigate
}
- "launch_app" | "activate_app" | "bring_to_front" | "set_window_frame" | "list_apps"
- | "list_windows" | "kill_app" => CursorAction::App,
+ "launch_app" | "activate_app" | "bring_to_front" | "set_window_frame" | "invoke_menu"
+ | "list_apps" | "list_windows" | "kill_app" => CursorAction::App,
"upload_file" | "download_file" | "copy_file" | "move_file" => CursorAction::Transfer,
@@ -257,7 +257,7 @@ pub fn classify_cursor_semantics(name: &str, args: &Value) -> Option<CursorSeman
let target = if name.starts_with("browser_") {
Some(CursorTarget::Browser)
- } else if name == "set_window_frame" {
+ } else if matches!(name, "set_window_frame" | "invoke_menu") {
Some(CursorTarget::Desktop)
} else if args
.get("element_index")
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/desktop.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/desktop.rs
index 3a7596bae3..93a3406591 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/desktop.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/desktop.rs
@@ -11,8 +11,9 @@ use crate::{
ActionResult, ClickInput, ClipboardReadInput, ClipboardReadOutput, ClipboardWriteInput,
ClipboardWriteOutput, CursorAction, CursorPositionOutput, CursorSemantics, DesktopStateOutput,
DragInput, GetCursorPositionInput, GetDesktopStateInput, GetScreenSizeInput, HotkeyInput,
- MoveCursorInput, Platform, PressKeyInput, SchemaMode, ScreenSizeOutput, ScrollInput,
- SetWindowFrameInput, ToolAnnotations, ToolContract, ToolInput, ToolOutput, TypeTextInput,
+ InvokeMenuInput, MoveCursorInput, Platform, PressKeyInput, SchemaMode, ScreenSizeOutput,
+ ScrollInput, SetWindowFrameInput, ToolAnnotations, ToolContract, ToolInput, ToolOutput,
+ TypeTextInput,
};
const ALL_PLATFORMS: [Platform; 3] = [Platform::Macos, Platform::Windows, Platform::Linux];
@@ -24,6 +25,7 @@ pub fn contracts() -> Vec<ToolContract> {
get_cursor_position(),
move_cursor(),
set_window_frame(),
+ invoke_menu(),
click(),
drag(),
scroll(),
@@ -215,6 +217,21 @@ fn set_window_frame() -> ToolContract {
)
}
+fn invoke_menu() -> ToolContract {
+ contract::<InvokeMenuInput, ActionResult>(
+ "invoke_menu",
+ "Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.",
+ &["menu.path.invoke", "accessibility.menu.native"],
+ ToolAnnotations {
+ read_only: false,
+ destructive: true,
+ idempotent: false,
+ open_world: true,
+ },
+ CursorAction::App,
+ )
+}
+
fn click() -> ToolContract {
contract::<ClickInput, ActionResult>(
"click",
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
index 42e349d8c3..3d0c8bf810 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
@@ -62,6 +62,15 @@ fn string_list_schema(generator: &mut SchemaGenerator) -> Schema {
Vec::<String>::json_schema(generator)
}
+fn menu_path_schema(_: &mut SchemaGenerator) -> Schema {
+ json_schema!({
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 16,
+ "items": { "type": "string", "minLength": 1, "maxLength": 200 }
+ })
+}
+
fn number_schema(_: &mut SchemaGenerator) -> Schema {
json_schema!({ "type": "number" })
}
@@ -440,6 +449,28 @@ impl ToolInput for SetWindowFrameInput {
const TOOL_NAME: &'static str = "set_window_frame";
}
+/// Exact, immediate-child application menu path to resolve and invoke through
+/// the operating system's accessibility API. Path labels are matched after
+/// trimming surrounding whitespace and otherwise remain case-sensitive.
+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq, uniffi::Record)]
+#[serde(deny_unknown_fields)]
+pub struct InvokeMenuInput {
+ #[schemars(schema_with = "positive_integer_schema")]
+ pub pid: u32,
+ #[schemars(schema_with = "positive_integer_schema")]
+ pub window_id: u64,
+ #[schemars(schema_with = "menu_path_schema")]
+ pub path: Vec<String>,
+ /// Optional session id.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schemars(schema_with = "string_schema")]
+ pub session: Option<String>,
+}
+
+impl ToolInput for InvokeMenuInput {
+ const TOOL_NAME: &'static str = "invoke_menu";
+}
+
impl ToolInput for MoveCursorInput {
const TOOL_NAME: &'static str = "move_cursor";
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs
index dbf3c3c560..2e723bb95f 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs
@@ -30,8 +30,8 @@ pub use inputs::{
CaptureScope, ClickButton, ClickInput, ClipboardReadInput, ClipboardWriteInput, DesktopScope,
DragInput, EndSessionInput, EscalateSessionInput, EscalationReason, GetAgentCursorStateInput,
GetCursorPositionInput, GetDesktopStateInput, GetScreenSizeInput, GetSessionStateInput,
- HotkeyInput, MoveCursorInput, PressKeyInput, ScrollBy, ScrollDirection, ScrollInput,
- SetAgentCursorEnabledInput, SetAgentCursorMotionInput, SetAgentCursorThemeInput,
+ HotkeyInput, InvokeMenuInput, MoveCursorInput, PressKeyInput, ScrollBy, ScrollDirection,
+ ScrollInput, SetAgentCursorEnabledInput, SetAgentCursorMotionInput, SetAgentCursorThemeInput,
SetWindowFrameInput, StartSessionInput, ToolInput, TypeTextInput,
};
pub use outputs::{
@@ -56,7 +56,7 @@ pub const TOOLS_LIST_SCHEMA_VERSION: &str = "1";
pub const CAPABILITY_VERSION: &str = "1";
/// Shape version for the checked-in generated client contract.
-pub const CONTRACT_VERSION: &str = "0.5.0";
+pub const CONTRACT_VERSION: &str = "0.6.0";
/// MCP protocol version used by current cua-driver clients.
pub const MCP_PROTOCOL_VERSION: &str = "2025-06-18";
@@ -82,6 +82,7 @@ pub const ACTION_RESULT_TOOLS: &[&str] = &[
"hotkey",
"set_value",
"set_window_frame",
+ "invoke_menu",
"browser_click",
"browser_pointer",
"browser_type",
@@ -292,7 +293,7 @@ mod tests {
let mut sorted = names.clone();
sorted.sort_unstable();
assert_eq!(names, sorted);
- assert_eq!(manifest.contract_version, "0.5.0");
+ assert_eq!(manifest.contract_version, "0.6.0");
assert!(manifest.experimental);
}
@@ -362,6 +363,7 @@ mod tests {
"get_desktop_state",
"get_screen_size",
"hotkey",
+ "invoke_menu",
"move_cursor",
"press_key",
"scroll",
@@ -405,6 +407,27 @@ mod tests {
);
}
+ #[test]
+ fn invoke_menu_requires_an_exact_nonempty_bounded_path() {
+ let contract = tool_contract("invoke_menu").expect("invoke_menu contract");
+ assert_eq!(
+ contract.input_schema["required"],
+ serde_json::json!(["pid", "window_id", "path"])
+ );
+ let path = &contract.input_schema["properties"]["path"];
+ assert_eq!(path["minItems"], serde_json::json!(1));
+ assert_eq!(path["maxItems"], serde_json::json!(16));
+ assert_eq!(path["items"]["minLength"], serde_json::json!(1));
+ assert_eq!(
+ contract.cursor_semantics.expect("cursor semantics").action,
+ CursorAction::App
+ );
+ assert_eq!(
+ contract.capabilities,
+ vec!["menu.path.invoke", "accessibility.menu.native"]
+ );
+ }
+
#[test]
fn list_windows_defines_nullable_higher_is_frontmost_z_index() {
assert!(tool_contract("list_windows").is_none());
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/authorization.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/authorization.rs
index fbb5efe41c..ef5437ac6f 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/authorization.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/authorization.rs
@@ -845,6 +845,7 @@ pub fn advertised_risk_for(tool: &str) -> RiskAssessment {
| "press_key"
| "hotkey"
| "set_value"
+ | "invoke_menu"
| "launch_app"
| "bring_to_front"
| "set_window_frame"
@@ -1503,6 +1504,14 @@ mod tests {
assert!(!risk.operation_sensitive);
}
+ #[test]
+ fn native_menu_invocation_matches_local_gui_action_risk() {
+ let risk = advertised_risk_for("invoke_menu");
+ assert_eq!(risk.class, RiskClass::R1);
+ assert_eq!(risk.enforcement, RiskEnforcement::MetadataOnly);
+ assert!(!risk.operation_sensitive);
+ }
+
#[test]
fn screenshot_file_output_is_classified_as_egress() {
for tool in ["get_desktop_state", "get_window_state"] {
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/element_query.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/element_query.rs
new file mode 100644
index 0000000000..4008ee6d64
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/element_query.rs
@@ -0,0 +1,63 @@
+//! Shared projection for query-filtered accessibility snapshots.
+//!
+//! Platform walkers keep their complete native cache so original handles stay
+//! resolvable. This module only projects the protocol response to the indexed
+//! rows rendered by the filtered Markdown tree.
+
+use serde_json::Value;
+use std::collections::HashSet;
+
+pub fn project_elements_for_query(
+ mut elements: Vec<Value>,
+ query: Option<&str>,
+ filtered_markdown: &str,
+) -> Vec<Value> {
+ if query.is_none() {
+ return elements;
+ }
+
+ let visible_indices: HashSet<u64> = filtered_markdown
+ .lines()
+ .filter_map(|line| {
+ let rendered = line.trim_start().strip_prefix("- [")?;
+ let (index, _) = rendered.split_once(']')?;
+ index.parse().ok()
+ })
+ .collect();
+ elements.retain(|entry| {
+ entry
+ .get("element_index")
+ .and_then(Value::as_u64)
+ .is_some_and(|index| visible_indices.contains(&index))
+ });
+ elements
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use serde_json::json;
+
+ #[test]
+ fn projection_preserves_original_indices_for_matches_and_ancestors() {
+ let elements = (0..5)
+ .map(|element_index| json!({ "element_index": element_index }))
+ .collect();
+ let markdown = "- [0] Window\n - [1] Window\n - [3] Left\n";
+ let projected = project_elements_for_query(elements, Some("Left"), markdown);
+ let indices: Vec<_> = projected
+ .iter()
+ .map(|entry| entry["element_index"].as_u64().unwrap())
+ .collect();
+ assert_eq!(indices, vec![0, 1, 3]);
+ }
+
+ #[test]
+ fn absent_query_returns_every_element() {
+ let elements = vec![json!({ "element_index": 7 })];
+ assert_eq!(
+ project_elements_for_query(elements.clone(), None, ""),
+ elements
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs
index dfd59a8de5..b0788ce36a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs
@@ -53,8 +53,9 @@ use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Mutex;
use std::sync::OnceLock;
-/// LRU cap of valid snapshots retained per pid. Past this point the
-/// oldest entry for the pid is evicted and its tokens go stale.
+/// LRU cap of valid snapshots retained per pid. Only the newest snapshot for
+/// each `(pid, window_id)` remains valid; the cap bounds snapshots retained for
+/// other windows owned by the same process.
///
/// Chosen at 8: enough for an agent that re-snapshots once per turn over
/// a multi-window session (open Slack, open Safari, swap to Cursor, …)
@@ -122,6 +123,11 @@ impl TokenRegistry {
let runtime_scope = current_runtime_scope();
let mut entries = self.by_runtime_and_pid.lock().unwrap();
let lane = entries.entry((runtime_scope, pid)).or_default();
+ // Every platform replaces its per-window element cache on the next
+ // snapshot. Retaining an older token for the same window would resolve
+ // that token against the new cache and could target a different row.
+ // Invalidate it at the same boundary as the cache replacement.
+ lane.retain(|entry| entry.window_id != window_id);
lane.push(SnapshotEntry {
snapshot_id: id,
window_id,
@@ -273,6 +279,14 @@ fn parse_token(token: &str) -> Option<(u32, usize)> {
Some((sid, idx))
}
+fn parse_snapshot_handle(snapshot_id: &str) -> Option<u32> {
+ let hex = snapshot_id.strip_prefix('s')?;
+ if hex.len() != 8 || hex.contains(':') {
+ return None;
+ }
+ u32::from_str_radix(hex, 16).ok()
+}
+
/// Process-global handle to the token registry. Used by every platform's
/// `get_window_state` (to register a fresh snapshot) and every element-
/// targeting tool (to resolve a passed-in token).
@@ -294,19 +308,17 @@ pub fn token_for(snapshot_id: u32, element_index: usize) -> String {
format_token(snapshot_id, element_index)
}
-/// Result of dispatching the `element_token` ↔ `element_index` precedence
-/// rule on a tool call's args. Returned by [`resolve_element_args`].
+/// Result of validating an `element_token` or snapshot-bound `element_index`
+/// on a tool call's args. Returned by [`resolve_element_args`].
#[derive(Debug, Clone)]
pub enum ResolvedElement {
/// Neither `element_token` nor `element_index` was supplied — the
/// tool should fall through to its non-element addressing mode
/// (typically pixel `x, y`) or error.
None,
- /// Resolved to `(window_id, element_index)`. The `window_id` may be
- /// `None` when the caller supplied only `element_index` without a
- /// `window_id` (legacy back-compat for tools that already handled
- /// that case); when the caller supplied a token, `window_id` is
- /// always the one the snapshot was taken against.
+ /// Resolved to `(window_id, element_index)`. Element targets always resolve
+ /// through the snapshot registry, so `window_id` is always present. The
+ /// optional shape remains temporarily to avoid duplicating tool plumbing.
Element {
window_id: Option<u32>,
element_index: usize,
@@ -317,24 +329,21 @@ pub enum ResolvedElement {
},
}
-/// Apply the Surface 6 precedence rule for tool args that accept both
-/// `element_index` and `element_token`. Returns either a stale/format
-/// error (already wrapped as a `ToolResult::error`) or the resolved
+/// Validate tool args that accept `element_index`, `snapshot_id`, and
+/// `element_token`. Returns either a structured refusal or the resolved
/// `(window_id, element_index)` pair.
///
/// Rule:
/// - **Neither**: returns [`ResolvedElement::None`]. The tool decides
/// whether to error or fall through to a pixel path.
-/// - **Only `element_index`**: legacy behaviour, unchanged. Returns
-/// `Element { window_id: <caller's window_id arg, if any>, element_index, via_token: false }`.
+/// - **Only `element_index`**: refuses with `snapshot_id_required`.
+/// - **`element_index` + `snapshot_id`**: resolves that exact registered
+/// snapshot and fails closed if it is stale.
/// - **Only `element_token`**: resolves through the registry. On stale
/// or malformed token, returns an error. On success returns
/// `Element { window_id: Some(<from snapshot>), element_index, via_token: true }`.
-/// - **Both supplied**: `element_token` takes precedence; the resolver
-/// verifies it matches `element_index` and logs a warning on
-/// disagreement (the integer is treated as advisory once a token is
-/// present). On stale or malformed token, returns an error — the
-/// integer is NOT used as a fallback (Surface 6 plan: "token wins").
+/// - **Token plus other target fields**: every supplied field must identify the
+/// same snapshot, window, and element. A conflict fails closed.
///
/// `args_window_id` is the `window_id` arg the caller already pulled
/// off the JSON via the existing `args.opt_u64("window_id")`. Passing
@@ -344,49 +353,81 @@ pub fn resolve_element_args(
pid: i32,
args_element_index: Option<usize>,
args_element_token: Option<&str>,
+ args_snapshot_id: Option<&str>,
args_window_id: Option<u32>,
tool_name: &str,
) -> Result<ResolvedElement, crate::protocol::ToolResult> {
- match (args_element_index, args_element_token) {
- (None, None) => Ok(ResolvedElement::None),
- (Some(idx), None) => Ok(ResolvedElement::Element {
- window_id: args_window_id,
- element_index: idx,
- via_token: false,
- }),
- (idx_opt, Some(tok)) => {
- // Token wins. Resolve through the registry; bail on stale
- // or malformed without falling back to the integer.
- let (wid, idx) = global().resolve(pid, tok).map_err(|message| {
- let code = if message.contains("another runtime generation") {
- "generation_mismatch"
- } else if message == STALE_TOKEN_ERROR {
- "stale_element_token"
- } else {
- "invalid_element_token"
- };
- crate::protocol::ToolResult::error(message.clone()).with_structured(
- serde_json::json!({
- "status": "refused",
- "refusal": {
- "code": code,
- "message": message,
- }
- }),
+ let refusal = |code: &str, message: String| {
+ crate::protocol::ToolResult::error(message.clone()).with_structured(serde_json::json!({
+ "status": "refused",
+ "refusal": { "code": code, "message": message }
+ }))
+ };
+ let resolve_token = |tok: &str| {
+ let (wid, idx) = global().resolve(pid, tok).map_err(|message| {
+ let code = if message.contains("another runtime generation") {
+ "generation_mismatch"
+ } else if message == STALE_TOKEN_ERROR {
+ "stale_element_token"
+ } else {
+ "invalid_element_token"
+ };
+ refusal(code, message)
+ })?;
+ Ok::<_, crate::protocol::ToolResult>((wid, idx))
+ };
+
+ match (args_element_index, args_element_token, args_snapshot_id) {
+ (None, None, None) => Ok(ResolvedElement::None),
+ (None, None, Some(_)) => Err(refusal(
+ "element_index_required",
+ format!("{tool_name}: snapshot_id requires element_index"),
+ )),
+ (Some(_), None, None) => Err(refusal(
+ "snapshot_id_required",
+ format!(
+ "{tool_name}: bare element_index is not accepted in Cua Driver 0.17; pass element_token or snapshot_id with element_index"
+ ),
+ )),
+ (Some(idx), None, Some(snapshot_handle)) => {
+ let snapshot_id = parse_snapshot_handle(snapshot_handle).ok_or_else(|| {
+ refusal(
+ "invalid_snapshot_id",
+ format!("{tool_name}: snapshot_id has invalid format"),
)
})?;
- if let Some(int_idx) = idx_opt {
- if int_idx != idx {
- // Disagreement is non-fatal — token wins, but we
- // log so the consumer can debug. Use eprintln so
- // the daemon's stderr captures it (the recording
- // path doesn't see this).
- eprintln!(
- "[cua-driver-rs] {tool_name}: element_token / element_index \
- disagree (token={tok} → idx={idx}, arg element_index={int_idx}); \
- token wins."
- );
- }
+ let token = format_token(snapshot_id, idx);
+ let (wid, resolved_idx) = resolve_token(&token)?;
+ if args_window_id.is_some_and(|arg_wid| arg_wid != wid) {
+ return Err(refusal(
+ "conflicting_element_target",
+ format!(
+ "{tool_name}: snapshot belongs to window_id {wid}, not {}",
+ args_window_id.unwrap()
+ ),
+ ));
+ }
+ Ok(ResolvedElement::Element {
+ window_id: Some(wid),
+ element_index: resolved_idx,
+ via_token: false,
+ })
+ }
+ (idx_opt, Some(tok), snapshot_opt) => {
+ let (wid, idx) = resolve_token(tok)?;
+ if idx_opt.is_some_and(|arg_idx| arg_idx != idx)
+ || args_window_id.is_some_and(|arg_wid| arg_wid != wid)
+ || snapshot_opt.is_some_and(|handle| {
+ parse_snapshot_handle(handle)
+ != parse_token(tok).map(|(snapshot_id, _)| snapshot_id)
+ })
+ {
+ return Err(refusal(
+ "conflicting_element_target",
+ format!(
+ "{tool_name}: element_token conflicts with element_index, snapshot_id, or window_id"
+ ),
+ ));
}
Ok(ResolvedElement::Element {
window_id: Some(wid),
@@ -488,32 +529,37 @@ mod tests {
}
#[test]
- fn next_snapshot_for_same_pid_keeps_old_until_lru_evicts() {
- // The contract is "previous snapshot is invalidated when a NEW
- // snapshot runs for the pid" — but we hold an LRU of size
- // LRU_CAP_PER_PID, so callers get a small grace window of recent
- // snapshots, not strictly the most recent one. This is what the
- // Surface 6 plan describes ("cap at e.g. 8 most recent").
+ fn next_snapshot_for_same_window_invalidates_old_immediately() {
let reg = fresh_registry();
let pid = 12;
let s1 = reg.register_snapshot(pid, 1, 5);
let s2 = reg.register_snapshot(pid, 1, 5);
- // Both should still resolve.
- let _ = reg
- .resolve(pid, &format_token(s1, 0))
- .expect("s1 still in LRU");
+ assert_eq!(
+ reg.resolve(pid, &format_token(s1, 0)).unwrap_err(),
+ STALE_TOKEN_ERROR
+ );
let _ = reg.resolve(pid, &format_token(s2, 0)).expect("s2 fresh");
}
+ #[test]
+ fn snapshots_for_different_windows_share_the_bounded_lru() {
+ let reg = fresh_registry();
+ let pid = 12;
+ let s1 = reg.register_snapshot(pid, 1, 5);
+ let s2 = reg.register_snapshot(pid, 2, 5);
+ assert_eq!(reg.resolve(pid, &format_token(s1, 0)).unwrap().0, 1);
+ assert_eq!(reg.resolve(pid, &format_token(s2, 0)).unwrap().0, 2);
+ }
+
#[test]
fn lru_eviction_invalidates_oldest_snapshot() {
let reg = fresh_registry();
let pid = 13;
// Fill the LRU.
let oldest = reg.register_snapshot(pid, 1, 5);
- for _ in 0..LRU_CAP_PER_PID {
+ for window_id in 2..=(LRU_CAP_PER_PID as u32 + 1) {
// Push LRU_CAP_PER_PID more, which evicts `oldest`.
- let _ = reg.register_snapshot(pid, 1, 5);
+ let _ = reg.register_snapshot(pid, window_id, 5);
}
// Lane size must respect the cap.
assert_eq!(reg.snapshot_count(pid), LRU_CAP_PER_PID);
@@ -587,32 +633,16 @@ mod tests {
// thin precedence layer on top.
#[test]
- fn element_index_alone_still_works() {
- // Surface 6 backward-compat regression guard: tools that only
- // see element_index keep returning the same shape.
- let resolved = resolve_element_args(
+ fn bare_element_index_is_refused() {
+ let result = resolve_element_args(
/* pid = */ 1,
/* element_index = */ Some(7),
/* element_token = */ None,
+ /* snapshot_id = */ None,
/* window_id = */ Some(99),
"click",
- )
- .expect("element_index-only must succeed");
- match resolved {
- ResolvedElement::Element {
- window_id,
- element_index,
- via_token,
- } => {
- assert_eq!(window_id, Some(99));
- assert_eq!(element_index, 7);
- assert!(
- !via_token,
- "element_index-only path must NOT report via_token"
- );
- }
- _ => panic!("expected Element, got {resolved:?}"),
- }
+ );
+ assert!(result.is_err());
}
#[test]
@@ -629,6 +659,7 @@ mod tests {
pid,
None,
Some(&token),
+ None,
// window_id arg intentionally omitted — the token carries it.
None,
"click",
@@ -649,35 +680,31 @@ mod tests {
}
#[test]
- fn both_provided_token_wins_disagree_warns() {
- // Both args supplied with disagreeing indices — token wins, no
- // error returned. We can't assert the stderr line content from
- // a unit test, but we CAN assert the returned indices come from
- // the token, not the integer.
+ fn conflicting_token_and_index_are_refused() {
let reg = global();
let pid = 0x7fff_0002_i32;
let snapshot_id = reg.register_snapshot(pid, 777, 5);
let token = format_token(snapshot_id, 3);
- let resolved = resolve_element_args(
- pid,
- Some(99), // disagrees with token (which says idx 3)
- Some(&token),
- None,
- "click",
- )
- .expect("disagreement still resolves; token wins");
- match resolved {
+ let result = resolve_element_args(pid, Some(99), Some(&token), None, None, "click");
+ assert!(result.is_err());
+ }
+
+ #[test]
+ fn snapshot_id_and_index_resolve_safely() {
+ let reg = global();
+ let pid = 0x7fff_0004_i32;
+ let snapshot_id = reg.register_snapshot(pid, 888, 5);
+ let handle = format!("s{snapshot_id:08x}");
+ let resolved = resolve_element_args(pid, Some(2), None, Some(&handle), Some(888), "click")
+ .expect("snapshot-bound index must resolve");
+ assert!(matches!(
+ resolved,
ResolvedElement::Element {
- window_id,
- element_index,
- via_token,
- } => {
- assert_eq!(window_id, Some(777));
- assert_eq!(element_index, 3, "token's idx wins over the integer arg");
- assert!(via_token);
+ window_id: Some(888),
+ element_index: 2,
+ via_token: false
}
- _ => panic!("expected Element, got {resolved:?}"),
- }
+ ));
}
#[test]
@@ -687,7 +714,8 @@ mod tests {
let pid = 0x7fff_0003_i32;
// Token references a snapshot that was never registered → stale.
let token = format_token(0xdead, 0);
- let err = resolve_element_args(pid, Some(0), Some(&token), Some(1), "click").unwrap_err();
+ let err =
+ resolve_element_args(pid, Some(0), Some(&token), None, Some(1), "click").unwrap_err();
// ToolResult::error wraps the message in a Content::Text — the
// assertion uses the protocol-level error_text accessor.
assert!(
@@ -698,7 +726,7 @@ mod tests {
#[test]
fn neither_returns_none() {
- let resolved = resolve_element_args(1, None, None, None, "click")
+ let resolved = resolve_element_args(1, None, None, None, None, "click")
.expect("neither arg returns None, not error");
assert!(matches!(resolved, ResolvedElement::None));
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index 08df5e915f..c220f56df5 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -57,6 +57,7 @@ pub mod cursor_events;
pub mod cursor_sampler;
pub mod daemon;
pub mod element_cache;
+pub mod element_query;
pub mod element_token;
pub mod expectation;
pub mod ffmpeg_install;
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
index 07f676128c..8671836fbe 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
@@ -6,8 +6,8 @@
//! legacy `app_state.json` and `screenshot.png` names remain post-action aliases.
//!
//! Schema mirrors the Swift/Windows reference `action.json`:
-//! { tool, arguments, result_summary, timestamp, t_ms_from_session_start,
-//! t_start_ms_from_session_start }
+//! { tool, arguments, result_summary, result_error, timestamp,
+//! t_ms_from_session_start, t_start_ms_from_session_start }
use std::path::{Path, PathBuf};
use std::sync::{Mutex, OnceLock};
@@ -584,6 +584,21 @@ impl RecordingSession {
pending: PendingTurn,
result_text: &str,
action_record: Option<&crate::action_record::ActionExecutionRecord>,
+ ) {
+ self.finish_turn_with_outcome(pending, result_text, action_record, false);
+ }
+
+ /// Finalize a turn while also recording whether dispatch returned an
+ /// error. A click-family call rejected before its target can be resolved
+ /// has no click point to annotate; retaining this bit lets the evidence
+ /// manifest distinguish that honest non-action from a missing marker on a
+ /// dispatched click.
+ pub fn finish_turn_with_outcome(
+ &self,
+ pending: PendingTurn,
+ result_text: &str,
+ action_record: Option<&crate::action_record::ActionExecutionRecord>,
+ result_is_error: bool,
) {
let mut inner = self.inner.lock().unwrap();
if !inner.enabled || inner.generation != pending.generation {
@@ -593,7 +608,7 @@ impl RecordingSession {
);
return;
}
- if let Err(error) = write_turn(pending, result_text, action_record) {
+ if let Err(error) = write_turn(pending, result_text, action_record, result_is_error) {
inner.last_error = Some(error.to_string());
}
}
@@ -691,6 +706,7 @@ fn write_evidence_manifest(
state_expected: bool,
click_expected: bool,
click_captured: bool,
+ click_not_applicable_classification: &'static str,
) -> anyhow::Result<()> {
let manifest = serde_json::json!({
"schema": "cua-turn-evidence/v1",
@@ -710,7 +726,14 @@ fn write_evidence_manifest(
after.screenshot_classification,
),
},
- "click": capture_status(click_captured, click_expected, None),
+ "click": if click_expected {
+ capture_status(click_captured, true, None)
+ } else {
+ serde_json::json!({
+ "status": "not_applicable",
+ "classification": click_not_applicable_classification,
+ })
+ },
});
write_json_atomic(&turn_dir.join("evidence.json"), &manifest)
}
@@ -737,6 +760,7 @@ fn write_turn(
pending: PendingTurn,
result_text: &str,
action_record: Option<&crate::action_record::ActionExecutionRecord>,
+ result_is_error: bool,
) -> anyhow::Result<()> {
let PendingTurn {
generation: _,
@@ -762,12 +786,15 @@ fn write_turn(
screenshot_classification: Some("privacy_suppressed"),
}
};
- let click_expected = matches!(tool_name.as_str(), "click" | "double_click" | "right_click");
+ let click_family = matches!(tool_name.as_str(), "click" | "double_click" | "right_click");
+ let refused_before_target_resolution = click_family && result_is_error && click_point.is_none();
+ let click_expected = click_family && !refused_before_target_resolution;
let mut payload = serde_json::json!({
"tool": tool_name,
"arguments": args,
"result_summary": result_text,
+ "result_error": result_is_error,
"timestamp": iso_now(),
"t_ms_from_session_start": now.saturating_sub(session_start_ms),
"t_start_ms_from_session_start": start_ms.saturating_sub(session_start_ms),
@@ -810,6 +837,11 @@ fn write_turn(
pid.is_some() && capture_visual_state,
click_expected,
click_captured,
+ if refused_before_target_resolution {
+ "action_refused_before_target_resolution"
+ } else {
+ "not_a_click_action"
+ },
)?;
Ok(())
@@ -1003,6 +1035,35 @@ mod tests {
assert_eq!(token_action["click_point"]["y"], 1.0);
assert!(token_turn.join("click.png").exists());
+ let stale_snapshot = crate::element_token::global().register_snapshot(1, 88, 1);
+ let stale_token = crate::element_token::token_for(stale_snapshot, 0);
+ let _newer_snapshot = crate::element_token::global().register_snapshot(1, 88, 1);
+ let pending = session
+ .begin_turn(
+ "click",
+ &serde_json::json!({"pid": 1, "element_token": stale_token}),
+ now_ms(),
+ )
+ .expect("stale-token refusal should reserve an evidence turn");
+ session.finish_turn_with_outcome(pending, "stale token", None, true);
+ let refused_turn = output_dir.join("turn-00003");
+ let refused_action: Value = serde_json::from_slice(
+ &std::fs::read(refused_turn.join("action.json")).expect("read refused action"),
+ )
+ .expect("parse refused action");
+ assert_eq!(refused_action["result_error"], true);
+ assert!(refused_action.get("click_point").is_none());
+ assert!(!refused_turn.join("click.png").exists());
+ let refused_manifest: Value = serde_json::from_slice(
+ &std::fs::read(refused_turn.join("evidence.json")).expect("read refused evidence"),
+ )
+ .expect("parse refused evidence");
+ assert_eq!(refused_manifest["click"]["status"], "not_applicable");
+ assert_eq!(
+ refused_manifest["click"]["classification"],
+ "action_refused_before_target_resolution"
+ );
+
let files = [
"action.json",
"app_state.json",
@@ -1020,6 +1081,11 @@ mod tests {
}
std::fs::remove_dir(directory).expect("remove turn fixture directory");
}
+ for file in files.into_iter().filter(|file| *file != "click.png") {
+ std::fs::remove_file(refused_turn.join(file))
+ .expect("remove refused turn fixture file");
+ }
+ std::fs::remove_dir(refused_turn).expect("remove refused turn fixture directory");
std::fs::remove_dir(&output_dir).expect("remove recording fixture directory");
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_tools.rs
index 3a032a25a9..43779ee71b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_tools.rs
@@ -56,10 +56,12 @@ impl Tool for StartRecordingTool {
expected artifact could not be captured.\n\
- `app_state.json` — post-action AX/UIA snapshot for the target pid.\n\
- `screenshot.png` — compatibility alias of `after.png`.\n\
- - `action.json` — tool name, full input arguments, result summary, pid, \
- click point (when applicable), ISO-8601 timestamp.\n\
- - `click.png` — for click-family actions only, `before.png` with a red \
- marker at the click point.\n\n\
+ - `action.json` — tool name, full input arguments, result summary, \
+ result-error flag, pid, click point (when applicable), ISO-8601 \
+ timestamp.\n\
+ - `click.png` — for dispatched click-family actions only, `before.png` \
+ with a red marker at the click point. A call refused before target \
+ resolution is explicitly not applicable instead.\n\n\
Turn folders are named `turn-00001/`, `turn-00002/`, etc. Turn \
numbering restarts at 1 each time recording is (re-)started.\n\n\
**Video is off by default.** Pass `record_video: true` to also \
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index 4dfb21dea9..6e2de746bf 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -1429,10 +1429,11 @@ impl ToolRegistry {
// stream stays the actual user-action sequence (not the meta
// start/stop frames).
if let Some(pending_turn) = pending_turn {
- self.recording.finish_turn_with_action(
+ self.recording.finish_turn_with_outcome(
pending_turn,
recording_result_text.as_deref().unwrap_or(""),
result.action_record.as_ref(),
+ result.is_error == Some(true),
);
}
@@ -3647,8 +3648,15 @@ resources:
);
let structured = DISPATCH_RUNTIME_SCOPE
.scope("runtime-b".to_owned(), async {
- crate::element_token::resolve_element_args(pid, None, Some(&token), None, "click")
- .unwrap_err()
+ crate::element_token::resolve_element_args(
+ pid,
+ None,
+ Some(&token),
+ None,
+ None,
+ "click",
+ )
+ .unwrap_err()
})
.await
.structured_content
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
index 7501e83350..c7e0bb1b51 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
@@ -87,9 +87,19 @@ pub fn scope_schema() -> Value {
pub fn element_index_schema() -> Value {
json!({
"type": "integer",
- "description": "Element index from the last get_window_state. REQUIRES \
- `pid` and `window_id` alongside it — element_index alone (no pid) \
- fails fast, it is not a silent no-op."
+ "description": "Element index from get_window_state. Cua Driver 0.17 \
+ requires the matching `snapshot_id` alongside it. Prefer \
+ `element_token`, which carries both values."
+ })
+}
+
+/// `snapshot_id` — the snapshot handle paired with a numeric element index.
+pub fn snapshot_id_schema() -> Value {
+ json!({
+ "type": "string",
+ "pattern": "^s[0-9a-f]{8}$",
+ "description": "Snapshot handle from get_window_state. Required when \
+ targeting by element_index; stale snapshots fail closed."
})
}
@@ -98,9 +108,9 @@ pub fn element_token_schema() -> Value {
json!({
"type": "string",
"description": "Opaque per-snapshot element handle from \
- `structuredContent.elements[].element_token`. Takes precedence over \
- element_index when both are supplied. Returns an explicit \"stale\" \
- error once a newer snapshot supersedes it — re-snapshot in that case."
+ `structuredContent.elements[].element_token`. If element_index, \
+ snapshot_id, or window_id are also supplied they must agree. Returns \
+ an explicit stale error once a newer snapshot supersedes it."
})
}
@@ -119,6 +129,7 @@ fn shared_param_canonical(name: &str) -> Option<Value> {
"scope" => scope_schema(),
"element_index" => element_index_schema(),
"element_token" => element_token_schema(),
+ "snapshot_id" => snapshot_id_schema(),
"capture_mode" => crate::capture_mode::capture_mode_schema(),
_ => return None,
};
diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
index 731a0a865a..a6c9c7d8c7 100644
--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
@@ -8,8 +8,8 @@
use cua_driver_contract::{
ActionResult, ClickInput, ClipboardReadInput, ClipboardWriteInput, DragInput, EndSessionInput,
EndSessionOutput, EscalateSessionInput, GetAgentCursorStateInput, GetCursorPositionInput,
- GetDesktopStateInput, GetScreenSizeInput, GetSessionStateInput, HotkeyInput, MoveCursorInput,
- PressKeyInput, ScrollInput, SessionStateOutput, SetAgentCursorEnabledInput,
+ GetDesktopStateInput, GetScreenSizeInput, GetSessionStateInput, HotkeyInput, InvokeMenuInput,
+ MoveCursorInput, PressKeyInput, ScrollInput, SessionStateOutput, SetAgentCursorEnabledInput,
SetAgentCursorMotionInput, SetAgentCursorThemeInput, SetWindowFrameInput, StartSessionInput,
StartSessionOutput, ToolInput, TypeTextInput, VerifyStateInput, VerifyStateOutput,
};
@@ -557,6 +557,7 @@ macro_rules! desktop_tool_methods {
verify_state: VerifyStateInput,
move_cursor: MoveCursorInput,
set_window_frame: SetWindowFrameInput,
+ invoke_menu: InvokeMenuInput,
click: ClickInput,
drag: DragInput,
scroll: ScrollInput,
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
index cf806696a9..23a42cab59 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
@@ -1574,6 +1574,22 @@ fn validate_one_turn(turn: &Path, cell_id: &str, errors: &mut Vec<String>) {
Some("click" | "double_click" | "right_click")
)
}) {
+ let refused_before_target_resolution = action.as_ref().is_some_and(|value| {
+ value["result_error"].as_bool() == Some(true) && value.get("click_point").is_none()
+ });
+ if refused_before_target_resolution {
+ let click = manifest.as_ref().map(|value| &value["click"]);
+ if !click.is_some_and(|value| {
+ value["status"].as_str() == Some("not_applicable")
+ && value["classification"].as_str()
+ == Some("action_refused_before_target_resolution")
+ }) {
+ errors.push(format!(
+ "invalid refused-click evidence for {cell_id}/{turn_name}: expected not_applicable/action_refused_before_target_resolution"
+ ));
+ }
+ return;
+ }
validate_capture_status(
manifest.as_ref(),
&["click"],
@@ -2049,6 +2065,55 @@ mod tests {
.any(|error| error.contains("turn-00001/click.png")));
}
+ #[test]
+ fn validator_accepts_click_refused_before_target_resolution_without_marker() {
+ let (root, case, result, turn) = complete_turn_fixture();
+ std::fs::write(
+ turn.join("action.json"),
+ br#"{
+ "tool":"click",
+ "arguments":{"pid":1,"element_token":"stale-token"},
+ "result_error":true
+ }"#,
+ )
+ .expect("write refused action");
+ std::fs::write(
+ turn.join("evidence.json"),
+ br#"{
+ "schema":"cua-turn-evidence/v1",
+ "before":{"state":{"status":"captured"},"screenshot":{"status":"captured"}},
+ "after":{"state":{"status":"captured"},"screenshot":{"status":"captured"}},
+ "click":{"status":"not_applicable","classification":"action_refused_before_target_resolution"}
+ }"#,
+ )
+ .expect("write refused evidence");
+ std::fs::remove_file(turn.join("click.png")).expect("remove inapplicable click marker");
+
+ validate_catalog(&[case], &[result], Some(root.path()), true)
+ .expect("a pre-target refusal must not invent click evidence");
+ }
+
+ #[test]
+ fn validator_rejects_successful_click_marked_not_applicable() {
+ let (root, case, result, turn) = complete_turn_fixture();
+ std::fs::write(
+ turn.join("evidence.json"),
+ br#"{
+ "schema":"cua-turn-evidence/v1",
+ "before":{"state":{"status":"captured"},"screenshot":{"status":"captured"}},
+ "after":{"state":{"status":"captured"},"screenshot":{"status":"captured"}},
+ "click":{"status":"not_applicable","classification":"action_refused_before_target_resolution"}
+ }"#,
+ )
+ .expect("write invalid evidence");
+
+ let errors = validate_catalog(&[case], &[result], Some(root.path()), true)
+ .expect_err("successful clicks still require click evidence");
+ assert!(errors
+ .iter()
+ .any(|error| error.contains("turn-00001/click")));
+ }
+
#[test]
fn validator_reports_capture_classification_for_missing_phase() {
let (root, case, result, turn) = complete_turn_fixture();
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs
index 38f8255ecb..2d1b3d1503 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs
@@ -103,6 +103,15 @@ impl ToolResponse {
.and_then(Value::as_bool)
.unwrap_or(false)
}
+
+ /// Snapshot handle paired with numeric element indices. Element-targeted
+ /// calls in the 0.17 contract must send this value with `element_index`.
+ pub fn snapshot_id(&self) -> &str {
+ self.structured
+ .get("snapshot_id")
+ .and_then(Value::as_str)
+ .expect("get_window_state response must carry snapshot_id")
+ }
}
#[cfg(test)]
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
index fd01636f02..6746c4fbb3 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
@@ -646,6 +646,10 @@ fn action_target_args(
let object = args.as_object_mut().expect("action arguments object");
if addressing == "ax" {
object.insert("element_index".to_owned(), serde_json::json!(index));
+ object.insert(
+ "snapshot_id".to_owned(),
+ serde_json::json!(state.snapshot_id()),
+ );
} else {
let origin = window_origin(fixture, state);
let scale = screenshot_scale(state);
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
index 5fe217f093..65fe2c6e27 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
@@ -103,6 +103,21 @@ fn snapshot_elements(driver: &mut McpDriver, pid: u32, window_id: u64) -> ToolRe
)
}
+fn element_token_by_id(snapshot: &ToolResponse, identifier: &str) -> String {
+ let index = element_index_by_id(snapshot.tree_text(), identifier)
+ .unwrap_or_else(|| panic!("{identifier} element_index not found"));
+ snapshot.structured()["elements"]
+ .as_array()
+ .and_then(|elements| {
+ elements
+ .iter()
+ .find(|element| element["element_index"].as_u64() == Some(index))
+ })
+ .and_then(|element| element["element_token"].as_str())
+ .unwrap_or_else(|| panic!("{identifier} element_token not found"))
+ .to_owned()
+}
+
fn element_pixel_frame(snapshot: &ToolResponse, identifier: &str) -> (f64, f64, f64, f64) {
let index = element_index_by_id(snapshot.tree_text(), identifier)
.unwrap_or_else(|| panic!("{identifier} element_index not found"));
@@ -214,14 +229,15 @@ fn harness_appkit_smoke() {
// and other AXStaticText leaves do NOT propagate
// setAccessibilityIdentifier into the AX tree's identifier slot, so
// we don't assert on ids for labels. We assert on text-presence for
- // those, and on AX ids only for actionable controls (Buttons,
- // TextFields).
+ // those, and on AX ids only for actionable controls whose AppKit
+ // identifiers are actually propagated (Buttons and TextFields).
+ // NSMenuItem behaves like the static leaves here: its title is
+ // exposed, but setAccessibilityIdentifier is not.
for aid in [
"wnd-main", // NSWindow
"btn-increment",
- "btn-reset", // NSButton
- "txt-input", // editable NSTextField
- "menu-test-item", // NSMenuItem (Mac-specific)
+ "btn-reset", // NSButton
+ "txt-input", // editable NSTextField
"btn-exit",
] {
assert!(
@@ -239,6 +255,10 @@ fn harness_appkit_smoke() {
// AXStaticText nodes — assert on their starting text instead of ids.
assert!(text.contains("counter=0"), "counter label missing");
assert!(text.contains("clicks=0"), "click_count label missing");
+ assert!(
+ text.contains("Harness Test Item"),
+ "AppKit menu item title missing"
+ );
assert!(
text.contains("last_action=none"),
"last_action label missing"
@@ -248,6 +268,141 @@ fn harness_appkit_smoke() {
);
}
+#[test]
+#[ignore]
+fn harness_appkit_query_projects_structured_elements() {
+ run_case(
+ native_readonly_case(
+ "appkit",
+ "query_projection",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, wid, driver| {
+ let response = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "query": "btn-increment",
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !response.is_error(),
+ "query snapshot failed: {}",
+ response.text()
+ );
+ let total = response.structured()["total_element_count"]
+ .as_u64()
+ .expect("total_element_count");
+ let returned = response.structured()["returned_element_count"]
+ .as_u64()
+ .expect("returned_element_count");
+ let elements = response.structured()["elements"]
+ .as_array()
+ .expect("projected elements");
+ assert_eq!(returned as usize, elements.len());
+ assert!(
+ returned < total,
+ "query did not compact {returned}/{total} elements"
+ );
+ assert!(has_id(response.tree_text(), "btn-increment"));
+ let _ = element_token_by_id(&response, "btn-increment");
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_appkit_stale_element_token_fails_closed() {
+ run_case(
+ native_readonly_case(
+ "appkit",
+ "stale_element_token",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, wid, driver| {
+ let first = snapshot_elements(driver, pid, wid);
+ let token = element_token_by_id(&first, "btn-increment");
+ let _newer = snapshot_elements(driver, pid, wid);
+ let refused = driver.call(
+ "click",
+ serde_json::json!({"pid": pid as i64, "element_token": token}),
+ );
+ assert!(
+ refused.is_error(),
+ "stale token was accepted: {}",
+ refused.text()
+ );
+ assert_eq!(
+ refused.structured()["refusal"]["code"].as_str(),
+ Some("stale_element_token")
+ );
+ let post = snapshot_elements(driver, pid, wid);
+ assert!(
+ post.tree_text().contains("counter=0"),
+ "stale click mutated counter"
+ );
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_appkit_invoke_menu_live_path() {
+ run_case(
+ native_foreground_case(
+ "appkit",
+ "invoke_menu",
+ Targeting::Ax,
+ DriverRoute::MacosAxAction,
+ ),
+ |pid, wid, driver| {
+ let refused = driver.call(
+ "invoke_menu",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": wid,
+ "path": ["Window", "Arrange", "Missing"]
+ }),
+ );
+ assert!(refused.is_error(), "missing menu path was accepted");
+ assert!(snapshot_elements(driver, pid, wid)
+ .tree_text()
+ .contains("menu_action=none"));
+
+ let invoked = driver.call(
+ "invoke_menu",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": wid,
+ "path": ["Window", "Arrange", "Left"]
+ }),
+ );
+ assert!(
+ !invoked.is_error(),
+ "invoke_menu failed: {}",
+ invoked.text()
+ );
+ assert_eq!(invoked.action_effect(), Some("unverifiable"));
+ std::thread::sleep(Duration::from_millis(300));
+ let post = snapshot_elements(driver, pid, wid);
+ assert!(
+ post.tree_text().contains("menu_action=window_arrange_left"),
+ "menu action did not reach fixture: {}",
+ post.tree_text()
+ );
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ },
+ );
+}
+
/// text_input: type_text into the NSTextField, verify the mirror label
/// shows the typed string. Exercises the AX type_text path
/// (AXSetAttribute on AXValue, or CGEvent fallback).
@@ -274,6 +429,7 @@ fn harness_appkit_text_input() {
"pid": pid as i64,
"window_id": wid,
"element_index": idx,
+ "snapshot_id": snap_pre.snapshot_id(),
"value": "hello-cua"
}),
);
@@ -316,6 +472,7 @@ fn harness_appkit_type_text_background() {
"type_text",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap_pre.snapshot_id(),
"text": "kbd-cua", "delivery_mode": "background"
}),
);
@@ -357,6 +514,7 @@ fn harness_appkit_scroll_foreground() {
"pid": pid as i64,
"window_id": wid,
"element_index": index,
+ "snapshot_id": pre.snapshot_id(),
"direction": "down",
"amount": 5,
"delivery_mode": "foreground"
@@ -396,6 +554,7 @@ fn harness_appkit_scroll_background() {
"pid": pid as i64,
"window_id": wid,
"element_index": index,
+ "snapshot_id": pre.snapshot_id(),
"direction": "down",
"amount": 5,
"delivery_mode": "background"
@@ -446,6 +605,7 @@ fn harness_appkit_counter() {
"pid": pid as i64,
"window_id": wid,
"element_index": idx,
+ "snapshot_id": snap_pre.snapshot_id(),
"action": "press",
"delivery_mode": "background"
}),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
index 4c195f6b57..0d838ee4af 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
@@ -85,6 +85,20 @@ fn element_index(state: &ToolResponse, name: &str) -> u64 {
})
}
+fn element_token(state: &ToolResponse, name: &str) -> String {
+ let index = element_index(state, name);
+ state.structured()["elements"]
+ .as_array()
+ .and_then(|elements| {
+ elements
+ .iter()
+ .find(|element| element["element_index"].as_u64() == Some(index))
+ })
+ .and_then(|element| element["element_token"].as_str())
+ .unwrap_or_else(|| panic!("{name:?} element_token not found"))
+ .to_owned()
+}
+
fn element_rect(
driver: &mut McpDriver,
pid: u32,
@@ -224,6 +238,132 @@ fn run_case(case: CaseSpec, test: impl FnOnce(u32, u64, &mut McpDriver) -> Obser
});
}
+#[test]
+#[ignore]
+fn harness_gtk3_query_projects_structured_elements() {
+ run_case(
+ native_readonly_case(
+ "gtk3",
+ "query_projection",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, window_id, driver| {
+ let response = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": window_id,
+ "query": "btn-increment",
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !response.is_error(),
+ "query snapshot failed: {}",
+ response.text()
+ );
+ let total = response.structured()["total_element_count"]
+ .as_u64()
+ .expect("total_element_count");
+ let returned = response.structured()["returned_element_count"]
+ .as_u64()
+ .expect("returned_element_count");
+ let elements = response.structured()["elements"]
+ .as_array()
+ .expect("projected elements");
+ assert_eq!(returned as usize, elements.len());
+ assert!(
+ returned < total,
+ "query did not compact {returned}/{total} elements"
+ );
+ let _ = element_token(&response, "btn-increment");
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_gtk3_stale_element_token_fails_closed() {
+ run_case(
+ native_readonly_case(
+ "gtk3",
+ "stale_element_token",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, window_id, driver| {
+ let first = snapshot(driver, pid, window_id);
+ let token = element_token(&first, "btn-increment");
+ let _newer = snapshot(driver, pid, window_id);
+ let refused = driver.call(
+ "click",
+ serde_json::json!({"pid": pid, "element_token": token}),
+ );
+ assert!(
+ refused.is_error(),
+ "stale token was accepted: {}",
+ refused.text()
+ );
+ assert_eq!(
+ refused.structured()["refusal"]["code"].as_str(),
+ Some("stale_element_token")
+ );
+ assert!(snapshot(driver, pid, window_id)
+ .tree_text()
+ .contains("counter=0"));
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_gtk3_invoke_menu_live_path() {
+ run_case(
+ native_foreground_case(
+ "gtk3",
+ "invoke_menu",
+ Targeting::Ax,
+ DriverRoute::LinuxAtSpiAction,
+ ),
+ |pid, window_id, driver| {
+ let refused = driver.call(
+ "invoke_menu",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": window_id,
+ "path": ["Window", "Arrange", "Missing"]
+ }),
+ );
+ assert!(refused.is_error(), "missing menu path was accepted");
+ assert!(snapshot(driver, pid, window_id)
+ .tree_text()
+ .contains("menu_action=none"));
+
+ let invoked = driver.call(
+ "invoke_menu",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": window_id,
+ "path": ["Window", "Arrange", "Left"]
+ }),
+ );
+ assert!(
+ !invoked.is_error(),
+ "invoke_menu failed: {}",
+ invoked.text()
+ );
+ assert_eq!(invoked.action_effect(), Some("unverifiable"));
+ wait_for_state(driver, pid, window_id, "menu_action=window_arrange_left");
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ },
+ );
+}
+
#[derive(Clone, Copy, Debug)]
enum Operation {
AxClick {
@@ -333,7 +473,8 @@ fn invoke_operation(
"click",
serde_json::json!({
"pid": pid as i64, "window_id": window_id,
- "element_index": index, "delivery_mode": mode
+ "element_index": index, "snapshot_id": pre.snapshot_id(),
+ "delivery_mode": mode
}),
),
expected,
@@ -350,7 +491,8 @@ fn invoke_operation(
"type_text",
serde_json::json!({
"pid": pid as i64, "window_id": window_id,
- "element_index": index, "text": text, "delivery_mode": mode
+ "element_index": index, "snapshot_id": pre.snapshot_id(),
+ "text": text, "delivery_mode": mode
}),
),
expected,
@@ -367,7 +509,8 @@ fn invoke_operation(
"set_value",
serde_json::json!({
"pid": pid as i64, "window_id": window_id,
- "element_index": index, "value": value
+ "element_index": index, "snapshot_id": pre.snapshot_id(),
+ "value": value
}),
),
expected,
@@ -451,6 +594,7 @@ fn invoke_operation(
args["y"] = serde_json::json!(y + height / 2.0);
} else {
args["element_index"] = serde_json::json!(element_index(&pre, target));
+ args["snapshot_id"] = serde_json::json!(pre.snapshot_id());
}
let response = driver.call("scroll", args);
if expect_refusal {
@@ -495,7 +639,8 @@ fn invoke_operation(
"click",
serde_json::json!({
"pid": pid as i64, "window_id": window_id,
- "element_index": index, "delivery_mode": mode
+ "element_index": index, "snapshot_id": pre.snapshot_id(),
+ "delivery_mode": mode
}),
);
assert!(
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_libreoffice_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_libreoffice_test.rs
index 53f401de65..d0481da70a 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_libreoffice_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_libreoffice_test.rs
@@ -324,7 +324,8 @@ fn harness_lo_vcl_font_color_expand_opens_picker() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid,
- "element_index": idx, "action": "expand"
+ "element_index": idx, "snapshot_id": snap.snapshot_id(),
+ "action": "expand"
}),
);
let resp_text = resp.text();
@@ -685,7 +686,8 @@ fn harness_lo_vcl_color_pick_green_end_to_end() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid,
- "element_index": fc_idx, "action": "expand"
+ "element_index": fc_idx, "snapshot_id": snap.snapshot_id(),
+ "action": "expand"
}),
);
let open_text = open_resp.text();
@@ -750,7 +752,7 @@ fn harness_lo_vcl_color_pick_green_end_to_end() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": picker_wid,
- "element_index": green_idx
+ "element_index": green_idx, "snapshot_id": psnap.snapshot_id()
}),
);
let pick_text = pick_resp.text();
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_swiftui_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_swiftui_test.rs
index 3ef0fa7438..4f296d3dad 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_swiftui_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_swiftui_test.rs
@@ -267,6 +267,7 @@ fn harness_swiftui_counter_background() {
"pid": pid as i64,
"window_id": wid,
"element_index": index,
+ "snapshot_id": pre.snapshot_id(),
"action": "press",
"delivery_mode": "background"
}),
@@ -303,6 +304,7 @@ fn harness_swiftui_set_value_background() {
"pid": pid as i64,
"window_id": wid,
"element_index": index,
+ "snapshot_id": pre.snapshot_id(),
"value": "swiftui-cua"
}),
);
@@ -353,6 +355,7 @@ fn harness_swiftui_popover_foreground() {
"pid": pid as i64,
"window_id": wid,
"element_index": trigger_idx,
+ "snapshot_id": snap_pre.snapshot_id(),
"action": "press",
"delivery_mode": "foreground"
}),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_winui3_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_winui3_test.rs
index 88d152bfbc..708176c124 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_winui3_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_winui3_test.rs
@@ -260,6 +260,7 @@ fn harness_winui3_type_text() {
"type_text",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"text": "winui3-typed", "delivery_mode": "background"
}),
);
@@ -293,6 +294,7 @@ fn harness_winui3_xaml_popup_open() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "background"
}),
);
@@ -335,6 +337,7 @@ fn harness_winui3_checkbox_toggle() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "background"
}),
);
@@ -374,6 +377,7 @@ fn harness_winui3_radio_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "background"
}),
);
@@ -425,6 +429,7 @@ fn harness_winui3_slider_set_value() {
"set_value",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"value": "42"
}),
);
@@ -471,6 +476,7 @@ fn harness_winui3_combo_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": combo_idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "background"
}),
);
@@ -488,6 +494,7 @@ fn harness_winui3_combo_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": item_idx,
+ "snapshot_id": snap2.snapshot_id(),
"delivery_mode": "background"
}),
);
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
index 56f3e54c2d..a7dd659a86 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
@@ -115,6 +115,21 @@ fn snapshot(driver: &mut McpDriver, pid: u32, window_id: u64) -> ToolResponse {
)
}
+fn element_token_by_id(snapshot: &ToolResponse, identifier: &str) -> String {
+ let index = ax::element_index_by_id(snapshot.tree_text(), identifier)
+ .unwrap_or_else(|| panic!("{identifier} element_index not found"));
+ snapshot.structured()["elements"]
+ .as_array()
+ .and_then(|elements| {
+ elements
+ .iter()
+ .find(|element| element["element_index"].as_u64() == Some(index))
+ })
+ .and_then(|element| element["element_token"].as_str())
+ .unwrap_or_else(|| panic!("{identifier} element_token not found"))
+ .to_owned()
+}
+
fn snapshot_lines_containing(text: &str, needles: &[&str]) -> String {
let mut lines = Vec::new();
for line in text.lines() {
@@ -278,6 +293,86 @@ fn harness_wpf_smoke() {
);
}
+#[test]
+#[ignore]
+fn harness_wpf_query_projects_structured_elements() {
+ run_case(
+ native_readonly_case(
+ "wpf",
+ "query_projection",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, wid, driver| {
+ let response = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "query": "btn-increment",
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !response.is_error(),
+ "query snapshot failed: {}",
+ response.text()
+ );
+ let total = response.structured()["total_element_count"]
+ .as_u64()
+ .expect("total_element_count");
+ let returned = response.structured()["returned_element_count"]
+ .as_u64()
+ .expect("returned_element_count");
+ let elements = response.structured()["elements"]
+ .as_array()
+ .expect("projected elements");
+ assert_eq!(returned as usize, elements.len());
+ assert!(
+ returned < total,
+ "query did not compact {returned}/{total} elements"
+ );
+ let _ = element_token_by_id(&response, "btn-increment");
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_wpf_stale_element_token_fails_closed() {
+ run_case(
+ native_readonly_case(
+ "wpf",
+ "stale_element_token",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ ),
+ |pid, wid, driver| {
+ let first = snapshot(driver, pid, wid);
+ let token = element_token_by_id(&first, "btn-increment");
+ let _newer = snapshot(driver, pid, wid);
+ let refused = driver.call(
+ "click",
+ serde_json::json!({"pid": pid as i64, "element_token": token}),
+ );
+ assert!(
+ refused.is_error(),
+ "stale token was accepted: {}",
+ refused.text()
+ );
+ assert_eq!(
+ refused.structured()["refusal"]["code"].as_str(),
+ Some("stale_element_token")
+ );
+ assert!(snapshot(driver, pid, wid).tree_text().contains("counter=0"));
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ },
+ );
+}
+
// ── shared driver session helper ─────────────────────────────────────────────
/// Spin up a fresh cua-driver + harness pair, run the closure, then tear
@@ -436,6 +531,7 @@ fn harness_wpf_counter_invoke() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": pre.snapshot_id(),
"delivery_mode": "background"
}),
)
@@ -551,6 +647,7 @@ fn harness_wpf_type_text() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -612,6 +709,7 @@ fn harness_wpf_set_value() {
"set_value",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"value": "via-uia-setvalue"
}),
)
@@ -673,6 +771,7 @@ fn harness_wpf_right_click() {
"right_click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -716,6 +815,7 @@ fn harness_wpf_double_click() {
"double_click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -852,6 +952,7 @@ fn harness_wpf_scroll() {
"pid": pid as i64,
"window_id": wid,
"element_index": idx,
+ "snapshot_id": pre.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -911,6 +1012,7 @@ fn harness_wpf_modal_messagebox() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -972,6 +1074,7 @@ fn harness_wpf_modal_messagebox() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": modal_wid, "element_index": cancel_idx,
+ "snapshot_id": modal_snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1004,6 +1107,7 @@ fn harness_wpf_owned_popup() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1063,6 +1167,7 @@ fn harness_wpf_layered_popup_capture() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1277,7 +1382,8 @@ fn harness_wpf_slider_increase_large() {
let resp = driver.call(
"click",
serde_json::json!({
- "pid": pid as i64, "window_id": wid, "element_index": idx
+ "pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id()
}),
);
println!("invoke IncreaseLarge #{i}: {}", resp.text());
@@ -1329,6 +1435,7 @@ fn harness_wpf_checkbox_toggle() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1368,6 +1475,7 @@ fn harness_wpf_radio_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1406,6 +1514,7 @@ fn harness_wpf_combo_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": combo_idx,
+ "snapshot_id": snap.snapshot_id(),
"action": "expand", "delivery_mode": "background"
}),
);
@@ -1419,6 +1528,7 @@ fn harness_wpf_combo_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": item_idx,
+ "snapshot_id": snap2.snapshot_id(),
"delivery_mode": "background"
}),
);
@@ -1457,6 +1567,7 @@ fn harness_wpf_listbox_select() {
"click",
serde_json::json!({
"pid": pid as i64, "window_id": wid, "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"delivery_mode": "foreground"
}),
);
@@ -1484,75 +1595,52 @@ fn harness_wpf_listbox_select() {
#[test]
#[ignore]
-fn harness_wpf_menu_invoke() {
+fn harness_wpf_invoke_menu_live_path() {
run_foreground_case(
- "menu_invoke",
+ "invoke_menu",
Targeting::Ax,
DriverRoute::UiaExpandCollapse,
Vec::new(),
|pid, wid, driver| {
focus_harness(driver, pid, wid);
- // Expand File menu first (UIA expand pattern on MenuItem)
- let snap = snapshot(driver, pid, wid);
- let file_idx = ax::element_index_by_id(snap.text(), "menu-file")
- .or_else(|| ax::element_index_containing(snap.text(), "File"))
- .unwrap_or_else(|| {
- panic!(
- "menu-file missing. Menu-related snapshot lines: {}",
- snapshot_lines_containing(snap.text(), &["menu", "file", "new", "open"])
- )
- });
- let expand = driver.call(
- "click",
+ let refused = driver.call(
+ "invoke_menu",
serde_json::json!({
- "pid": pid as i64, "window_id": wid, "element_index": file_idx,
- "action": "expand",
- "delivery_mode": "foreground"
+ "pid": pid, "window_id": wid,
+ "path": ["Window", "Arrange", "Missing"]
}),
);
- assert!(
- !expand.is_error(),
- "File menu expand failed: {}",
- expand.text()
- );
- std::thread::sleep(Duration::from_millis(400));
+ assert!(refused.is_error(), "missing menu path was accepted");
+ assert!(snapshot(driver, pid, wid)
+ .tree_text()
+ .contains("menu_action=none"));
- // Re-snapshot so menu-file-new is in the cache (it materialized
- // when the menu expanded).
- let snap2 = snapshot(driver, pid, wid);
- let new_idx = ax::element_index_by_id(snap2.text(), "menu-file-new")
- .or_else(|| ax::element_index_containing(snap2.text(), "New"))
- .unwrap_or_else(|| {
- panic!(
- "menu-file-new missing after expand. Menu-related snapshot lines: {}",
- snapshot_lines_containing(snap2.text(), &["menu", "file", "new", "open"])
- )
- });
let invoke = driver.call(
- "click",
+ "invoke_menu",
serde_json::json!({
- "pid": pid as i64, "window_id": wid, "element_index": new_idx,
- "delivery_mode": "foreground"
+ "pid": pid, "window_id": wid,
+ "path": ["Window", "Arrange", "Left"]
}),
);
assert!(
!invoke.is_error(),
- "File > New invoke failed: {}",
+ "Window > Arrange > Left invoke failed: {}",
invoke.text()
);
+ assert_eq!(invoke.action_effect(), Some("unverifiable"));
std::thread::sleep(Duration::from_millis(500));
let post = snapshot(driver, pid, wid);
assert!(
- post.text().contains("menu_action=file_new"),
- "File>New didn't invoke: {}",
- post.text()
+ post.tree_text().contains("menu_action=window_arrange_left"),
+ "Window>Arrange>Left didn't invoke: {}",
+ post.tree_text()
.lines()
.filter(|l| l.contains("menu_action="))
.collect::<Vec<_>>()
.join(" / ")
);
- println!("✅ harness_wpf_menu_invoke: menu_action=file_new");
+ println!("✅ harness_wpf_invoke_menu_live_path: menu_action=window_arrange_left");
Vec::new()
},
);
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/installed_app_textedit_macos_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/installed_app_textedit_macos_test.rs
index 8d6cffc3b9..edf11916f6 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/installed_app_textedit_macos_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/installed_app_textedit_macos_test.rs
@@ -68,7 +68,7 @@ fn background_type_on_native_cocoa_is_ax_verified() {
// so prefer visible windows and select the one that actually exposes
// the editor instead of assuming `windows[0]` is the document.
windows.sort_by_key(|window| !window["is_on_screen"].as_bool().unwrap_or(false));
- let (wid, el) = windows
+ let (wid, el, snapshot_id) = windows
.iter()
.filter_map(|window| window["window_id"].as_u64())
.find_map(|window_id| {
@@ -88,7 +88,7 @@ fn background_type_on_native_cocoa_is_ax_verified() {
.find(|element| element["role"] == "AXTextArea")
.and_then(|element| element["element_index"].as_u64())
})
- .map(|element_index| (window_id, element_index))
+ .map(|element_index| (window_id, element_index, state.snapshot_id().to_owned()))
})
.expect("TextEdit opened no window containing an AXTextArea");
@@ -103,6 +103,7 @@ fn background_type_on_native_cocoa_is_ax_verified() {
"type_text",
serde_json::json!({
"pid": pid, "window_id": wid, "element_index": el,
+ "snapshot_id": snapshot_id,
"text": "ladder", "delivery_mode": "background"
}),
)
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_element_token_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_element_token_test.rs
index 79b62f8650..a179be813a 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_element_token_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_element_token_test.rs
@@ -1,5 +1,4 @@
-//! Integration tests for Surface 6 — opaque `element_token` alongside
-//! the existing integer `element_index`.
+//! Protocol integration tests for snapshot-bound element targeting.
//!
//! These exercise the MCP protocol surface (schema + capability claims)
//! by spawning the real `cua-driver` binary and walking `tools/list`.
@@ -7,11 +6,6 @@
//! `cua-driver-core::element_token` and the per-platform tool modules —
//! this file is only for what's visible across the JSON-RPC boundary.
-// element_token resolution is exercised on macOS/Linux only; gate the whole
-// file (helpers included) so Windows excludes it instead of warning about the
-// then-unused tools/list helpers.
-#![cfg(any(target_os = "macos", target_os = "linux"))]
-
use cua_driver_testkit::RawDriver;
/// Spawn the driver, send initialize + tools/list, return the parsed
@@ -29,12 +23,9 @@ fn fetch_tools_list() -> Option<serde_json::Value> {
Some(driver.recv())
}
-/// Surface 6: Every tool that accepts the opaque `element_token` arg
-/// must (a) advertise the field in its inputSchema and (b) claim the
-/// `accessibility.element_tokens` capability. These together let
-/// consumers detect support before issuing a token-carrying call.
+/// Every element-addressed tool advertises both safe target forms: an opaque
+/// token, or an integer paired with its snapshot id.
#[test]
-#[cfg(any(target_os = "macos", target_os = "linux"))]
fn token_accepting_tools_advertise_element_token_in_schema_and_capabilities() {
let resp = match fetch_tools_list() {
Some(r) => r,
@@ -63,7 +54,7 @@ fn token_accepting_tools_advertise_element_token_in_schema_and_capabilities() {
.find(|t| t["name"].as_str() == Some(tool_name))
.unwrap_or_else(|| panic!("tool {tool_name:?} missing from tools/list"));
- // (a) inputSchema.properties.element_token present, type=string.
+ // (a) inputSchema exposes the three related target fields.
let props = tool["inputSchema"]["properties"]
.as_object()
.unwrap_or_else(|| panic!("{tool_name} has no inputSchema.properties"));
@@ -78,6 +69,16 @@ fn token_accepting_tools_advertise_element_token_in_schema_and_capabilities() {
Some("string"),
"{tool_name} element_token must be type:string"
);
+ assert_eq!(
+ props["element_index"]["type"].as_str(),
+ Some("integer"),
+ "{tool_name} element_index must remain available only as a snapshot-bound pair"
+ );
+ assert_eq!(
+ props["snapshot_id"]["type"].as_str(),
+ Some("string"),
+ "{tool_name} must advertise snapshot_id for safe integer targeting"
+ );
// (b) capabilities array includes `accessibility.element_tokens`.
let caps: Vec<&str> = tool["capabilities"]
@@ -94,11 +95,10 @@ fn token_accepting_tools_advertise_element_token_in_schema_and_capabilities() {
}
}
-/// Surface 6: `get_window_state` claims `accessibility.element_tokens`
+/// `get_window_state` claims `accessibility.element_tokens`
/// because it EMITS the tokens (the other side of the contract from
/// the action tools above).
#[test]
-#[cfg(any(target_os = "macos", target_os = "linux"))]
fn get_window_state_claims_element_tokens_capability() {
let resp = match fetch_tools_list() {
Some(r) => r,
@@ -122,39 +122,27 @@ fn get_window_state_claims_element_tokens_capability() {
);
}
-/// Surface 6 hard constraint: `element_index` MUST still be present on
-/// every token-accepting tool's schema — the new `element_token` field
-/// is additive, not a replacement. This is the regression guard for
-/// the back-compat contract.
+/// The native menu operation is a path contract, never another entry point for
+/// mutable snapshot indices.
#[test]
-#[cfg(any(target_os = "macos", target_os = "linux"))]
-fn element_index_field_still_present_on_token_accepting_tools() {
- let resp = match fetch_tools_list() {
- Some(r) => r,
- None => return,
+fn invoke_menu_is_path_only_and_claims_native_menu_capability() {
+ let Some(resp) = fetch_tools_list() else {
+ return;
};
let tools = resp["result"]["tools"].as_array().expect("tools array");
-
- const TOKEN_TOOLS: &[&str] = &[
- "click",
- "double_click",
- "right_click",
- "scroll",
- "type_text",
- "press_key",
- "set_value",
- ];
-
- for tool_name in TOKEN_TOOLS {
- let tool = tools
- .iter()
- .find(|t| t["name"].as_str() == Some(tool_name))
- .unwrap_or_else(|| panic!("{tool_name} missing"));
- let props = tool["inputSchema"]["properties"].as_object().unwrap();
- assert!(
- props.contains_key("element_index"),
- "Surface 6 hard constraint: {tool_name}.element_index must \
- stay on the schema — element_token is purely additive"
- );
- }
+ let tool = tools
+ .iter()
+ .find(|tool| tool["name"].as_str() == Some("invoke_menu"))
+ .expect("invoke_menu missing");
+ let properties = tool["inputSchema"]["properties"]
+ .as_object()
+ .expect("invoke_menu properties");
+ assert!(properties.contains_key("path"));
+ assert!(!properties.contains_key("element_index"));
+ assert!(!properties.contains_key("element_token"));
+ let capabilities = tool["capabilities"].as_array().expect("capabilities");
+ assert!(capabilities.iter().any(|value| value == "menu.path.invoke"));
+ assert!(capabilities
+ .iter()
+ .any(|value| value == "accessibility.menu.native"));
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
index f0cfcec9ee..fd43fd0aea 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
@@ -943,6 +943,10 @@ fn set_value_via_element_index() {
let element_count = resp["result"]["structuredContent"]["element_count"]
.as_u64()
.unwrap_or(0);
+ let snapshot_id = resp["result"]["structuredContent"]["snapshot_id"]
+ .as_str()
+ .expect("get_window_state snapshot_id")
+ .to_owned();
if element_count == 0 {
eprintln!("No AX elements — skipping set_value test");
return;
@@ -951,7 +955,7 @@ fn set_value_via_element_index() {
// set_value on element 0 — this is the document / text area in a new TextEdit document.
d.send(&serde_json::json!({
"jsonrpc":"2.0","id":6,"method":"tools/call",
- "params":{"name":"set_value","arguments":{"pid": pid, "window_id": wid, "element_index": 0, "value": "cua-test-value"}}
+ "params":{"name":"set_value","arguments":{"pid": pid, "window_id": wid, "element_index": 0, "snapshot_id": snapshot_id, "value": "cua-test-value"}}
}));
let resp = d.recv();
// set_value may fail if element 0 is not settable; accept tool-level errors but not protocol errors.
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
index f921f7e6aa..4a6c4d0605 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
@@ -1814,6 +1814,7 @@ fn run_native_omnibox_select_all(spec: &BrowserSpec) {
"pid": fixture.pid as i64,
"window_id": fixture.window_id,
"element_index": index,
+ "snapshot_id": snapshot.snapshot_id(),
"value": initial,
}),
);
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
index 915af05240..9b0b70e3a7 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
@@ -719,11 +719,21 @@ fn format_value(v: f64) -> String {
/// (`perform_action`, `set_value`, `get_element_bounds`, snapshot bounds);
/// any divergence would desync indices between the snapshot and the operations.
fn is_indexable(v: &Visited) -> bool {
- is_indexable_capabilities(!v.actions.is_empty(), v.has_editable, v.has_value)
+ is_indexable_capabilities(
+ !v.actions.is_empty(),
+ v.has_editable,
+ v.has_value,
+ v.enabled,
+ )
}
-fn is_indexable_capabilities(has_action: bool, has_editable: bool, has_value: bool) -> bool {
- has_action || has_editable || has_value
+fn is_indexable_capabilities(
+ has_action: bool,
+ has_editable: bool,
+ has_value: bool,
+ enabled: Option<bool>,
+) -> bool {
+ (has_action || has_editable || has_value) && enabled != Some(false)
}
// ── Public (sync) entry points ───────────────────────────────────────────────
@@ -1318,6 +1328,109 @@ fn activation_index(role: &str, actions: &[String]) -> Option<usize> {
})
}
+fn is_menu_role(role: &str) -> bool {
+ role.trim().to_ascii_lowercase().contains("menu")
+}
+
+/// Find one exact visible menu lineage in a flattened pre-order AT-SPI walk.
+/// Unlabelled menu containers are transparent; all labelled menu ancestors
+/// must match the requested prefix, so duplicate labels elsewhere fail closed.
+fn exact_menu_path_matches(visited: &[Visited<'_>], path: &[String]) -> Vec<usize> {
+ let mut parent_at_depth: Vec<Option<usize>> = Vec::new();
+ let mut parents = vec![None; visited.len()];
+ for (index, node) in visited.iter().enumerate() {
+ parents[index] = if node.depth == 0 {
+ None
+ } else {
+ (0..node.depth)
+ .rev()
+ .find_map(|depth| parent_at_depth.get(depth).copied().flatten())
+ };
+ while parent_at_depth.len() <= node.depth {
+ parent_at_depth.push(None);
+ }
+ parent_at_depth[node.depth] = Some(index);
+ for deeper in (node.depth + 1)..parent_at_depth.len() {
+ parent_at_depth[deeper] = None;
+ }
+ }
+
+ visited
+ .iter()
+ .enumerate()
+ .filter_map(|(index, node)| {
+ if !is_menu_role(&node.role)
+ || node.enabled == Some(false)
+ || node.name.trim() != path.last().map(String::as_str).unwrap_or("")
+ {
+ return None;
+ }
+ let mut lineage = Vec::new();
+ let mut cursor = Some(index);
+ while let Some(current) = cursor {
+ let ancestor = &visited[current];
+ if is_menu_role(&ancestor.role) && !ancestor.name.trim().is_empty() {
+ lineage.push(ancestor.name.trim());
+ }
+ cursor = parents[current];
+ }
+ lineage.reverse();
+ lineage.dedup();
+ (lineage == path.iter().map(String::as_str).collect::<Vec<_>>()).then_some(index)
+ })
+ .collect()
+}
+
+/// Resolve and invoke an application menu one live hierarchy level at a time.
+/// Every hop re-walks AT-SPI after the preceding menu has materialized; no
+/// snapshot index is retained across mutations.
+pub fn invoke_menu_path(pid: u32, path: &[String]) -> Result<()> {
+ bounded(
+ async {
+ let conn = shared_connection().await?;
+ for depth in 0..path.len() {
+ let visited = collect_visited(conn, pid)
+ .await?
+ .ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?;
+ let matches = exact_menu_path_matches(&visited, &path[..=depth]);
+ let target_index = match matches.as_slice() {
+ [index] => *index,
+ [] => anyhow::bail!("menu path segment {depth} was not found"),
+ _ => anyhow::bail!("menu path segment {depth} is ambiguous"),
+ };
+ let target = &visited[target_index];
+ if target.enabled == Some(false) {
+ anyhow::bail!("menu path segment {depth} is disabled");
+ }
+ let chosen = activation_index(&target.role, &target.actions).ok_or_else(|| {
+ anyhow!("menu path segment {depth} has no safe activation action")
+ })?;
+ let proxies = target
+ .acc
+ .proxies()
+ .await
+ .map_err(|error| anyhow!("interface proxies unavailable: {error}"))?;
+ let action = proxies
+ .action()
+ .await
+ .map_err(|error| anyhow!("Action unavailable: {error}"))?;
+ let accepted = action
+ .do_action(chosen as i32)
+ .await
+ .map_err(|error| anyhow!("doAction failed: {error}"))?;
+ if !accepted {
+ anyhow::bail!("menu path segment {depth} rejected its native action");
+ }
+ if depth + 1 != path.len() {
+ tokio::time::sleep(Duration::from_millis(80)).await;
+ }
+ }
+ Ok(())
+ },
+ || Err(anyhow!("invoke_menu timed out for pid {pid}")),
+ )
+}
+
pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool)> {
bounded(
async {
@@ -2374,10 +2487,11 @@ mod coord_tests {
#[test]
fn editable_only_nodes_are_addressable() {
- assert!(is_indexable_capabilities(false, true, false));
- assert!(is_indexable_capabilities(true, false, false));
- assert!(is_indexable_capabilities(false, false, true));
- assert!(!is_indexable_capabilities(false, false, false));
+ assert!(is_indexable_capabilities(false, true, false, Some(true)));
+ assert!(is_indexable_capabilities(true, false, false, None));
+ assert!(is_indexable_capabilities(false, false, true, Some(true)));
+ assert!(!is_indexable_capabilities(false, false, false, Some(true)));
+ assert!(!is_indexable_capabilities(true, false, false, Some(false)));
}
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 2b59dc9e6d..12ea98659d 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -3,7 +3,8 @@
use async_trait::async_trait;
use cua_driver_contract::{
ClickButton, ClickInput, DragInput, GetCursorPositionInput, GetDesktopStateInput,
- GetScreenSizeInput, HotkeyInput, MoveCursorInput, PressKeyInput, ScrollInput, TypeTextInput,
+ GetScreenSizeInput, HotkeyInput, InvokeMenuInput, MoveCursorInput, PressKeyInput, ScrollInput,
+ TypeTextInput,
};
use cua_driver_core::{
protocol::ToolResult,
@@ -581,7 +582,10 @@ impl Tool for GetWindowStateTool {
`parent_index`, `depth`). The markdown `tree_markdown` stays \
available and unchanged in shape for existing text-parsing \
callers — but new fields will only be added to the structured \
- side.\n\n\
+ side. Set `query` to project BOTH representations to matching \
+ rows plus their ancestor chain while preserving original indices. \
+ `total_element_count` reports the complete snapshot and \
+ `returned_element_count` reports the projection.\n\n\
Always returns BOTH the element tree AND a screenshot — ground on \
both and cross-check (the tree lies on some surfaces). Choose the \
modality at ACTION time: an element ax action \
@@ -602,7 +606,7 @@ impl Tool for GetWindowStateTool {
"description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string",
"description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},
- "query":{"type":"string"},
+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
"max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."}
},"additionalProperties":false}),
@@ -669,12 +673,13 @@ impl Tool for GetWindowStateTool {
.and_then(|value| value.as_bool())
== Some(true);
let state = self.state.clone();
+ let query_for_walk = query.clone();
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
let tree_result = Some(crate::atspi::walk_tree_bounded(
pid,
xid,
- query.as_deref(),
+ query_for_walk.as_deref(),
max_elements,
max_depth,
));
@@ -821,6 +826,13 @@ impl Tool for GetWindowStateTool {
Some(entry)
})
.collect();
+ let elements = cua_driver_core::element_query::project_elements_for_query(
+ elements,
+ query.as_deref(),
+ &tr.tree_markdown,
+ );
+ structured["total_element_count"] = json!(count);
+ structured["returned_element_count"] = json!(elements.len());
structured["elements"] = json!(elements);
// Surface 6: snapshot id mirror for debug correlation.
if let Some(snapshot_id) = snapshot_id {
@@ -1857,6 +1869,7 @@ impl Tool for ClickTool {
"y":{"type":"number"},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
// Shape matches the shared button_schema() canon (string +
// [left,right,middle]); kept inline to carry the Linux/Wayland
// back-compat prose the click button-schema test asserts on.
@@ -1965,6 +1978,7 @@ impl Tool for ClickTool {
pid as i32,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg.map(|v| v as u32),
"click",
) {
@@ -2332,6 +2346,7 @@ impl Tool for TypeTextTool {
"text":{"type":"string"},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click."},
"y":{"type":"number","description":"Screenshot-pixel Y of the field (see x)."},
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
@@ -2387,6 +2402,7 @@ impl Tool for TypeTextTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"type_text",
) {
@@ -2948,6 +2964,7 @@ impl Tool for PressKeyTool {
"modifiers":{"type":"array","items":{"type":"string"}},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index."},
"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
@@ -3010,6 +3027,7 @@ impl Tool for PressKeyTool {
pid as i32,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg.map(|v| v as u32),
"press_key",
) {
@@ -3255,6 +3273,7 @@ impl Tool for HotkeyTool {
"description":"Modifier(s) + one non-modifier key, e.g. [\"ctrl\",\"c\"]."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
@@ -3313,6 +3332,7 @@ impl Tool for HotkeyTool {
pid as i32,
element_index_arg,
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg.map(|value| value as u32),
"hotkey",
) {
@@ -3561,6 +3581,7 @@ impl Tool for SetValueTool {
"window_id":{"type":"integer","description":"Required when element_index is used; optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"value":{"type":"string"}
},"additionalProperties":false
}),
@@ -3583,6 +3604,7 @@ impl Tool for SetValueTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"set_value",
) {
@@ -3652,6 +3674,7 @@ impl Tool for ScrollTool {
"window_id":{"type":"integer"},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index."},
"y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index."},
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
@@ -3714,6 +3737,7 @@ impl Tool for ScrollTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"scroll",
) {
@@ -4041,6 +4065,7 @@ impl Tool for DoubleClickTool {
"y":{"type":"number"},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false}),
@@ -4071,6 +4096,7 @@ impl Tool for DoubleClickTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"double_click",
) {
@@ -4272,6 +4298,7 @@ impl Tool for RightClickTool {
"y":{"type":"number"},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"modifier": cua_driver_core::tool_schema::modifier_schema(),
"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
@@ -4303,6 +4330,7 @@ impl Tool for RightClickTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"right_click",
) {
@@ -6923,6 +6951,145 @@ fn kill_app_stale_process_refusal(message: String) -> ToolResult {
}))
}
+// ── invoke_menu (Linux) ──────────────────────────────────────────────────────
+
+pub struct InvokeMenuTool;
+
+static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
+ if path.is_empty() || path.len() > 16 {
+ return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ }
+ path.into_iter()
+ .enumerate()
+ .map(|(index, segment)| {
+ let segment = segment.trim();
+ if segment.is_empty() {
+ Err(format!("invoke_menu: path segment {index} is empty"))
+ } else {
+ Ok(segment.to_owned())
+ }
+ })
+ .collect()
+}
+
+fn menu_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": { "code": "menu_path_unavailable", "message": message }
+ }))
+}
+
+#[async_trait]
+impl Tool for InvokeMenuTool {
+ fn def(&self) -> &ToolDef {
+ INVOKE_MENU_DEF.get_or_init(|| {
+ let contract =
+ cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
+ EvidenceKind, RequestedDelivery,
+ };
+
+ let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let path = match normalized_menu_path(input.path) {
+ Ok(path) => path,
+ Err(error) => return menu_refusal(error),
+ };
+ let pid = input.pid;
+ let window_id = input.window_id;
+ if !crate::wayland::list_windows_dispatch(Some(pid))
+ .iter()
+ .any(|window| window.xid == window_id && window.pid == Some(pid))
+ {
+ return menu_refusal("invoke_menu: window_id does not belong to pid".into());
+ }
+
+ let activation = if crate::wayland::is_wayland() {
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::activate_window_for_input_target(window_id, Some(pid))
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => None,
+ Ok(Err(error)) => {
+ return menu_refusal(format!(
+ "invoke_menu: target window could not be activated: {error}"
+ ))
+ }
+ Err(error) => {
+ return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
+ }
+ }
+ } else {
+ match tokio::task::spawn_blocking(move || {
+ crate::input::x11_activate_window_persistent(window_id)
+ })
+ .await
+ {
+ Ok(Ok(prior)) => Some(prior),
+ Ok(Err(error)) => {
+ return menu_refusal(format!(
+ "invoke_menu: target window could not be activated: {error}"
+ ))
+ }
+ Err(error) => {
+ return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
+ }
+ }
+ };
+
+ let outcome = tokio::task::spawn_blocking(move || {
+ let result = crate::atspi::native::invoke_menu_path(pid, &path);
+ if let Some(Some(prior_window)) = activation {
+ let _ = crate::input::x11_activate_window_persistent(prior_window);
+ }
+ result
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(())) => ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",
+ )
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::LinuxAtSpiAction,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Foreground)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"
+ .into(),
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),
+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
+ }
+ }
+}
+
// ── set_window_frame (Linux) ─────────────────────────────────────────────────
pub struct SetWindowFrameTool;
@@ -7264,6 +7431,7 @@ pub fn build_registry_with_provider(
let pid_window_candidates: WindowTargetCandidates = Arc::new(pid_window_target_candidates);
r.register(pid_window_guarded(BringToFrontTool, &pid_window_candidates));
r.register(Box::new(SetWindowFrameTool));
+ r.register(Box::new(InvokeMenuTool));
r.register(pid_window_guarded(
ClickTool {
state: state.clone(),
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs
index 2a780d1cf9..bc8a05e093 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs
@@ -223,6 +223,7 @@ contract_stub_tool!(
SetAgentCursorThemeTool,
"set_agent_cursor_theme"
);
+contract_stub_tool!(invoke_menu_m, InvokeMenuTool, "invoke_menu");
stub_tool!(
check_perms_m,
@@ -324,6 +325,7 @@ pub fn build_registry() -> cua_driver_core::tool::ToolRegistry {
r.register(Box::new(PressKeyTool));
r.register(Box::new(HotkeyTool));
r.register(Box::new(SetValueTool));
+ r.register(Box::new(InvokeMenuTool));
r.register(Box::new(ScrollTool));
r.register(Box::new(ScreenshotTool));
r.register(Box::new(GetScreenSizeTool));
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
index a54b03aab7..f50f240905 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
@@ -466,6 +466,26 @@ pub unsafe fn focused_element_of_pid(pid: i32) -> Option<AXUIElementRef> {
Some(value as AXUIElementRef)
}
+/// Return the CGWindowID of the application's focused AX window.
+///
+/// This is a narrow read-only proof used before global keyboard delivery: an
+/// already focused exact window must not be re-activated, because doing so can
+/// make a focus-proxy renderer drop its current key target.
+pub fn focused_window_id_of_pid(pid: i32) -> Option<u32> {
+ unsafe {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ return None;
+ }
+ let window = copy_element_attr(app, "AXFocusedWindow");
+ CFRelease(app as CFTypeRef);
+ let window = window?;
+ let window_id = ax_get_window_id(window);
+ CFRelease(window as CFTypeRef);
+ window_id
+ }
+}
+
/// Get the children of an AX element.
///
/// # Safety
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
index 7696962e45..822b3866cc 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
@@ -146,8 +146,8 @@ fn role_supports_value_addressing(role: &str) -> bool {
)
}
-fn is_addressable(actions_present: bool, value_settable: bool) -> bool {
- actions_present || value_settable
+fn is_addressable(actions_present: bool, value_settable: bool, enabled: Option<bool>) -> bool {
+ (actions_present || value_settable) && enabled != Some(false)
}
pub struct TreeWalkResult {
@@ -458,7 +458,16 @@ unsafe fn walk_element(
let value_settable = actions.is_empty()
&& role_supports_value_addressing(&role)
&& is_attribute_settable(element, "AXValue");
- let is_actionable = is_addressable(!actions.is_empty(), value_settable);
+ // A closed submenu can keep its descendants in AXChildren while reporting
+ // those controls disabled. Never assign such a row a live element index:
+ // the same native state also causes dispatch to refuse it, and exposing an
+ // index for it invites agents to retain an unusable menu target.
+ let enabled = if !actions.is_empty() || value_settable {
+ copy_bool_attr(element, "AXEnabled")
+ } else {
+ None
+ };
+ let is_actionable = is_addressable(!actions.is_empty(), value_settable, enabled);
if !is_actionable && !has_content && role != "AXWindow" && role != "AXSheet" {
let children = copy_children(element);
@@ -497,7 +506,7 @@ unsafe fn walk_element(
.filter(|v| !v.is_empty()),
min_value: copy_number_attr(element, "AXMinValue"),
max_value: copy_number_attr(element, "AXMaxValue"),
- enabled: copy_bool_attr(element, "AXEnabled"),
+ enabled,
selected: copy_bool_attr(element, "AXSelected"),
});
let node = if is_actionable {
@@ -756,9 +765,10 @@ mod tests {
#[test]
fn writable_value_controls_are_addressable_without_actions() {
- assert!(is_addressable(false, true));
- assert!(is_addressable(true, false));
- assert!(!is_addressable(false, false));
+ assert!(is_addressable(false, true, Some(true)));
+ assert!(is_addressable(true, false, None));
+ assert!(!is_addressable(false, false, Some(true)));
+ assert!(!is_addressable(true, false, Some(false)));
for role in [
"AXTextField",
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
index a65072c42b..4fcd051231 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
@@ -490,6 +490,17 @@ pub fn with_foreground_hid_activation(
if !get_process_psn_for_window(target_wid, target_pid, &mut target_psn) {
anyhow::bail!("could not resolve target window for foreground HID delivery");
}
+
+ let focused_window_id = crate::ax::bindings::focused_window_id_of_pid(target_pid);
+ if preserves_exact_existing_focus(prev_ok, prev_psn, target_psn, focused_window_id, target_wid)
+ {
+ // Re-activating an already key exact window can clear Chromium's
+ // renderer focus even though WindowServer keeps the app frontmost.
+ // The AX window proof lets us deliver directly without weakening the
+ // exact-window guard or disturbing the current key target.
+ return action();
+ }
+
let activated = unsafe { set_front(target_psn.as_ptr() as *const c_void, target_wid, 0x400) };
if activated != 0 {
anyhow::bail!("WindowServer rejected foreground HID activation");
@@ -506,6 +517,18 @@ pub fn with_foreground_hid_activation(
result
}
+fn preserves_exact_existing_focus(
+ previous_process_known: bool,
+ previous_psn: [u8; 8],
+ target_psn: [u8; 8],
+ focused_window_id: Option<u32>,
+ target_window_id: u32,
+) -> bool {
+ previous_process_known
+ && previous_psn == target_psn
+ && focused_window_id == Some(target_window_id)
+}
+
/// Activate `target_pid`'s window `target_wid` for NSMenu key dispatch, run `action`,
/// then immediately restore the prior frontmost process.
///
@@ -556,3 +579,44 @@ pub fn with_menu_shortcut_activation(
result?;
Ok(true)
}
+
+#[cfg(test)]
+mod tests {
+ use super::preserves_exact_existing_focus;
+
+ #[test]
+ fn exact_existing_focus_avoids_reactivation() {
+ let psn = [1, 2, 3, 4, 5, 6, 7, 8];
+ assert!(preserves_exact_existing_focus(true, psn, psn, Some(42), 42));
+ }
+
+ #[test]
+ fn process_or_window_uncertainty_requires_guarded_activation() {
+ let target = [1, 2, 3, 4, 5, 6, 7, 8];
+ let other = [8, 7, 6, 5, 4, 3, 2, 1];
+ assert!(!preserves_exact_existing_focus(
+ false,
+ target,
+ target,
+ Some(42),
+ 42
+ ));
+ assert!(!preserves_exact_existing_focus(
+ true,
+ other,
+ target,
+ Some(42),
+ 42
+ ));
+ assert!(!preserves_exact_existing_focus(
+ true,
+ target,
+ target,
+ Some(41),
+ 42
+ ));
+ assert!(!preserves_exact_existing_focus(
+ true, target, target, None, 42
+ ));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index aec9ce0047..17dd6fcff3 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -118,16 +118,16 @@ fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "click".into(),
description:
- "Click against a target pid. **Prefer `element_index` over pixel \
- coordinates** — element_index works on backgrounded / minimized / hidden / \
- off-Space windows, surfaces a stable handle that survives rebuilds, and tells \
+ "Click against a target pid. **Prefer `element_token` over pixel \
+ coordinates** — the token works on backgrounded / minimized / hidden / \
+ off-Space windows, identifies one exact snapshot element, and tells \
you what you're clicking via the cached element's role + label. Reach for \
`x, y` only when the target is a canvas / video / WebGL / custom-drawn surface \
that doesn't appear in the AX tree.\n\n\
Two addressing modes:\n\n\
- - element_index + window_id (from last get_window_state): AX action path. \
+ - element_token, or element_index + snapshot_id (from get_window_state): AX action path. \
Works on backgrounded/hidden windows. No cursor move, no focus steal. \
- element_index cache is scoped per (pid, window_id) and is replaced by the \
+ The snapshot cache is scoped per (pid, window_id) and is replaced by the \
next snapshot of the same window — re-snapshot every turn before clicking.\n\n\
- x, y (window-local screenshot pixels, top-left origin of the PNG returned \
by get_window_state): CGEvent path. Synthesizes mouse events and posts to \
@@ -155,8 +155,9 @@ fn def() -> &'static ToolDef {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
"window_id": { "type": "integer", "description": "Target window ID. Required for element_index. Optional when element_token is supplied (the token carries it)." },
- "element_index": { "type": "integer", "description": "Element index from last get_window_state. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with \"Missing required integer field: pid\"; it is not a silent no-op." },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token` of the last get_window_state. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded — re-snapshot in that case." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x": { "type": "number", "description": "X in screenshot pixels, read straight off the image you were handed — no scaling math needed. With pid+window_id (capture_scope=window): window-local pixels from the get_window_state PNG (top-left origin). Windowless (no pid/window_id, capture_scope=desktop): pixels from the get_desktop_state PNG (the native full-display image). Either way, the pixel you read IS the pixel that gets clicked; the driver undoes the Retina backing scale + any downscale internally." },
"y": { "type": "number", "description": "Y in screenshot pixels (see x). Window-local from get_window_state, or full-display from get_desktop_state under capture_scope=desktop." },
"action": { "type": "string", "description": "AX action: press, show_menu, pick, confirm, cancel, open." },
@@ -337,6 +338,7 @@ impl Tool for ClickTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"click",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
index 6ae4f53313..b47de84c48 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
@@ -43,8 +43,9 @@ fn def() -> &'static ToolDef {
"x": { "type": "number", "description": "Screen X coordinate (pixel path)." },
"y": { "type": "number", "description": "Screen Y coordinate (pixel path)." },
"window_id": { "type": "integer", "description": "CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it)." },
- "element_index": { "type": "integer", "description": "Element index from last get_window_state. Uses AX path. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with \"Missing required integer field: pid\"; it is not a silent no-op." },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
"additionalProperties": false
@@ -82,6 +83,7 @@ impl Tool for DoubleClickTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"double_click",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index c4c5e218b0..dc7231bbc1 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -60,9 +60,11 @@ fn def() -> &'static ToolDef {
the requested WindowServer bounds. `px_frame_mismatch` or \
`px_capture_unavailable` omits an unprovable screenshot/pixel frame \
instead of guessing a transform; the truthful AX payload remains available.\n\n\
- Optional `query` filters the tree_markdown to matching lines plus their ancestor \
- chain (case-insensitive substring). The element_index values are unchanged — \
- filtering only trims the rendered Markdown.\n\n\
+ Optional `query` projects both tree_markdown and structured `elements` to \
+ matching lines plus their ancestor chain (case-insensitive substring). The \
+ element_index values are unchanged, the complete snapshot remains actionable, \
+ and `element_count` continues to report its total size; \
+ `filtered_element_count` reports the projected response size.\n\n\
Optional `max_elements` / `max_depth` bound the AX walk to mitigate \
context-window blow-up on Electron / Obsidian / large web apps that \
produce 10k+ element trees. When applied, BOTH the markdown \
@@ -75,7 +77,7 @@ fn def() -> &'static ToolDef {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
"window_id": { "type": "integer", "description": "Target window ID from list_windows." },
- "query": { "type": "string", "description": "Case-insensitive filter for tree_markdown." },
+ "query": { "type": "string", "description": "Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values." },
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
"include_screenshot": {
"type": "boolean",
@@ -506,6 +508,12 @@ impl Tool for GetWindowStateTool {
(None, Some(r)) if scope_matched => build_elements_array(&r.nodes),
_ => Vec::new(),
};
+ let elements_json = cua_driver_core::element_query::project_elements_for_query(
+ elements_json,
+ query.as_deref(),
+ &tree_md,
+ );
+ let filtered_element_count = elements_json.len();
// The structured array intentionally contains only actionable nodes,
// and AX child reads can fail independently of the element/depth caps.
// Until the walker exposes a proof over the projected search domain,
@@ -516,6 +524,8 @@ impl Tool for GetWindowStateTool {
"window_id": window_id,
"pid": pid,
"element_count": element_count,
+ "total_element_count": element_count,
+ "returned_element_count": filtered_element_count,
"elements_complete": elements_complete,
"tree_markdown": tree_md,
"elements": elements_json,
@@ -524,6 +534,9 @@ impl Tool for GetWindowStateTool {
Issue #22865: use `max_elements` / `max_depth` to bound the \
AX walk on apps with very large trees."
});
+ if query.is_some() {
+ structured["filtered_element_count"] = serde_json::json!(filtered_element_count);
+ }
// Surface 6: an opaque snapshot identifier consumers can log
// alongside the per-element tokens for debug correlation. Same value
// embedded in every `element_token` emitted in `elements[]` above.
@@ -847,6 +860,13 @@ pub(crate) fn build_elements_array(nodes: &[crate::ax::tree::AXNode]) -> Vec<ser
out
}
+/// Keep the structured response aligned with a query-filtered markdown tree.
+///
+/// The AX walker deliberately keeps the complete node/cache snapshot so the
+/// original element indices remain valid. The rendered markdown already holds
+/// the exact matching rows and ancestor chain, so use its indices as the
+/// projection source of truth instead of duplicating query matching over the
+/// structured fields.
#[cfg(test)]
mod window_scope_contract_tests {
use super::*;
@@ -968,6 +988,7 @@ mod window_scope_contract_tests {
mod tests {
use super::*;
use crate::ax::tree::AXNode;
+ use cua_driver_core::element_query::project_elements_for_query;
fn node(
idx: Option<usize>,
@@ -1047,6 +1068,62 @@ mod tests {
);
}
+ #[test]
+ fn query_projection_keeps_only_rendered_actionable_rows() {
+ let nodes = vec![
+ node(Some(0), "AXWindow", Some("Document"), 0, None, None),
+ node(Some(1), "AXMenuItem", Some("Window"), 1, Some(0), None),
+ node(
+ Some(2),
+ "AXMenuItem",
+ Some("Move & Resize"),
+ 2,
+ Some(1),
+ None,
+ ),
+ node(Some(3), "AXMenuItem", Some("Left"), 3, Some(2), None),
+ node(Some(4), "AXButton", Some("Unrelated"), 1, Some(0), None),
+ ];
+ let elements = build_elements_array(&nodes);
+ let filtered_markdown = concat!(
+ "- [0] AXWindow \"Document\"\n",
+ " - [1] AXMenuItem \"Window\"\n",
+ " - [2] AXMenuItem \"Move & Resize\"\n",
+ " - [3] AXMenuItem \"Left\"\n",
+ );
+
+ let projected = project_elements_for_query(elements, Some("Left"), filtered_markdown);
+ let indices: Vec<u64> = projected
+ .iter()
+ .map(|entry| entry["element_index"].as_u64().unwrap())
+ .collect();
+
+ assert_eq!(indices, vec![0, 1, 2, 3]);
+ }
+
+ #[test]
+ fn query_projection_returns_no_elements_when_markdown_has_no_match() {
+ let nodes = vec![node(Some(0), "AXButton", Some("Unrelated"), 0, None, None)];
+ let elements = build_elements_array(&nodes);
+
+ let projected = project_elements_for_query(elements, Some("zoomLeft"), "");
+
+ assert!(projected.is_empty());
+ }
+
+ #[test]
+ fn unfiltered_projection_preserves_every_element() {
+ let nodes = vec![
+ node(Some(0), "AXButton", Some("One"), 0, None, None),
+ node(Some(1), "AXButton", Some("Two"), 0, None, None),
+ ];
+ let elements = build_elements_array(&nodes);
+
+ let projected = project_elements_for_query(elements, None, "");
+
+ assert_eq!(projected.len(), 2);
+ }
+
#[test]
fn elements_shape_carries_role_label_frame_parent_depth() {
let nodes = vec![node(
@@ -1225,9 +1302,8 @@ mod tests {
assert_eq!(elements[2]["label"], "from-id");
}
- /// Surface 6: every element entry must carry a non-empty
- /// `element_token` alongside its integer `element_index`. The
- /// integer field stays unchanged — the token is purely additive.
+ /// Every element entry carries a non-empty snapshot-bound
+ /// `element_token` alongside its numeric `element_index`.
#[test]
fn build_elements_array_with_token_emits_element_token_per_row() {
let reg = cua_driver_core::element_token::global();
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs
new file mode 100644
index 0000000000..c41b738ecb
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs
@@ -0,0 +1,291 @@
+use async_trait::async_trait;
+use core_foundation::base::{CFRelease, CFTypeRef};
+use cua_driver_contract::InvokeMenuInput;
+use cua_driver_core::{
+ action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
+ EvidenceKind, RequestedDelivery,
+ },
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::Value;
+use std::time::Duration;
+
+use crate::ax::bindings::{
+ copy_action_names, copy_bool_attr, copy_children, copy_element_attr, copy_string_attr,
+ kAXErrorSuccess, perform_action, AXUIElementCreateApplication, AXUIElementRef,
+ AXUIElementSetMessagingTimeout,
+};
+
+pub struct InvokeMenuTool;
+
+const AX_MESSAGING_TIMEOUT_SECONDS: f32 = 2.0;
+
+unsafe fn set_messaging_timeout(element: AXUIElementRef) {
+ let _ = AXUIElementSetMessagingTimeout(element, AX_MESSAGING_TIMEOUT_SECONDS);
+}
+
+static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn def() -> &'static ToolDef {
+ DEF.get_or_init(|| {
+ let contract =
+ cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+}
+
+fn normalized_path(path: Vec<String>) -> Result<Vec<String>, String> {
+ if path.is_empty() || path.len() > 16 {
+ return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ }
+ path.into_iter()
+ .enumerate()
+ .map(|(index, segment)| {
+ let segment = segment.trim();
+ if segment.is_empty() {
+ Err(format!("invoke_menu: path segment {index} is empty"))
+ } else {
+ Ok(segment.to_owned())
+ }
+ })
+ .collect()
+}
+
+/// Return the semantic menu children of an AX node. AppKit inserts an
+/// untitled `AXMenu` container between a menu-bar/submenu item and its items;
+/// callers express paths in visible labels, so that container is transparent.
+unsafe fn semantic_children(parent: AXUIElementRef) -> Vec<AXUIElementRef> {
+ let mut out = Vec::new();
+ for child in copy_children(parent) {
+ if copy_string_attr(child, "AXRole").as_deref() == Some("AXMenu") {
+ out.extend(copy_children(child));
+ CFRelease(child as CFTypeRef);
+ } else {
+ out.push(child);
+ }
+ }
+ out
+}
+
+unsafe fn resolve_exact_prefix(
+ menu_bar: AXUIElementRef,
+ prefix: &[String],
+) -> Result<AXUIElementRef, String> {
+ let mut current = menu_bar;
+ let mut owns_current = false;
+
+ for (depth, segment) in prefix.iter().enumerate() {
+ let children = semantic_children(current);
+ if owns_current {
+ CFRelease(current as CFTypeRef);
+ }
+
+ let mut matches = Vec::new();
+ for child in children {
+ let title = copy_string_attr(child, "AXTitle").unwrap_or_default();
+ if title.trim() == segment {
+ matches.push(child);
+ } else {
+ CFRelease(child as CFTypeRef);
+ }
+ }
+
+ if matches.len() != 1 {
+ let match_count = matches.len();
+ for candidate in matches {
+ CFRelease(candidate as CFTypeRef);
+ }
+ return Err(if match_count == 0 {
+ format!("invoke_menu: path segment {depth} was not found")
+ } else {
+ format!("invoke_menu: path segment {depth} is ambiguous")
+ });
+ }
+ current = matches.pop().expect("one exact match");
+ owns_current = true;
+ }
+
+ if owns_current {
+ Ok(current)
+ } else {
+ Err("invoke_menu: path is empty".into())
+ }
+}
+
+fn choose_action(actions: &[String], final_segment: bool) -> Option<&'static str> {
+ let supports = |name: &str| actions.iter().any(|action| action == name);
+ let order: &[&str] = if final_segment {
+ &["AXPress", "AXPick", "AXConfirm"]
+ } else {
+ &["AXPress", "AXPick", "AXShowMenu", "AXOpen"]
+ };
+ order.iter().copied().find(|action| supports(action))
+}
+
+unsafe fn invoke_path(pid: i32, path: &[String]) -> Result<(), String> {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ return Err("invoke_menu: target application is unavailable".into());
+ }
+ set_messaging_timeout(app);
+
+ let result = (|| {
+ for depth in 0..path.len() {
+ // Resolve from the live app root for every hop. Opening a menu can
+ // replace its AX objects and reorder unrelated snapshot indices.
+ let menu_bar = copy_element_attr(app, "AXMenuBar")
+ .ok_or_else(|| "invoke_menu: target exposes no AXMenuBar".to_owned())?;
+ set_messaging_timeout(menu_bar);
+ let target = resolve_exact_prefix(menu_bar, &path[..=depth]);
+ CFRelease(menu_bar as CFTypeRef);
+ let target = target?;
+ set_messaging_timeout(target);
+
+ if copy_bool_attr(target, "AXEnabled") == Some(false) {
+ CFRelease(target as CFTypeRef);
+ return Err(format!("invoke_menu: path segment {depth} is disabled"));
+ }
+ let actions = copy_action_names(target);
+ let action = choose_action(&actions, depth + 1 == path.len()).ok_or_else(|| {
+ format!("invoke_menu: path segment {depth} has no usable native menu action")
+ });
+ let action = match action {
+ Ok(action) => action,
+ Err(error) => {
+ CFRelease(target as CFTypeRef);
+ return Err(error);
+ }
+ };
+ let error = perform_action(target, action);
+ CFRelease(target as CFTypeRef);
+ if error != kAXErrorSuccess {
+ return Err(format!(
+ "invoke_menu: native action for path segment {depth} failed with AX error {error}"
+ ));
+ }
+ if depth + 1 != path.len() {
+ std::thread::sleep(Duration::from_millis(80));
+ }
+ }
+ Ok(())
+ })();
+
+ CFRelease(app as CFTypeRef);
+ result
+}
+
+fn refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(serde_json::json!({
+ "status": "refused",
+ "refusal": { "code": "menu_path_unavailable", "message": message }
+ }))
+}
+
+#[async_trait]
+impl Tool for InvokeMenuTool {
+ fn def(&self) -> &ToolDef {
+ def()
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let input: InvokeMenuInput =
+ match cua_driver_core::tool_args::parse_typed_input("invoke_menu", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let path = match normalized_path(input.path) {
+ Ok(path) => path,
+ Err(error) => return refusal(error),
+ };
+ let pid = match i32::try_from(input.pid) {
+ Ok(pid) => pid,
+ Err(_) => return refusal("invoke_menu: pid is out of range".into()),
+ };
+ let window_id = match u32::try_from(input.window_id) {
+ Ok(window_id) => window_id,
+ Err(_) => return refusal("invoke_menu: window_id is out of range".into()),
+ };
+ if !crate::windows::all_windows()
+ .iter()
+ .any(|window| window.pid == pid && window.window_id == window_id)
+ {
+ return refusal("invoke_menu: window_id does not belong to pid".into());
+ }
+
+ let outcome = tokio::task::spawn_blocking(move || {
+ let prior_frontmost = crate::apps::frontmost_pid();
+ let needs_activation = prior_frontmost != Some(pid);
+ if needs_activation
+ && !crate::input::skylight::set_front_process_persistently(pid, window_id)
+ && !crate::apps::activate_pid(pid)
+ {
+ return Err("invoke_menu: target application could not be activated".into());
+ }
+ if needs_activation {
+ std::thread::sleep(Duration::from_millis(120));
+ }
+ let result = unsafe { invoke_path(pid, &path) };
+ if needs_activation {
+ if let Some(prior_pid) = prior_frontmost {
+ let _ = crate::apps::activate_pid(prior_pid);
+ }
+ }
+ result
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(())) => ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",
+ )
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::MacosAxAction,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Foreground)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and AX accepted the final action".into(),
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ Ok(Err(error)) => refusal(error),
+ Err(error) => refusal(format!("invoke_menu: blocking task failed: {error}")),
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn path_normalization_rejects_empty_segments() {
+ assert_eq!(
+ normalized_path(vec![" Window ".into(), " Left ".into()]).unwrap(),
+ vec!["Window", "Left"]
+ );
+ assert!(normalized_path(vec!["Window".into(), " ".into()]).is_err());
+ }
+
+ #[test]
+ fn action_selection_is_explicit_and_ordered() {
+ let actions = vec!["AXShowMenu".into(), "AXPress".into()];
+ assert_eq!(choose_action(&actions, false), Some("AXPress"));
+ assert_eq!(choose_action(&actions, true), Some("AXPress"));
+ assert_eq!(choose_action(&["AXShowMenu".into()], true), None);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index 2ddcb81b0c..ba22178f91 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -7,6 +7,7 @@ mod double_click;
mod drag;
mod get_window_state;
mod hotkey;
+mod invoke_menu;
mod kill_app;
mod launch_app;
mod list_apps;
@@ -730,6 +731,7 @@ pub fn register_all(
&pid_window_candidates,
));
registry.register(Box::new(set_window_frame::SetWindowFrameTool));
+ registry.register(Box::new(invoke_menu::InvokeMenuTool));
registry.register(pid_window_guarded(
click::ClickTool::new(state.clone()),
&pid_window_candidates,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index 6649bdf921..2e9f63a977 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -30,16 +30,14 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "press_key".into(),
- description: "Press and release a single key, delivered to the target pid via \
- CGEventPostToPid. Follows the same `delivery_mode` ladder as click/type_text \
+ description: "Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text \
— it does NOT raise the window by default:\n\
• `background` (default): post to the pid WITHOUT fronting/raising — the \
auth-message path (Chromium-safe). With element_index it focuses that AX \
element first. `window_id` only targets; it does not raise.\n\
- • `foreground`: briefly front the window (NSMenu path, < 1 ms) so native \
- menu key-equivalents dispatch, then restore prior frontmost — the explicit \
- escalation for menu shortcuts an app drops in the background. Requires \
- window_id (and no element_index).\n\n\
+ • `foreground`: guard and briefly front the exact window, send a genuine HID \
+ key transition so Chromium content and native menu equivalents both receive \
+ it, then restore prior frontmost. Requires window_id (and no element_index).\n\n\
A key press is never driver-verifiable → effect:\"unverifiable\"; confirm via \
screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, \
home, end, pageup, pagedown, f1-f12, plus any letter or digit. \
@@ -57,8 +55,9 @@ fn def() -> &'static ToolDef {
"description": "Modifier keys: cmd, shift, option/alt, ctrl, fn."
},
"window_id": { "type": "integer", "description": "Target window. Required for delivery_mode:\"foreground\". Does NOT itself raise the window — raising is gated on delivery_mode." },
- "element_index": { "type": "integer" },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x": { "type": "number", "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index." },
"y": { "type": "number", "description": "Screenshot-pixel Y (see x)." },
"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the key to the frontmost application." },
@@ -125,6 +124,7 @@ impl Tool for PressKeyTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"press_key",
) {
@@ -234,16 +234,19 @@ impl Tool for PressKeyTool {
tokio::task::spawn_blocking(move || {
let m: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- // foreground rung: NSMenu activation (raise+restore), only when
- // explicitly requested and addressing a window without an element.
- // Skipped when px-focus already fronted/clicked the target.
+ // Foreground rung: keep the exact target frontmost through a genuine
+ // physical HID key down/up pair, then restore. PID-routed events without the
+ // authentication envelope reach NSMenu, but Chromium/Electron may
+ // silently discard them even while frontmost; the guarded HID route
+ // is accepted by both. Skipped when px-focus already handled the
+ // target-specific foreground transition.
if fg && !px_focus {
if let Some(wid) = window_id {
if element_index.is_none() {
- crate::input::skylight::with_menu_shortcut_activation(
+ crate::input::skylight::with_foreground_hid_activation(
pid as libc::pid_t,
wid,
- || crate::input::keyboard::press_key_no_auth(pid, &key, &m),
+ || crate::input::keyboard::press_key_bare_global(&key, &m),
)?;
return Ok(());
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
index 6abbe8f9fd..28787bbaea 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
@@ -45,14 +45,9 @@ fn def() -> &'static ToolDef {
"properties": {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
- "element_index": {
- "type": "integer",
- "description": "Element index from last get_window_state. Routes through AXShowMenu. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with \"Missing required integer field: pid\"; it is not a silent no-op."
- },
- "element_token": {
- "type": "string",
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded."
- },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"window_id": {
"type": "integer",
"description": "CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it)."
@@ -108,6 +103,7 @@ impl Tool for RightClickTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"right_click",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
index 986dbed249..0930888257 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
@@ -90,8 +90,9 @@ fn def() -> &'static ToolDef {
"description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Default: 3."
},
"window_id": { "type": "integer" },
- "element_index": { "type": "integer", "description": "Element from last get_window_state. Routes through the pixel-wheel path AT this element's center — use it to scroll a nested overflow region you located in the AX tree." },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x": { "type": "number", "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion." },
"y": { "type": "number", "description": "Window-local screenshot Y. See `x`." },
"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates." },
@@ -185,6 +186,7 @@ impl Tool for ScrollTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"scroll",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
index 3617c67e3b..5d8c8595cb 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
@@ -70,8 +70,9 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used; optional when element_token is supplied (the token carries it)."
},
- "element_index": { "type": "integer", "description": "Element index from last get_window_state. Must be supplied unless element_token is provided. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with \"Missing required integer field: pid\"; it is not a silent no-op." },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"value": {
"type": "string",
"description": "New value. AX will coerce to the element's native type."
@@ -113,6 +114,7 @@ impl Tool for SetValueTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"set_value",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 3cd684cb42..893e03009e 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -93,14 +93,9 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it)."
},
- "element_index": {
- "type": "integer",
- "description": "Element index from last get_window_state. Directs the write to a specific field. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with \"Missing required integer field: pid\"; it is not a silent no-op."
- },
- "element_token": {
- "type": "string",
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded."
- },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x": { "type": "number", "description": "Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click." },
"y": { "type": "number", "description": "Screenshot-pixel Y of the field (see x)." },
"delay_ms": {
@@ -208,6 +203,7 @@ impl Tool for TypeTextTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"type_text",
) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
index 5edbb1c8e2..0f390c9878 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
@@ -34,8 +34,9 @@ fn def() -> &'static ToolDef {
"text": { "type": "string", "description": "Text to type." },
"delay_ms": { "type": "integer", "description": "Milliseconds between characters (default 30)." },
"window_id": { "type": "integer", "description": "Window ID for element focus. Optional when element_token is supplied." },
- "element_index": { "type": "integer", "description": "Element to focus before typing." },
- "element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"type_chars_only": { "type": "boolean", "description": "Skip AX focus, type directly. Default false." }
},
"additionalProperties": false
@@ -76,6 +77,7 @@ impl Tool for TypeTextCharsTool {
pid,
element_index_arg,
element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
window_id_arg,
"type_text_chars",
) {
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index a416756c55..2927ff2994 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -238,8 +238,8 @@ async fn track_overlay_drag(
}
use cua_driver_contract::{
ClickButton, ClickInput, DragInput, GetCursorPositionInput, GetDesktopStateInput,
- GetScreenSizeInput, HotkeyInput, MoveCursorInput, PressKeyInput, ScrollDirection, ScrollInput,
- TypeTextInput,
+ GetScreenSizeInput, HotkeyInput, InvokeMenuInput, MoveCursorInput, PressKeyInput,
+ ScrollDirection, ScrollInput, TypeTextInput,
};
use cua_driver_core::{
protocol::ToolResult,
@@ -906,9 +906,10 @@ impl Tool for GetWindowStateTool {
picks a window implicitly.\n\n\
`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. \
The driver does not auto-fall-back to a different window.\n\n\
- Set `query` to a case-insensitive substring to filter `tree_markdown` down to \
- matching lines plus their ancestor chain. element_index values and \
- `element_count` are unchanged — the filter only trims the rendered Markdown.\n\n\
+ Set `query` to a case-insensitive substring to project BOTH `tree_markdown` \
+ and `structuredContent.elements` to matching rows plus their ancestor chain. \
+ Original element indices are preserved. `total_element_count` reports the \
+ complete snapshot; `returned_element_count` reports the projection.\n\n\
Always returns BOTH the element tree AND a screenshot — ground on both \
and cross-check (the tree lies on some surfaces). Choose the modality at \
ACTION time: an element ax action (element_index/element_token → \
@@ -938,7 +939,7 @@ impl Tool for GetWindowStateTool {
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},
- "query":{"type":"string","description":"Optional case-insensitive substring. When set, `tree_markdown` only contains lines that match plus their ancestor chain; element indices and `element_count` are unchanged."},
+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
"max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."}
},"additionalProperties":false}),
@@ -1209,6 +1210,13 @@ impl Tool for GetWindowStateTool {
Some(entry)
})
.collect();
+ let elements = cua_driver_core::element_query::project_elements_for_query(
+ elements,
+ query.as_deref(),
+ &tr.tree_markdown,
+ );
+ structured["total_element_count"] = json!(count);
+ structured["returned_element_count"] = json!(elements.len());
structured["elements"] = json!(elements);
// Surface 6: snapshot id mirror for debug correlation.
if let Some(snapshot_id) = snapshot_id {
@@ -2621,6 +2629,7 @@ impl Tool for ClickTool {
"window_id":{"type":"integer","description":"HWND for the window whose get_window_state produced the element_index. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"X in window-local screenshot pixels — same space as the PNG get_window_state returns. Must be provided together with y."},
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"button": cua_driver_core::tool_schema::button_schema(),
@@ -2749,6 +2758,7 @@ impl Tool for ClickTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"click",
) {
@@ -3609,6 +3619,7 @@ impl Tool for TypeTextTool {
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
"element_index":{"type":"integer","description":"Element index from the last get_window_state for the same (pid, window_id). When supplied, type_text (1) writes via UIA ValuePattern.SetValue on that element (works for WPF/WinForms/UWP/XAML without focus steal) and (2) confirms by reading that element's value back by handle — focus-independent. A matching read-back produces effect:\"confirmed\" with value_readback evidence; an unchanged or unreadable value remains effect:\"unverifiable\". Strongly preferred over typing into 'whatever is focused' (no element_index), which falls back to PostMessage WM_CHAR plus a foreground-only focused-element read-back. Requires window_id."},
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Window-local screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the UIA/WM_CHAR path can't reach. Read straight off the get_window_state PNG, same convention as click."},
"y":{"type":"number","description":"Window-local screenshot-pixel Y of the field (see x)."},
"delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."},
@@ -3676,6 +3687,7 @@ impl Tool for TypeTextTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"type_text",
) {
@@ -4339,7 +4351,8 @@ impl Tool for PressKeyTool {
"key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},
"modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},
"element_index":{"type":"integer","description":"Optional element_index from the last get_window_state; focuses that UIA element before the key is delivered."},
- "element_token":{"type":"string","description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the UIA path can't focus. Pass with y, no element_index."},
"y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
"window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},
@@ -4400,6 +4413,7 @@ impl Tool for PressKeyTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"press_key",
) {
@@ -5016,6 +5030,7 @@ impl Tool for SetValueTool {
"window_id":{"type":"integer","description":"HWND of the window. Required when element_index is used; optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"value":{"type":"string","description":"New value. UIA will coerce to the element's native type."}
},"additionalProperties":false
}),
@@ -5043,6 +5058,7 @@ impl Tool for SetValueTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"set_value",
) {
@@ -5214,6 +5230,7 @@ impl Tool for ScrollTool {
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},
"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
@@ -5297,6 +5314,7 @@ impl Tool for ScrollTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"scroll",
) {
@@ -5779,6 +5797,7 @@ impl Tool for DoubleClickTool {
"window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"modifier": cua_driver_core::tool_schema::modifier_schema(),
@@ -5802,6 +5821,7 @@ impl Tool for DoubleClickTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"double_click",
) {
@@ -6117,6 +6137,7 @@ impl Tool for RightClickTool {
"window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"modifier": cua_driver_core::tool_schema::modifier_schema(),
@@ -6140,6 +6161,7 @@ impl Tool for RightClickTool {
pid as i32,
args.opt_u64("element_index").map(|v| v as usize),
args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
args.opt_u64("window_id").map(|v| v as u32),
"right_click",
) {
@@ -8002,6 +8024,164 @@ impl Tool for TypeTextCharsTool {
// keyed on pid. Marked `destructive: true` so MCP clients with permission
// gating prompt before invoking.
+// ── invoke_menu ───────────────────────────────────────────────────────────
+
+pub struct InvokeMenuTool;
+static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
+ if path.is_empty() || path.len() > 16 {
+ return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ }
+ path.into_iter()
+ .enumerate()
+ .map(|(index, segment)| {
+ let segment = segment.trim();
+ if segment.is_empty() {
+ Err(format!("invoke_menu: path segment {index} is empty"))
+ } else {
+ Ok(segment.to_owned())
+ }
+ })
+ .collect()
+}
+
+fn menu_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": { "code": "menu_path_unavailable", "message": message }
+ }))
+}
+
+/// Activate a window for a visible native-menu operation. Windows may reject
+/// the first foreground request after the test/user session has been idle even
+/// when we attach to the current foreground thread. A reserved no-name key has
+/// no application meaning but makes this process the owner of the most recent
+/// input, allowing a bounded retry without sending a real shortcut or text to
+/// whichever application currently has focus.
+fn activate_window_for_menu(hwnd: windows::Win32::Foundation::HWND) -> Result<bool, String> {
+ use windows::Win32::UI::Input::KeyboardAndMouse::{
+ keybd_event, KEYBD_EVENT_FLAGS, KEYEVENTF_KEYUP,
+ };
+
+ if unsafe { crate::input::force_foreground_attached(hwnd) } {
+ return Ok(false);
+ }
+
+ const VK_NONAME: u8 = 0xFC;
+ unsafe {
+ keybd_event(VK_NONAME, 0, KEYBD_EVENT_FLAGS(0), 0);
+ keybd_event(VK_NONAME, 0, KEYEVENTF_KEYUP, 0);
+ }
+ for _ in 0..3 {
+ if unsafe { crate::input::force_foreground_attached(hwnd) } {
+ return Ok(true);
+ }
+ std::thread::sleep(std::time::Duration::from_millis(25));
+ }
+ Err("invoke_menu: target window could not be activated".to_owned())
+}
+
+#[async_trait]
+impl Tool for InvokeMenuTool {
+ fn def(&self) -> &ToolDef {
+ INVOKE_MENU_DEF.get_or_init(|| {
+ let contract =
+ cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
+ EvidenceKind, RequestedDelivery,
+ };
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow,
+ };
+
+ let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let path = match normalized_menu_path(input.path) {
+ Ok(path) => path,
+ Err(error) => return menu_refusal(error),
+ };
+ let hwnd_value = input.window_id;
+ let hwnd = HWND(hwnd_value as usize as *mut _);
+ if !unsafe { IsWindow(hwnd) }.as_bool() {
+ return menu_refusal("invoke_menu: window_id is closed, stale, or invalid".into());
+ }
+ let mut owner_pid = 0_u32;
+ unsafe { GetWindowThreadProcessId(hwnd, Some(&mut owner_pid)) };
+ if owner_pid != input.pid {
+ return menu_refusal("invoke_menu: window_id does not belong to pid".into());
+ }
+
+ let outcome = tokio::task::spawn_blocking(move || {
+ // `HWND` wraps a raw pointer and is intentionally not `Send`.
+ // Move the integer handle into the blocking worker and rebuild the
+ // process-local wrapper there instead of carrying it across threads.
+ let hwnd = HWND(hwnd_value as usize as *mut _);
+ let prior = unsafe { GetForegroundWindow() };
+ let needs_activation = prior != hwnd;
+ let used_activation_retry = if needs_activation {
+ activate_window_for_menu(hwnd)?
+ } else {
+ false
+ };
+ if needs_activation {
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ }
+ let result = crate::uia::invoke_menu_path(hwnd_value, &path);
+ if needs_activation && !prior.0.is_null() {
+ let _ = unsafe { crate::input::force_foreground_attached(prior) };
+ }
+ result.map(|()| used_activation_retry)
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(used_activation_retry)) => ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",
+ )
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::WindowsUiaInvoke,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Foreground)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and UI Automation accepted the final action"
+ .into(),
+ })
+ .detail(if used_activation_retry {
+ "target activation required a bounded reserved-key foreground-lock retry"
+ } else {
+ "target was already foreground or activated without a foreground-lock retry"
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ Ok(Err(error)) => menu_refusal(error),
+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
+ }
+ }
+}
+
// ── set_window_frame ──────────────────────────────────────────────────────
pub struct SetWindowFrameTool;
@@ -8886,6 +9066,7 @@ pub fn build_registry_with_provider(
let pid_window_candidates: WindowTargetCandidates = Arc::new(pid_window_target_candidates);
r.register(pid_window_guarded(BringToFrontTool, &pid_window_candidates));
r.register(Box::new(SetWindowFrameTool));
+ r.register(Box::new(InvokeMenuTool));
r.register(Box::new(DebugWindowInfoTool));
r.register(pid_window_guarded(
ClickTool {
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs
index d81eb02971..d87386eca2 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs
@@ -113,7 +113,7 @@ stub_tool!(
ClickTool,
"click",
"Click at (x, y) or on an AX/UIA element by element_index + window_id.",
- serde_json::json!({"type":"object","required":[],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},"window_id":{"type":"integer"},"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"x":{"type":"number"},"y":{"type":"number"},"button": cua_driver_core::tool_schema::button_schema(),"modifier": cua_driver_core::tool_schema::modifier_schema(),"from_zoom":{"type":"boolean"}},"additionalProperties":false})
+ serde_json::json!({"type":"object","required":[],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},"window_id":{"type":"integer"},"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),"x":{"type":"number"},"y":{"type":"number"},"button": cua_driver_core::tool_schema::button_schema(),"modifier": cua_driver_core::tool_schema::modifier_schema(),"from_zoom":{"type":"boolean"}},"additionalProperties":false})
);
stub_tool!(
@@ -212,6 +212,7 @@ contract_stub_tool!(
SetAgentCursorThemeTool,
"set_agent_cursor_theme"
);
+contract_stub_tool!(invoke_menu_m, InvokeMenuTool, "invoke_menu");
stub_tool!(
check_perms_m,
@@ -316,6 +317,7 @@ pub fn build_registry() -> cua_driver_core::tool::ToolRegistry {
r.register(Box::new(PressKeyTool));
r.register(Box::new(HotkeyTool));
r.register(Box::new(SetValueTool));
+ r.register(Box::new(InvokeMenuTool));
r.register(Box::new(ScrollTool));
r.register(Box::new(ScreenshotTool));
r.register(Box::new(GetScreenSizeTool));
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
index 2eb71bcb06..193d89809d 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
@@ -15,6 +15,7 @@ use windows::Win32::System::Com::{
};
use windows::Win32::UI::Accessibility::{
CUIAutomation, IUIAutomation, IUIAutomationCacheRequest, IUIAutomationElement,
+ IUIAutomationExpandCollapsePattern, IUIAutomationInvokePattern,
IUIAutomationSelectionItemPattern, IUIAutomationTogglePattern, ToggleState_Off, ToggleState_On,
TreeScope_Children, TreeScope_Subtree, UIA_AutomationIdPropertyId,
UIA_BoundingRectanglePropertyId, UIA_ControlTypePropertyId, UIA_ExpandCollapsePatternId,
@@ -102,6 +103,132 @@ pub fn walk_tree(hwnd: u64, query: Option<&str>) -> UiaTreeResult {
walk_tree_bounded(hwnd, query, DEFAULT_MAX_TOTAL_ELEMENTS, DEFAULT_MAX_DEPTH)
}
+fn is_menu_control(control_type: &str) -> bool {
+ matches!(control_type, "Menu" | "MenuBar" | "MenuItem")
+}
+
+fn exact_menu_path_matches(nodes: &[UiaNode], path: &[String]) -> Vec<usize> {
+ let by_element_index: std::collections::HashMap<usize, usize> = nodes
+ .iter()
+ .enumerate()
+ .filter_map(|(node_index, node)| node.element_index.map(|index| (index, node_index)))
+ .collect();
+
+ nodes
+ .iter()
+ .enumerate()
+ .filter_map(|(node_index, node)| {
+ if !is_menu_control(&node.control_type)
+ || node.enabled == Some(false)
+ || node.name.as_deref().map(str::trim) != path.last().map(String::as_str)
+ {
+ return None;
+ }
+ let mut lineage = Vec::new();
+ let mut cursor = Some(node_index);
+ while let Some(current) = cursor {
+ let ancestor = &nodes[current];
+ if is_menu_control(&ancestor.control_type) {
+ if let Some(name) = ancestor.name.as_deref().map(str::trim) {
+ if !name.is_empty() {
+ lineage.push(name);
+ }
+ }
+ }
+ cursor = ancestor
+ .parent_element_index
+ .and_then(|index| by_element_index.get(&index).copied());
+ }
+ lineage.reverse();
+ lineage.dedup();
+ (lineage == path.iter().map(String::as_str).collect::<Vec<_>>()).then_some(node_index)
+ })
+ .collect()
+}
+
+unsafe fn release_walk_nodes(nodes: Vec<UiaNode>) {
+ use windows::Win32::UI::Accessibility::IAccessible;
+ for node in nodes {
+ if node.element_ptr == 0 {
+ continue;
+ }
+ if node.msaa_role.is_some() {
+ drop(IAccessible::from_raw(node.element_ptr as *mut _));
+ } else {
+ drop(IUIAutomationElement::from_raw(node.element_ptr as *mut _));
+ }
+ }
+}
+
+unsafe fn invoke_menu_element(element_ptr: usize, final_segment: bool) -> Result<(), String> {
+ let element =
+ std::mem::ManuallyDrop::new(IUIAutomationElement::from_raw(element_ptr as *mut _));
+ if !final_segment {
+ if let Ok(pattern) = element.GetCurrentPattern(UIA_ExpandCollapsePatternId) {
+ if let Ok(expand) = pattern.cast::<IUIAutomationExpandCollapsePattern>() {
+ return expand
+ .Expand()
+ .map_err(|error| format!("ExpandCollapse.Expand failed: {error}"));
+ }
+ }
+ }
+ if let Ok(pattern) = element.GetCurrentPattern(UIA_InvokePatternId) {
+ if let Ok(invoke) = pattern.cast::<IUIAutomationInvokePattern>() {
+ return invoke
+ .Invoke()
+ .map_err(|error| format!("InvokePattern.Invoke failed: {error}"));
+ }
+ }
+ if final_segment {
+ if let Ok(pattern) = element.GetCurrentPattern(UIA_SelectionItemPatternId) {
+ if let Ok(selection) = pattern.cast::<IUIAutomationSelectionItemPattern>() {
+ return selection
+ .Select()
+ .map_err(|error| format!("SelectionItem.Select failed: {error}"));
+ }
+ }
+ }
+ Err("element exposes no usable native menu pattern".into())
+}
+
+/// Resolve and invoke an exact application menu path from fresh UIA state at
+/// every hop. No cached element index survives a menu mutation.
+pub fn invoke_menu_path(hwnd: u64, path: &[String]) -> Result<(), String> {
+ for depth in 0..path.len() {
+ let result = walk_tree(hwnd, None);
+ let matches = exact_menu_path_matches(&result.nodes, &path[..=depth]);
+ let target_index = match matches.as_slice() {
+ [index] => *index,
+ [] => {
+ unsafe { release_walk_nodes(result.nodes) };
+ return Err(format!("menu path segment {depth} was not found"));
+ }
+ _ => {
+ unsafe { release_walk_nodes(result.nodes) };
+ return Err(format!("menu path segment {depth} is ambiguous"));
+ }
+ };
+ let target = &result.nodes[target_index];
+ let error = if target.enabled == Some(false) {
+ Some(format!("menu path segment {depth} is disabled"))
+ } else if target.msaa_role.is_some() {
+ Some(format!(
+ "menu path segment {depth} is exposed only through MSAA, not UI Automation"
+ ))
+ } else {
+ unsafe { invoke_menu_element(target.element_ptr, depth + 1 == path.len()) }.err()
+ };
+ unsafe { release_walk_nodes(result.nodes) };
+ if let Some(error) = error {
+ return Err(format!("menu path segment {depth}: {error}"));
+ }
+ if depth + 1 != path.len() {
+ std::thread::sleep(std::time::Duration::from_millis(80));
+ }
+ }
+ Ok(())
+}
+
/// Walk the UIA tree with caller-supplied caps. `max_elements`/`max_depth`
/// truncate the walk and the rendered markdown identically. Issue #22865:
/// caps protect against Electron / large web apps that produce 10k+
diff --git a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
index f0643a3f39..bd3ea9e7e8 100755
--- a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
+++ b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
@@ -55,6 +55,22 @@ class HarnessWindow(Gtk.Window):
root.set_border_width(12)
scroller.add(root)
+ # Native application menu used by invoke_menu E2E. The three-level
+ # hierarchy exercises live path re-resolution after each submenu opens.
+ menubar = Gtk.MenuBar()
+ window_item = Gtk.MenuItem(label="Window")
+ window_menu = Gtk.Menu()
+ arrange_item = Gtk.MenuItem(label="Arrange")
+ arrange_menu = Gtk.Menu()
+ left_item = Gtk.MenuItem(label="Left")
+ left_item.connect("activate", self.on_native_menu_left)
+ arrange_menu.append(left_item)
+ arrange_item.set_submenu(arrange_menu)
+ window_menu.append(arrange_item)
+ window_item.set_submenu(window_menu)
+ menubar.append(window_item)
+ root.pack_start(menubar, False, False, 0)
+
# ── counter ───────────────────────────────────────────────────────
self.counter_label = Gtk.Label(label="counter=0", xalign=0)
root.pack_start(self.counter_label, False, False, 0)
@@ -210,6 +226,10 @@ class HarnessWindow(Gtk.Window):
self._menu_action = lbl
self.ctx_status.set_text(f"menu_action={lbl}")
+ def on_native_menu_left(self, _w):
+ self._menu_action = "window_arrange_left"
+ self.ctx_status.set_text("menu_action=window_arrange_left")
+
def on_key_press(self, _w, ev):
key = (Gdk.keyval_name(ev.keyval) or "unknown").lower()
ctrl = bool(ev.state & Gdk.ModifierType.CONTROL_MASK)
diff --git a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
index 7961427bd5..ced302f9d8 100644
--- a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
+++ b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
@@ -363,6 +363,10 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate {
menuActionLabel.stringValue = "menu_action=\(sender.title)"
}
+ @objc func onArrangeLeft(_ sender: NSMenuItem) {
+ menuActionLabel.stringValue = "menu_action=window_arrange_left"
+ }
+
func controlTextDidChange(_ obj: Notification) {
guard let field = obj.object as? NSTextField else { return }
if field === textInput {
@@ -405,7 +409,7 @@ final class ClickTargetButton: NSButton {
// MARK: - Menu bar (Mac-specific scenario: ns_menubar)
-func installMenuBar() {
+func installMenuBar(target: HarnessWindowController) {
let main = NSMenu()
let appItem = NSMenuItem()
main.addItem(appItem)
@@ -418,7 +422,25 @@ func installMenuBar() {
action: #selector(NSApplication.terminate(_:)),
keyEquivalent: "q"))
appItem.submenu = appMenu
+
+ let windowItem = NSMenuItem(title: "Window", action: nil, keyEquivalent: "")
+ let windowMenu = NSMenu(title: "Window")
+ let arrangeItem = NSMenuItem(title: "Arrange", action: nil, keyEquivalent: "")
+ let arrangeMenu = NSMenu(title: "Arrange")
+ let leftItem = NSMenuItem(
+ title: "Left",
+ action: #selector(HarnessWindowController.onArrangeLeft(_:)),
+ keyEquivalent: ""
+ )
+ leftItem.target = target
+ leftItem.setAccessibilityIdentifier("menu-window-arrange-left")
+ arrangeMenu.addItem(leftItem)
+ arrangeItem.submenu = arrangeMenu
+ windowMenu.addItem(arrangeItem)
+ windowItem.submenu = windowMenu
+ main.addItem(windowItem)
NSApp.mainMenu = main
+ NSApp.windowsMenu = windowMenu
}
// MARK: - Entry
@@ -428,8 +450,8 @@ struct CuaAppKitHarness {
static func main() {
let app = NSApplication.shared
app.setActivationPolicy(.regular)
- installMenuBar()
let controller = HarnessWindowController()
+ installMenuBar(target: controller)
controller.show()
app.activate(ignoringOtherApps: true)
app.run()
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
index a117d288df..eb1f4dfef1 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
@@ -102,6 +102,15 @@
AutomationProperties.AutomationId="menu-edit-copy"
Click="OnMenuEditCopy"/>
</MenuItem>
+ <MenuItem Header="_Window"
+ AutomationProperties.AutomationId="menu-window">
+ <MenuItem Header="_Arrange"
+ AutomationProperties.AutomationId="menu-window-arrange">
+ <MenuItem Header="_Left"
+ AutomationProperties.AutomationId="menu-window-arrange-left"
+ Click="OnMenuWindowArrangeLeft"/>
+ </MenuItem>
+ </MenuItem>
</Menu>
<Button x:Name="BtnContextTarget"
AutomationProperties.AutomationId="btn-context-target"
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
index 9a66907483..155fbeebe8 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
@@ -276,6 +276,7 @@ public partial class MainWindow : Window
private void OnMenuFileNew(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=file_new";
private void OnMenuFileOpen(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=file_open";
private void OnMenuEditCopy(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=edit_copy";
+ private void OnMenuWindowArrangeLeft(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=window_arrange_left";
private void OnCtxAction(object sender, RoutedEventArgs e)
{
diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
index 222f26bddd..bec6290e08 100644
--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
@@ -1875,6 +1875,65 @@ const FfiConverterTypeHotkeyInput = (() => {
return new FFIConverter();
})();
+/**
+ * Exact, immediate-child application menu path to resolve and invoke through
+ * the operating system's accessibility API. Path labels are matched after
+ * trimming surrounding whitespace and otherwise remain case-sensitive.
+ */
+export type InvokeMenuInput = {
+ pid: number,
+ windowId: bigint,
+ path: Array<string>,
+ /**
+ * Optional session id.
+ */
+ session?: string
+}
+
+/**
+ * Generated factory for {@link InvokeMenuInput} record objects.
+ */
+export const InvokeMenuInput = (() => {
+ const defaults = () => ({
+ });
+ const create = (() => {
+ return uniffiCreateRecord<InvokeMenuInput, ReturnType<typeof defaults>>(defaults);
+ })();
+ return Object.freeze({
+ create,
+ new: create,
+ defaults: () => Object.freeze(defaults()) as Partial<InvokeMenuInput>,
+ });
+})();
+
+const FfiConverterTypeInvokeMenuInput = (() => {
+ type TypeName = InvokeMenuInput;
+ class FFIConverter extends AbstractFfiConverterByteArray<TypeName> {
+ read(from: RustBuffer): TypeName {
+ return {
+ pid: FfiConverterUInt32.read(from),
+ windowId: FfiConverterUInt64.read(from),
+ path: FfiConverterSequenceString.read(from),
+ session: FfiConverterOptionalString.read(from)
+ };
+ }
+ write(value: TypeName, into: RustBuffer): void {
+ FfiConverterUInt32.write(value.pid, into);
+ FfiConverterUInt64.write(value.windowId, into);
+ FfiConverterSequenceString.write(value.path, into);
+ FfiConverterOptionalString.write(value.session, into);
+ }
+ allocationSize(value: TypeName): number {
+ return FfiConverterUInt32.allocationSize(value.pid) +
+ FfiConverterUInt64.allocationSize(value.windowId) +
+ FfiConverterSequenceString.allocationSize(value.path) +
+ FfiConverterOptionalString.allocationSize(value.session);
+
+ }
+ };
+ return new FFIConverter();
+})();
+
export type MoveCursorInput = {
x: number,
y: number,
@@ -3276,6 +3335,7 @@ export default Object.freeze({
FfiConverterTypeGetScreenSizeInput,
FfiConverterTypeGetSessionStateInput,
FfiConverterTypeHotkeyInput,
+ FfiConverterTypeInvokeMenuInput,
FfiConverterTypeMoveCursorInput,
FfiConverterTypePlatform,
FfiConverterTypePredicateOutcome,
diff --git a/libs/cua-driver/typescript/src/native/cua_driver_sdk-ffi.ts b/libs/cua-driver/typescript/src/native/cua_driver_sdk-ffi.ts
index 2f85eefb17..32cc75651f 100644
--- a/libs/cua-driver/typescript/src/native/cua_driver_sdk-ffi.ts
+++ b/libs/cua-driver/typescript/src/native/cua_driver_sdk-ffi.ts
@@ -483,6 +483,11 @@ const DEFINITIONS = {
ret: FfiType.Handle,
hasRustCallStatus: false,
},
+ "uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu": {
+ args: [FfiType.Handle, FfiType.RustBuffer],
+ ret: FfiType.Handle,
+ hasRustCallStatus: false,
+ },
"uniffi_cua_driver_sdk_fn_method_cuadriver_is_available": {
args: [FfiType.Handle],
ret: FfiType.Int8,
@@ -633,6 +638,11 @@ const DEFINITIONS = {
ret: FfiType.Handle,
hasRustCallStatus: false,
},
+ "uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu": {
+ args: [FfiType.Handle, FfiType.RustBuffer],
+ ret: FfiType.Handle,
+ hasRustCallStatus: false,
+ },
"uniffi_cua_driver_sdk_fn_method_cuadriversession_move_cursor": {
args: [FfiType.Handle, FfiType.RustBuffer],
ret: FfiType.Handle,
@@ -898,6 +908,11 @@ const DEFINITIONS = {
ret: FfiType.UInt16,
hasRustCallStatus: false,
},
+ "uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu": {
+ args: [],
+ ret: FfiType.UInt16,
+ hasRustCallStatus: false,
+ },
"uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available": {
args: [],
ret: FfiType.UInt16,
@@ -1048,6 +1063,11 @@ const DEFINITIONS = {
ret: FfiType.UInt16,
hasRustCallStatus: false,
},
+ "uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu": {
+ args: [],
+ ret: FfiType.UInt16,
+ hasRustCallStatus: false,
+ },
"uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor": {
args: [],
ret: FfiType.UInt16,
@@ -1313,6 +1333,7 @@ interface NativeModuleInterface {
uniffi_cua_driver_sdk_fn_method_cuadriver_get_screen_size(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriver_get_session_state(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriver_hotkey(uniffiSelf: bigint, input: Uint8Array): bigint;
+ uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriver_is_available(uniffiSelf: bigint, uniffi_out_err: UniffiRustCallStatus): number;
uniffi_cua_driver_sdk_fn_method_cuadriver_list_tools_json(uniffiSelf: bigint): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriver_metadata(uniffiSelf: bigint): bigint;
@@ -1343,6 +1364,7 @@ interface NativeModuleInterface {
uniffi_cua_driver_sdk_fn_method_cuadriversession_get_screen_size(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriversession_get_session_state(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriversession_hotkey(uniffiSelf: bigint, input: Uint8Array): bigint;
+ uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriversession_move_cursor(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriversession_press_key(uniffiSelf: bigint, input: Uint8Array): bigint;
uniffi_cua_driver_sdk_fn_method_cuadriversession_scroll(uniffiSelf: bigint, input: Uint8Array): bigint;
@@ -1396,6 +1418,7 @@ interface NativeModuleInterface {
uniffi_cua_driver_sdk_checksum_method_cuadriver_get_screen_size(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriver_get_session_state(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey(): number;
+ uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriver_list_tools_json(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriver_metadata(): number;
@@ -1426,6 +1449,7 @@ interface NativeModuleInterface {
uniffi_cua_driver_sdk_checksum_method_cuadriversession_get_screen_size(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriversession_get_session_state(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey(): number;
+ uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriversession_press_key(): number;
uniffi_cua_driver_sdk_checksum_method_cuadriversession_scroll(): number;
diff --git a/libs/cua-driver/typescript/src/native/cua_driver_sdk.ts b/libs/cua-driver/typescript/src/native/cua_driver_sdk.ts
index 95142bec00..14f851059d 100644
--- a/libs/cua-driver/typescript/src/native/cua_driver_sdk.ts
+++ b/libs/cua-driver/typescript/src/native/cua_driver_sdk.ts
@@ -6,12 +6,12 @@
import nativeModule from "./cua_driver_sdk-ffi.js";
import { type UniffiRustFutureContinuationCallback, type UniffiForeignFutureDroppedCallback, type UniffiForeignFutureDroppedCallbackStruct, type UniffiVTableCallbackInterfaceCuaDriverSdkDriverActivityObserver, type UniffiForeignFutureResultRustBuffer, type UniffiForeignFutureCompleterustBuffer, type UniffiVTableCallbackInterfaceCuaDriverSdkDriverAuthorizationHost,
} from "./cua_driver_sdk-ffi.js";
-import { type ActionResult, type ClickInput, type ClipboardReadInput, type ClipboardWriteInput, type DragInput, type EndSessionInput, type EndSessionOutput, type EscalateSessionInput, type GetAgentCursorStateInput, type GetCursorPositionInput, type GetDesktopStateInput, type GetScreenSizeInput, type GetSessionStateInput, type HotkeyInput, type MoveCursorInput, type PressKeyInput, type ScrollInput, type SessionStateOutput, type SetAgentCursorEnabledInput, type SetAgentCursorMotionInput, type SetAgentCursorThemeInput, type SetWindowFrameInput, type StartSessionInput, type StartSessionOutput, type TypeTextInput, type VerifyStateInput, type VerifyStateOutput,
+import { type ActionResult, type ClickInput, type ClipboardReadInput, type ClipboardWriteInput, type DragInput, type EndSessionInput, type EndSessionOutput, type EscalateSessionInput, type GetAgentCursorStateInput, type GetCursorPositionInput, type GetDesktopStateInput, type GetScreenSizeInput, type GetSessionStateInput, type HotkeyInput, type InvokeMenuInput, type MoveCursorInput, type PressKeyInput, type ScrollInput, type SessionStateOutput, type SetAgentCursorEnabledInput, type SetAgentCursorMotionInput, type SetAgentCursorThemeInput, type SetWindowFrameInput, type StartSessionInput, type StartSessionOutput, type TypeTextInput, type VerifyStateInput, type VerifyStateOutput,
} from "./cua_driver_contract.js";
import { type FfiConverter, type UniffiByteArray, type UniffiGcObject, type UniffiHandle, type UniffiObjectFactory, type UniffiReferenceHolder, type UniffiRustCallStatus, AbstractFfiConverterByteArray, FfiConverterArray, FfiConverterBool, FfiConverterInt32, FfiConverterObject, FfiConverterObjectWithCallbacks, FfiConverterOptional, FfiConverterUInt32, FfiConverterUInt64, FfiConverterUInt8, RustBuffer, UniffiAbstractObject, UniffiEnum, UniffiError, UniffiInternalError, UniffiResult, UniffiRustCaller, destructorGuardSymbol, pointerLiteralSymbol, uniffiCreateFfiConverterString, uniffiCreateRecord, uniffiRustCallAsync, uniffiTraitInterfaceCall, uniffiTraitInterfaceCallAsyncWithError, uniffiTypeNameSymbol, variantOrdinalSymbol,
} from "@ubjs/core";
import uniffiCuaDriverContractModule from "./cua_driver_contract.js";
-const { FfiConverterTypeActionResult, FfiConverterTypeClickInput, FfiConverterTypeClipboardReadInput, FfiConverterTypeClipboardWriteInput, FfiConverterTypeDragInput, FfiConverterTypeEndSessionInput, FfiConverterTypeEndSessionOutput, FfiConverterTypeEscalateSessionInput, FfiConverterTypeGetAgentCursorStateInput, FfiConverterTypeGetCursorPositionInput, FfiConverterTypeGetDesktopStateInput, FfiConverterTypeGetScreenSizeInput, FfiConverterTypeGetSessionStateInput, FfiConverterTypeHotkeyInput, FfiConverterTypeMoveCursorInput, FfiConverterTypePressKeyInput, FfiConverterTypeScrollInput, FfiConverterTypeSessionStateOutput, FfiConverterTypeSetAgentCursorEnabledInput, FfiConverterTypeSetAgentCursorMotionInput, FfiConverterTypeSetAgentCursorThemeInput, FfiConverterTypeSetWindowFrameInput, FfiConverterTypeStartSessionInput, FfiConverterTypeStartSessionOutput, FfiConverterTypeTypeTextInput, FfiConverterTypeVerifyStateInput, FfiConverterTypeVerifyStateOutput } = uniffiCuaDriverContractModule.converters;
+const { FfiConverterTypeActionResult, FfiConverterTypeClickInput, FfiConverterTypeClipboardReadInput, FfiConverterTypeClipboardWriteInput, FfiConverterTypeDragInput, FfiConverterTypeEndSessionInput, FfiConverterTypeEndSessionOutput, FfiConverterTypeEscalateSessionInput, FfiConverterTypeGetAgentCursorStateInput, FfiConverterTypeGetCursorPositionInput, FfiConverterTypeGetDesktopStateInput, FfiConverterTypeGetScreenSizeInput, FfiConverterTypeGetSessionStateInput, FfiConverterTypeHotkeyInput, FfiConverterTypeInvokeMenuInput, FfiConverterTypeMoveCursorInput, FfiConverterTypePressKeyInput, FfiConverterTypeScrollInput, FfiConverterTypeSessionStateOutput, FfiConverterTypeSetAgentCursorEnabledInput, FfiConverterTypeSetAgentCursorMotionInput, FfiConverterTypeSetAgentCursorThemeInput, FfiConverterTypeSetWindowFrameInput, FfiConverterTypeStartSessionInput, FfiConverterTypeStartSessionOutput, FfiConverterTypeTypeTextInput, FfiConverterTypeVerifyStateInput, FfiConverterTypeVerifyStateOutput } = uniffiCuaDriverContractModule.converters;
const uniffiCaller = new UniffiRustCaller(() => ({ code: 0 }));
const uniffiIsDebug =
@@ -2549,6 +2549,7 @@ export interface CuaDriverLike {
getScreenSize(input: GetScreenSizeInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
getSessionState(input: GetSessionStateInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<SessionStateOutput>;
hotkey(input: HotkeyInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
+ invokeMenu(input: InvokeMenuInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
isAvailable(): boolean;
/**
* Canonical tool inventory for MCP and other protocol adapters.
@@ -3279,6 +3280,38 @@ private constructor(pointer: UniffiHandle) {
}
}
+ async invokeMenu(input: InvokeMenuInput, asyncOpts_?: { signal: AbortSignal }): Promise<ToolResult> /*throws*/ {
+ const __stack = uniffiIsDebug ? new Error().stack : undefined;
+ try {
+ return await uniffiRustCallAsync(
+ /*rustCaller:*/ uniffiCaller,
+ /*rustFutureFunc:*/ () => {
+ return nativeModule().uniffi_cua_driver_sdk_fn_method_cuadriver_invoke_menu(
+ uniffiTypeCuaDriverObjectFactory.clonePointer(this),FfiConverterTypeInvokeMenuInput.lower(input, nativeModule().rustbuffer_alloc)
+ );
+ },
+ /*pollFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_poll_rust_buffer,
+ /*cancelFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_cancel_rust_buffer,
+ /*completeFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_complete_rust_buffer,
+ /*freeFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_free_rust_buffer,
+ // Async returns always go through the JS-side converter: the
+ // FFI symbol returns the future handle (u64), and the user-level
+ // RustBuffer comes back via the shared `rust_future_complete_*`
+ // export. The bytes the runtime hands back must be deserialized
+ // here using the per-callable return-type converter.
+ /*liftFunc:*/ FfiConverterTypeToolResult.lift.bind(FfiConverterTypeToolResult),
+ /*liftString:*/ FfiConverterString.lift.bind(FfiConverterString),
+ /*asyncOpts:*/ asyncOpts_,
+ /*errorHandler:*/ FfiConverterTypeDriverError.lift.bind(FfiConverterTypeDriverError)
+ );
+ } catch (__error: any) {
+ if (uniffiIsDebug && __error instanceof Error) {
+ __error.stack = __stack;
+ }
+ throw __error;
+ }
+ }
+
isAvailable(): boolean {
return FfiConverterBool.lift(uniffiCaller.rustCall(
/*caller:*/ (callStatus) => {
@@ -3848,6 +3881,7 @@ export interface CuaDriverSessionLike {
getScreenSize(input: GetScreenSizeInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
getSessionState(input: GetSessionStateInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<SessionStateOutput>;
hotkey(input: HotkeyInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
+ invokeMenu(input: InvokeMenuInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
moveCursor(input: MoveCursorInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
pressKey(input: PressKeyInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
scroll(input: ScrollInput, asyncOpts_?: { signal: AbortSignal }) /*throws*/: Promise<ToolResult>;
@@ -4311,6 +4345,38 @@ private constructor(pointer: UniffiHandle) {
}
}
+ async invokeMenu(input: InvokeMenuInput, asyncOpts_?: { signal: AbortSignal }): Promise<ToolResult> /*throws*/ {
+ const __stack = uniffiIsDebug ? new Error().stack : undefined;
+ try {
+ return await uniffiRustCallAsync(
+ /*rustCaller:*/ uniffiCaller,
+ /*rustFutureFunc:*/ () => {
+ return nativeModule().uniffi_cua_driver_sdk_fn_method_cuadriversession_invoke_menu(
+ uniffiTypeCuaDriverSessionObjectFactory.clonePointer(this),FfiConverterTypeInvokeMenuInput.lower(input, nativeModule().rustbuffer_alloc)
+ );
+ },
+ /*pollFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_poll_rust_buffer,
+ /*cancelFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_cancel_rust_buffer,
+ /*completeFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_complete_rust_buffer,
+ /*freeFunc:*/ nativeModule().ffi_cua_driver_sdk_rust_future_free_rust_buffer,
+ // Async returns always go through the JS-side converter: the
+ // FFI symbol returns the future handle (u64), and the user-level
+ // RustBuffer comes back via the shared `rust_future_complete_*`
+ // export. The bytes the runtime hands back must be deserialized
+ // here using the per-callable return-type converter.
+ /*liftFunc:*/ FfiConverterTypeToolResult.lift.bind(FfiConverterTypeToolResult),
+ /*liftString:*/ FfiConverterString.lift.bind(FfiConverterString),
+ /*asyncOpts:*/ asyncOpts_,
+ /*errorHandler:*/ FfiConverterTypeDriverError.lift.bind(FfiConverterTypeDriverError)
+ );
+ } catch (__error: any) {
+ if (uniffiIsDebug && __error instanceof Error) {
+ __error.stack = __stack;
+ }
+ throw __error;
+ }
+ }
+
async moveCursor(input: MoveCursorInput, asyncOpts_?: { signal: AbortSignal }): Promise<ToolResult> /*throws*/ {
const __stack = uniffiIsDebug ? new Error().stack : undefined;
try {
@@ -5533,6 +5599,9 @@ function uniffiEnsureInitialized() {
if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey() !== 53333) {
throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriver_hotkey");
}
+ if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu() !== 13118) {
+ throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriver_invoke_menu");
+ }
if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available() !== 42961) {
throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriver_is_available");
}
@@ -5623,6 +5692,9 @@ function uniffiEnsureInitialized() {
if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey() !== 12820) {
throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriversession_hotkey");
}
+ if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu() !== 2742) {
+ throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriversession_invoke_menu");
+ }
if (nativeModule().uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor() !== 35768) {
throw new UniffiInternalError.ApiChecksumMismatch("uniffi_cua_driver_sdk_checksum_method_cuadriversession_move_cursor");
}
diff --git a/libs/cua-driver/typescript/test/electron-main-fixture.mjs b/libs/cua-driver/typescript/test/electron-main-fixture.mjs
index 78cffe0f00..dad16d5711 100644
--- a/libs/cua-driver/typescript/test/electron-main-fixture.mjs
+++ b/libs/cua-driver/typescript/test/electron-main-fixture.mjs
@@ -27,7 +27,7 @@ const server = net.createServer(socket => {
const request = JSON.parse(buffer.split("\\n", 1)[0]);
const result = request.method === "metadata" ? {
driver_version: ${JSON.stringify(packageVersion)},
- contract_version: hostBundleId.endsWith(".failure") ? "incompatible" : "0.5.0",
+ contract_version: hostBundleId.endsWith(".failure") ? "incompatible" : "0.6.0",
tools_list_schema_version: "1",
capability_version: "1",
mcp_protocol_version: "2025-06-18",
diff --git a/libs/cua-driver/typescript/test/native-daemon-fixture.mjs b/libs/cua-driver/typescript/test/native-daemon-fixture.mjs
index 2fb0bdb7f1..a403c2c020 100644
--- a/libs/cua-driver/typescript/test/native-daemon-fixture.mjs
+++ b/libs/cua-driver/typescript/test/native-daemon-fixture.mjs
@@ -18,7 +18,7 @@ const server = net.createServer((connection) => {
ok: true,
result: {
driver_version: "0.12.6",
- contract_version: "0.5.0",
+ contract_version: "0.6.0",
tools_list_schema_version: "1",
capability_version: "1",
mcp_protocol_version: "2025-06-18",
diff --git a/libs/cua-driver/typescript/test/native-loader.test.mjs b/libs/cua-driver/typescript/test/native-loader.test.mjs
index c65388e062..7ffdf54f22 100644
--- a/libs/cua-driver/typescript/test/native-loader.test.mjs
+++ b/libs/cua-driver/typescript/test/native-loader.test.mjs
@@ -53,7 +53,7 @@ const server = net.createServer(socket => {
const request = JSON.parse(buffer.split("\\n", 1)[0]);
const result = request.method === "metadata" ? {
driver_version: "0.10.0",
- contract_version: "0.5.0",
+ contract_version: "0.6.0",
tools_list_schema_version: "1",
capability_version: "1",
mcp_protocol_version: "2025-06-18",
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index 9e4164f42e..5669a6cdb6 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -318,6 +318,9 @@ if [[ "${SUITE}" == native || "${SUITE}" == all ]]; then
--ignored --nocapture --test-threads=1
for appkit_test in \
harness_appkit_smoke \
+ harness_appkit_query_projects_structured_elements \
+ harness_appkit_stale_element_token_fails_closed \
+ harness_appkit_invoke_menu_live_path \
harness_appkit_text_input \
harness_appkit_type_text_background \
harness_appkit_scroll_foreground \