Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): remove lean-read failure modes and make batching the default path (#4812)

Francesco Bonaccib86959306a92
diff --git a/.github/workflows/smoke-cua-driver-live.yml b/.github/workflows/smoke-cua-driver-live.yml
index a8718f3234..90b93d3901 100644
--- a/.github/workflows/smoke-cua-driver-live.yml
+++ b/.github/workflows/smoke-cua-driver-live.yml
@@ -15,6 +15,8 @@ on:
- "libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs"
- "libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs"
- "libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs"
+ - "libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs"
+ - "libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs"
- "libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs"
- "libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs"
- "libs/cua-driver/rust/crates/platform-windows/src/win32/combo.rs"
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
index 7fa9a6f530..d41ba3faed 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
@@ -14,16 +14,16 @@ Served by `cua-driver mcp`; see [MCP tools](/cua-driver/reference/mcp-tools) for
## `run_actions`
-Run several action tools in ONE call, in order, and stop at the first failure. Use it when you already know the next few actions and do not need to look at the app between them (fill three fields, click, then read the result). Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.
+The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.
-Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.
+Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.
Effect: destructive. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="run_actions--delay_ms"></span>`delay_ms` | `integer` | | Pause between steps in milliseconds (not after the last). Default 0. Range: 0 to 2000. |
-| <span id="run_actions--observe"></span>`observe` | `object` | | Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to include_screenshot=false and max_elements=200 to stay cheap; pass include_screenshot=true or a larger max_elements to widen it. Omit to read nothing. |
+| <span id="run_actions--observe"></span>`observe` | `object \| boolean` | | Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since="latest" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. |
| <span id="run_actions--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Items: 1 to 32. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
index 9ce7df6058..2c4aa08a9b 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
@@ -119,5 +119,5 @@ Windows-only convenience: `button: "left"|"right"|"middle"` switches the mouse b
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="click--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="click--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="click--target"></span>`target` | `window target \| desktop target` | | Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/double-and-right-click.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/double-and-right-click.mdx
index 24455f9d90..ed6b055f51 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/double-and-right-click.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/double-and-right-click.mdx
@@ -87,7 +87,7 @@ Exactly one of `element_token` or (`x` AND `y`) must be provided. `pid` is requi
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="double_click--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="double_click--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="double_click--pid"></span>`pid` | `integer` | required | Target process ID. |
| <span id="double_click--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
@@ -169,7 +169,7 @@ Exactly one of `element_token` or (`x` AND `y`) must be provided. `pid` is requi
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="right_click--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="right_click--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="right_click--pid"></span>`pid` | `integer` | required | Target process ID. |
| <span id="right_click--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
index 7b350adb01..ec564c9691 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
@@ -149,7 +149,7 @@ Run several action tools in one call. [Reference](/cua-driver/reference/mcp-tool
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="run_actions"></span>[`run_actions`](/cua-driver/reference/mcp-tools/batch#run_actions) | Run several action tools in ONE call, in order, and stop at the first failure. | macOS, Linux, Windows |
+| <span id="run_actions"></span>[`run_actions`](/cua-driver/reference/mcp-tools/batch#run_actions) | The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. | macOS, Linux, Windows |
## Page tools
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
index 43d5896e23..9edd11b006 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/keyboard.mdx
@@ -31,7 +31,7 @@ Web content (browser tabs, Slack, VS Code): AXValue is not proof the DOM saw the
| --- | --- | --- | --- |
| <span id="type_text--macos--delay_ms"></span>`delay_ms` | `integer` | | Ms between characters in the CGEvent fallback. Default 30. Range: 0 to 200. |
| <span id="type_text--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default "background": AX insert, then CGEvent keys, no focus steal. "foreground": front the window, type, restore; last resort for surfaces (WhatsApp/Catalyst) where background keys do not land. |
-| <span id="type_text--macos--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="type_text--macos--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="type_text--macos--pid"></span>`pid` | `integer` | | Target process ID. |
| <span id="type_text--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "desktop" with no pid/window_id types into the frontmost app. |
| <span id="type_text--macos--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. |
@@ -51,7 +51,7 @@ Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.
| --- | --- | --- | --- |
| <span id="type_text--linux--coordinate_frame"></span>`coordinate_frame` | `"window" \| "desktop"` | | Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing. |
| <span id="type_text--linux--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | `"background"` | Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus: Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window; a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. |
-| <span id="type_text--linux--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="type_text--linux--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="type_text--linux--pid"></span>`pid` | `integer` | | Target process ID. Omit with scope "desktop" to type into the focused application. |
| <span id="type_text--linux--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | Use "desktop" with no pid to type into the focused application. Default "window". |
| <span id="type_text--linux--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
index a5ea771eba..7f7d18f7f8 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/keys-and-shortcuts.mdx
@@ -98,7 +98,7 @@ Key vocabulary: return, tab, escape, up/down/left/right, space, delete, home, en
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="press_key--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="press_key--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="press_key--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
**Example arguments**
@@ -187,7 +187,7 @@ Recognized modifiers: ctrl/control, shift, alt, win/windows. Non-modifier keys u
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="hotkey--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="hotkey--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="hotkey--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="hotkey--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index 1ae3197c4c..f57eaff4ac 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -205,7 +205,7 @@ Note: `element_token` is accepted for cross-platform parity but currently no-op
| --- | --- | --- | --- |
| <span id="scroll--by"></span>`by` | `"line" \| "page"` | | Scroll granularity. Default: line. |
| <span id="scroll--direction"></span>`direction` | `"up" \| "down" \| "left" \| "right"` | required | Scroll direction. |
-| <span id="scroll--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="scroll--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="scroll--target"></span>`target` | `window target \| desktop target` | | Preferred per-call target: an exact window (`kind="window"`, `pid`, `window_id`) or the primary desktop (`kind="desktop"`, `display_id="primary"`). |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/screen.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/screen.mdx
index da8362afd3..7fdec9a476 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/screen.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/screen.mdx
@@ -96,7 +96,7 @@ Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide.
+Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).
After a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.
@@ -114,7 +114,7 @@ After a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordin
</Tab>
<Tab value="Linux">
-Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide.
+Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).
After a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.
@@ -134,7 +134,7 @@ After a zoom, pass from_zoom=true to click/type_text to auto-translate coordinat
Zoom into a rectangular region of a window screenshot at full (native) resolution. Use this when `get_window_state` returned a resized image and you need to read small text, identify icons, or verify UI details.
-Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot returned by `get_window_state` (i.e. the resized image if `max_image_dimension` is active). The maximum zoom region width is 500 px in scaled-image coordinates.
+Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted). Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot returned by `get_window_state` (i.e. the resized image if `max_image_dimension` is active). The maximum zoom region width is 500 px in scaled-image coordinates.
A 20% padding is automatically added on every side of the requested region so the target remains visible even if the caller's coordinates are slightly off.
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
index b6bcf8ef61..b14d297894 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/values-and-clipboard.mdx
@@ -80,7 +80,7 @@ For free-form text entry that the target accepts via keystrokes only, prefer `ty
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="set_value--element_token"></span>`element_token` | `string` | | Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
+| <span id="set_value--element_token"></span>`element_token` | `string` | | Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is "s0000002a:11" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it. |
| <span id="set_value--pid"></span>`pid` | `integer` | required | Target process ID. |
**Example arguments**
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
index 84aabb5cfd..2579763216 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
@@ -32,9 +32,9 @@ Snapshot a window: the accessibility tree plus a screenshot. By default the tree
START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).
-DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.
+DIFF READS (use them for every re-read after acting): `since:"latest"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.
-A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
+A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.
@@ -145,7 +145,7 @@ Windows requires no special permissions.
| --- | --- | --- | --- |
| <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |
| <span id="get_window_state--full_output"></span>`full_output` | `boolean` | | Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`. |
-| <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. |
+| <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window, or "latest" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. |
| <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |
| <span id="get_window_state--tree_format"></span>`tree_format` | `"markdown" \| "elements" \| "both"` | | Which tree representation to return. Default "markdown": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. "elements": the structured `elements` array only (per-element element_token, frame, parent_index, value). "both": the two together, as before; roughly doubles the size. |
| <span id="get_window_state--verbose"></span>`verbose` | `boolean` | | Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context. |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 9fc884e7b9..2fd332b2df 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -123,7 +123,7 @@ For authorized foreground input, use the Cua action's
When a cua-driver call surprises you, diagnose cua-driver first:
- **Large tree or repeated reads?** `get_window_state` returns one compact
- markdown tree capped at 250 nodes; pass `since:<snapshot_id>` for a diff
+ markdown tree capped at 250 nodes; pass `since:"latest"` for a diff
(`no change since …; focused element is …` names the focused row on macOS)
and `tree_format:"elements"` for structured rows. `full_output:true` restores
the old payload (both forms, `_note`, `background_input`, 2 000 nodes).
@@ -163,8 +163,10 @@ response, and only switch _dispatch rung_ on a real signal:
1. **Re-snapshot and read the tree diff**: a changed `AXValue`, a new
element, a collapsed menu, a disabled button. If the tree shows the
change, you're done. When you only need the tree diff and don't need
- fresh pixels, pass `include_screenshot:false` to skip the grab: a
- **perf** knob, not a mode flip.
+ fresh pixels, pass `since:"latest"` and `include_screenshot:false` to
+ skip the grab: a **perf** knob, not a mode flip. The window keeps its
+ last screenshot's pixel frame while its size is unchanged, so `x,y`
+ from that screenshot still work afterwards.
2. **Trust the screenshot and do an element px action** when the tree
**lies**: the action response carried `effect:"suspected_noop"`, the
re-snapshot came back `degraded` (empty tree), or the tree looks
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index f48a9e0ab9..b6571346a9 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -37,10 +37,11 @@ Operate one exact target, observe it narrowly, act by element, and verify the us
The MCP server instructions carry the same rules, so they apply even when this file is not opened.
-1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`, and pass `include_screenshot:false` when the tree is enough. Widen only if the target is missing.
-2. Act by `element_token`. A narrowed read keeps the full snapshot actionable, so you can act on a match without re-reading. Use pixels only for surfaces missing from the tree.
-3. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.
-4. Batch known steps in one `run_actions` call (stops at the first failure, one optional bounded read at the end); see [Workflow](WORKFLOW.md#batch-known-actions).
+1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`. Widen only if the target is missing.
+2. Act by `element_token`: row `[N]` of a read is `<snapshot_id>:N` (for example `"s0000002a:11"`, not `"11"`). Use pixels only for surfaces missing from the tree.
+3. Act through `run_actions` with `observe:true`, even for a single step. It runs the steps, stops at the first failure, and returns only what changed since your last read, so you do not need a separate read after acting; see [Workflow](WORKFLOW.md#batch-known-actions).
+4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (on macOS it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.
+5. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.
## Act
@@ -50,7 +51,7 @@ The MCP server instructions carry the same rules, so they apply even when this f
| Find or open the requested app | `list_apps`, `list_windows`, `launch_app` | Current platform guide below |
| Observe one window | `get_window_state({pid, window_id, query})` | [Workflow](WORKFLOW.md) |
| Act on a control | `click` / `type_text` with a fresh `element_token` and exact window target | [Workflow](WORKFLOW.md) |
-| Run several decided actions in one call | `run_actions({steps:[{tool,args},...], observe?})`; stops at the first failure | [Workflow](WORKFLOW.md#batch-known-actions) |
+| Act and see what changed in one call | `run_actions({steps:[{tool,args},...], observe:true})`; stops at the first failure | [Workflow](WORKFLOW.md#batch-known-actions) |
| Use pixels when semantics cannot reach it | Fresh target screenshot, then `x,y` on the same target | [Workflow](WORKFLOW.md) |
| Verify the outcome | `verify_state({pid, window_id, expect})` or a fresh snapshot read by the agent | [Workflow](WORKFLOW.md) |
| Operate the authorized desktop | `get_desktop_state` → input with `target:{kind:"desktop",display_id:"primary"}` → `get_desktop_state` | [Workflow](WORKFLOW.md), [Linux](LINUX.md) on Wayland |
@@ -68,7 +69,7 @@ Check the installed version and advertised schema before using unfamiliar parame
1. Select the exact target on each action. A session is lifecycle metadata, not capture scope or permission authority.
2. Observe before input and verify the outcome at checkpoints, not after every action. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.
-3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`. Reads are bounded and return one tree form by default; follow up with `since:<snapshot_id>` instead of re-reading a large window.
+3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`. Reads are bounded and return one tree form by default; follow up with `since:"latest"` instead of re-reading a large window.
4. Keep background window actions non-interfering. Foreground delivery and desktop input require authorization for visible control; an unavailable route is not permission to escalate.
5. Never infer pixels from a missing image, a different window, or an unaccounted-for resized preview. Capture failure and an empty accessibility tree are different failures.
6. Keep one controller for a shared desktop. Distinct sessions/cursors do not isolate focus, keyboard input, application state, or snapshot caches.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
index 7b540b5f35..543c7baae8 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
@@ -41,10 +41,11 @@ By default the tree comes back **once**, as compact `tree_markdown` (`tree_forma
- Use `query` to project matching rows plus ancestors without renumbering their indices.
- A read walks at most 250 nodes by default. When it stops there the response says `Tree truncated at max_elements=…` (`truncated:true`, `truncation_hint`). Pass a larger `max_elements`, or narrow with `query` / `max_depth`. Truncation does not prove absence.
-- After the first read, pass `since:<snapshot_id>` to get only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.
+- After the first read, pass `since:"latest"` (or `since:<snapshot_id>`) to get only what changed. `latest` is this window's most recent read with the same `query`, `max_elements` and `max_depth`. You get: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.
- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.
-- Use `include_screenshot:false` only when tree-only observation is enough; it cannot ground a pixel action.
-- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. Check the installed schema first.
+- `include_screenshot:false` returns the tree only. On macOS the window keeps the pixel frame of its last screenshot while its size is unchanged, so `x,y` read off that screenshot still work; elsewhere, and after a resize, take a new screenshot before a pixel action.
+- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. On macOS it keeps the current snapshot's rows and tokens valid. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.
+- Row numbers belong to one read. A narrower re-read (smaller `max_elements`, `max_depth`, another `query`) can stop before a row you saw earlier, and `<new snapshot_id>:N` for that row is then out of range. Use rows printed in the read whose `snapshot_id` you use.
- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.
- Use `screenshot_out_file` to save a PNG instead of inlining it, then actually read the image. Use an absolute, run-scoped output path.
@@ -52,7 +53,7 @@ The accessibility model may lag or disagree with rendered state: Electron text s
## Act once
-Use an opaque `element_token` from the latest snapshot of the intended window. Do not derive or edit tokens. A later snapshot can invalidate a pending action, including when another agent observes the same window.
+Use the `element_token` `<snapshot_id>:N` for row `[N]` of the latest snapshot of the intended window, for example `"s0000002a:11"`. A bare `"11"` is accepted only together with `window_id` and then means row 11 of that window's current snapshot. Do not mix a row number from one read with the `snapshot_id` of another. A later snapshot can invalidate a pending action, including when another agent observes the same window.
Example CLI window action, with IDs and token replaced from the preceding response:
@@ -78,7 +79,7 @@ Keep `delivery_mode:"background"` as the default for window input. The route may
## Batch known actions
-When the next several actions are already decided and nothing between them needs a look, send them as one `run_actions` call instead of one call per action. The batch runs the same tools in order, stops at the first failure, and returns per-step status plus at most one bounded observation. Typical fit: fill several fields, then press a button, then read the result. Do not batch across a point where the answer decides the next step, or when a step reshuffles the window and invalidates element tokens used by later steps (use pixel targets after it, or split the batch there).
+`run_actions` is the default way to act. When the next action or actions are decided, send them as one `run_actions` call with `observe:true` instead of one call per action followed by a read. Even a single step pays off: the action and the look at its result become one call. The batch runs the same tools in order, stops at the first failure, and returns per-step status plus at most one bounded observation. Typical fit: fill several fields, then press a button, then read the result. Do not batch across a point where the answer decides the next step, or when a step reshuffles the window and invalidates element tokens used by later steps (use pixel targets after it, or split the batch there).
```bash
cua-driver run_actions '{"session":"run-1","steps":[
@@ -86,13 +87,13 @@ cua-driver run_actions '{"session":"run-1","steps":[
{"tool":"set_value","args":{"pid":844,"element_token":"s0000002a:15","value":"Lovelace"}},
{"tool":"click","args":{"target":{"kind":"window","pid":844,"window_id":10725},"element_token":"s0000002a:21"}},
{"tool":"press_key","args":{"pid":844,"key":"return"}}
- ],"delay_ms":100,"observe":{"max_elements":120}}'
+ ],"delay_ms":100,"observe":true}'
```
-- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`; `args` are exactly that tool's arguments. Run `describe run_actions` and `describe <tool>` for schemas. Up to 32 steps.
+- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag` (`press` and `type` are accepted for `press_key` and `type_text`); `args` are exactly that tool's arguments. Observation tools cannot run inside a batch: use `observe`. Run `describe run_actions` and `describe <tool>` for schemas. Up to 32 steps.
- Every step is validated before the first runs, so a malformed step changes nothing. Each step then passes the same session, permission, capability-manifest and approval checks as a direct call; a batch grants nothing a single call lacks, and a refused step ends the batch like any other failure.
- A batch has one session. Set `session` on `run_actions`; a step may repeat it but not name another.
-- `observe` is optional and reads once, after the last executed step (also after a failure): `get_window_state` arguments, `pid`/`window_id` taken from the last step that names both, `include_screenshot:false` and `max_elements:200` unless you override. Omit `observe` to read nothing.
+- `observe` is optional and reads once, after the last executed step (also after a failure). Pass `true` or `get_window_state` arguments; `pid`/`window_id` come from the last step that names both. Defaults: `since:"latest"` (only what changed since your last read of that window with the same view; a full read if there is none), `include_screenshot:false`, `max_elements:250`. Pass `include_screenshot:true` to see the window, or `since:null` for a full read. Omit `observe` to read nothing.
- Read `steps[].ok` and `failed_step`. Steps before a failure did run and are not rolled back; steps after it did not. Observe before repairing, as for a single `unverifiable` action.
## Pixel coordinates
@@ -101,7 +102,7 @@ Ground window actions on the PNG from that exact `get_window_state`; ground desk
The harness may downsample the displayed preview independently of the returned PNG. Use the returned dimensions and the original file. If measuring on a resized preview, account for its exact scale in both axes; do not assume the preview is native resolution. Do not guess from accessibility frames or another app's geometry.
-After movement, resize, navigation, or a competing desktop interaction, reobserve. When using `zoom`, read its schema and preserve the `from_zoom` mapping on the supported follow-up action. A manually cropped image requires its crop offset; an untracked crop is not an action coordinate source.
+After movement, resize, navigation, or a competing desktop interaction, reobserve. `zoom` takes a region as corners, `x1,y1` (top-left) to `x2,y2` (bottom-right); `x,y,width,height` is converted. Preserve the `from_zoom` mapping on the supported follow-up action. A manually cropped image requires its crop offset; an untracked crop is not an action coordinate source.
For tiny targets, inspect at full resolution or annotate a copy without changing its dimensions. Keep the raw evidence unmodified. `debug_image_out`, where advertised, captures an action diagnostic; it is not a pre-action approval step.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
index 315d086cf3..4ad16c1453 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
@@ -41,8 +41,17 @@ pub const BATCHABLE_TOOLS: &[&str] = &[
pub const MAX_STEPS: usize = 32;
/// Longest pause between two steps.
pub const MAX_DELAY_MS: u64 = 2_000;
-/// Default `max_elements` for the end-of-batch observation.
-const DEFAULT_OBSERVE_MAX_ELEMENTS: u64 = 200;
+/// Default `max_elements` for the end-of-batch observation: the same budget as
+/// a plain read, so its automatic `since` diff finds that read as a baseline.
+const DEFAULT_OBSERVE_MAX_ELEMENTS: usize = crate::window_state_view::DEFAULT_MAX_ELEMENTS;
+/// Step tool names models reach for that mean a batchable tool.
+const TOOL_ALIASES: &[(&str, &str)] = &[
+ ("press", "press_key"),
+ ("key", "press_key"),
+ ("keypress", "press_key"),
+ ("type", "type_text"),
+ ("set", "set_value"),
+];
const MESSAGE_LIMIT: usize = 300;
const OBSERVE_TOOL: &str = "get_window_state";
@@ -63,9 +72,12 @@ impl Tool for RunActionsTool {
fn def(&self) -> &ToolDef {
DEF.get_or_init(|| ToolDef {
name: RUN_ACTIONS_TOOL.into(),
- description: "Run several action tools in ONE call, in order, and stop at the first \
- failure. Use it when you already know the next few actions and do not need to \
- look at the app between them (fill three fields, click, then read the result). \
+ description: "The default way to act: run one or more action tools in ONE call, in \
+ order, stop at the first failure, and (with `observe`) get what changed in the \
+ same response. Use it whenever you know the next action(s) and would otherwise \
+ re-read the window after them: fill three fields, click, and see the result in \
+ one call instead of five. Even a single step plus `observe:true` replaces an \
+ action call followed by a get_window_state call. \
Each step is `{tool, args}` where `tool` is one of click, double_click, \
right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` \
are exactly that tool's arguments. Every step goes through the same session, \
@@ -76,8 +88,10 @@ impl Tool for RunActionsTool {
invalidate later element targets, so put element-targeted actions before the \
actions that reshuffle the window, or use pixel targets after them.\n\n\
Returns per-step status (`ok` or the error message) and, when `observe` is \
- given, ONE bounded get_window_state read after the last executed step. \
- Without `observe` nothing is read. The batch uses one session: steps may \
+ given, ONE bounded get_window_state read after the last executed step. That \
+ read is a `since:\"latest\"` diff by default: only the rows that changed \
+ since your last read of the window, plus a new snapshot_id. Without \
+ `observe` nothing is read. The batch uses one session: steps may \
omit `session` or repeat the batch's. `delay_ms` pauses between steps (max \
2000). At most 32 steps."
.into(),
@@ -108,7 +122,8 @@ impl Tool for RunActionsTool {
},
"observe": {
"type": "object",
- "description": "Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to include_screenshot=false and max_elements=200 to stay cheap; pass include_screenshot=true or a larger max_elements to widen it. Omit to read nothing."
+ "type": ["object", "boolean"],
+ "description": "Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since=\"latest\" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing."
}
},
"additionalProperties": false
@@ -224,7 +239,7 @@ impl Plan {
}
let observe = match object.get("observe") {
- None | Some(Value::Null) => None,
+ None | Some(Value::Null) | Some(Value::Bool(false)) => None,
Some(value) => Some(parse_observe(registry, value, &steps, session.as_deref())?),
};
Ok(Self {
@@ -354,6 +369,10 @@ fn parse_step(
.get("tool")
.and_then(Value::as_str)
.ok_or_else(|| PlanError::step(index, "`tool` must be a string"))?;
+ let name = TOOL_ALIASES
+ .iter()
+ .find(|(alias, _)| *alias == name)
+ .map_or(name, |(_, tool)| *tool);
let tool = BATCHABLE_TOOLS
.iter()
.copied()
@@ -384,13 +403,15 @@ fn parse_observe(
steps: &[Step],
session: Option<&str>,
) -> Result<Value, PlanError> {
- let mut args = value
- .as_object()
- .cloned()
- .map(Value::Object)
- .ok_or_else(|| {
- PlanError::batch("`observe` must be an object of get_window_state arguments")
- })?;
+ let mut args = match value {
+ Value::Bool(true) => Value::Object(Map::new()),
+ Value::Object(object) => Value::Object(object.clone()),
+ _ => {
+ return Err(PlanError::batch(
+ "`observe` must be true or an object of get_window_state arguments",
+ ))
+ }
+ };
prepare_args(&mut args, session)
.map_err(|message| PlanError::batch(format!("observe: {message}")))?;
let object = args.as_object_mut().expect("object");
@@ -418,6 +439,21 @@ fn parse_observe(
object
.entry("max_elements")
.or_insert_with(|| json!(DEFAULT_OBSERVE_MAX_ELEMENTS));
+ // Diff against the caller's last read of the window unless it asked for
+ // a full read (`since:null` or `full_output:true`).
+ let full_output = object.get("full_output").and_then(Value::as_bool) == Some(true);
+ match object.get("since") {
+ Some(Value::Null) => {
+ object.remove("since");
+ }
+ None if !full_output => {
+ object.insert(
+ "since".to_owned(),
+ json!(crate::window_state_view::SINCE_LATEST),
+ );
+ }
+ _ => {}
+ }
validate_against_schema(registry, OBSERVE_TOOL, &args)
.map_err(|message| PlanError::batch(format!("observe: {message}")))?;
Ok(args)
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
index e80c6a01ea..7ac21d3b81 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
@@ -65,6 +65,8 @@ impl Harness {
"text": {"type": "string"},
"fail": {"type": "boolean"},
"include_screenshot": {"type": "boolean"},
+ "since": {"type": "string"},
+ "full_output": {"type": "boolean"},
"max_elements": {"type": "integer", "minimum": 1}
},
"additionalProperties": false
@@ -283,7 +285,14 @@ async fn observation_inherits_the_window_and_stays_bounded() {
assert_eq!(observed["pid"], 42);
assert_eq!(observed["window_id"], 7);
assert_eq!(observed["include_screenshot"], false);
- assert_eq!(observed["max_elements"], 200);
+ assert_eq!(
+ observed["max_elements"], 250,
+ "the plain-read budget, so `latest` finds it"
+ );
+ assert_eq!(
+ observed["since"], "latest",
+ "the observation is a diff by default"
+ );
harness
.run(json!({
@@ -394,3 +403,56 @@ async fn the_batch_is_advertised_with_its_schema() {
.collect();
assert_eq!(listed, super::BATCHABLE_TOOLS);
}
+
+#[tokio::test]
+async fn observe_true_reads_a_diff_and_since_null_reads_in_full() {
+ let harness = Harness::new();
+ let step = json!([{"tool": "click", "args": {"pid": 42, "window_id": 7}}]);
+
+ let result = harness.run(json!({"steps": step, "observe": true})).await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let observed = harness.last("get_window_state");
+ assert_eq!(observed["window_id"], 7);
+ assert_eq!(observed["since"], "latest");
+
+ harness
+ .run(json!({"steps": step, "observe": {"since": null}}))
+ .await;
+ assert!(harness.last("get_window_state").get("since").is_none());
+
+ harness
+ .run(json!({"steps": step, "observe": {"full_output": true}}))
+ .await;
+ assert!(harness.last("get_window_state").get("since").is_none());
+
+ harness
+ .run(json!({"steps": step, "observe": {"since": "s0000002a"}}))
+ .await;
+ assert_eq!(harness.last("get_window_state")["since"], "s0000002a");
+
+ harness.run(json!({"steps": step, "observe": false})).await;
+ assert_eq!(
+ harness.hits("get_window_state"),
+ 4,
+ "observe:false reads nothing"
+ );
+}
+
+#[tokio::test]
+async fn common_tool_name_slips_map_to_the_batchable_tool() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"tool": "press", "args": {"pid": 1}},
+ {"tool": "type", "args": {"pid": 1, "text": "x"}},
+ ]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ assert_eq!(harness.hits("press_key"), 1);
+ assert_eq!(harness.hits("type_text"), 1);
+
+ let refused = harness
+ .run(json!({"steps": [{"tool": "get_window_state", "args": {"pid": 1}}]}))
+ .await;
+ assert_eq!(refused.is_error, Some(true));
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
index 315711a4af..5d76845f96 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
@@ -383,10 +383,10 @@ fn agent_instructions() -> String {
Efficient loop:
1. `list_windows` or `launch_app` for `pid` and `window_id`.
-2. Read narrowly: `get_window_state` with `query:"Save"` (or `max_elements`) returns ~2K chars vs ~42K for a full snapshot. Widen only if the target is missing; `include_screenshot:false` when the tree suffices.
-3. Act by `element_token` ({tree_kind}) from the latest snapshot; pixel `x,y` only for surfaces missing from the tree. A new snapshot stales older tokens.
-4. Verify at checkpoints, not after every action: `verify_state(pid, window_id, expect)` or one targeted read. `unknown` and `effect:"unverifiable"` are not success.
-5. Batch known steps: `run_actions({{steps:[{{tool,args}}]}})` stops at the first failure.
+2. Read narrowly: `get_window_state` with `query:"Save"` (or `max_elements`). Re-read with `since:"latest"` for only what changed. For just a picture, `include_accessibility_tree:false` (tokens stay valid).
+3. Act by `element_token` ({tree_kind}): row `[N]` is `<snapshot_id>:N`. Pixel `x,y` only for surfaces missing from the tree.
+4. Act through `run_actions({{steps:[{{tool,args}}],observe:true}})`, even for one step: it stops at the first failure and returns what changed, so no separate re-read.
+5. Verify at checkpoints, not after every action: `verify_state(pid, window_id, expect)`. `unknown` and `effect:"unverifiable"` are not success.
Use the narrowest semantic route first: `set_window_frame` plus `list_windows` for geometry, typed browser tools for pages, clipboard tools for the clipboard. Stay in the background; `delivery_mode:"foreground"` only if refused or unverifiable.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs
index ad613a7e2d..944f54e8dc 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs
@@ -1,5 +1,6 @@
use crate::element_token::{
- format_snapshot_id, parse_token, refusal, ResolvedElement, LRU_CAP_PER_PID, STALE_TOKEN_ERROR,
+ format_snapshot_id, parse_token, refusal, token_for, ResolvedElement, LRU_CAP_PER_PID,
+ STALE_TOKEN_ERROR,
};
use crate::protocol::ToolResult;
use std::any::Any;
@@ -26,6 +27,9 @@ struct Snapshot<S> {
window_id: u64,
screenshot_owner: Option<String>,
screenshot_scale: Option<f64>,
+ /// Window size in points when the screenshot was taken. A tree-only read
+ /// keeps the screenshot frame only while the window still has this size.
+ screenshot_window_size: Option<(f64, f64)>,
zoom: Option<ZoomContext>,
semantic: bool,
payload: S,
@@ -68,12 +72,68 @@ impl ZoomContext {
}
}
-fn screenshot_context_refusal(pid: Option<i32>, window_id: Option<u64>) -> ToolResult {
- ToolResult::error(
- "No current snapshot for this window contains a screenshot owned by this session. Call get_window_state with a screenshot on the same connection before using pixels.",
- )
- .with_structured(serde_json::json!({
+/// Why a pixel action found no screenshot frame for its window.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum MissingFrame {
+ /// This pid has no snapshot of the window at all in this runtime.
+ WindowNeverRead,
+ /// The window's current snapshot carries no screenshot frame: the last
+ /// read skipped the screenshot after the window changed size, or no
+ /// screenshot was ever taken of it.
+ NoScreenshot,
+ /// The current screenshot frame belongs to another session.
+ OtherSession,
+ /// No window was named and the process's snapshots disagree.
+ Ambiguous,
+}
+
+impl MissingFrame {
+ fn as_str(self) -> &'static str {
+ match self {
+ Self::WindowNeverRead => "window_never_read",
+ Self::NoScreenshot => "no_screenshot",
+ Self::OtherSession => "other_session",
+ Self::Ambiguous => "ambiguous_window",
+ }
+ }
+}
+
+fn screenshot_context_refusal(
+ pid: Option<i32>,
+ window_id: Option<u64>,
+ why: MissingFrame,
+) -> ToolResult {
+ let read = match (pid, window_id) {
+ (Some(pid), Some(window_id)) => {
+ format!("get_window_state(pid:{pid}, window_id:{window_id})")
+ }
+ (None, Some(window_id)) => format!("get_window_state(pid, window_id:{window_id})"),
+ _ => "get_window_state(pid, window_id)".to_owned(),
+ };
+ let window = window_id.map_or_else(|| "this window".to_owned(), |id| format!("window {id}"));
+ let message = match why {
+ MissingFrame::WindowNeverRead => format!(
+ "No screenshot of {window} in this session: it has not been read yet. Call {read} \
+ and take x,y from that window's own screenshot; pixels read off another window's \
+ screenshot do not apply to it."
+ ),
+ MissingFrame::NoScreenshot => format!(
+ "No screenshot frame for {window}: its latest read had no screenshot and the window \
+ changed size since the last one (or it was never captured). Call {read} with the \
+ default screenshot and take x,y from that image."
+ ),
+ MissingFrame::OtherSession => format!(
+ "The current screenshot of {window} belongs to another session. Call {read} on \
+ this connection before using pixels."
+ ),
+ MissingFrame::Ambiguous => format!(
+ "Pass window_id: this process has several windows without one shared screenshot \
+ frame. Then call {read} if that window has no screenshot yet."
+ ),
+ };
+ ToolResult::error(message).with_structured(serde_json::json!({
"code": "screenshot_context_missing",
+ "reason": why.as_str(),
"pid": pid,
"window_id": window_id,
}))
@@ -121,6 +181,82 @@ fn zoom_context_refusal(pid: i32, window_id: Option<u64>) -> ToolResult {
}))
}
+/// The snapshot a bare row number refers to: the current snapshot of the
+/// named window, or the process's only snapshot when no window is named.
+fn row_snapshot<S: SnapshotPayload>(
+ pid: i32,
+ lane: &[Snapshot<S>],
+ window_id: Option<u64>,
+ row: usize,
+) -> Result<&Snapshot<S>, ToolResult> {
+ let found = match window_id {
+ Some(window_id) => lane.iter().find(|snapshot| snapshot.window_id == window_id),
+ None => match lane {
+ [only] => Some(only),
+ _ => None,
+ },
+ };
+ found.ok_or_else(|| {
+ let current = lane
+ .iter()
+ .map(|snapshot| {
+ format!(
+ "\"{}\" (window {})",
+ token_for(snapshot.id, row),
+ snapshot.window_id
+ )
+ })
+ .collect::<Vec<_>>();
+ let message = if current.is_empty() {
+ format!(
+ "element_token \"{row}\" is a row number, and pid {pid} has no current \
+ snapshot. Call get_window_state, then pass \"<snapshot_id>:{row}\"."
+ )
+ } else {
+ format!(
+ "element_token \"{row}\" is a row number, not a token. Pass the full token, \
+ <snapshot_id>:{row}: {}; or add window_id.",
+ current.join(", ")
+ )
+ };
+ refusal("invalid_element_token", message)
+ })
+}
+
+fn invalid_token_refusal<S>(token: &str, lane: &[Snapshot<S>]) -> ToolResult {
+ let example = lane.last().map_or_else(
+ || "s0000002a:11".to_owned(),
+ |snapshot| token_for(snapshot.id, 11),
+ );
+ refusal(
+ "invalid_element_token",
+ format!(
+ "element_token \"{token}\" has invalid format: use <snapshot_id>:<row>, e.g. \
+ \"{example}\" for row [11] of the read whose snapshot_id is {}.",
+ example.split(':').next().unwrap_or_default()
+ ),
+ )
+}
+
+fn out_of_range_refusal<S: SnapshotPayload>(snapshot: &Snapshot<S>, row: usize) -> ToolResult {
+ let snapshot_id = format_snapshot_id(snapshot.id);
+ let rows = match snapshot.payload.len() {
+ 0 => "no rows".to_owned(),
+ n => format!("rows [0] to [{}] only", n - 1),
+ };
+ refusal(
+ "invalid_element_token",
+ format!(
+ "element_token {snapshot_id}:{row} is out of range: snapshot {snapshot_id} has \
+ {rows} ({} element(s)). Row numbers belong to one read: a read with a smaller \
+ max_elements, max_depth or another query can stop before row [{row}]. Use a row \
+ printed in {snapshot_id}, or re-read with a larger max_elements (or a query that \
+ matches the target) and use that read's rows.",
+ snapshot.payload.len()
+ ),
+ )
+}
+
pub struct SnapshotStore<S: SnapshotPayload> {
runtime_scope: String,
inner: Mutex<HashMap<i32, Vec<Snapshot<S>>>>,
@@ -155,7 +291,74 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
session: Option<&str>,
screenshot_scale: Option<f64>,
) -> Option<(u32, Vec<u32>)> {
- self.publish_snapshot(pid, window_id, payload, session, screenshot_scale, true)
+ self.publish_snapshot(
+ pid,
+ window_id,
+ payload,
+ session,
+ (screenshot_scale, None),
+ true,
+ )
+ }
+
+ /// [`Self::publish_for_session`] that also records the window's size in
+ /// points (`window_size`), so later reads can tell whether the screenshot
+ /// frame still applies.
+ ///
+ /// * With a `screenshot_scale`, the size is stored with the new frame.
+ /// * Without one (a tree-only read), the new snapshot keeps the replaced
+ /// snapshot's screenshot frame when that frame belongs to the same
+ /// session and was taken at the same window size. Window-relative pixels
+ /// off that screenshot still land where they did, so a pixel action
+ /// after a tree-only re-read is not refused. A resized window, an
+ /// unknown size or another session's frame retires the frame as before.
+ pub fn publish_sized_for_session(
+ &self,
+ pid: i32,
+ window_id: u64,
+ payload: S,
+ session: Option<&str>,
+ screenshot_scale: Option<f64>,
+ window_size: Option<(f64, f64)>,
+ ) -> Option<(u32, Vec<u32>)> {
+ self.publish_snapshot(
+ pid,
+ window_id,
+ payload,
+ session,
+ (screenshot_scale, window_size),
+ true,
+ )
+ }
+
+ /// Refresh the screenshot frame of the window's current snapshot in place,
+ /// keeping its id, its elements and their tokens. Used by a
+ /// screenshot-only read (no tree walk): it adds a new image but no new
+ /// element indices, so the tokens the caller holds stay valid. Returns the
+ /// kept snapshot id, or `None` when the window has no snapshot with
+ /// elements (the caller then publishes a new one).
+ pub fn refresh_screenshot_for_session(
+ &self,
+ pid: i32,
+ window_id: u64,
+ session: Option<&str>,
+ screenshot_scale: f64,
+ window_size: Option<(f64, f64)>,
+ ) -> Option<u32> {
+ let mut inner = self.inner.lock().unwrap();
+ if session.is_some_and(crate::session::is_session_ended) {
+ return None;
+ }
+ let snapshot = inner
+ .get_mut(&pid)?
+ .iter_mut()
+ .find(|snapshot| snapshot.window_id == window_id)
+ .filter(|snapshot| snapshot.semantic && !snapshot.payload.is_empty())?;
+ snapshot.screenshot_owner = session.map(str::to_owned);
+ snapshot.screenshot_scale = Some(screenshot_scale);
+ snapshot.screenshot_window_size = window_size;
+ snapshot.zoom = None;
+ Some(snapshot.id)
}
/// Publish screenshot/capture state without claiming that an accessibility
@@ -168,7 +371,14 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
session: Option<&str>,
screenshot_scale: Option<f64>,
) -> Option<(u32, Vec<u32>)> {
- self.publish_snapshot(pid, window_id, payload, session, screenshot_scale, false)
+ self.publish_snapshot(
+ pid,
+ window_id,
+ payload,
+ session,
+ (screenshot_scale, None),
+ false,
+ )
}
fn publish_snapshot(
@@ -177,7 +387,9 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
window_id: u64,
payload: S,
session: Option<&str>,
- screenshot_scale: Option<f64>,
+ // The new screenshot's scale, if any, and the window size in points
+ // at this read.
+ (screenshot_scale, window_size): (Option<f64>, Option<(f64, f64)>),
semantic: bool,
) -> Option<(u32, Vec<u32>)> {
let (id, retired) = {
@@ -190,6 +402,21 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
if let Some(position) = lane.iter().position(|entry| entry.window_id == window_id) {
retired.push(lane.remove(position));
}
+ // A new screenshot brings its own frame. A tree-only read keeps the
+ // replaced snapshot's frame while the window keeps its size.
+ let (scale, frame_size) = match (screenshot_scale, window_size) {
+ (Some(scale), size) => (Some(scale), size),
+ (None, Some(now)) => retired
+ .first()
+ .filter(|previous| previous.screenshot_owner.as_deref() == session)
+ .and_then(|previous| {
+ let then = previous.screenshot_window_size?;
+ let same = (then.0 - now.0).abs() < 0.5 && (then.1 - now.1).abs() < 0.5;
+ same.then_some((previous.screenshot_scale?, then))
+ })
+ .map_or((None, None), |(scale, size)| (Some(scale), Some(size))),
+ (None, None) => (None, None),
+ };
if lane.len() == LRU_CAP_PER_PID {
retired.push(lane.remove(0));
}
@@ -198,7 +425,8 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
id,
window_id,
screenshot_owner: session.map(str::to_owned),
- screenshot_scale,
+ screenshot_scale: scale,
+ screenshot_window_size: frame_size,
zoom: None,
semantic,
payload,
@@ -221,21 +449,46 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
) -> Result<ScreenshotContext, ToolResult> {
let inner = self.inner.lock().unwrap();
let lane = inner.get(&pid).map(Vec::as_slice).unwrap_or_default();
- let context = match window_id {
- Some(window_id) => lane
- .iter()
- .find(|snapshot| snapshot.window_id == window_id)
- .and_then(|snapshot| snapshot.screenshot(session)),
+ match window_id {
+ Some(window_id) => {
+ let snapshot = lane
+ .iter()
+ .find(|snapshot| snapshot.window_id == window_id)
+ .ok_or_else(|| {
+ screenshot_context_refusal(
+ Some(pid),
+ Some(window_id),
+ MissingFrame::WindowNeverRead,
+ )
+ })?;
+ snapshot.screenshot(session).ok_or_else(|| {
+ let why = if snapshot.screenshot_scale.is_some() {
+ MissingFrame::OtherSession
+ } else {
+ MissingFrame::NoScreenshot
+ };
+ screenshot_context_refusal(Some(pid), Some(window_id), why)
+ })
+ }
None => {
let mut contexts = lane.iter().map(|snapshot| snapshot.screenshot(session));
- contexts.next().flatten().filter(|first| {
- contexts.all(|context| {
- context.is_some_and(|context| (context.scale - first.scale).abs() < 1e-9)
+ let why = match lane {
+ [] => MissingFrame::WindowNeverRead,
+ [_] => MissingFrame::NoScreenshot,
+ _ => MissingFrame::Ambiguous,
+ };
+ contexts
+ .next()
+ .flatten()
+ .filter(|first| {
+ contexts.all(|context| {
+ context
+ .is_some_and(|context| (context.scale - first.scale).abs() < 1e-9)
+ })
})
- })
+ .ok_or_else(|| screenshot_context_refusal(Some(pid), None, why))
}
- };
- context.ok_or_else(|| screenshot_context_refusal(Some(pid), window_id))
+ }
}
/// The native-image / delivered-image scale for a window-relative pixel
@@ -279,7 +532,11 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
});
match (matches.next(), matches.next()) {
(Some(found), None) => Ok(found),
- _ => Err(screenshot_context_refusal(None, Some(window_id))),
+ _ => Err(screenshot_context_refusal(
+ None,
+ Some(window_id),
+ MissingFrame::WindowNeverRead,
+ )),
}
}
@@ -398,35 +655,34 @@ impl<S: SnapshotPayload> SnapshotStore<S> {
else {
return Ok(ResolvedElement::None);
};
- let (snapshot_id, element_index) = parse_token(token).ok_or_else(|| {
- refusal(
- "invalid_element_token",
- "element_token has invalid format".into(),
- )
- })?;
+ let window_arg = args["window_id"].as_u64();
let inner = self.inner.lock().unwrap();
let lane = inner.get(&pid).map(Vec::as_slice).unwrap_or_default();
- let Some(snapshot) = lane.iter().find(|snapshot| snapshot.id == snapshot_id) else {
- return Err(stale_token_refusal(pid, lane));
+ let (snapshot, element_index) = match parse_token(token) {
+ Some((snapshot_id, element_index)) => {
+ let Some(snapshot) = lane.iter().find(|snapshot| snapshot.id == snapshot_id) else {
+ return Err(stale_token_refusal(pid, lane));
+ };
+ (snapshot, element_index)
+ }
+ // A bare row number ("11") is the `[11]` of a markdown read. It
+ // names a row of the window's current snapshot, the one a fresh
+ // read just printed, so resolve it there when the window is known.
+ None => match token.trim().parse::<usize>() {
+ Ok(row) => (row_snapshot(pid, lane, window_arg, row)?, row),
+ Err(_) => return Err(invalid_token_refusal(token, lane)),
+ },
};
- if args["window_id"]
- .as_u64()
- .is_some_and(|window_id| window_id != snapshot.window_id)
- {
+ if window_arg.is_some_and(|window_id| window_id != snapshot.window_id) {
return Err(refusal(
"conflicting_element_target",
"element_token conflicts with window_id".into(),
));
}
- let element = snapshot.payload.retain(element_index).ok_or_else(|| {
- refusal(
- "invalid_element_token",
- format!(
- "element_token element_index {element_index} out of range (snapshot had {} elements)",
- snapshot.payload.len()
- ),
- )
- })?;
+ let element = snapshot
+ .payload
+ .retain(element_index)
+ .ok_or_else(|| out_of_range_refusal(snapshot, element_index))?;
Ok(ResolvedElement::Element {
window_id: snapshot.window_id,
element_index,
@@ -841,6 +1097,170 @@ mod tests {
);
}
+ #[test]
+ fn tree_only_read_keeps_the_screenshot_frame_while_the_window_keeps_its_size() {
+ let cache = SnapshotStore::new();
+ let size = Some((900.0, 680.0));
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("a"), Some(2.0), size);
+ cache.publish_sized_for_session(10, 20, Payload(vec![1, 2]), Some("a"), None, size);
+ assert_eq!(scale(&cache, 20, "a"), Some(2.0), "same size: frame kept");
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("a"), None, size);
+ assert_eq!(
+ scale(&cache, 20, "a"),
+ Some(2.0),
+ "kept across repeated tree reads"
+ );
+
+ cache.publish_sized_for_session(
+ 10,
+ 20,
+ Payload(vec![1]),
+ Some("a"),
+ None,
+ Some((901.0, 680.0)),
+ );
+ assert_eq!(scale(&cache, 20, "a"), None, "resized: frame retired");
+
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("a"), Some(1.0), size);
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("b"), None, size);
+ assert_eq!(scale(&cache, 20, "a"), None, "another session's tree read");
+ assert_eq!(
+ scale(&cache, 20, "b"),
+ None,
+ "never inherits a foreign frame"
+ );
+
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("a"), Some(1.0), size);
+ cache.publish_sized_for_session(10, 20, Payload(vec![1]), Some("a"), None, None);
+ assert_eq!(scale(&cache, 20, "a"), None, "unknown size: frame retired");
+ }
+
+ #[test]
+ fn screenshot_only_read_refreshes_the_frame_and_keeps_tokens() {
+ let cache = SnapshotStore::new();
+ assert_eq!(
+ cache.refresh_screenshot_for_session(10, 20, Some("a"), 1.0, None),
+ None,
+ "nothing to refresh"
+ );
+ let (id, _) = cache
+ .publish_sized_for_session(10, 20, Payload(vec![5, 6, 7]), Some("a"), None, None)
+ .unwrap();
+ assert_eq!(scale(&cache, 20, "a"), None);
+ assert_eq!(
+ cache.refresh_screenshot_for_session(10, 20, Some("a"), 2.0, Some((10.0, 10.0))),
+ Some(id)
+ );
+ assert_eq!(scale(&cache, 20, "a"), Some(2.0));
+ let token = serde_json::json!({ "element_token": token_for(id, 2) });
+ assert!(matches!(
+ cache.resolve(10, &token).unwrap(),
+ ResolvedElement::Element { element: 7, .. }
+ ));
+
+ // A capture-only snapshot has no rows to keep: publish a new one.
+ cache.publish_sized_for_session(10, 21, Payload(vec![]), Some("a"), Some(1.0), None);
+ assert_eq!(
+ cache.refresh_screenshot_for_session(10, 21, Some("a"), 1.0, None),
+ None
+ );
+ }
+
+ #[test]
+ fn missing_frame_refusals_say_why() {
+ let cache = SnapshotStore::new();
+ let reason = |cache: &SnapshotStore<Payload>, window: u64| {
+ let refused = cache
+ .screenshot_context(10, Some(window), Some("a"))
+ .unwrap_err();
+ let structured = refused.structured_content.clone().unwrap();
+ assert_eq!(structured["code"], "screenshot_context_missing");
+ (
+ structured["reason"].as_str().unwrap().to_owned(),
+ crate::snapshot_test_support::text(&refused),
+ )
+ };
+ let (why, text) = reason(&cache, 20);
+ assert_eq!(why, "window_never_read");
+ assert!(
+ text.contains("get_window_state(pid:10, window_id:20)"),
+ "{text}"
+ );
+ assert!(text.contains("another window's"), "{text}");
+
+ cache.publish_for_session(10, 20, Payload(vec![1]), Some("a"), None);
+ assert_eq!(reason(&cache, 20).0, "no_screenshot");
+
+ cache.publish_for_session(10, 20, Payload(vec![1]), Some("b"), Some(1.0));
+ assert_eq!(reason(&cache, 20).0, "other_session");
+ }
+
+ #[test]
+ fn bare_row_number_resolves_against_the_named_windows_current_snapshot() {
+ let cache = SnapshotStore::new();
+ let first = cache.publish(10, 20, Payload(vec![100, 101, 102]));
+ cache.publish(10, 21, Payload(vec![200, 201]));
+
+ let row = serde_json::json!({ "element_token": "2", "window_id": 20 });
+ assert!(matches!(
+ cache.resolve(10, &row).unwrap(),
+ ResolvedElement::Element {
+ window_id: 20,
+ element_index: 2,
+ element: 102
+ }
+ ));
+
+ // Two windows and no window_id: refused, naming the full tokens.
+ let refused = cache
+ .resolve(10, &serde_json::json!({ "element_token": "1" }))
+ .unwrap_err();
+ let text = crate::snapshot_test_support::text(&refused);
+ assert!(text.contains("is a row number, not a token"), "{text}");
+ assert!(
+ text.contains(&format!("\"{}\" (window 20)", token_for(first, 1))),
+ "{text}"
+ );
+
+ // One snapshot for the pid: the row is unambiguous.
+ let only = SnapshotStore::new();
+ only.publish(11, 30, Payload(vec![7, 8]));
+ assert!(matches!(
+ only.resolve(11, &serde_json::json!({ "element_token": "1" }))
+ .unwrap(),
+ ResolvedElement::Element { element: 8, .. }
+ ));
+
+ // Garbage still fails, with the expected shape spelled out.
+ let refused = cache
+ .resolve(10, &serde_json::json!({ "element_token": "row11" }))
+ .unwrap_err();
+ let text = crate::snapshot_test_support::text(&refused);
+ assert!(text.contains("use <snapshot_id>:<row>"), "{text}");
+ }
+
+ #[test]
+ fn out_of_range_row_explains_that_rows_belong_to_one_read() {
+ let cache = SnapshotStore::new();
+ let id = cache.publish(10, 20, Payload(vec![0; 44]));
+ let refused = cache
+ .resolve(
+ 10,
+ &serde_json::json!({ "element_token": token_for(id, 48) }),
+ )
+ .unwrap_err();
+ let text = crate::snapshot_test_support::text(&refused);
+ assert!(
+ text.contains("rows [0] to [43] only (44 element(s))"),
+ "{text}"
+ );
+ assert!(text.contains("smaller max_elements"), "{text}");
+ assert_eq!(
+ refused.structured_content.unwrap()["refusal"]["code"],
+ "invalid_element_token"
+ );
+ }
+
#[test]
fn window_relative_pixels_require_a_current_snapshot() {
let cache = SnapshotStore::<Payload>::new();
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_test_support.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_test_support.rs
index bbfeb7a063..ed2235a29b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_test_support.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_test_support.rs
@@ -18,3 +18,15 @@ pub(crate) fn cache() -> Arc<SnapshotStore<Payload>> {
register_runtime_store(&cache);
cache
}
+
+/// The first text block of a tool result.
+pub(crate) fn text(result: &crate::protocol::ToolResult) -> String {
+ result
+ .content
+ .iter()
+ .find_map(|content| match content {
+ crate::protocol::Content::Text { text, .. } => Some(text.clone()),
+ _ => None,
+ })
+ .unwrap_or_default()
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index e2c60174a3..87021e4025 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -196,17 +196,88 @@ impl ToolDef {
}
}
-/// First argument name absent from the tool's advertised closed schema.
+/// The argument names absent from the tool's advertised closed schema, as one
+/// refusal message that also lists the accepted names, so the caller can fix
+/// every name in one retry instead of guessing one at a time.
fn unknown_argument(def: &ToolDef, args: &Value) -> Option<String> {
let schema = advertised_runtime_input_schema(&def.name, &def.input_schema);
if schema["additionalProperties"] != false {
return None;
}
let properties = schema.get("properties").and_then(Value::as_object);
- args.as_object()?
+ let unknown: Vec<&str> = args
+ .as_object()?
.keys()
- .find(|name| !properties.is_some_and(|properties| properties.contains_key(*name)))
- .cloned()
+ .filter(|name| !properties.is_some_and(|properties| properties.contains_key(*name)))
+ .map(String::as_str)
+ .collect();
+ if unknown.is_empty() {
+ return None;
+ }
+ let accepted: Vec<&str> = properties
+ .map(|properties| {
+ properties
+ .keys()
+ .map(String::as_str)
+ .filter(|name| *name != "session")
+ .collect()
+ })
+ .unwrap_or_default();
+ let noun = if unknown.len() == 1 {
+ "argument"
+ } else {
+ "arguments"
+ };
+ Some(format!(
+ "{noun} {}; accepted: {}",
+ unknown.join(", "),
+ accepted.join(", ")
+ ))
+}
+
+/// `zoom` takes a region as two corners (`x1,y1`-`x2,y2`). Models reach for
+/// `x, y, width, height` (and pass numbers as strings), so rewrite that shape
+/// into corners before the closed schema is checked. Explicit corners win.
+fn normalize_zoom_args(tool_name: &str, args: &mut Value) {
+ if tool_name != "zoom" {
+ return;
+ }
+ let Some(arguments) = args.as_object_mut() else {
+ return;
+ };
+ for key in [
+ "x1", "y1", "x2", "y2", "x", "y", "width", "height", "w", "h",
+ ] {
+ if let Some(number) = arguments
+ .get(key)
+ .and_then(Value::as_str)
+ .and_then(|text| text.trim().parse::<f64>().ok())
+ {
+ arguments.insert(key.to_owned(), serde_json::json!(number));
+ }
+ }
+ for (short, long) in [("w", "width"), ("h", "height")] {
+ if !arguments.contains_key(long) {
+ if let Some(value) = arguments.remove(short) {
+ arguments.insert(long.to_owned(), value);
+ }
+ }
+ }
+ let num = |arguments: &serde_json::Map<String, Value>, key: &str| {
+ arguments.get(key).and_then(Value::as_f64)
+ };
+ for (origin, extent, first, second) in [("x", "width", "x1", "x2"), ("y", "height", "y1", "y2")]
+ {
+ if arguments.contains_key(first) || arguments.contains_key(second) {
+ continue;
+ }
+ if let (Some(start), Some(size)) = (num(arguments, origin), num(arguments, extent)) {
+ arguments.remove(origin);
+ arguments.remove(extent);
+ arguments.insert(first.to_owned(), serde_json::json!(start));
+ arguments.insert(second.to_owned(), serde_json::json!(start + size));
+ }
+ }
}
pub(crate) fn advertised_runtime_input_schema(tool_name: &str, schema: &Value) -> Value {
@@ -1287,6 +1358,7 @@ impl ToolRegistry {
// Normalize deprecated public argument spellings before any policy,
// consent, recording, or implementation layer interprets the call.
normalize_delivery_mode_args(tool.def(), &mut args);
+ normalize_zoom_args(resolved_name, &mut args);
let unknown_argument = unknown_argument(tool.def(), &args);
if let Err(result) = crate::action_target::normalize_action_target(resolved_name, &mut args)
{
@@ -1451,10 +1523,10 @@ impl ToolRegistry {
);
}
- if let Some(name) = unknown_argument {
+ if let Some(detail) = unknown_argument {
return protected_refusal(
"invalid_arguments",
- &format!("{resolved_name}: unknown argument {name}"),
+ &format!("{resolved_name}: unknown {detail}"),
);
}
@@ -6466,3 +6538,77 @@ mod capability_tests {
}));
}
}
+
+#[cfg(test)]
+mod argument_shape_tests {
+ use super::{normalize_zoom_args, unknown_argument, ToolDef};
+ use serde_json::json;
+
+ fn zoom_def() -> ToolDef {
+ ToolDef {
+ name: "zoom".into(),
+ description: String::new(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "window_id": {"type": "integer"}, "pid": {"type": "integer"},
+ "x1": {"type": "number"}, "y1": {"type": "number"},
+ "x2": {"type": "number"}, "y2": {"type": "number"}
+ },
+ "additionalProperties": false
+ }),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ }
+ }
+
+ #[test]
+ fn zoom_accepts_origin_and_size_as_corners() {
+ // The exact shape models sent in the v035 bench (numbers as strings).
+ let mut args = json!({"pid": 1, "window_id": 928, "x": "0", "y": "100", "width": "1180", "height": "640"});
+ normalize_zoom_args("zoom", &mut args);
+ assert_eq!(
+ args,
+ json!({"pid": 1, "window_id": 928, "x1": 0.0, "y1": 100.0, "x2": 1180.0, "y2": 740.0})
+ );
+ assert_eq!(unknown_argument(&zoom_def(), &args), None);
+
+ let mut short = json!({"window_id": 1, "x": 10, "y": 20, "w": 5, "h": 6});
+ normalize_zoom_args("zoom", &mut short);
+ assert_eq!(
+ short,
+ json!({"window_id": 1, "x1": 10.0, "y1": 20.0, "x2": 15.0, "y2": 26.0})
+ );
+
+ let mut corners = json!({"window_id": 1, "x1": "1", "y1": 2, "x2": 3, "y2": 4});
+ normalize_zoom_args("zoom", &mut corners);
+ assert_eq!(
+ corners,
+ json!({"window_id": 1, "x1": 1.0, "y1": 2, "x2": 3, "y2": 4})
+ );
+
+ let mut other_tool = json!({"x": 1, "width": 2});
+ normalize_zoom_args("click", &mut other_tool);
+ assert_eq!(other_tool, json!({"x": 1, "width": 2}));
+ }
+
+ #[test]
+ fn unknown_arguments_are_all_named_with_the_accepted_ones() {
+ let detail = unknown_argument(
+ &zoom_def(),
+ &json!({"window_id": 1, "height": 2, "width": 3}),
+ )
+ .unwrap();
+ assert_eq!(
+ detail,
+ "arguments height, width; accepted: pid, window_id, x1, x2, y1, y2"
+ );
+ let detail = unknown_argument(&zoom_def(), &json!({"window_id": 1, "height": 2})).unwrap();
+ assert!(
+ detail.starts_with("argument height; accepted: "),
+ "{detail}"
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
index e9420949ec..b3cd09a75d 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs
@@ -96,10 +96,11 @@ pub fn element_token_schema() -> Value {
json!({
"type": "string",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
- "description": "Opaque per-snapshot element handle from \
- `structuredContent.elements[].element_token`. Returns an explicit \
- stale error naming the current snapshots once a newer read \
- supersedes it."
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: \
+ row `[11]` of a read whose snapshot_id is s0000002a is \
+ \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). \
+ Returns an explicit stale error naming the current snapshots once a \
+ newer read supersedes it."
})
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
index 330b427a41..a2c9ec4a26 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
@@ -81,7 +81,7 @@ pub fn schema_properties() -> Value {
},
"since": {
"type": "string",
- "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise."
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise."
},
"verbose": {
"type": "boolean",
@@ -135,7 +135,13 @@ impl ViewOptions {
};
let since = match args.get("since") {
None | Some(Value::Null) => None,
- Some(Value::String(s)) if !s.trim().is_empty() => Some(s.trim().to_owned()),
+ // Models sometimes spell "no baseline" as the string "null".
+ Some(Value::String(s))
+ if matches!(s.trim().to_ascii_lowercase().as_str(), "" | "null" | "none") =>
+ {
+ None
+ }
+ Some(Value::String(s)) => Some(s.trim().to_owned()),
Some(_) => {
return Err(ToolResult::error(
"since must be a snapshot_id string from an earlier get_window_state.",
@@ -217,10 +223,21 @@ pub fn apply(
.and_then(Value::as_str)
.map(str::to_owned);
- let outcome = opts
- .since
- .as_deref()
- .map(|since| resolve_since(since, snapshot_id.as_deref(), &md, ctx));
+ // `since:"latest"` names this window's most recent read of the same view.
+ let since_resolved = opts.since.as_deref().map(|since| {
+ if since.eq_ignore_ascii_case(SINCE_LATEST) {
+ latest_snapshot(ctx, snapshot_id.as_deref()).unwrap_or_default()
+ } else {
+ since.to_owned()
+ }
+ });
+ let outcome = since_resolved.as_deref().map(|since| {
+ if since.is_empty() {
+ SinceOutcome::Fallback("no_baseline")
+ } else {
+ resolve_since(since, snapshot_id.as_deref(), &md, ctx)
+ }
+ });
if let Some(sid) = &snapshot_id {
remember(sid, ctx, &md);
}
@@ -238,7 +255,7 @@ pub fn apply(
let body;
match outcome {
Some(SinceOutcome::Diff(report)) => {
- let since = opts.since.as_deref().unwrap_or_default();
+ let since = since_resolved.as_deref().unwrap_or_default();
let focus = if report.is_empty() {
ctx.focus_probe.and_then(|probe| probe())
} else {
@@ -270,11 +287,25 @@ pub fn apply(
if let Some(obj) = structured.as_object_mut() {
obj.remove("tree_markdown");
}
+ if structured.get("truncated").and_then(Value::as_bool) == Some(true) {
+ if let (Some(sid), Some(last)) = (
+ &snapshot_id,
+ parse_rows(&md).iter().filter_map(|row| row.index).max(),
+ ) {
+ header_extra.push_str(&format!(
+ "\nthis read stops at row [{last}]: rows past it from a larger read are not in {sid}"
+ ));
+ }
+ }
body = text;
}
other => {
if let Some(SinceOutcome::Fallback(reason)) = &other {
let since = opts.since.as_deref().unwrap_or_default();
+ let since = match since_resolved.as_deref() {
+ Some(resolved) if !resolved.is_empty() => resolved,
+ _ => since,
+ };
structured["since"] = json!(since);
structured["since_status"] = json!(reason);
header_extra.push_str(&format!(
@@ -297,9 +328,9 @@ pub fn apply(
body = md.clone();
}
}
- if opts.tree_format == TreeFormat::Markdown && snapshot_id.is_some() {
- header_extra
- .push_str("\nelement_token for row [N] = <snapshot_id>:N (e.g. s0000002a:N)");
+ if let (TreeFormat::Markdown, Some(sid)) = (opts.tree_format, &snapshot_id) {
+ let truncated = structured.get("truncated").and_then(Value::as_bool) == Some(true);
+ header_extra.push_str(&token_hint(sid, &md, truncated));
}
}
}
@@ -320,6 +351,48 @@ pub fn apply(
rewrite_tree_block(content, &md, &header_extra, &body);
}
+/// `since` value that names the window's most recent read of the same view.
+pub const SINCE_LATEST: &str = "latest";
+
+/// The header line telling the caller how to address rows, with this read's
+/// real snapshot id and, for a truncated read, where its rows end.
+fn token_hint(snapshot_id: &str, md: &str, truncated: bool) -> String {
+ let rows = parse_rows(md);
+ let example = rows
+ .iter()
+ .filter_map(|row| row.index)
+ .find(|index| *index > 0)
+ .unwrap_or(0);
+ let mut hint = format!(
+ "\nelement_token for row [N] = {snapshot_id}:N (row [{example}] is \"{snapshot_id}:{example}\")"
+ );
+ if truncated {
+ if let Some(last) = rows.iter().filter_map(|row| row.index).max() {
+ hint.push_str(&format!(
+ "; this read stops at row [{last}], so rows past it from a larger read are not in {snapshot_id}"
+ ));
+ }
+ }
+ hint
+}
+
+/// The most recent remembered read of this window with the same view key,
+/// other than the read being shaped now.
+fn latest_snapshot(ctx: &ViewContext<'_>, current_id: Option<&str>) -> Option<String> {
+ let guard = store().lock().unwrap_or_else(|e| e.into_inner());
+ guard
+ .iter()
+ .rev()
+ .find(|s| {
+ s.pid == ctx.pid
+ && s.window_id == ctx.window_id
+ && s.key == ctx.key
+ && Some(s.snapshot_id.as_str()) != current_id
+ && s.at.elapsed() <= STORE_TTL
+ })
+ .map(|s| s.snapshot_id.clone())
+}
+
fn fallback_text(reason: &str) -> &'static str {
match reason {
"unknown_snapshot" => "unknown or expired snapshot_id",
@@ -328,6 +401,9 @@ fn fallback_text(reason: &str) -> &'static str {
"too_much_changed" => "most of the tree changed",
"diff_too_large" => "the trees are too large to compare",
"no_snapshot" => "this read produced no snapshot to compare against",
+ "no_baseline" => {
+ "no earlier read of this window with the same query, max_elements and max_depth"
+ }
_ => "cannot diff",
}
}
@@ -930,10 +1006,94 @@ mod tests {
assert_eq!(s["tree_format"], "markdown");
let text = text_of(&content);
assert!(text.starts_with("window_id=7 pid=9 size=10x10 elements=4 snapshot_id=s000000a1\n"));
- assert!(text.contains("<snapshot_id>:N"));
+ assert!(text.contains("element_token for row [N] = s000000a1:N"));
+ assert!(text.contains("row [1] is \"s000000a1:1\""));
+ assert!(!text.contains("this read stops at row"));
assert!(text.ends_with(OLD));
}
+ #[test]
+ fn truncated_read_names_its_last_row() {
+ let (mut content, mut s) = payload("s000000f4", OLD);
+ s["truncated"] = json!(true);
+ s["truncation_reason"] = json!("timeout");
+ apply(&opts(json!({})), &ctx(7101), &mut content, &mut s);
+ let last = parse_rows(OLD)
+ .iter()
+ .filter_map(|row| row.index)
+ .max()
+ .unwrap();
+ assert!(text_of(&content).contains(&format!(
+ "this read stops at row [{last}], so rows past it from a larger read are not in s000000f4"
+ )));
+ }
+
+ #[test]
+ fn since_null_strings_mean_no_baseline_and_truncated_diffs_say_where_they_stop() {
+ for spelled in ["null", "None", " "] {
+ assert_eq!(opts(json!({ "since": spelled })).since, None, "{spelled:?}");
+ }
+ let window = 7103;
+ let (mut c1, mut s1) = payload("s000000f5", OLD);
+ apply(&opts(json!({})), &ctx(window), &mut c1, &mut s1);
+ let (mut c2, mut s2) = payload("s000000f6", OLD);
+ s2["truncated"] = json!(true);
+ apply(
+ &opts(json!({"since": "latest"})),
+ &ctx(window),
+ &mut c2,
+ &mut s2,
+ );
+ assert_eq!(s2["since_status"], "no_change");
+ assert!(text_of(&c2).contains(
+ "this read stops at row [3]: rows past it from a larger read are not in s000000f6"
+ ));
+ }
+
+ #[test]
+ fn since_latest_diffs_against_the_last_read_of_the_same_view() {
+ let window = 7102;
+ // No earlier read: full read, stated.
+ let (mut c0, mut s0) = payload("s000000f0", OLD);
+ apply(
+ &opts(json!({"since": "latest"})),
+ &ctx(window),
+ &mut c0,
+ &mut s0,
+ );
+ assert_eq!(s0["since_status"], "no_baseline");
+ assert_eq!(s0["tree_markdown"], OLD);
+ assert!(text_of(&c0).contains("since=latest: no earlier read of this window"));
+
+ // A read with another view is not the baseline; the last same-view read is.
+ let mut narrow = ctx(window);
+ narrow.key.query = Some("Save".into());
+ let (mut c1, mut s1) = payload("s000000f1", OLD);
+ apply(&opts(json!({})), &narrow, &mut c1, &mut s1);
+
+ let edited = OLD.replace("\"a\"", "\"ab\"");
+ let (mut c2, mut s2) = payload("s000000f2", &edited);
+ apply(
+ &opts(json!({"since": "latest"})),
+ &ctx(window),
+ &mut c2,
+ &mut s2,
+ );
+ assert_eq!(s2["since_status"], "diff");
+ assert_eq!(s2["since"], "s000000f0");
+ assert!(text_of(&c2).contains("since s000000f0:"));
+
+ let (mut c3, mut s3) = payload("s000000f3", &edited);
+ apply(
+ &opts(json!({"since": "LATEST"})),
+ &ctx(window),
+ &mut c3,
+ &mut s3,
+ );
+ assert_eq!(s3["since_status"], "no_change");
+ assert_eq!(s3["since"], "s000000f2");
+ }
+
#[test]
fn full_output_is_untouched() {
let (mut content, mut s) = payload("s000000a2", OLD);
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 167765331c..de4e06c3cb 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -13483,7 +13483,8 @@ impl Tool for ZoomTool {
ZOOM_DEF.get_or_init(|| ToolDef {
name: "zoom".into(),
description: "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in \
- screenshot pixels, with 20% padding. Output is at most 500 px wide.\n\n\
+ screenshot pixels, with 20% padding. Output is at most 500 px wide. \
+ Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\n\
After a zoom, pass from_zoom=true to click/type_text to auto-translate \
coordinates back to full-window space. Coordinate actions return \
`screenshot_context_missing` when no current snapshot contains a \
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index e69b4d46ad..2f120cd150 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -31,9 +31,9 @@ fn def() -> &'static ToolDef {
\n\
START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\
\n\
- DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\
+ DIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\
\n\
- A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
+ A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
\n\
Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({
@@ -472,23 +472,52 @@ impl Tool for GetWindowStateTool {
.map(|r| r.tree_markdown.clone())
.unwrap_or_default();
- let snapshot_payload = prepared_snapshot.or_else(|| {
- screenshot_resize_scale
- .is_some()
- .then(|| crate::ax::snapshot::AxSnapshot::from_nodes(&[]))
- });
- let (snapshot_id, replaced) = snapshot_payload
- .filter(|_| scope_matched && !observation_only)
- .and_then(|payload| {
- self.state.snapshots.publish_for_session(
+ // Window size in points: the captured bounds, or, for a tree-only
+ // read, a cheap WindowServer lookup. A tree-only read keeps the last
+ // screenshot's pixel frame while the window keeps this size.
+ let window_size = screenshot_frame
+ .as_ref()
+ .map(|(bounds, _)| (bounds.width, bounds.height))
+ .or_else(|| {
+ (prepared_snapshot.is_some() && !observation_only)
+ .then(|| crate::windows::window_bounds_by_id(window_id))
+ .flatten()
+ .map(|bounds| (bounds.width, bounds.height))
+ });
+ // A screenshot-only read adds an image but no new element rows, so it
+ // refreshes the current snapshot's frame and keeps its tokens valid.
+ let refreshed = match (&prepared_snapshot, screenshot_resize_scale) {
+ (None, Some(scale)) if scope_matched && !observation_only => {
+ self.state.snapshots.refresh_screenshot_for_session(
pid,
u64::from(window_id),
- payload,
session_id.as_deref(),
- screenshot_resize_scale,
+ scale,
+ window_size,
)
- })
- .unzip();
+ }
+ _ => None,
+ };
+ let snapshot_payload = prepared_snapshot.or_else(|| {
+ (screenshot_resize_scale.is_some() && refreshed.is_none())
+ .then(|| crate::ax::snapshot::AxSnapshot::from_nodes(&[]))
+ });
+ let (snapshot_id, replaced) = match refreshed {
+ Some(id) => (Some(id), None),
+ None => snapshot_payload
+ .filter(|_| scope_matched && !observation_only)
+ .and_then(|payload| {
+ self.state.snapshots.publish_sized_for_session(
+ pid,
+ u64::from(window_id),
+ payload,
+ session_id.as_deref(),
+ screenshot_resize_scale,
+ window_size,
+ )
+ })
+ .unzip(),
+ };
let capture_id = match (snapshot_id, screenshot.as_ref()) {
(Some(_), Some((png, _, width, height, native_width, native_height, _, _))) => {
match self.state.capture_bindings.publish_window(
@@ -588,6 +617,14 @@ impl Tool for GetWindowStateTool {
if let Some(capture_id) = capture_id {
structured["capture_id"] = serde_json::json!(capture_id);
}
+ if let Some(id) = refreshed {
+ let sid = cua_driver_core::element_token::format_snapshot_id(id);
+ content.push(Content::text(format!(
+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
+ ({sid}:N) stay valid."
+ )));
+ structured["screenshot_refreshed_snapshot"] = serde_json::json!(true);
+ }
// Best-effort-background ladder, rung (2). Both rungs point the agent at
// the same next move: an empty AX tree means element_index has nothing
// to bind to, so the deliberate action is an element px action — read
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/zoom.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/zoom.rs
index 6b5677be61..777c0f22cf 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/zoom.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/zoom.rs
@@ -20,7 +20,7 @@ fn def() -> &'static ToolDef {
name: "zoom".into(),
description: "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot \
pixel coordinates, with 20% padding added on each side. The output image is at most \
- 500 px wide.\n\n\
+ 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\n\
After a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates \
back to full-window space. Coordinate actions return `screenshot_context_missing` \
when no current snapshot contains a screenshot owned by this session. \
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 0195969933..6565815d8b 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -8999,6 +8999,7 @@ impl Tool for ZoomTool {
description: "Zoom into a rectangular region of a window screenshot at full (native) \
resolution. Use this when `get_window_state` returned a resized image and you \
need to read small text, identify icons, or verify UI details.\n\n\
+ Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted). \
Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot \
returned by `get_window_state` (i.e. the resized image if `max_image_dimension` \
is active). The maximum zoom region width is 500 px in scaled-image coordinates.\n\n\
diff --git a/scripts/ci/cua-driver-live-smoke/smoke.py b/scripts/ci/cua-driver-live-smoke/smoke.py
index c2af262507..4f5cfb060f 100644
--- a/scripts/ci/cua-driver-live-smoke/smoke.py
+++ b/scripts/ci/cua-driver-live-smoke/smoke.py
@@ -402,6 +402,10 @@ def run_checks(smoke):
# 7. element_token alone, no pid: the token names its pid.
token_without_pid_check(smoke)
+ # 7b. Lean-read ergonomics from the v035 bench: since:"latest", a bare row
+ # number as element_token, run_actions observe:true, zoom x/y/width/height.
+ lean_read_ergonomics_check(smoke)
+
# 8. The drop-down combo with delivery_mode foreground (last because a
# broken route may leave a popup open), then a click by token must still
# land, which it does not while a popup holds the pointer grab.
@@ -476,6 +480,54 @@ def token_without_pid_check(smoke):
)
+def lean_read_ergonomics_check(smoke):
+ check = "since:latest, bare row token, run_actions observe:true, zoom x/y/width/height"
+ problems = []
+ smoke.read()
+ latest = smoke.read(since="latest")
+ if latest.structured.get("since_status") not in ("diff", "no_change"):
+ problems.append(f"since=latest gave since_status={latest.structured.get('since_status')!r}")
+
+ state = smoke.read()
+ before = count_value(tree(state))
+ increment = find_row(tree(state), BUTTON, "Increment")
+ row_args = {"pid": smoke.pid, "window_id": smoke.window_id, "element_token": str(increment)}
+ click = smoke.call("click", row_args)
+ after = smoke.wait_for_text(f"Count: {(before or 0) + 1}")
+ if click.is_error:
+ problems.append(f"click {json.dumps(row_args)} -> {click.first_line()}")
+ elif f"Count: {(before or 0) + 1}" not in tree(after):
+ problems.append(f"bare row click ok but the count did not move from {before}")
+
+ increment = find_row(tree(after), BUTTON, "Increment")
+ batch = smoke.call("run_actions", {
+ "steps": [{"tool": "click", "args": {"pid": smoke.pid, "window_id": smoke.window_id,
+ "element_token": token(after, increment)}}],
+ "observe": True,
+ })
+ observed = ((batch.structured.get("observation") or {}).get("state") or {})
+ if batch.is_error or observed.get("since_status") not in ("diff", "no_change"):
+ problems.append(
+ f"run_actions observe:true -> is_error={batch.is_error}, "
+ f"since_status={observed.get('since_status')!r}: {batch.first_line()}"
+ )
+
+ smoke.read()
+ zoom = smoke.call("zoom", {"pid": smoke.pid, "window_id": smoke.window_id,
+ "x": "0", "y": "0", "width": "120", "height": "60"})
+ if zoom.is_error and "unknown argument" in zoom.text:
+ problems.append(f"zoom x/y/width/height -> {zoom.first_line()}")
+ smoke.record(
+ check,
+ "fail" if problems else "pass",
+ "; ".join(problems)
+ or f"since=latest -> {latest.structured.get('since_status')}; click element_token "
+ f"{str(increment)!r} + window_id -> Count moved; observe:true -> "
+ f"{observed.get('since_status')}; zoom x/y/width/height -> "
+ f"{'ok' if not zoom.is_error else zoom.first_line()}",
+ )
+
+
def unknown_option_check(smoke):
"""set_value with an option the drop-down does not have is refused and
changes nothing."""
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
index 35e2a2589d..7b2a64f569 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
@@ -106,7 +106,7 @@
"type": "string"
},
"since": {
- "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
"type": "string"
},
"timeout_ms": {
@@ -542,7 +542,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -667,7 +667,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -733,7 +733,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1242,7 +1242,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1366,7 +1366,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1489,7 +1489,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1609,7 +1609,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1695,7 +1695,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -2400,7 +2400,7 @@
"read_only": true
},
{
- "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide.\n\nAfter a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.",
+ "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\nAfter a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.",
"destructive": false,
"idempotent": true,
"input_schema": {
@@ -3467,7 +3467,7 @@
"read_only": false
},
{
- "description": "Run several action tools in ONE call, in order, and stop at the first failure. Use it when you already know the next few actions and do not need to look at the app between them (fill three fields, click, then read the result). Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3480,8 +3480,11 @@
"type": "integer"
},
"observe": {
- "description": "Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to include_screenshot=false and max_elements=200 to stay cheap; pass include_screenshot=true or a larger max_elements to widen it. Omit to read nothing.",
- "type": "object"
+ "description": "Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since=\"latest\" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing.",
+ "type": [
+ "object",
+ "boolean"
+ ]
},
"session": {
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index c8947f4d6c..c967bfaf1e 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -106,7 +106,7 @@
"type": "string"
},
"since": {
- "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
"type": "string"
},
"timeout_ms": {
@@ -541,7 +541,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -652,7 +652,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -701,7 +701,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -912,7 +912,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1027,7 +1027,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1149,7 +1149,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1260,7 +1260,7 @@
"additionalProperties": false,
"properties": {
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1330,7 +1330,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -2036,7 +2036,7 @@
"read_only": true
},
{
- "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide.\n\nAfter a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.",
+ "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\nAfter a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.",
"destructive": false,
"idempotent": true,
"input_schema": {
@@ -3103,7 +3103,7 @@
"read_only": false
},
{
- "description": "Run several action tools in ONE call, in order, and stop at the first failure. Use it when you already know the next few actions and do not need to look at the app between them (fill three fields, click, then read the result). Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3116,8 +3116,11 @@
"type": "integer"
},
"observe": {
- "description": "Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to include_screenshot=false and max_elements=200 to stay cheap; pass include_screenshot=true or a larger max_elements to widen it. Omit to read nothing.",
- "type": "object"
+ "description": "Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since=\"latest\" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing.",
+ "type": [
+ "object",
+ "boolean"
+ ]
},
"session": {
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index 4499e6a9d1..8b81bd6161 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -106,7 +106,7 @@
"type": "string"
},
"since": {
- "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
"type": "string"
},
"timeout_ms": {
@@ -580,7 +580,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -691,7 +691,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -751,7 +751,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1079,7 +1079,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1200,7 +1200,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1318,7 +1318,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -1388,7 +1388,7 @@
"type": "string"
},
"element_token": {
- "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
"type": "string"
},
@@ -2092,7 +2092,7 @@
"read_only": true
},
{
- "description": "Zoom into a rectangular region of a window screenshot at full (native) resolution. Use this when `get_window_state` returned a resized image and you need to read small text, identify icons, or verify UI details.\n\nCoordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot returned by `get_window_state` (i.e. the resized image if `max_image_dimension` is active). The maximum zoom region width is 500 px in scaled-image coordinates.\n\nA 20% padding is automatically added on every side of the requested region so the target remains visible even if the caller's coordinates are slightly off.\n\nAfter a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.\n\nWindows-specific: `window_id` is the canonical addressing field (HWND). `pid` is preferred when available; otherwise the driver resolves the unique current snapshot owned by this session and window.",
+ "description": "Zoom into a rectangular region of a window screenshot at full (native) resolution. Use this when `get_window_state` returned a resized image and you need to read small text, identify icons, or verify UI details.\n\nCorners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted). Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot returned by `get_window_state` (i.e. the resized image if `max_image_dimension` is active). The maximum zoom region width is 500 px in scaled-image coordinates.\n\nA 20% padding is automatically added on every side of the requested region so the target remains visible even if the caller's coordinates are slightly off.\n\nAfter a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.\n\nWindows-specific: `window_id` is the canonical addressing field (HWND). `pid` is preferred when available; otherwise the driver resolves the unique current snapshot owned by this session and window.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -3159,7 +3159,7 @@
"read_only": false
},
{
- "description": "Run several action tools in ONE call, in order, and stop at the first failure. Use it when you already know the next few actions and do not need to look at the app between them (fill three fields, click, then read the result). Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3172,8 +3172,11 @@
"type": "integer"
},
"observe": {
- "description": "Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to include_screenshot=false and max_elements=200 to stay cheap; pass include_screenshot=true or a larger max_elements to widen it. Omit to read nothing.",
- "type": "object"
+ "description": "Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since=\"latest\" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing.",
+ "type": [
+ "object",
+ "boolean"
+ ]
},
"session": {
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",