Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): slim get_window_state and add since diffs (CUA-1192, CUA-1193) (#4743)

Francesco Bonacci45775a9244df
diff --git a/docs/content/docs/cua-driver/reference/cli/mcp.mdx b/docs/content/docs/cua-driver/reference/cli/mcp.mdx
index 5b312e3f0a..f0b51f89e7 100644
--- a/docs/content/docs/cua-driver/reference/cli/mcp.mdx
+++ b/docs/content/docs/cua-driver/reference/cli/mcp.mdx
@@ -142,6 +142,12 @@ cua-driver call click '{"pid":844,"x":100,"y":200}'
# Save the screenshot from a tool response
cua-driver call get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file state.png
+
+# Read only what changed since an earlier snapshot_id
+cua-driver call get_window_state '{"pid":844,"window_id":10725,"since":"s0000002a"}'
+
+# Return the structured elements instead of the markdown tree
+cua-driver call get_window_state '{"pid":844,"window_id":10725,"tree_format":"elements"}'
```
## `cua-driver manifest`
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
index b5d019a317..f1f9436acc 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
@@ -69,7 +69,7 @@ Screenshots, accessibility trees, verification and visual regions of a window. [
| Tool | Description | Platforms |
| --- | --- | --- |
-| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |
+| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Snapshot a window: the accessibility tree plus a screenshot. | macOS, Linux, Windows |
| <span id="get_accessibility_tree"></span>[`get_accessibility_tree`](/cua-driver/reference/mcp-tools/window-state#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
| <span id="verify_state"></span>[`verify_state`](/cua-driver/reference/mcp-tools/window-state#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
| <span id="parse_visual_regions"></span>[`parse_visual_regions`](/cua-driver/reference/mcp-tools/window-state#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
index 6e25873936..84aabb5cfd 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
@@ -14,7 +14,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
-| [`get_window_state`](#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |
+| [`get_window_state`](#get_window_state) | Snapshot a window: the accessibility tree plus a screenshot. | macOS, Linux, Windows |
| [`get_accessibility_tree`](#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
| [`verify_state`](#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
| [`parse_visual_regions`](#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
@@ -28,9 +28,11 @@ Effect: read-only. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.
+Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:"markdown"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:"elements"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `"both"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.
-START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.
+START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).
+
+DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.
A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
@@ -44,7 +46,7 @@ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `own
| <span id="get_window_state--macos--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture. |
| <span id="get_window_state--macos--max_depth"></span>`max_depth` | `integer` | | Cap on walk depth. Default 25. Minimum: 1. |
| <span id="get_window_state--macos--max_dimension"></span>`max_dimension` | `integer` | | Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved): the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. Minimum: 1. |
-| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000. Minimum: 1. |
+| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut. Minimum: 1. |
| <span id="get_window_state--macos--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior. Minimum: 0. |
| <span id="get_window_state--macos--pid"></span>`pid` | `integer` | required | Target process ID. |
| <span id="get_window_state--macos--query"></span>`query` | `string` | | Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read. |
@@ -55,15 +57,17 @@ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `own
</Tab>
<Tab value="Linux">
-Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `set_value`, etc.
+Walk a running app's AT-SPI tree and return it ONCE, as compact Markdown by default (`tree_format:"markdown"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:"elements"` for the structured `elements` array instead (explicit `element_token`, `role`, `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`), or `"both"` for both (about twice the size). Set `query` to project the tree to matching rows plus their ancestor chain while preserving original indices; `total_element_count` reports the complete snapshot.
+
+Size: a read is bounded by default (`max_elements` 250 actionable nodes). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the walker's own limits (≤5 000 nodes).
-PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.
+DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Linux yet.
Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.
The mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title); the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.
-Optional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.
+Optional `max_elements` / `max_depth` bound the AT-SPI walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250.
TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.
@@ -75,11 +79,11 @@ POPUP MENUS: a context menu / popover / combo list is an override-redirect windo
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="get_window_state--linux--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |
+| <span id="get_window_state--linux--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |
| <span id="get_window_state--linux--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action). |
| <span id="get_window_state--linux--max_depth"></span>`max_depth` | `integer` | | Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps. Minimum: 1. |
| <span id="get_window_state--linux--max_dimension"></span>`max_dimension` | `integer` | | Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted. Minimum: 1. |
-| <span id="get_window_state--linux--max_elements"></span>`max_elements` | `integer` | | Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees. Minimum: 1. |
+| <span id="get_window_state--linux--max_elements"></span>`max_elements` | `integer` | | Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. |
| <span id="get_window_state--linux--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior. Minimum: 0. |
| <span id="get_window_state--linux--pid"></span>`pid` | `integer` | required | Process ID that owns the window. |
| <span id="get_window_state--linux--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |
@@ -90,17 +94,19 @@ POPUP MENUS: a context menu / popover / combo list is an override-redirect windo
</Tab>
<Tab value="Windows">
-Walk a running app's UIA tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `scroll`, etc.
+Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:"markdown"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:"elements"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `"both"` for both (about twice the size).
-INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.
+Size: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).
-PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of UIA patterns exposed as actions, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side.
+DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.
+
+INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.
The UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about: the driver never picks a window implicitly.
`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.
-Set `query` to a case-insensitive substring to project BOTH `tree_markdown` and `structuredContent.elements` to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.
+Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.
Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.
@@ -108,7 +114,7 @@ The mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk e
Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).
-Optional `max_elements` / `max_depth` bound the UIA walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤5 000 elements, depth ≤25).
+Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.
CHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.
@@ -118,11 +124,11 @@ Windows requires no special permissions.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="get_window_state--windows--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |
+| <span id="get_window_state--windows--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |
| <span id="get_window_state--windows--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action). |
| <span id="get_window_state--windows--max_depth"></span>`max_depth` | `integer` | | Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees. Minimum: 1. |
| <span id="get_window_state--windows--max_dimension"></span>`max_dimension` | `integer` | | Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins. Minimum: 1. |
-| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees. Minimum: 1. |
+| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. |
| <span id="get_window_state--windows--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default. Minimum: 0. |
| <span id="get_window_state--windows--pid"></span>`pid` | `integer` | required | Process ID from `list_apps`. |
| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |
@@ -138,7 +144,11 @@ Windows requires no special permissions.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |
+| <span id="get_window_state--full_output"></span>`full_output` | `boolean` | | Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`. |
+| <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. |
| <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |
+| <span id="get_window_state--tree_format"></span>`tree_format` | `"markdown" \| "elements" \| "both"` | | Which tree representation to return. Default "markdown": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. "elements": the structured `elements` array only (per-element element_token, frame, parent_index, value). "both": the two together, as before; roughly doubles the size. |
+| <span id="get_window_state--verbose"></span>`verbose` | `boolean` | | Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context. |
**Example arguments**
diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.json
index 6a668a2623..80d7ca449f 100644
--- a/libs/cua-driver/contract/manifest.json
+++ b/libs/cua-driver/contract/manifest.json
@@ -1652,6 +1652,10 @@
"input_schema": {
"additionalProperties": false,
"properties": {
+ "full_output": {
+ "description": "Restore the previous full response (both representations, all\nmetadata, platform walk limits).",
+ "type": "boolean"
+ },
"include_accessibility_tree": {
"type": "boolean"
},
@@ -1689,12 +1693,29 @@
"session": {
"type": "string"
},
+ "since": {
+ "description": "Return only what changed since this earlier `snapshot_id`.",
+ "type": "string"
+ },
"timeout_ms": {
"description": "Wall-clock budget for the accessibility walk in milliseconds\n(default 1000 on every platform). A walk that runs out returns a\npartial tree flagged `truncated` rather than failing.",
"maximum": 120000,
"minimum": 100,
"type": "integer"
},
+ "tree_format": {
+ "description": "Tree representation to return: `markdown` (default), `elements`, or\n`both`.",
+ "enum": [
+ "markdown",
+ "elements",
+ "both"
+ ],
+ "type": "string"
+ },
+ "verbose": {
+ "description": "Include `_note` and the full `background_input` report.",
+ "type": "boolean"
+ },
"window_id": {
"minimum": 1,
"type": "integer"
@@ -1919,6 +1940,18 @@
"null"
]
},
+ "since": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "since_status": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
"snapshot_id": {
"type": [
"string",
@@ -1932,6 +1965,18 @@
"null"
]
},
+ "tree_diff": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "tree_format": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
"tree_markdown": {
"type": [
"string",
@@ -1944,6 +1989,12 @@
"null"
]
},
+ "truncation_hint": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
"truncation_reason": {
"type": [
"string",
diff --git a/libs/cua-driver/python/src/cua_driver/_native_contract.py b/libs/cua-driver/python/src/cua_driver/_native_contract.py
index 9a3c21e3b0..d269f746ad 100644
--- a/libs/cua-driver/python/src/cua_driver/_native_contract.py
+++ b/libs/cua-driver/python/src/cua_driver/_native_contract.py
@@ -4157,7 +4157,7 @@ class _UniffiFfiConverterTypeGetSessionStateInput(_UniffiConverterRustBuffer):
@dataclass
class GetWindowStateInput:
- def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int], timeout_ms:typing.Optional[int] = _DEFAULT):
+ def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int], timeout_ms:typing.Optional[int] = _DEFAULT, tree_format:typing.Optional[str] = _DEFAULT, since:typing.Optional[str] = _DEFAULT, verbose:typing.Optional[bool] = _DEFAULT, full_output:typing.Optional[bool] = _DEFAULT):
self.pid = pid
self.window_id = window_id
self.session = session
@@ -4173,12 +4173,28 @@ class GetWindowStateInput:
self.timeout_ms = None
else:
self.timeout_ms = timeout_ms
+ if tree_format is _DEFAULT:
+ self.tree_format = None
+ else:
+ self.tree_format = tree_format
+ if since is _DEFAULT:
+ self.since = None
+ else:
+ self.since = since
+ if verbose is _DEFAULT:
+ self.verbose = None
+ else:
+ self.verbose = verbose
+ if full_output is _DEFAULT:
+ self.full_output = None
+ else:
+ self.full_output = full_output
def __str__(self):
- return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={}, timeout_ms={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension, self.timeout_ms)
+ return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={}, timeout_ms={}, tree_format={}, since={}, verbose={}, full_output={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension, self.timeout_ms, self.tree_format, self.since, self.verbose, self.full_output)
def __eq__(self, other):
if self.pid != other.pid:
return False
@@ -4204,6 +4220,14 @@ class GetWindowStateInput:
return False
if self.timeout_ms != other.timeout_ms:
return False
+ if self.tree_format != other.tree_format:
+ return False
+ if self.since != other.since:
+ return False
+ if self.verbose != other.verbose:
+ return False
+ if self.full_output != other.full_output:
+ return False
return True
class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):
@@ -4222,6 +4246,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):
max_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),
max_image_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),
timeout_ms=_UniffiFfiConverterOptionalUInt32.read(buf),
+ tree_format=_UniffiFfiConverterOptionalString.read(buf),
+ since=_UniffiFfiConverterOptionalString.read(buf),
+ verbose=_UniffiFfiConverterOptionalBoolean.read(buf),
+ full_output=_UniffiFfiConverterOptionalBoolean.read(buf),
)
@staticmethod
@@ -4238,6 +4266,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):
_UniffiFfiConverterOptionalUInt32.check_lower(value.max_dimension)
_UniffiFfiConverterOptionalUInt32.check_lower(value.max_image_dimension)
_UniffiFfiConverterOptionalUInt32.check_lower(value.timeout_ms)
+ _UniffiFfiConverterOptionalString.check_lower(value.tree_format)
+ _UniffiFfiConverterOptionalString.check_lower(value.since)
+ _UniffiFfiConverterOptionalBoolean.check_lower(value.verbose)
+ _UniffiFfiConverterOptionalBoolean.check_lower(value.full_output)
@staticmethod
def write(value, buf):
@@ -4253,6 +4285,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):
_UniffiFfiConverterOptionalUInt32.write(value.max_dimension, buf)
_UniffiFfiConverterOptionalUInt32.write(value.max_image_dimension, buf)
_UniffiFfiConverterOptionalUInt32.write(value.timeout_ms, buf)
+ _UniffiFfiConverterOptionalString.write(value.tree_format, buf)
+ _UniffiFfiConverterOptionalString.write(value.since, buf)
+ _UniffiFfiConverterOptionalBoolean.write(value.verbose, buf)
+ _UniffiFfiConverterOptionalBoolean.write(value.full_output, buf)
@dataclass
class HotkeyInput:
@@ -8195,7 +8231,7 @@ class _UniffiFfiConverterSequenceTypeSnapshotImage(_UniffiConverterRustBuffer):
@dataclass
class WindowStateOutput:
- def __init__(self, *, pid:int, window_id:int, snapshot_id:typing.Optional[str], app_name:typing.Optional[str], window_title:typing.Optional[str], tree_markdown:typing.Optional[str], elements:typing.Optional[typing.List[WindowElement]], element_count:typing.Optional[int], total_element_count:typing.Optional[int], returned_element_count:typing.Optional[int], filtered_element_count:typing.Optional[int], elements_complete:typing.Optional[bool], degraded:typing.Optional[bool], degraded_reason:typing.Optional[str], truncated:typing.Optional[bool], truncation_reason:typing.Optional[str], screenshot_width:typing.Optional[int], screenshot_height:typing.Optional[int], screenshot_scale:typing.Optional[float], screenshot_mime_type:typing.Optional[str], screenshot_file_path:typing.Optional[str], screenshot_frame_valid:typing.Optional[bool], window_bounds:typing.Optional[WindowBounds], images:typing.List[SnapshotImage]):
+ def __init__(self, *, pid:int, window_id:int, snapshot_id:typing.Optional[str], app_name:typing.Optional[str], window_title:typing.Optional[str], tree_markdown:typing.Optional[str], elements:typing.Optional[typing.List[WindowElement]], element_count:typing.Optional[int], total_element_count:typing.Optional[int], returned_element_count:typing.Optional[int], filtered_element_count:typing.Optional[int], elements_complete:typing.Optional[bool], degraded:typing.Optional[bool], degraded_reason:typing.Optional[str], truncated:typing.Optional[bool], truncation_reason:typing.Optional[str], truncation_hint:typing.Optional[str], tree_format:typing.Optional[str], since:typing.Optional[str], since_status:typing.Optional[str], tree_diff:typing.Optional[str], screenshot_width:typing.Optional[int], screenshot_height:typing.Optional[int], screenshot_scale:typing.Optional[float], screenshot_mime_type:typing.Optional[str], screenshot_file_path:typing.Optional[str], screenshot_frame_valid:typing.Optional[bool], window_bounds:typing.Optional[WindowBounds], images:typing.List[SnapshotImage]):
self.pid = pid
self.window_id = window_id
self.snapshot_id = snapshot_id
@@ -8212,6 +8248,11 @@ class WindowStateOutput:
self.degraded_reason = degraded_reason
self.truncated = truncated
self.truncation_reason = truncation_reason
+ self.truncation_hint = truncation_hint
+ self.tree_format = tree_format
+ self.since = since
+ self.since_status = since_status
+ self.tree_diff = tree_diff
self.screenshot_width = screenshot_width
self.screenshot_height = screenshot_height
self.screenshot_scale = screenshot_scale
@@ -8225,7 +8266,7 @@ class WindowStateOutput:
def __str__(self):
- return "WindowStateOutput(pid={}, window_id={}, snapshot_id={}, app_name={}, window_title={}, tree_markdown={}, elements={}, element_count={}, total_element_count={}, returned_element_count={}, filtered_element_count={}, elements_complete={}, degraded={}, degraded_reason={}, truncated={}, truncation_reason={}, screenshot_width={}, screenshot_height={}, screenshot_scale={}, screenshot_mime_type={}, screenshot_file_path={}, screenshot_frame_valid={}, window_bounds={}, images={})".format(self.pid, self.window_id, self.snapshot_id, self.app_name, self.window_title, self.tree_markdown, self.elements, self.element_count, self.total_element_count, self.returned_element_count, self.filtered_element_count, self.elements_complete, self.degraded, self.degraded_reason, self.truncated, self.truncation_reason, self.screenshot_width, self.screenshot_height, self.screenshot_scale, self.screenshot_mime_type, self.screenshot_file_path, self.screenshot_frame_valid, self.window_bounds, self.images)
+ return "WindowStateOutput(pid={}, window_id={}, snapshot_id={}, app_name={}, window_title={}, tree_markdown={}, elements={}, element_count={}, total_element_count={}, returned_element_count={}, filtered_element_count={}, elements_complete={}, degraded={}, degraded_reason={}, truncated={}, truncation_reason={}, truncation_hint={}, tree_format={}, since={}, since_status={}, tree_diff={}, screenshot_width={}, screenshot_height={}, screenshot_scale={}, screenshot_mime_type={}, screenshot_file_path={}, screenshot_frame_valid={}, window_bounds={}, images={})".format(self.pid, self.window_id, self.snapshot_id, self.app_name, self.window_title, self.tree_markdown, self.elements, self.element_count, self.total_element_count, self.returned_element_count, self.filtered_element_count, self.elements_complete, self.degraded, self.degraded_reason, self.truncated, self.truncation_reason, self.truncation_hint, self.tree_format, self.since, self.since_status, self.tree_diff, self.screenshot_width, self.screenshot_height, self.screenshot_scale, self.screenshot_mime_type, self.screenshot_file_path, self.screenshot_frame_valid, self.window_bounds, self.images)
def __eq__(self, other):
if self.pid != other.pid:
return False
@@ -8259,6 +8300,16 @@ class WindowStateOutput:
return False
if self.truncation_reason != other.truncation_reason:
return False
+ if self.truncation_hint != other.truncation_hint:
+ return False
+ if self.tree_format != other.tree_format:
+ return False
+ if self.since != other.since:
+ return False
+ if self.since_status != other.since_status:
+ return False
+ if self.tree_diff != other.tree_diff:
+ return False
if self.screenshot_width != other.screenshot_width:
return False
if self.screenshot_height != other.screenshot_height:
@@ -8297,6 +8348,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):
degraded_reason=_UniffiFfiConverterOptionalString.read(buf),
truncated=_UniffiFfiConverterOptionalBoolean.read(buf),
truncation_reason=_UniffiFfiConverterOptionalString.read(buf),
+ truncation_hint=_UniffiFfiConverterOptionalString.read(buf),
+ tree_format=_UniffiFfiConverterOptionalString.read(buf),
+ since=_UniffiFfiConverterOptionalString.read(buf),
+ since_status=_UniffiFfiConverterOptionalString.read(buf),
+ tree_diff=_UniffiFfiConverterOptionalString.read(buf),
screenshot_width=_UniffiFfiConverterOptionalUInt32.read(buf),
screenshot_height=_UniffiFfiConverterOptionalUInt32.read(buf),
screenshot_scale=_UniffiFfiConverterOptionalFloat64.read(buf),
@@ -8325,6 +8381,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):
_UniffiFfiConverterOptionalString.check_lower(value.degraded_reason)
_UniffiFfiConverterOptionalBoolean.check_lower(value.truncated)
_UniffiFfiConverterOptionalString.check_lower(value.truncation_reason)
+ _UniffiFfiConverterOptionalString.check_lower(value.truncation_hint)
+ _UniffiFfiConverterOptionalString.check_lower(value.tree_format)
+ _UniffiFfiConverterOptionalString.check_lower(value.since)
+ _UniffiFfiConverterOptionalString.check_lower(value.since_status)
+ _UniffiFfiConverterOptionalString.check_lower(value.tree_diff)
_UniffiFfiConverterOptionalUInt32.check_lower(value.screenshot_width)
_UniffiFfiConverterOptionalUInt32.check_lower(value.screenshot_height)
_UniffiFfiConverterOptionalFloat64.check_lower(value.screenshot_scale)
@@ -8352,6 +8413,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):
_UniffiFfiConverterOptionalString.write(value.degraded_reason, buf)
_UniffiFfiConverterOptionalBoolean.write(value.truncated, buf)
_UniffiFfiConverterOptionalString.write(value.truncation_reason, buf)
+ _UniffiFfiConverterOptionalString.write(value.truncation_hint, buf)
+ _UniffiFfiConverterOptionalString.write(value.tree_format, buf)
+ _UniffiFfiConverterOptionalString.write(value.since, buf)
+ _UniffiFfiConverterOptionalString.write(value.since_status, buf)
+ _UniffiFfiConverterOptionalString.write(value.tree_diff, buf)
_UniffiFfiConverterOptionalUInt32.write(value.screenshot_width, buf)
_UniffiFfiConverterOptionalUInt32.write(value.screenshot_height, buf)
_UniffiFfiConverterOptionalFloat64.write(value.screenshot_scale, buf)
diff --git a/libs/cua-driver/python/tests/test_remote_channel.py b/libs/cua-driver/python/tests/test_remote_channel.py
index d07f2d7a00..aed628981f 100644
--- a/libs/cua-driver/python/tests/test_remote_channel.py
+++ b/libs/cua-driver/python/tests/test_remote_channel.py
@@ -346,6 +346,8 @@ class NativeWindowTests(unittest.IsolatedAsyncioTestCase):
"max_depth": 3,
"max_dimension": 800,
"max_image_dimension": 1600,
+ # The typed SDK keeps the full response unless told otherwise.
+ "full_output": True,
},
)
diff --git a/libs/cua-driver/rust/CHANGELOG.md b/libs/cua-driver/rust/CHANGELOG.md
index 98a9eafd0f..bb529c4e5f 100644
--- a/libs/cua-driver/rust/CHANGELOG.md
+++ b/libs/cua-driver/rust/CHANGELOG.md
@@ -2,6 +2,10 @@
## Unreleased
+### Features
+
+* **cua-driver:** `get_window_state` returns one tree representation by default (compact markdown; `tree_format` selects `elements` or `both`), walks at most 250 nodes unless `max_elements` says otherwise and says when it truncated, and omits `_note` / `background_input` unless `verbose:true`. `full_output:true` restores the previous full response. New `since:<snapshot_id>` returns only added, changed and removed rows (CUA-1192, CUA-1193). The typed SDKs keep the full response unless a shape is requested.
+
### Bug Fixes
* **cua-driver:** deliver X11 key-down before the tap delay and finish background keyboard delivery before closing the input connection.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
index 56ab893861..025f0b2ea7 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
@@ -77,7 +77,7 @@ sentence naming the call that targets each one
(`get_window_state(pid, window_id=<that id>)`). When one overlaps the window
the screenshot is taken from the screen (`screenshot_composited:true`) so the
menu or dialog is visible; the window's own drawable never shows them. The
-payload states its frame (`coordinate_frame:"window"`, `frame_note`): `x`/`y`
+payload states its frame (`coordinate_frame:"window"`, plus `frame_note` with `verbose:true`): `x`/`y`
of the pointer tools are pixels of THIS screenshot. Closed menus are listed
with a `description` ("closed menu with N items…") and are not walked;
click the menu (a real press) and read the popup by its window_id.
@@ -212,6 +212,12 @@ doctor` now probes `org.a11y.Bus` for real (not just "is there a bus?")
daemon as root against a user session is the Linux analogue of the Windows
"Session 0" isolation problem.
+`get_window_state` returns one compact markdown tree capped at 250 actionable
+nodes by default (`full_output:true` restores 5 000 nodes and both forms), and
+`since:<snapshot_id>` returns only added/changed/removed rows. The focused
+element is not reported on Linux yet, so an unchanged read says only
+`no change since …`.
+
An empty AT-SPI walk is now surfaced honestly: `get_window_state` sets
`degraded: true` + a `degraded_reason` (instead of a bare `elements: []`) so a
caller can tell "this window genuinely has no controls" apart from "the a11y
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index a8cf28bdb9..9fc884e7b9 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -122,6 +122,11 @@ For authorized foreground input, use the Cua action's
When a cua-driver call surprises you, diagnose cua-driver first:
+- **Large tree or repeated reads?** `get_window_state` returns one compact
+ markdown tree capped at 250 nodes; pass `since:<snapshot_id>` for a diff
+ (`no change since …; focused element is …` names the focused row on macOS)
+ and `tree_format:"elements"` for structured rows. `full_output:true` restores
+ the old payload (both forms, `_note`, `background_input`, 2 000 nodes).
- **Empty `tree_markdown`?** `get_window_state` returns **both** the
AX tree and a screenshot by default: there's nothing to configure and
no capture mode to pick. An empty tree means the surface isn't AX (a
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 05cb1225a0..4b78f68900 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -67,7 +67,7 @@ Check the installed version and advertised schema before using unfamiliar parame
1. Select the exact target on each action. A session is lifecycle metadata, not capture scope or permission authority.
2. Observe before input and verify the outcome at checkpoints, not after every action. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.
-3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`.
+3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`. Reads are bounded and return one tree form by default; follow up with `since:<snapshot_id>` instead of re-reading a large window.
4. Keep background window actions non-interfering. Foreground delivery and desktop input require authorization for visible control; an unavailable route is not permission to escalate.
5. Never infer pixels from a missing image, a different window, or an unaccounted-for resized preview. Capture failure and an empty accessibility tree are different failures.
6. Keep one controller for a shared desktop. Distinct sessions/cursors do not isolate focus, keyboard input, application state, or snapshot caches.
@@ -79,7 +79,7 @@ Check the installed version and advertised schema before using unfamiliar parame
| --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Missing binary, mismatched daemon, unknown tool/field | [Runtime preflight](RUNTIME.md#preflight-and-transport) |
| Stale token or ambiguous window | Refresh `list_windows` / `get_window_state`; choose the intended live target |
-| Large or sparse tree | [Bounded observation](WORKFLOW.md#observe) |
+| Large, truncated, or sparse tree; repeated reads | [Bounded observation and `since` diffs](WORKFLOW.md#observe) |
| `surface_identity_unproven` or screenshot permission wait | [Wayland capture recovery](LINUX.md#capture-recovery) |
| `background_unavailable` | Verify current state; ask before foreground/desktop control if not already authorized |
| Text did not visibly change | Reobserve before retrying; [text and value semantics](WORKFLOW.md#act-once) |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index 10bc995f2e..3f1a1012ed 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -376,6 +376,11 @@ When a cua-driver surprises you, diagnose cua-driver first:
click in.
- **`Invalid window handle (0x80070578)`?** The HWND you passed is
stale (window closed, recreated). Re-resolve via `list_windows`.
+- **Large tree or repeated reads?** `get_window_state` returns one compact
+ markdown tree capped at 250 nodes (`full_output:true` restores 5 000 nodes
+ and both forms). Pass `since:<snapshot_id>` for added/changed/removed rows;
+ the focused element is not reported on Windows yet, so an unchanged read
+ says only `no change since …`.
- **Empty `tree_markdown` / sparse UIA tree?** Some apps populate
their UIA tree lazily on first call; retry `get_window_state`
once. If still empty, the app has no UIA provider: fall back to an
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
index 80ea3c4536..0086d3e6f7 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
@@ -37,10 +37,12 @@ These objects are argument fragments for tools advertising `target`, not standal
`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Start with `query` (or `max_elements`) instead of a full read, and verify at checkpoints rather than after every action. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.
-Prefer `structuredContent.elements` in MCP (the CLI prints structured fields directly) over parsing `tree_markdown`. Rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.
+By default the tree comes back **once**, as compact `tree_markdown` (`tree_format:"markdown"`). A row `[N]` is addressed with `element_token` `<snapshot_id>:N`; the `snapshot_id` is in the response header and in `structuredContent`. Use `tree_format:"elements"` for the structured `elements` array when you need explicit tokens, frames, parents, or values, or `"both"` for the old double payload (about twice the size). `full_output:true` restores the previous full response, including the platform's walk limits. In `elements`, rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.
- Use `query` to project matching rows plus ancestors without renumbering their indices.
-- Use `max_elements` / `max_depth` to bound the walk, and compare returned/total counts. Truncation does not prove absence.
+- A read walks at most 250 nodes by default. When it stops there the response says `Tree truncated at max_elements=…` (`truncated:true`, `truncation_hint`). Pass a larger `max_elements`, or narrow with `query` / `max_depth`. Truncation does not prove absence.
+- After the first read, pass `since:<snapshot_id>` to get only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.
+- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.
- Use `include_screenshot:false` only when tree-only observation is enough; it cannot ground a pixel action.
- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. Check the installed schema first.
- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs
index ee76c55116..25fb314eb5 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs
@@ -104,6 +104,32 @@ pub struct GetWindowStateInput {
#[schemars(schema_with = "timeout_ms_schema")]
#[uniffi(default = None)]
pub timeout_ms: Option<u32>,
+ /// Tree representation to return: `markdown` (default), `elements`, or
+ /// `both`.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schemars(schema_with = "tree_format_schema")]
+ #[uniffi(default = None)]
+ pub tree_format: Option<String>,
+ /// Return only what changed since this earlier `snapshot_id`.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schemars(schema_with = "string_schema")]
+ #[uniffi(default = None)]
+ pub since: Option<String>,
+ /// Include `_note` and the full `background_input` report.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schemars(schema_with = "bool_schema")]
+ #[uniffi(default = None)]
+ pub verbose: Option<bool>,
+ /// Restore the previous full response (both representations, all
+ /// metadata, platform walk limits).
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ #[schemars(schema_with = "bool_schema")]
+ #[uniffi(default = None)]
+ pub full_output: Option<bool>,
+}
+
+fn tree_format_schema(_: &mut schemars::SchemaGenerator) -> schemars::Schema {
+ schemars::json_schema!({"type":"string", "enum":["markdown","elements","both"]})
}
/// Bounds of the accessibility-walk budget, shared with every live backend
@@ -303,6 +329,24 @@ pub struct WindowStateOutput {
pub truncated: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub truncation_reason: Option<String>,
+ /// Plain-language line saying the tree was cut at `max_elements` and how
+ /// to read more.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub truncation_hint: Option<String>,
+ /// Which representation this response carries.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub tree_format: Option<String>,
+ /// The `since` snapshot_id the caller asked to diff against.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub since: Option<String>,
+ /// `diff`, `no_change`, or the reason a full read was returned instead
+ /// (`unknown_snapshot`, `other_window`, `view_changed`, `too_much_changed`,
+ /// `diff_too_large`, `no_snapshot`).
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub since_status: Option<String>,
+ /// Added/changed/removed rows against `since`, one per line.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub tree_diff: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub screenshot_width: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index 33217a6c7b..09d691965e 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -113,6 +113,7 @@ pub mod video_ffmpeg;
pub mod walk_budget;
pub mod window_inspection;
pub mod window_observation;
+pub mod window_state_view;
pub mod window_target;
pub use cua_driver_contract::{CaptureScope, EscalationReason, TOOL_INVOCATION_FAILED_CODE};
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
index c78c09135b..324832362f 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
@@ -217,10 +217,15 @@ pub fn truncation_note(
Some(other) => other.to_owned(),
None => "the walk stopped early".to_owned(),
};
+ let remedy = if reason == Some("node_budget") {
+ "retry with a larger max_elements"
+ } else {
+ "retry with a larger timeout_ms (e.g. 5000)"
+ };
format!(
"⚠️ PARTIAL TREE: {why} after {visited} node(s) ({pending} discovered but not visited). \
Every element listed is real; elements after the cut are missing. If the element you \
- need is absent, retry with a larger timeout_ms (e.g. 5000) or narrow with query / max_depth."
+ need is absent, {remedy} or narrow with query / max_depth."
)
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
new file mode 100644
index 0000000000..330b427a41
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
@@ -0,0 +1,1074 @@
+//! Response shaping for `get_window_state`, shared by every platform backend.
+//!
+//! A backend walks its accessibility API and builds the complete legacy
+//! payload (structured `elements`, `tree_markdown`, a text block with the same
+//! markdown, plus metadata). [`apply`] then reduces that payload to what a
+//! model needs:
+//!
+//! * ONE tree representation (`tree_format`, default compact markdown),
+//! * trimmed metadata (`_note`, `background_input`) unless `verbose`,
+//! * a stated truncation line when the walk hit the `max_elements` budget,
+//! * with `since: <snapshot_id>`, only the rows that changed against that
+//! snapshot, falling back to a full read when the snapshot is unknown.
+//!
+//! `full_output: true` restores the previous payload byte for byte (both
+//! representations, all metadata, the platform walk caps).
+//!
+//! The diff works on the rendered markdown rows, which every backend emits in
+//! the same `- [N] Role "label" ...` shape, so it also sees display-only rows
+//! (static text) that the structured `elements` array omits.
+
+use crate::protocol::{Content, ToolResult};
+use serde_json::{json, Value};
+use std::collections::VecDeque;
+use std::sync::{Mutex, OnceLock};
+use std::time::{Duration, Instant};
+
+/// Default accessibility-node budget for a model-facing read. The platform
+/// walkers default to 2 000 (macOS) or 5 000 (Windows, Linux) nodes, which
+/// renders 40-100K characters for an ordinary window.
+pub const DEFAULT_MAX_ELEMENTS: usize = 250;
+
+/// Snapshots remembered per (pid, window) for `since`.
+const STORE_PER_WINDOW: usize = 8;
+/// Snapshots remembered across all windows.
+const STORE_TOTAL: usize = 32;
+/// A remembered snapshot stops being a valid `since` baseline after this long.
+const STORE_TTL: Duration = Duration::from_secs(15 * 60);
+/// Largest old-rows x new-rows product the line diff will compute.
+const DIFF_CELL_LIMIT: usize = 6_000_000;
+/// A diff larger than this share (percent) of the full markdown is not worth
+/// it: return the full read instead.
+const DIFF_MAX_PERCENT_OF_FULL: usize = 60;
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum TreeFormat {
+ /// Compact markdown only (default).
+ Markdown,
+ /// Structured `elements` only.
+ Elements,
+ /// Both, as before this change.
+ Both,
+}
+
+impl TreeFormat {
+ pub fn as_str(self) -> &'static str {
+ match self {
+ TreeFormat::Markdown => "markdown",
+ TreeFormat::Elements => "elements",
+ TreeFormat::Both => "both",
+ }
+ }
+}
+
+/// Caller-controlled shape of the response.
+#[derive(Clone, Debug)]
+pub struct ViewOptions {
+ pub tree_format: TreeFormat,
+ pub verbose: bool,
+ pub full_output: bool,
+ pub since: Option<String>,
+}
+
+/// JSON schema fragments for the shared arguments, so every backend declares
+/// the same properties.
+pub fn schema_properties() -> Value {
+ json!({
+ "tree_format": {
+ "type": "string",
+ "enum": ["markdown", "elements", "both"],
+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size."
+ },
+ "since": {
+ "type": "string",
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise."
+ },
+ "verbose": {
+ "type": "boolean",
+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context."
+ },
+ "full_output": {
+ "type": "boolean",
+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`."
+ }
+ })
+}
+
+/// Add the shared `tree_format` / `since` / `verbose` / `full_output`
+/// properties to a backend's `get_window_state` input schema.
+pub fn extend_input_schema(mut schema: Value) -> Value {
+ if let (Some(props), Some(extra)) = (
+ schema.get_mut("properties").and_then(Value::as_object_mut),
+ schema_properties().as_object().cloned(),
+ ) {
+ props.extend(extra);
+ }
+ schema
+}
+
+impl ViewOptions {
+ pub fn from_args(args: &Value) -> Result<Self, ToolResult> {
+ let full_output = args.get("full_output").and_then(Value::as_bool) == Some(true);
+ let tree_format = match args.get("tree_format") {
+ None | Some(Value::Null) => {
+ if full_output {
+ TreeFormat::Both
+ } else {
+ TreeFormat::Markdown
+ }
+ }
+ Some(Value::String(s)) => match s.as_str() {
+ "markdown" => TreeFormat::Markdown,
+ "elements" => TreeFormat::Elements,
+ "both" => TreeFormat::Both,
+ other => {
+ return Err(ToolResult::error(format!(
+ "tree_format must be one of \"markdown\", \"elements\", \"both\"; got \"{other}\"."
+ )))
+ }
+ },
+ Some(_) => {
+ return Err(ToolResult::error(
+ "tree_format must be one of \"markdown\", \"elements\", \"both\".",
+ ))
+ }
+ };
+ let since = match args.get("since") {
+ None | Some(Value::Null) => None,
+ Some(Value::String(s)) if !s.trim().is_empty() => Some(s.trim().to_owned()),
+ Some(_) => {
+ return Err(ToolResult::error(
+ "since must be a snapshot_id string from an earlier get_window_state.",
+ ))
+ }
+ };
+ Ok(Self {
+ tree_format,
+ verbose: args.get("verbose").and_then(Value::as_bool) == Some(true),
+ full_output,
+ since: if full_output { None } else { since },
+ })
+ }
+
+ /// Node budget for the walk. An explicit `max_elements` always wins;
+ /// otherwise a model-facing read gets [`DEFAULT_MAX_ELEMENTS`], while
+ /// `full_output` and internal observation reads keep the platform default.
+ pub fn max_elements(
+ &self,
+ args: &Value,
+ platform_default: usize,
+ observation_only: bool,
+ ) -> usize {
+ if let Some(n) = args.get("max_elements").and_then(Value::as_u64) {
+ return n.max(1) as usize;
+ }
+ if self.full_output || observation_only {
+ platform_default
+ } else {
+ DEFAULT_MAX_ELEMENTS.min(platform_default)
+ }
+ }
+}
+
+/// What a `since` read resolved to.
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum SinceOutcome {
+ Diff(DiffReport),
+ Fallback(&'static str),
+}
+
+/// Inputs that identify the view a snapshot was read with. A diff is only
+/// meaningful between two reads of the same view.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ViewKey {
+ pub query: Option<String>,
+ pub max_elements: usize,
+ pub max_depth: Option<usize>,
+}
+
+pub struct ViewContext<'a> {
+ pub pid: i64,
+ pub window_id: u64,
+ pub key: ViewKey,
+ /// Describes the focused element, called only for a `no change` answer.
+ pub focus_probe: Option<&'a dyn Fn() -> Option<String>>,
+}
+
+/// Shape a finished `get_window_state` payload in place.
+pub fn apply(
+ opts: &ViewOptions,
+ ctx: &ViewContext<'_>,
+ content: &mut [Content],
+ structured: &mut Value,
+) {
+ if opts.full_output {
+ return;
+ }
+ trim_metadata(opts, structured);
+ let Some(md) = structured
+ .get("tree_markdown")
+ .and_then(Value::as_str)
+ .map(str::to_owned)
+ else {
+ return;
+ };
+ let snapshot_id = structured
+ .get("snapshot_id")
+ .and_then(Value::as_str)
+ .map(str::to_owned);
+
+ let outcome = opts
+ .since
+ .as_deref()
+ .map(|since| resolve_since(since, snapshot_id.as_deref(), &md, ctx));
+ if let Some(sid) = &snapshot_id {
+ remember(sid, ctx, &md);
+ }
+
+ // Header text added to the first line of the tree block.
+ let mut header_extra = String::new();
+ if let Some(sid) = &snapshot_id {
+ header_extra.push_str(&format!(" snapshot_id={sid}"));
+ }
+ let truncation = truncation_line(structured, &ctx.key);
+ if let Some(line) = &truncation {
+ structured["truncation_hint"] = json!(line);
+ }
+
+ let body;
+ match outcome {
+ Some(SinceOutcome::Diff(report)) => {
+ let since = opts.since.as_deref().unwrap_or_default();
+ let focus = if report.is_empty() {
+ ctx.focus_probe.and_then(|probe| probe())
+ } else {
+ None
+ };
+ let text = report.render(since, snapshot_id.as_deref(), focus.as_deref());
+ structured["since"] = json!(since);
+ structured["since_status"] = json!(if report.is_empty() {
+ "no_change"
+ } else {
+ "diff"
+ });
+ structured["tree_diff"] = json!(text);
+ structured["diff_counts"] = json!({
+ "added": report.added.len(),
+ "changed": report.changed.len(),
+ "removed": report.removed.len(),
+ });
+ if let Some(focus) = focus {
+ structured["focused_element"] = json!(focus);
+ }
+ let keep: std::collections::HashSet<u64> = report
+ .added
+ .iter()
+ .chain(report.changed.iter())
+ .filter_map(|row| row.index)
+ .collect();
+ retain_elements(structured, opts.tree_format, Some(&keep));
+ if let Some(obj) = structured.as_object_mut() {
+ obj.remove("tree_markdown");
+ }
+ body = text;
+ }
+ other => {
+ if let Some(SinceOutcome::Fallback(reason)) = &other {
+ let since = opts.since.as_deref().unwrap_or_default();
+ structured["since"] = json!(since);
+ structured["since_status"] = json!(reason);
+ header_extra.push_str(&format!(
+ "\nsince={since}: {}; full read follows.",
+ fallback_text(reason)
+ ));
+ }
+ match opts.tree_format {
+ TreeFormat::Markdown => {
+ body = md.clone();
+ retain_elements(structured, TreeFormat::Markdown, None);
+ }
+ TreeFormat::Elements => {
+ body = "(tree is in structuredContent.elements; pass tree_format \"markdown\" for the compact text rendering)".to_owned();
+ if let Some(obj) = structured.as_object_mut() {
+ obj.remove("tree_markdown");
+ }
+ }
+ TreeFormat::Both => {
+ body = md.clone();
+ }
+ }
+ if opts.tree_format == TreeFormat::Markdown && snapshot_id.is_some() {
+ header_extra
+ .push_str("\nelement_token for row [N] = <snapshot_id>:N (e.g. s0000002a:N)");
+ }
+ }
+ }
+ structured["tree_format"] = json!(opts.tree_format.as_str());
+
+ if let Some(line) = truncation {
+ // Backends that already print the long PARTIAL TREE note keep it.
+ let already_noted = content.iter().any(|c| match c {
+ Content::Text { text, .. } => text.contains("PARTIAL TREE"),
+ _ => false,
+ });
+ if !already_noted {
+ header_extra.push('\n');
+ header_extra.push_str(&line);
+ }
+ }
+
+ rewrite_tree_block(content, &md, &header_extra, &body);
+}
+
+fn fallback_text(reason: &str) -> &'static str {
+ match reason {
+ "unknown_snapshot" => "unknown or expired snapshot_id",
+ "other_window" => "that snapshot belongs to a different window",
+ "view_changed" => "query, max_elements or max_depth differ from that snapshot's read",
+ "too_much_changed" => "most of the tree changed",
+ "diff_too_large" => "the trees are too large to compare",
+ "no_snapshot" => "this read produced no snapshot to compare against",
+ _ => "cannot diff",
+ }
+}
+
+/// Replace the markdown inside the platform's text block with `body`, and add
+/// `header_extra` to the end of that block's first line.
+fn rewrite_tree_block(content: &mut [Content], md: &str, header_extra: &str, body: &str) {
+ if md.is_empty() {
+ return;
+ }
+ for part in content.iter_mut() {
+ let Content::Text { text, .. } = part else {
+ continue;
+ };
+ let Some(pos) = text.find(md) else { continue };
+ let prefix = &text[..pos];
+ let suffix = &text[pos + md.len()..];
+ let mut out = String::with_capacity(text.len());
+ match prefix.find('\n') {
+ Some(nl) => {
+ out.push_str(&prefix[..nl]);
+ out.push_str(header_extra);
+ out.push_str(&prefix[nl..]);
+ }
+ None => {
+ out.push_str(prefix);
+ out.push_str(header_extra);
+ out.push('\n');
+ }
+ }
+ out.push_str(body);
+ out.push_str(suffix);
+ *text = out;
+ return;
+ }
+}
+
+fn trim_metadata(opts: &ViewOptions, structured: &mut Value) {
+ if opts.verbose {
+ return;
+ }
+ let degraded = structured.get("degraded").and_then(Value::as_bool) == Some(true);
+ if let Some(obj) = structured.as_object_mut() {
+ obj.remove("_note");
+ // Linux restates the pixel-coordinate convention on every read; the
+ // tool description already carries it.
+ obj.remove("frame_note");
+ if !degraded {
+ obj.remove("background_input");
+ }
+ }
+}
+
+/// Keep `elements` per the requested format. `keep` limits them to the given
+/// element indices (diff reads).
+fn retain_elements(
+ structured: &mut Value,
+ format: TreeFormat,
+ keep: Option<&std::collections::HashSet<u64>>,
+) {
+ let Some(obj) = structured.as_object_mut() else {
+ return;
+ };
+ if format == TreeFormat::Markdown {
+ obj.remove("elements");
+ return;
+ }
+ if let (Some(keep), Some(Value::Array(elements))) = (keep, obj.get_mut("elements")) {
+ elements.retain(|e| {
+ e.get("element_index")
+ .and_then(Value::as_u64)
+ .is_some_and(|i| keep.contains(&i))
+ });
+ }
+}
+
+fn truncation_line(structured: &Value, key: &ViewKey) -> Option<String> {
+ if structured.get("truncated").and_then(Value::as_bool) != Some(true) {
+ return None;
+ }
+ let reason = structured
+ .get("truncation_reason")
+ .and_then(Value::as_str)
+ .unwrap_or("");
+ if reason != "node_budget" {
+ return None;
+ }
+ let pending = structured
+ .get("nodes_pending")
+ .and_then(Value::as_u64)
+ .unwrap_or(0);
+ Some(format!(
+ "Tree truncated at max_elements={} ({pending} more node(s) not shown). Pass a larger max_elements, or query/max_depth to reach the rest.",
+ key.max_elements
+ ))
+}
+
+// ── snapshot memory for `since` ──────────────────────────────────────────
+
+#[derive(Clone)]
+struct StoredSnapshot {
+ snapshot_id: String,
+ pid: i64,
+ window_id: u64,
+ key: ViewKey,
+ markdown: String,
+ at: Instant,
+}
+
+fn store() -> &'static Mutex<VecDeque<StoredSnapshot>> {
+ static STORE: OnceLock<Mutex<VecDeque<StoredSnapshot>>> = OnceLock::new();
+ STORE.get_or_init(|| Mutex::new(VecDeque::new()))
+}
+
+fn remember(snapshot_id: &str, ctx: &ViewContext<'_>, md: &str) {
+ let mut guard = store().lock().unwrap_or_else(|e| e.into_inner());
+ guard.retain(|s| s.snapshot_id != snapshot_id);
+ guard.push_back(StoredSnapshot {
+ snapshot_id: snapshot_id.to_owned(),
+ pid: ctx.pid,
+ window_id: ctx.window_id,
+ key: ctx.key.clone(),
+ markdown: md.to_owned(),
+ at: Instant::now(),
+ });
+ let same_window = |s: &StoredSnapshot| s.pid == ctx.pid && s.window_id == ctx.window_id;
+ while guard.iter().filter(|s| same_window(s)).count() > STORE_PER_WINDOW {
+ if let Some(pos) = guard.iter().position(same_window) {
+ guard.remove(pos);
+ }
+ }
+ while guard.len() > STORE_TOTAL {
+ guard.pop_front();
+ }
+}
+
+fn resolve_since(
+ since: &str,
+ current_id: Option<&str>,
+ md: &str,
+ ctx: &ViewContext<'_>,
+) -> SinceOutcome {
+ if current_id.is_none() {
+ return SinceOutcome::Fallback("no_snapshot");
+ }
+ let found = {
+ let mut guard = store().lock().unwrap_or_else(|e| e.into_inner());
+ guard.retain(|s| s.at.elapsed() <= STORE_TTL);
+ guard.iter().find(|s| s.snapshot_id == since).cloned()
+ };
+ let Some(old) = found else {
+ return SinceOutcome::Fallback("unknown_snapshot");
+ };
+ if old.pid != ctx.pid || old.window_id != ctx.window_id {
+ return SinceOutcome::Fallback("other_window");
+ }
+ if old.key != ctx.key {
+ return SinceOutcome::Fallback("view_changed");
+ }
+ match diff_markdown(&old.markdown, md) {
+ Err(DiffError::TooLarge) => SinceOutcome::Fallback("diff_too_large"),
+ Ok(report) => {
+ if !report.is_empty() && report.render_len() * 100 > md.len() * DIFF_MAX_PERCENT_OF_FULL
+ {
+ SinceOutcome::Fallback("too_much_changed")
+ } else {
+ SinceOutcome::Diff(report)
+ }
+ }
+ }
+}
+
+// ── markdown row diff ────────────────────────────────────────────────────
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+struct Row {
+ indent: usize,
+ index: Option<u64>,
+ /// Row text without indentation and without the `[N]` index.
+ key: String,
+ /// Role plus identifier or label; pairs a removed row with its changed
+ /// replacement.
+ identity: String,
+ /// Row text as rendered, index included, without indentation.
+ text: String,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq, Default)]
+pub struct DiffReport {
+ added: Vec<Row>,
+ changed: Vec<Row>,
+ removed: Vec<Row>,
+ reindexed: Vec<(u64, u64, i64)>,
+}
+
+#[derive(Debug, PartialEq, Eq)]
+enum DiffError {
+ TooLarge,
+}
+
+fn parse_rows(md: &str) -> Vec<Row> {
+ let mut rows: Vec<Row> = Vec::new();
+ for line in md.lines() {
+ let trimmed = line.trim_start();
+ if let Some(rest) = trimmed.strip_prefix("- ") {
+ let indent = line.len() - trimmed.len();
+ let (index, after) = match rest.strip_prefix('[') {
+ Some(r) => match r.split_once(']') {
+ Some((n, tail)) if n.parse::<u64>().is_ok() => {
+ (n.parse::<u64>().ok(), tail.trim_start())
+ }
+ _ => (None, rest),
+ },
+ None => (None, rest),
+ };
+ rows.push(Row {
+ indent,
+ index,
+ identity: identity_of(indent, after),
+ key: after.to_owned(),
+ text: rest.to_owned(),
+ });
+ } else if let Some(last) = rows.last_mut() {
+ if !line.trim().is_empty() {
+ // Continuation of a multi-line row (an action name with a
+ // newline in it).
+ last.key.push('\n');
+ last.key.push_str(line);
+ last.text.push('\n');
+ last.text.push_str(line);
+ }
+ }
+ }
+ rows
+}
+
+fn identity_of(indent: usize, after_index: &str) -> String {
+ let role = after_index.split_whitespace().next().unwrap_or("");
+ let id = after_index.find("id=").map(|p| {
+ let tail = &after_index[p + 3..];
+ let end = tail
+ .find(|c: char| c.is_whitespace() || c == ']')
+ .unwrap_or(tail.len());
+ &tail[..end]
+ });
+ // A quoted string right after the role is the title; after `=` it is the
+ // value, which changes and so cannot identify the row.
+ let label = after_index
+ .split_once(char::is_whitespace)
+ .map(|(_, tail)| tail.trim_start())
+ .and_then(|tail| tail.strip_prefix('"'))
+ .and_then(|tail| tail.split('"').next());
+ format!("{indent}|{role}|{}", id.or(label).unwrap_or(""))
+}
+
+fn diff_markdown(old_md: &str, new_md: &str) -> Result<DiffReport, DiffError> {
+ let old = parse_rows(old_md);
+ let new = parse_rows(new_md);
+ let ops = diff_ops(&old, &new)?;
+
+ let mut report = DiffReport::default();
+ let mut dels: Vec<usize> = Vec::new();
+ let mut inss: Vec<usize> = Vec::new();
+ // Unchanged rows whose index moved, grouped into runs of equal delta.
+ let mut run: Option<(u64, u64, i64)> = None;
+
+ let flush_hunk = |dels: &mut Vec<usize>, inss: &mut Vec<usize>, report: &mut DiffReport| {
+ let mut used = vec![false; dels.len()];
+ for &i in inss.iter() {
+ let pair = dels
+ .iter()
+ .enumerate()
+ .find(|(d, &o)| !used[*d] && old[o].identity == new[i].identity);
+ match pair {
+ Some((d, _)) => {
+ used[d] = true;
+ report.changed.push(new[i].clone());
+ }
+ None => report.added.push(new[i].clone()),
+ }
+ }
+ for (d, &o) in dels.iter().enumerate() {
+ if !used[d] {
+ report.removed.push(old[o].clone());
+ }
+ }
+ dels.clear();
+ inss.clear();
+ };
+
+ for op in ops {
+ match op {
+ Op::Del(o) => dels.push(o),
+ Op::Ins(n) => inss.push(n),
+ Op::Equal(o, n) => {
+ flush_hunk(&mut dels, &mut inss, &mut report);
+ if let (Some(oi), Some(ni)) = (old[o].index, new[n].index) {
+ let delta = ni as i64 - oi as i64;
+ match (&mut run, delta) {
+ (Some((_, last_old, d)), delta2) if *d == delta2 && delta2 != 0 => {
+ *last_old = oi;
+ }
+ _ => {
+ if let Some(done) = run.take() {
+ report.reindexed.push(done);
+ }
+ if delta != 0 {
+ run = Some((oi, oi, delta));
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ flush_hunk(&mut dels, &mut inss, &mut report);
+ if let Some(done) = run.take() {
+ report.reindexed.push(done);
+ }
+ Ok(report)
+}
+
+enum Op {
+ Equal(usize, usize),
+ Del(usize),
+ Ins(usize),
+}
+
+fn diff_ops(old: &[Row], new: &[Row]) -> Result<Vec<Op>, DiffError> {
+ let mut prefix = 0;
+ while prefix < old.len()
+ && prefix < new.len()
+ && old[prefix].key == new[prefix].key
+ && old[prefix].indent == new[prefix].indent
+ {
+ prefix += 1;
+ }
+ let mut suffix = 0;
+ while suffix < old.len() - prefix
+ && suffix < new.len() - prefix
+ && old[old.len() - 1 - suffix].key == new[new.len() - 1 - suffix].key
+ && old[old.len() - 1 - suffix].indent == new[new.len() - 1 - suffix].indent
+ {
+ suffix += 1;
+ }
+ let a = &old[prefix..old.len() - suffix];
+ let b = &new[prefix..new.len() - suffix];
+ let (n, m) = (a.len(), b.len());
+ if n.saturating_mul(m) > DIFF_CELL_LIMIT {
+ return Err(DiffError::TooLarge);
+ }
+ let same = |x: &Row, y: &Row| x.key == y.key && x.indent == y.indent;
+
+ let mut ops: Vec<Op> = (0..prefix).map(|i| Op::Equal(i, i)).collect();
+ // dp[i][j] = LCS length of a[i..] and b[j..].
+ let width = m + 1;
+ let mut dp = vec![0u32; (n + 1) * width];
+ for i in (0..n).rev() {
+ for j in (0..m).rev() {
+ dp[i * width + j] = if same(&a[i], &b[j]) {
+ dp[(i + 1) * width + j + 1] + 1
+ } else {
+ dp[(i + 1) * width + j].max(dp[i * width + j + 1])
+ };
+ }
+ }
+ let (mut i, mut j) = (0, 0);
+ while i < n && j < m {
+ if same(&a[i], &b[j]) {
+ ops.push(Op::Equal(prefix + i, prefix + j));
+ i += 1;
+ j += 1;
+ } else if dp[(i + 1) * width + j] >= dp[i * width + j + 1] {
+ ops.push(Op::Del(prefix + i));
+ i += 1;
+ } else {
+ ops.push(Op::Ins(prefix + j));
+ j += 1;
+ }
+ }
+ while i < n {
+ ops.push(Op::Del(prefix + i));
+ i += 1;
+ }
+ while j < m {
+ ops.push(Op::Ins(prefix + j));
+ j += 1;
+ }
+ for k in 0..suffix {
+ ops.push(Op::Equal(old.len() - suffix + k, new.len() - suffix + k));
+ }
+ Ok(ops)
+}
+
+impl DiffReport {
+ pub fn is_empty(&self) -> bool {
+ self.added.is_empty() && self.changed.is_empty() && self.removed.is_empty()
+ }
+
+ fn render_len(&self) -> usize {
+ self.added
+ .iter()
+ .chain(&self.changed)
+ .chain(&self.removed)
+ .map(|r| r.text.len() + 3)
+ .sum()
+ }
+
+ fn render(&self, since: &str, snapshot_id: Option<&str>, focus: Option<&str>) -> String {
+ let new_id = snapshot_id.unwrap_or("the new snapshot");
+ if self.is_empty() {
+ let mut out = format!("no change since {since}");
+ if let Some(focus) = focus {
+ out.push_str(&format!("; focused element is {focus}"));
+ }
+ out.push_str(&format!(
+ ". Rows and indices are as before; use snapshot_id {new_id} in element_tokens ({new_id}:N)."
+ ));
+ return out;
+ }
+ let mut out = format!(
+ "since {since}: {} added, {} changed, {} removed. Use element_tokens {new_id}:N for + and ~ rows and unchanged rows; - ids are from {since}.\n",
+ self.added.len(),
+ self.changed.len(),
+ self.removed.len()
+ );
+ for row in &self.added {
+ out.push_str(&format!("+ {}\n", row.text));
+ }
+ for row in &self.changed {
+ out.push_str(&format!("~ {}\n", row.text));
+ }
+ for row in &self.removed {
+ out.push_str(&format!("- {}\n", row.text));
+ }
+ const MAX_RUNS: usize = 20;
+ for (first_old, last_old, delta) in self.reindexed.iter().take(MAX_RUNS) {
+ out.push_str(&format!(
+ "reindexed: [{first_old}-{last_old}] -> [{}-{}]\n",
+ *first_old as i64 + delta,
+ *last_old as i64 + delta
+ ));
+ }
+ if self.reindexed.len() > MAX_RUNS {
+ out.push_str("reindexed: more shifted ranges not listed; pass full_output or omit since for a fresh index.\n");
+ }
+ out.truncate(out.trim_end().len());
+ out
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn opts(args: Value) -> ViewOptions {
+ ViewOptions::from_args(&args).unwrap()
+ }
+
+ const OLD: &str = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Cancel\" [id=cancel actions=[press]]";
+
+ #[test]
+ fn identical_trees_report_no_change() {
+ let report = diff_markdown(OLD, OLD).unwrap();
+ assert!(report.is_empty());
+ let text = report.render(
+ "s00000001",
+ Some("s00000002"),
+ Some("[1] AXTextField \"Name\""),
+ );
+ assert!(text.starts_with("no change since s00000001; focused element is [1]"));
+ assert!(text.contains("s00000002:N"));
+ }
+
+ #[test]
+ fn value_edit_is_a_change_and_removal_and_addition_are_reported() {
+ let new = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"ab\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Help\" [id=help actions=[press]]";
+ let report = diff_markdown(OLD, new).unwrap();
+ assert_eq!(report.changed.len(), 1);
+ assert_eq!(report.changed[0].index, Some(1));
+ assert_eq!(report.added.len(), 1);
+ assert_eq!(report.removed.len(), 1);
+ assert_eq!(report.removed[0].index, Some(3));
+ let text = report.render("sA", Some("sB"), None);
+ assert!(text.contains("~ [1] AXTextField \"Name\" = \"ab\""));
+ assert!(text.contains("+ [3] AXButton \"Help\""));
+ assert!(text.contains("- [3] AXButton \"Cancel\""));
+ }
+
+ #[test]
+ fn inserted_row_reports_reindexed_range() {
+ let new = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXCheckBox \"New\" [id=new]\n - [3] AXButton \"Save\" [id=save actions=[press]]\n - [4] AXButton \"Cancel\" [id=cancel actions=[press]]";
+ let report = diff_markdown(OLD, new).unwrap();
+ assert_eq!(report.added.len(), 1);
+ assert!(report.removed.is_empty());
+ assert_eq!(report.reindexed, vec![(2, 3, 1)]);
+ assert!(report
+ .render("sA", Some("sB"), None)
+ .contains("reindexed: [2-3] -> [3-4]"));
+ }
+
+ #[test]
+ fn display_only_text_changes_are_seen() {
+ let new = OLD.replace("hello", "goodbye");
+ let report = diff_markdown(OLD, &new).unwrap();
+ assert_eq!(report.changed.len(), 1);
+ assert_eq!(report.changed[0].index, None);
+ }
+
+ #[test]
+ fn multiline_action_rows_stay_one_row() {
+ let md = "- [0] AXWindow\n - [1] AXButton \"x\" [actions=[press,name:move\ntarget:0x0\nselector:(null)]]\n - [2] AXButton \"y\"";
+ assert_eq!(parse_rows(md).len(), 3);
+ assert!(diff_markdown(md, md).unwrap().is_empty());
+ }
+
+ #[test]
+ fn options_parse_and_defaults() {
+ let o = opts(json!({}));
+ assert_eq!(o.tree_format, TreeFormat::Markdown);
+ assert!(!o.verbose && !o.full_output && o.since.is_none());
+ assert_eq!(
+ opts(json!({"full_output": true})).tree_format,
+ TreeFormat::Both
+ );
+ assert_eq!(
+ opts(json!({"tree_format": "elements"})).tree_format,
+ TreeFormat::Elements
+ );
+ assert!(ViewOptions::from_args(&json!({"tree_format": "xml"})).is_err());
+ assert!(ViewOptions::from_args(&json!({"since": 3})).is_err());
+ // full_output is a full read, never a diff.
+ assert!(opts(json!({"full_output": true, "since": "s00000001"}))
+ .since
+ .is_none());
+ }
+
+ #[test]
+ fn max_elements_precedence() {
+ let o = opts(json!({}));
+ assert_eq!(
+ o.max_elements(&json!({}), 2000, false),
+ DEFAULT_MAX_ELEMENTS
+ );
+ assert_eq!(
+ o.max_elements(&json!({"max_elements": 900}), 2000, false),
+ 900
+ );
+ assert_eq!(o.max_elements(&json!({}), 2000, true), 2000);
+ let full = opts(json!({"full_output": true}));
+ assert_eq!(full.max_elements(&json!({}), 5000, false), 5000);
+ }
+
+ fn payload(sid: &str, md: &str) -> (Vec<Content>, Value) {
+ let content = vec![
+ Content::image_png("AAAA".into()),
+ Content::text(format!("window_id=7 pid=9 size=10x10 elements=4\n\n{md}")),
+ ];
+ let structured = json!({
+ "window_id": 7, "pid": 9, "snapshot_id": sid,
+ "tree_markdown": md,
+ "elements": [{"element_index": 1}, {"element_index": 2}],
+ "_note": "n", "background_input": {"routes": []},
+ "element_count": 4
+ });
+ (content, structured)
+ }
+
+ fn ctx(window_id: u64) -> ViewContext<'static> {
+ ViewContext {
+ pid: 9,
+ window_id,
+ key: ViewKey {
+ query: None,
+ max_elements: 250,
+ max_depth: None,
+ },
+ focus_probe: None,
+ }
+ }
+
+ fn text_of(content: &[Content]) -> String {
+ content
+ .iter()
+ .find_map(|c| match c {
+ Content::Text { text, .. } => Some(text.clone()),
+ _ => None,
+ })
+ .unwrap()
+ }
+
+ #[test]
+ fn default_read_is_markdown_only_without_noise() {
+ let (mut content, mut s) = payload("s000000a1", OLD);
+ apply(&opts(json!({})), &ctx(7001), &mut content, &mut s);
+ assert!(s.get("elements").is_none());
+ assert!(s.get("_note").is_none());
+ assert!(s.get("background_input").is_none());
+ assert_eq!(s["tree_markdown"], OLD);
+ assert_eq!(s["tree_format"], "markdown");
+ let text = text_of(&content);
+ assert!(text.starts_with("window_id=7 pid=9 size=10x10 elements=4 snapshot_id=s000000a1\n"));
+ assert!(text.contains("<snapshot_id>:N"));
+ assert!(text.ends_with(OLD));
+ }
+
+ #[test]
+ fn full_output_is_untouched() {
+ let (mut content, mut s) = payload("s000000a2", OLD);
+ let before = s.clone();
+ let text_before = text_of(&content);
+ apply(
+ &opts(json!({"full_output": true})),
+ &ctx(7002),
+ &mut content,
+ &mut s,
+ );
+ assert_eq!(s, before);
+ assert_eq!(text_of(&content), text_before);
+ assert_eq!(s["elements"], before["elements"]);
+ assert_eq!(s["_note"], before["_note"]);
+ assert_eq!(s["background_input"], before["background_input"]);
+ assert_eq!(s["tree_markdown"], OLD);
+ }
+
+ #[test]
+ fn elements_format_drops_markdown() {
+ let (mut content, mut s) = payload("s000000a3", OLD);
+ apply(
+ &opts(json!({"tree_format": "elements"})),
+ &ctx(7003),
+ &mut content,
+ &mut s,
+ );
+ assert!(s.get("tree_markdown").is_none());
+ assert_eq!(s["elements"].as_array().unwrap().len(), 2);
+ assert!(!text_of(&content).contains("AXButton"));
+ }
+
+ #[test]
+ fn since_returns_diff_then_falls_back() {
+ let window = 7004;
+ let (mut c1, mut s1) = payload("s000000b1", OLD);
+ apply(&opts(json!({})), &ctx(window), &mut c1, &mut s1);
+
+ // Unchanged tree.
+ let (mut c2, mut s2) = payload("s000000b2", OLD);
+ apply(
+ &opts(json!({"since": "s000000b1"})),
+ &ctx(window),
+ &mut c2,
+ &mut s2,
+ );
+ assert_eq!(s2["since_status"], "no_change");
+ assert!(s2.get("tree_markdown").is_none());
+ assert!(text_of(&c2).contains("no change since s000000b1"));
+
+ // One edit, chained against the previous snapshot.
+ let edited = OLD.replace("\"a\"", "\"ab\"");
+ let (mut c3, mut s3) = payload("s000000b3", &edited);
+ apply(
+ &opts(json!({"since": "s000000b2"})),
+ &ctx(window),
+ &mut c3,
+ &mut s3,
+ );
+ assert_eq!(s3["since_status"], "diff");
+ assert_eq!(s3["diff_counts"]["changed"], 1);
+ assert!(text_of(&c3).contains("~ [1] AXTextField"));
+ assert!(!text_of(&c3).contains("AXButton \"Save\""));
+ assert!(s3.get("elements").is_none());
+
+ // Unknown snapshot falls back to the full read.
+ let (mut c4, mut s4) = payload("s000000b4", &edited);
+ apply(
+ &opts(json!({"since": "s0000dead"})),
+ &ctx(window),
+ &mut c4,
+ &mut s4,
+ );
+ assert_eq!(s4["since_status"], "unknown_snapshot");
+ assert_eq!(s4["tree_markdown"], edited);
+ assert!(text_of(&c4).contains("unknown or expired snapshot_id; full read follows"));
+
+ // Another window's snapshot is not a baseline.
+ let (mut c5, mut s5) = payload("s000000b5", &edited);
+ apply(
+ &opts(json!({"since": "s000000b4"})),
+ &ctx(window + 1),
+ &mut c5,
+ &mut s5,
+ );
+ assert_eq!(s5["since_status"], "other_window");
+ }
+
+ #[test]
+ fn a_rewritten_tree_falls_back_to_full() {
+ let window = 7005;
+ let (mut c1, mut s1) = payload("s000000c1", OLD);
+ apply(&opts(json!({})), &ctx(window), &mut c1, &mut s1);
+ let other = "- [0] AXWindow \"Other\"\n - [1] AXList \"x\"\n - [2] AXButton \"Q\"\n - [3] AXButton \"R\"\n - [4] AXButton \"S\"";
+ let (mut c2, mut s2) = payload("s000000c2", other);
+ apply(
+ &opts(json!({"since": "s000000c1"})),
+ &ctx(window),
+ &mut c2,
+ &mut s2,
+ );
+ assert_eq!(s2["since_status"], "too_much_changed");
+ assert_eq!(s2["tree_markdown"], other);
+ }
+
+ #[test]
+ fn node_budget_truncation_is_stated() {
+ let (mut content, mut s) = payload("s000000d1", OLD);
+ s["truncated"] = json!(true);
+ s["truncation_reason"] = json!("node_budget");
+ s["nodes_pending"] = json!(40);
+ apply(&opts(json!({})), &ctx(7006), &mut content, &mut s);
+ let text = text_of(&content);
+ assert!(text.contains("Tree truncated at max_elements=250 (40 more node(s) not shown)"));
+ assert!(s["truncation_hint"]
+ .as_str()
+ .unwrap()
+ .contains("larger max_elements"));
+ }
+
+ #[test]
+ fn degraded_snapshots_keep_background_input() {
+ let (mut content, mut s) = payload("s000000e1", OLD);
+ s["degraded"] = json!(true);
+ apply(&opts(json!({})), &ctx(7007), &mut content, &mut s);
+ assert!(s.get("background_input").is_some());
+ let (mut content, mut s) = payload("s000000e2", OLD);
+ apply(
+ &opts(json!({"verbose": true})),
+ &ctx(7008),
+ &mut content,
+ &mut s,
+ );
+ assert!(s.get("background_input").is_some());
+ assert!(s.get("_note").is_some());
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
index ccadcd58a0..77925cc654 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
@@ -861,6 +861,10 @@ fn sdk_window_input(fixture: &Fixture) -> GetWindowStateInput {
max_depth: None,
max_dimension: None,
max_image_dimension: None,
+ tree_format: None,
+ since: None,
+ verbose: None,
+ full_output: None,
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
index 1aee34b717..11e7205cfc 100644
--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs
@@ -755,8 +755,15 @@ macro_rules! define_native_window_methods {
pub async fn get_window_state(
&self,
- input: GetWindowStateInput,
+ mut input: GetWindowStateInput,
) -> Result<WindowStateOutput, DriverError> {
+ // The typed snapshot is read by programs, not budgeted
+ // against a model context: keep the full response (both tree
+ // representations, platform walk limits) unless the caller
+ // picks a leaner shape.
+ if input.tree_format.is_none() && input.full_output.is_none() {
+ input.full_output = Some(true);
+ }
self.invoke_typed(GetWindowStateInput::TOOL_NAME, input)
.await?
.window_state_success()
diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs
index c7c2eeb045..7e2f7878ad 100644
--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs
@@ -285,7 +285,26 @@ async fn typed_snapshot_request_preserves_target_and_session() {
assert_eq!(request["name"], "get_window_state");
assert_eq!(
request["args"],
- json!({"pid":42,"window_id":73,"session":"run-1","include_screenshot":true,"max_elements":10})
+ // The typed SDK keeps the full response unless the caller picks a
+ // leaner shape.
+ json!({"pid":42,"window_id":73,"session":"run-1","include_screenshot":true,"max_elements":10,"full_output":true})
+ );
+}
+
+#[tokio::test]
+#[cfg(unix)]
+async fn typed_snapshot_request_forwards_an_explicit_tree_shape() {
+ let (_directory, socket, server) = serve_once(json!({"ok":true,"result":snapshot_envelope()}));
+ let driver = CuaDriver::connect(Some(socket)).unwrap();
+ let input: GetWindowStateInput = serde_json::from_value(json!({
+ "pid":42,"window_id":73,"tree_format":"markdown","since":"s0000002a"
+ }))
+ .unwrap();
+ driver.get_window_state(input).await.unwrap();
+ let request = server.join().unwrap();
+ assert_eq!(
+ request["args"],
+ json!({"pid":42,"window_id":73,"tree_format":"markdown","since":"s0000002a"})
);
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs
index 8b03458596..ef3fa70c06 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs
@@ -111,6 +111,7 @@ impl Driver for CliDriver {
}
};
+ let args = crate::driver::with_full_window_state(tool, args);
if let Some(mut stdin) = child.stdin.take() {
let _ = writeln!(stdin, "{}", serde_json::to_string(&args).unwrap());
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs
index f0c21717e9..1848db36e0 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs
@@ -18,3 +18,21 @@ pub trait Driver {
pub trait BehaviorRecording {
fn start_behavior_recording(&mut self);
}
+
+/// Harness scenarios read both `elements` and `tree_markdown` from
+/// `get_window_state`. The tool's model-facing default is now one compact
+/// markdown tree, so request the full response unless the scenario chose a
+/// shape itself (`tree_format`, `full_output`, or `since`).
+pub fn with_full_window_state(tool: &str, mut args: Value) -> Value {
+ if tool == "get_window_state" {
+ if let Some(map) = args.as_object_mut() {
+ if !map.contains_key("tree_format")
+ && !map.contains_key("full_output")
+ && !map.contains_key("since")
+ {
+ map.insert("full_output".into(), Value::Bool(true));
+ }
+ }
+ }
+ args
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
index e1fbcb4994..f01eda3527 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
@@ -487,6 +487,7 @@ impl McpDriver {
let id = self.next_id;
self.next_id += 1;
let timeout = call_timeout(self.recording_started, tool, cfg!(target_os = "macos"));
+ let args = crate::driver::with_full_window_state(tool, args);
self.send(serde_json::json!({
"jsonrpc": "2.0", "id": id, "method": "tools/call",
"params": { "name": tool, "arguments": args }
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
index 2bf67c3539..4c2faff25e 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
@@ -4608,6 +4608,8 @@ const CLI_EXAMPLES: &[(&str, &[(&str, &str)])] = &[
("cua-driver call list_apps", "Call a tool with no arguments"),
("cua-driver call click '{\"pid\":844,\"x\":100,\"y\":200}'", "Click at window coordinates"),
("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725}' --screenshot-out-file state.png", "Save the screenshot from a tool response"),
+ ("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"since\":\"s0000002a\"}'", "Read only what changed since an earlier snapshot_id"),
+ ("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"tree_format\":\"elements\"}'", "Return the structured elements instead of the markdown tree"),
]),
("serve", &[
("cua-driver serve", "Run the daemon in the foreground"),
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index ca0dec422a..2f971479b1 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1005,6 +1005,9 @@ pub struct GetWindowStateTool {
const COLD_START_WALK_TIMEOUT_MS: u64 = 2_000;
+/// The AT-SPI walker's own node budget when the caller gives none.
+const LINUX_DEFAULT_MAX_ELEMENTS: usize = 5_000;
+
fn linux_snapshot_timeout_ms(timeout: Option<&Value>, has_prior_snapshot: bool) -> u64 {
cua_driver_core::tool_schema::resolve_timeout_ms_with_first_snapshot_grace(
timeout,
@@ -1020,24 +1023,31 @@ impl Tool for GetWindowStateTool {
fn def(&self) -> &ToolDef {
GWS_DEF.get_or_init(|| ToolDef {
name: "get_window_state".into(),
- description: "Walk a running app's AT-SPI tree and return BOTH a \
- structured `elements` array (preferred) AND a Markdown rendering of \
- the same tree (back-compat). Every actionable element is tagged \
- with [element_index N] in the markdown and as `element_index` in \
- the structured array; pass each element's `element_token` to \
- `click`, `type_text`, `set_value`, etc.\n\n\
- PREFERRED CONSUMERS read `structuredContent.elements` (one entry \
- per indexed row with `element_index`, `role`, `label`, `value`, \
- `enabled`, `selected`, `actions` (names of AT-SPI actions exposed \
- by the element, omitted when empty), \
- `frame: {x,y,w,h}` when AT-SPI reports usable bounds, \
- `parent_index`, `depth`). The markdown `tree_markdown` stays \
- available and unchanged in shape for existing text-parsing \
- callers — but new fields will only be added to the structured \
- side. Set `query` to project BOTH representations to matching \
- rows plus their ancestor chain while preserving original indices. \
- `total_element_count` reports the complete snapshot and \
- `returned_element_count` reports the projection.\n\n\
+ description: "Walk a running app's AT-SPI tree and return it ONCE, as \
+ compact Markdown by default (`tree_format:\"markdown\"`). Every actionable \
+ element is tagged `[N]` (an element_index); its element_token is \
+ `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and \
+ returned in structuredContent. Pass `tree_format:\"elements\"` for the \
+ structured `elements` array instead (explicit `element_token`, `role`, \
+ `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when \
+ AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for \
+ both (about twice the size). Set `query` to project the tree to matching rows \
+ plus their ancestor chain while preserving original indices; \
+ `total_element_count` reports the complete snapshot.\n\n\
+ Size: a read is bounded by default (`max_elements` 250 actionable nodes). \
+ When the walk stops at that budget the response says `Tree truncated at \
+ max_elements=…` and `truncated: true`; pass a larger `max_elements`, or \
+ `query` / `max_depth`, to reach the rest. `_note` is omitted unless \
+ `verbose:true`. `full_output:true` restores the previous full response: both \
+ representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\n\
+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \
+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
+ back to a full read and `since_status` says why. The focused element is not \
+ reported on Linux yet.\n\n\
Always returns BOTH the element tree AND a screenshot — ground on \
both and cross-check (the tree lies on some surfaces). Choose the \
modality at ACTION time: an element ax action \
@@ -1055,11 +1065,9 @@ impl Tool for GetWindowStateTool {
`include_screenshot:false` is an error. Optional `max_dimension` \
caps the returned screenshot's long edge in pixels for a cheap \
thumbnail.\n\n\
- Optional `max_elements` / `max_depth` bound the AT-SPI walk to \
- mitigate context-window blow-up on Electron / large web apps \
- that produce 10k+ element trees. When applied, BOTH \
- the markdown and the structured elements are truncated \
- identically. Omit both for current default behaviour.\n\n\
+ Optional `max_elements` / `max_depth` bound the AT-SPI walk \
+ (Electron / large web apps produce 10k+ element trees). Markdown and \
+ structured elements are truncated identically. Default: `max_elements` 250.\n\n\
TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI \
walk. When omitted, a window's first snapshot gets a 2000 ms \
cold-start budget so Chromium/Electron can finish publishing its \
@@ -1085,24 +1093,24 @@ impl Tool for GetWindowStateTool {
popup's own AT-SPI toplevel so its menu items get element indices \
(then click them by element_token). Omitting window_id while a popup \
of this pid is open walks that popup.".into(),
- input_schema: json!({"type":"object","required":["pid"],"properties":{
+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"pid":{"type":"integer","description":"Process ID that owns the window."},
"window_id":{"type":"integer","description":"Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window."},
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
"include_accessibility_tree":{"type":"boolean",
- "description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
+ "description":"Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
"include_screenshot":{"type":"boolean",
"description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string",
"description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},
"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
- "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."},
+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},
"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted."},
"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior."}
- },"additionalProperties":false}),
+ },"additionalProperties":false})),
read_only: true, destructive: false, idempotent: false, open_world: false,
})
}
@@ -1184,6 +1192,10 @@ impl Tool for GetWindowStateTool {
// time: an element ax action (element_token) or element px action (x,y).
// We don't even read the arg; it stays in the schema only so old callers
// don't trip additionalProperties:false.
+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
+ Ok(view) => view,
+ Err(refusal) => return refusal,
+ };
let query = args.opt_str("query");
let session_id = args.opt_str("_session_id");
// include_screenshot (default true) — the perf opt-out. The tree+screenshot
@@ -1210,10 +1222,15 @@ impl Tool for GetWindowStateTool {
});
// Optional caps — when omitted, the AT-SPI walker uses its built-in
// defaults (#22865).
- let max_elements = args
- .get("max_elements")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as usize);
+ let max_elements = Some(
+ view.max_elements(
+ &args,
+ LINUX_DEFAULT_MAX_ELEMENTS,
+ args.get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true),
+ ),
+ );
let max_depth = args
.get("max_depth")
.and_then(|v| v.as_u64())
@@ -1752,6 +1769,24 @@ impl Tool for GetWindowStateTool {
.with_structured(structured);
}
+ if !observation_only {
+ cua_driver_core::window_state_view::apply(
+ &view,
+ &cua_driver_core::window_state_view::ViewContext {
+ pid: i64::from(pid),
+ window_id: xid,
+ key: cua_driver_core::window_state_view::ViewKey {
+ query: query.clone(),
+ max_elements: max_elements.unwrap_or(LINUX_DEFAULT_MAX_ELEMENTS),
+ max_depth,
+ },
+ focus_probe: None,
+ },
+ &mut content,
+ &mut structured,
+ );
+ }
+
ToolResult {
content,
is_error: None,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index 2d1f2e4687..e69b4d46ad 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -27,14 +27,16 @@ const AX_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_se
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "get_window_state".into(),
- description: "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\
+ description: "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\
\n\
- START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\
+ START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\
+ \n\
+ DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\
\n\
A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
\n\
Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
- input_schema: serde_json::json!({
+ input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({
"type": "object",
"required": ["pid", "window_id"],
"properties": {
@@ -58,7 +60,7 @@ fn def() -> &'static ToolDef {
"max_elements": {
"type": "integer",
"minimum": 1,
- "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000."
+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut."
},
"max_depth": {
"type": "integer",
@@ -78,7 +80,7 @@ fn def() -> &'static ToolDef {
}
},
"additionalProperties": false
- }),
+ })),
read_only: true,
destructive: false,
idempotent: false,
@@ -157,6 +159,10 @@ impl Tool for GetWindowStateTool {
}
}
+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
+ Ok(view) => view,
+ Err(refusal) => return refusal,
+ };
let query = args.opt_str("query");
let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {
// Expand ~ prefix.
@@ -228,11 +234,11 @@ impl Tool for GetWindowStateTool {
// the AX walker (#22865). minimum:1 keyed in the schema, but defend
// against 0 here as well so a misbehaving client can't disable the
// walk entirely.
- let max_elements = args
- .get("max_elements")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as usize)
- .unwrap_or(crate::ax::tree::DEFAULT_MAX_ELEMENTS);
+ let max_elements = view.max_elements(
+ &args,
+ crate::ax::tree::DEFAULT_MAX_ELEMENTS,
+ observation_only,
+ );
let max_depth = args
.get("max_depth")
.and_then(|v| v.as_u64())
@@ -720,6 +726,27 @@ impl Tool for GetWindowStateTool {
cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::MayBeIncomplete,
),
);
+ if !observation_only {
+ let focus_probe = || focused_element_description(pid, tree_result.as_ref());
+ cua_driver_core::window_state_view::apply(
+ &view,
+ &cua_driver_core::window_state_view::ViewContext {
+ pid: i64::from(pid),
+ window_id: u64::from(window_id),
+ key: cua_driver_core::window_state_view::ViewKey {
+ query: query.clone(),
+ max_elements,
+ max_depth: args
+ .get("max_depth")
+ .and_then(|v| v.as_u64())
+ .map(|v| v as usize),
+ },
+ focus_probe: Some(&focus_probe),
+ },
+ &mut content,
+ &mut structured,
+ );
+ }
ToolResult {
content,
is_error: None,
@@ -729,6 +756,49 @@ impl Tool for GetWindowStateTool {
}
}
+/// Describe the app's focused UI element as `[N] Role "title"` when it is one
+/// of the indexed rows of this walk. Called only for a `since` read that found
+/// no change.
+fn focused_element_description(
+ pid: i32,
+ tree: Option<&crate::ax::tree::TreeWalkResult>,
+) -> Option<String> {
+ use crate::ax::bindings::{copy_element_attr, AXUIElementCreateApplication};
+ use core_foundation::base::{CFEqual, CFRelease, CFTypeRef};
+ let tree = tree?;
+ unsafe {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ return None;
+ }
+ let focused = copy_element_attr(app, "AXFocusedUIElement");
+ CFRelease(app as CFTypeRef);
+ let focused = focused?;
+ let found = tree.nodes.iter().find(|node| {
+ node.element_index.is_some()
+ && node.element_ptr != 0
+ && CFEqual(node.element_ptr as CFTypeRef, focused as CFTypeRef) != 0
+ });
+ let described = found.map(|node| {
+ let label = node
+ .title
+ .as_deref()
+ .or(node.description.as_deref())
+ .or(node.identifier.as_deref());
+ match label {
+ Some(label) => format!(
+ "[{}] {} \"{label}\"",
+ node.element_index.unwrap_or_default(),
+ node.role
+ ),
+ None => format!("[{}] {}", node.element_index.unwrap_or_default(), node.role),
+ }
+ });
+ CFRelease(focused as CFTypeRef);
+ described
+ }
+}
+
/// Turn an unresolvable window scope into a structured refusal, or `None` when
/// the scope is one the caller can still be served (issue #2237).
///
@@ -1150,6 +1220,31 @@ mod window_scope_contract_tests {
"description must document the both-false error"
);
}
+
+ #[test]
+ fn schema_advertises_single_tree_and_diff_controls() {
+ let d = def();
+ let props = &d.input_schema["properties"];
+ assert_eq!(
+ props["tree_format"]["enum"],
+ serde_json::json!(["markdown", "elements", "both"])
+ );
+ for name in ["since", "verbose", "full_output"] {
+ assert!(props.get(name).is_some(), "schema must advertise {name}");
+ }
+ assert_eq!(d.input_schema["additionalProperties"], false);
+ for needle in [
+ "at most 250 nodes",
+ "since:",
+ "full_output:true",
+ "tree_format",
+ ] {
+ assert!(
+ d.description.contains(needle),
+ "description must mention {needle}"
+ );
+ }
+ }
}
#[cfg(test)]
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 5bb612937c..b5152da090 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -1164,30 +1164,39 @@ impl Tool for GetWindowStateTool {
// Windows-specific adaptations (UIA instead of AX; no SkyLight
// Space validation; no per-call `javascript` field — that's a
// macOS AppleScript hook).
- description: "Walk a running app's UIA tree and return BOTH a structured \
- `elements` array (preferred) AND a Markdown rendering of the same tree \
- (back-compat). Every actionable element is tagged with [element_index N] \
- in the markdown and as `element_index` in the structured array; pass each \
- element's `element_token` to `click`, `type_text`, `scroll`, etc.\n\n\
+ description: "Walk a running app's UIA tree and return it ONCE, as compact \
+ Markdown by default (`tree_format:\"markdown\"`). Every actionable element is \
+ tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where \
+ `snapshot_id` is printed in the response header and returned in structuredContent. \
+ Pass `tree_format:\"elements\"` for the structured `elements` array instead \
+ (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), \
+ or `\"both\"` for both (about twice the size).\n\n\
+ Size: a read is bounded by default (`max_elements` 250). When the walk stops at \
+ that budget the response says `Tree truncated at max_elements=…` and `truncated: \
+ true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. \
+ `_note` is omitted unless `verbose:true`. `full_output:true` restores the \
+ previous full response: both representations, all metadata, and the platform's \
+ walk limits (≤5 000 elements, depth ≤25).\n\n\
+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \
+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
+ back to a full read and `since_status` says why. The focused element is not \
+ reported on Windows yet.\n\n\
INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \
any element action against that window. The next snapshot of the same \
(pid, window_id) replaces this one, stales its element tokens, and lists the \
replaced ids in `invalidated_snapshot_ids`.\n\n\
- PREFERRED CONSUMERS read `structuredContent.elements` (one entry per \
- indexed row with `element_index`, `role`, `label`, `value`, `enabled`, \
- `selected`, `actions` (names of UIA patterns exposed as actions, \
- omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \
- `tree_markdown` stays available \
- and unchanged in shape for existing text-parsing callers — but new \
- fields will only be added to the structured side.\n\n\
The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \
window bounds reported come from the same `window_id`. This is the source of \
truth for which window the caller intends to reason about — the driver never \
picks a window implicitly.\n\n\
`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. \
The driver does not auto-fall-back to a different window.\n\n\
- Set `query` to a case-insensitive substring to project BOTH `tree_markdown` \
- and `structuredContent.elements` to matching rows plus their ancestor chain. \
+ Set `query` to a case-insensitive substring to project the tree to matching \
+ rows plus their ancestor chain. \
Original element indices are preserved. `total_element_count` reports the \
complete snapshot; `returned_element_count` reports the projection.\n\n\
Always returns BOTH the element tree AND a screenshot — ground on both \
@@ -1206,11 +1215,9 @@ impl Tool for GetWindowStateTool {
Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a \
single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of \
timing out at 4s with per-property RPCs).\n\n\
- Optional `max_elements` / `max_depth` bound the UIA walk to mitigate \
- context-window blow-up on Electron / large web apps that produce 10k+ \
- element trees. When applied, BOTH the markdown and the structured \
- elements are truncated identically. Omit both for current default behaviour \
- (≤5 000 elements, depth ≤25).\n\n\
+ Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \
+ apps produce 10k+ element trees). Markdown and structured elements are \
+ truncated identically. Default: `max_elements` 250, depth ≤25.\n\n\
CHROMIUM COVERAGE: a browser-owned permission bubble can be \
composited outside the requested native window. Chromium-family \
snapshots therefore describe this limit in structuredContent.capture_coverage. \
@@ -1220,21 +1227,21 @@ impl Tool for GetWindowStateTool {
This is separate from page JavaScript dialogs, which remain on \
browser_dialog.\n\n\
Windows requires no special permissions.".into(),
- input_schema: json!({"type":"object","required":["pid","window_id"],"properties":{
+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid","window_id"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"pid":{"type":"integer","description":"Process ID from `list_apps`."},
"window_id":{"type":"integer","description":"HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array."},
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
- "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
+ "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},
"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
- "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."},
+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},
"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default."},
"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins."}
- },"additionalProperties":false}),
+ },"additionalProperties":false})),
// Swift annotation: idempotent: false (each call is a fresh snapshot).
read_only: true, destructive: false, idempotent: false, open_world: false,
})
@@ -1333,17 +1340,23 @@ impl Tool for GetWindowStateTool {
// time: an element ax action (element_index) or an element px action (x,y).
// We don't read the arg; it stays in the schema only so old callers don't
// trip additionalProperties:false.
+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
+ Ok(view) => view,
+ Err(refusal) => return refusal,
+ };
let query = args.opt_str("query");
let session_id = args.opt_str("_session_id");
let screenshot_out_file = args.opt_str("screenshot_out_file");
// Optional caps — when omitted, fall back to the walker's built-in
// defaults (#22865). minimum:1 enforced in the schema, but defend
// against 0 here too.
- let max_elements = args
- .get("max_elements")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as usize)
- .unwrap_or(crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS);
+ let max_elements = view.max_elements(
+ &args,
+ crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,
+ args.get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true),
+ );
let max_depth = args
.get("max_depth")
.and_then(|v| v.as_u64())
@@ -1748,6 +1761,27 @@ impl Tool for GetWindowStateTool {
.with_structured(structured);
}
+ if !observation_only {
+ cua_driver_core::window_state_view::apply(
+ &view,
+ &cua_driver_core::window_state_view::ViewContext {
+ pid: i64::from(pid),
+ window_id: hwnd,
+ key: cua_driver_core::window_state_view::ViewKey {
+ query: query.clone(),
+ max_elements,
+ max_depth: args
+ .get("max_depth")
+ .and_then(|v| v.as_u64())
+ .map(|v| v as usize),
+ },
+ focus_probe: None,
+ },
+ &mut content,
+ &mut structured,
+ );
+ }
+
ToolResult {
content,
is_error: None,
diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
index c63b6e3051..4a32a7a606 100644
--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
@@ -2852,7 +2852,25 @@ export type GetWindowStateInput = {
* (default 1000 on every platform). A walk that runs out returns a
* partial tree flagged `truncated` rather than failing.
*/
- timeoutMs?: number
+ timeoutMs?: number,
+ /**
+ * Tree representation to return: `markdown` (default), `elements`, or
+ * `both`.
+ */
+ treeFormat?: string,
+ /**
+ * Return only what changed since this earlier `snapshot_id`.
+ */
+ since?: string,
+ /**
+ * Include `_note` and the full `background_input` report.
+ */
+ verbose?: boolean,
+ /**
+ * Restore the previous full response (both representations, all
+ * metadata, platform walk limits).
+ */
+ fullOutput?: boolean
}
/**
@@ -2860,7 +2878,11 @@ export type GetWindowStateInput = {
*/
export const GetWindowStateInput = (() => {
const defaults = () => ({
- timeoutMs: undefined
+ timeoutMs: undefined,
+ treeFormat: undefined,
+ since: undefined,
+ verbose: undefined,
+ fullOutput: undefined
});
const create = (() => {
return uniffiCreateRecord<GetWindowStateInput, ReturnType<typeof defaults>>(defaults);
@@ -2888,7 +2910,11 @@ const FfiConverterTypeGetWindowStateInput = (() => {
maxDepth: FfiConverterOptionalUInt32.read(from),
maxDimension: FfiConverterOptionalUInt32.read(from),
maxImageDimension: FfiConverterOptionalUInt32.read(from),
- timeoutMs: FfiConverterOptionalUInt32.read(from)
+ timeoutMs: FfiConverterOptionalUInt32.read(from),
+ treeFormat: FfiConverterOptionalString.read(from),
+ since: FfiConverterOptionalString.read(from),
+ verbose: FfiConverterOptionalBoolean.read(from),
+ fullOutput: FfiConverterOptionalBoolean.read(from)
};
}
write(value: TypeName, into: RustBuffer): void {
@@ -2904,6 +2930,10 @@ const FfiConverterTypeGetWindowStateInput = (() => {
FfiConverterOptionalUInt32.write(value.maxDimension, into);
FfiConverterOptionalUInt32.write(value.maxImageDimension, into);
FfiConverterOptionalUInt32.write(value.timeoutMs, into);
+ FfiConverterOptionalString.write(value.treeFormat, into);
+ FfiConverterOptionalString.write(value.since, into);
+ FfiConverterOptionalBoolean.write(value.verbose, into);
+ FfiConverterOptionalBoolean.write(value.fullOutput, into);
}
allocationSize(value: TypeName): number {
return FfiConverterUInt32.allocationSize(value.pid) +
@@ -2917,7 +2947,11 @@ const FfiConverterTypeGetWindowStateInput = (() => {
FfiConverterOptionalUInt32.allocationSize(value.maxDepth) +
FfiConverterOptionalUInt32.allocationSize(value.maxDimension) +
FfiConverterOptionalUInt32.allocationSize(value.maxImageDimension) +
- FfiConverterOptionalUInt32.allocationSize(value.timeoutMs);
+ FfiConverterOptionalUInt32.allocationSize(value.timeoutMs) +
+ FfiConverterOptionalString.allocationSize(value.treeFormat) +
+ FfiConverterOptionalString.allocationSize(value.since) +
+ FfiConverterOptionalBoolean.allocationSize(value.verbose) +
+ FfiConverterOptionalBoolean.allocationSize(value.fullOutput);
}
};
@@ -6275,6 +6309,29 @@ export type WindowStateOutput = {
degradedReason?: string,
truncated?: boolean,
truncationReason?: string,
+ /**
+ * Plain-language line saying the tree was cut at `max_elements` and how
+ * to read more.
+ */
+ truncationHint?: string,
+ /**
+ * Which representation this response carries.
+ */
+ treeFormat?: string,
+ /**
+ * The `since` snapshot_id the caller asked to diff against.
+ */
+ since?: string,
+ /**
+ * `diff`, `no_change`, or the reason a full read was returned instead
+ * (`unknown_snapshot`, `other_window`, `view_changed`, `too_much_changed`,
+ * `diff_too_large`, `no_snapshot`).
+ */
+ sinceStatus?: string,
+ /**
+ * Added/changed/removed rows against `since`, one per line.
+ */
+ treeDiff?: string,
screenshotWidth?: number,
screenshotHeight?: number,
screenshotScale?: number,
@@ -6325,6 +6382,11 @@ const FfiConverterTypeWindowStateOutput = (() => {
degradedReason: FfiConverterOptionalString.read(from),
truncated: FfiConverterOptionalBoolean.read(from),
truncationReason: FfiConverterOptionalString.read(from),
+ truncationHint: FfiConverterOptionalString.read(from),
+ treeFormat: FfiConverterOptionalString.read(from),
+ since: FfiConverterOptionalString.read(from),
+ sinceStatus: FfiConverterOptionalString.read(from),
+ treeDiff: FfiConverterOptionalString.read(from),
screenshotWidth: FfiConverterOptionalUInt32.read(from),
screenshotHeight: FfiConverterOptionalUInt32.read(from),
screenshotScale: FfiConverterOptionalFloat64.read(from),
@@ -6352,6 +6414,11 @@ const FfiConverterTypeWindowStateOutput = (() => {
FfiConverterOptionalString.write(value.degradedReason, into);
FfiConverterOptionalBoolean.write(value.truncated, into);
FfiConverterOptionalString.write(value.truncationReason, into);
+ FfiConverterOptionalString.write(value.truncationHint, into);
+ FfiConverterOptionalString.write(value.treeFormat, into);
+ FfiConverterOptionalString.write(value.since, into);
+ FfiConverterOptionalString.write(value.sinceStatus, into);
+ FfiConverterOptionalString.write(value.treeDiff, into);
FfiConverterOptionalUInt32.write(value.screenshotWidth, into);
FfiConverterOptionalUInt32.write(value.screenshotHeight, into);
FfiConverterOptionalFloat64.write(value.screenshotScale, into);
@@ -6378,6 +6445,11 @@ const FfiConverterTypeWindowStateOutput = (() => {
FfiConverterOptionalString.allocationSize(value.degradedReason) +
FfiConverterOptionalBoolean.allocationSize(value.truncated) +
FfiConverterOptionalString.allocationSize(value.truncationReason) +
+ FfiConverterOptionalString.allocationSize(value.truncationHint) +
+ FfiConverterOptionalString.allocationSize(value.treeFormat) +
+ FfiConverterOptionalString.allocationSize(value.since) +
+ FfiConverterOptionalString.allocationSize(value.sinceStatus) +
+ FfiConverterOptionalString.allocationSize(value.treeDiff) +
FfiConverterOptionalUInt32.allocationSize(value.screenshotWidth) +
FfiConverterOptionalUInt32.allocationSize(value.screenshotHeight) +
FfiConverterOptionalFloat64.allocationSize(value.screenshotScale) +
diff --git a/libs/cua-driver/typescript/test/native-loader.test.mjs b/libs/cua-driver/typescript/test/native-loader.test.mjs
index 71b315e075..37a0d9f049 100644
--- a/libs/cua-driver/typescript/test/native-loader.test.mjs
+++ b/libs/cua-driver/typescript/test/native-loader.test.mjs
@@ -266,6 +266,7 @@ test(
pid: 42, window_id: 123, session: "node-run", query: "Save",
include_screenshot: true, include_accessibility_tree: true,
max_elements: 10, max_depth: 3, max_dimension: 800, max_image_dimension: 1600,
+ full_output: true,
})
assert.deepEqual(requests[5].args, {
target: { kind: "window", pid: 42, window_id: 123 },
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
index e1bd320e28..53d48a7aa1 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `set_value`, etc.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.\n\nTIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.\n\nSCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\nPOPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_token). Omitting window_id while a popup of this pid is open walks that popup.",
+ "description": "Walk a running app's AT-SPI tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `role`, `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for both (about twice the size). Set `query` to project the tree to matching rows plus their ancestor chain while preserving original indices; `total_element_count` reports the complete snapshot.\n\nSize: a read is bounded by default (`max_elements` 250 actionable nodes). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Linux yet.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250.\n\nTIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.\n\nSCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\nPOPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_token). Omitting window_id while a popup of this pid is open walks that popup.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -57,8 +57,12 @@
],
"type": "string"
},
+ "full_output": {
+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
+ "type": "boolean"
+ },
"include_accessibility_tree": {
- "description": "Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
+ "description": "Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
"type": "boolean"
},
"include_screenshot": {
@@ -76,7 +80,7 @@
"type": "integer"
},
"max_elements": {
- "description": "Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees.",
+ "description": "Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window.",
"minimum": 1,
"type": "integer"
},
@@ -101,6 +105,10 @@
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
"type": "string"
},
+ "since": {
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "type": "string"
+ },
"timeout_ms": {
"default": 1000,
"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",
@@ -108,6 +116,19 @@
"minimum": 100,
"type": "integer"
},
+ "tree_format": {
+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",
+ "enum": [
+ "markdown",
+ "elements",
+ "both"
+ ],
+ "type": "string"
+ },
+ "verbose": {
+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",
+ "type": "boolean"
+ },
"window_id": {
"description": "Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window.",
"type": "integer"
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 68dbcffc03..e19656720a 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\nSTART NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -57,6 +57,10 @@
],
"type": "string"
},
+ "full_output": {
+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
+ "type": "boolean"
+ },
"include_accessibility_tree": {
"description": "Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error.",
"type": "boolean"
@@ -76,7 +80,7 @@
"type": "integer"
},
"max_elements": {
- "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000.",
+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut.",
"minimum": 1,
"type": "integer"
},
@@ -101,6 +105,10 @@
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.",
"type": "string"
},
+ "since": {
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "type": "string"
+ },
"timeout_ms": {
"default": 1000,
"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",
@@ -108,6 +116,19 @@
"minimum": 100,
"type": "integer"
},
+ "tree_format": {
+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",
+ "enum": [
+ "markdown",
+ "elements",
+ "both"
+ ],
+ "type": "string"
+ },
+ "verbose": {
+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",
+ "type": "boolean"
+ },
"window_id": {
"description": "Target window ID from list_windows.",
"type": "integer"
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index bbf00bc656..c49a8a33de 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Walk a running app's UIA tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `scroll`, etc.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of UIA patterns exposed as actions, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project BOTH `tree_markdown` and `structuredContent.elements` to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤5 000 elements, depth ≤25).\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -57,8 +57,12 @@
],
"type": "string"
},
+ "full_output": {
+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
+ "type": "boolean"
+ },
"include_accessibility_tree": {
- "description": "Default true — walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
+ "description": "Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
"type": "boolean"
},
"include_screenshot": {
@@ -76,7 +80,7 @@
"type": "integer"
},
"max_elements": {
- "description": "Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees.",
+ "description": "Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window.",
"minimum": 1,
"type": "integer"
},
@@ -101,6 +105,10 @@
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
"type": "string"
},
+ "since": {
+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",
+ "type": "string"
+ },
"timeout_ms": {
"default": 1000,
"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",
@@ -108,6 +116,19 @@
"minimum": 100,
"type": "integer"
},
+ "tree_format": {
+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",
+ "enum": [
+ "markdown",
+ "elements",
+ "both"
+ ],
+ "type": "string"
+ },
+ "verbose": {
+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",
+ "type": "boolean"
+ },
"window_id": {
"description": "HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array.",
"type": "integer"
diff --git a/scripts/docs-generators/cli-specs/cua-driver.json b/scripts/docs-generators/cli-specs/cua-driver.json
index d812e0025e..282830899f 100644
--- a/scripts/docs-generators/cli-specs/cua-driver.json
+++ b/scripts/docs-generators/cli-specs/cua-driver.json
@@ -147,6 +147,14 @@
{
"command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725}' --screenshot-out-file state.png",
"description": "Save the screenshot from a tool response"
+ },
+ {
+ "command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"since\":\"s0000002a\"}'",
+ "description": "Read only what changed since an earlier snapshot_id"
+ },
+ {
+ "command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"tree_format\":\"elements\"}'",
+ "description": "Return the structured elements instead of the markdown tree"
}
],
"flags": [],