Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): slim get_window_state and add since diffs (CUA-1192, CUA-1193) (#4743)
Francesco Bonacci45775a9244df
diff --git a/docs/content/docs/cua-driver/reference/cli/mcp.mdx b/docs/content/docs/cua-driver/reference/cli/mcp.mdxindex 5b312e3f0a..f0b51f89e7 100644--- a/docs/content/docs/cua-driver/reference/cli/mcp.mdx+++ b/docs/content/docs/cua-driver/reference/cli/mcp.mdx@@ -142,6 +142,12 @@ cua-driver call click '{"pid":844,"x":100,"y":200}'# Save the screenshot from a tool responsecua-driver call get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file state.png++# Read only what changed since an earlier snapshot_id+cua-driver call get_window_state '{"pid":844,"window_id":10725,"since":"s0000002a"}'++# Return the structured elements instead of the markdown tree+cua-driver call get_window_state '{"pid":844,"window_id":10725,"tree_format":"elements"}'```## `cua-driver manifest`diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdxindex b5d019a317..f1f9436acc 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx@@ -69,7 +69,7 @@ Screenshots, accessibility trees, verification and visual regions of a window. [| Tool | Description | Platforms || --- | --- | --- |-| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |+| <span id="get_window_state"></span>[`get_window_state`](/cua-driver/reference/mcp-tools/window-state#get_window_state) | Snapshot a window: the accessibility tree plus a screenshot. | macOS, Linux, Windows || <span id="get_accessibility_tree"></span>[`get_accessibility_tree`](/cua-driver/reference/mcp-tools/window-state#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows || <span id="verify_state"></span>[`verify_state`](/cua-driver/reference/mcp-tools/window-state#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows || <span id="parse_visual_regions"></span>[`parse_visual_regions`](/cua-driver/reference/mcp-tools/window-state#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdxindex 6e25873936..84aabb5cfd 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx@@ -14,7 +14,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';| Tool | Description | Platforms || --- | --- | --- |-| [`get_window_state`](#get_window_state) | Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. | macOS, Linux, Windows |+| [`get_window_state`](#get_window_state) | Snapshot a window: the accessibility tree plus a screenshot. | macOS, Linux, Windows || [`get_accessibility_tree`](#get_accessibility_tree) | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows || [`verify_state`](#verify_state) | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows || [`parse_visual_regions`](#parse_visual_regions) | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |@@ -28,9 +28,11 @@ Effect: read-only. Platforms: macOS, Linux, Windows.<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist><Tab value="macOS">-Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.+Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:"markdown"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:"elements"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `"both"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.-START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.+START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).++DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.@@ -44,7 +46,7 @@ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `own| <span id="get_window_state--macos--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture. || <span id="get_window_state--macos--max_depth"></span>`max_depth` | `integer` | | Cap on walk depth. Default 25. Minimum: 1. || <span id="get_window_state--macos--max_dimension"></span>`max_dimension` | `integer` | | Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved): the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. Minimum: 1. |-| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000. Minimum: 1. |+| <span id="get_window_state--macos--max_elements"></span>`max_elements` | `integer` | | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut. Minimum: 1. || <span id="get_window_state--macos--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior. Minimum: 0. || <span id="get_window_state--macos--pid"></span>`pid` | `integer` | required | Target process ID. || <span id="get_window_state--macos--query"></span>`query` | `string` | | Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read. |@@ -55,15 +57,17 @@ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `own</Tab><Tab value="Linux">-Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `set_value`, etc.+Walk a running app's AT-SPI tree and return it ONCE, as compact Markdown by default (`tree_format:"markdown"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:"elements"` for the structured `elements` array instead (explicit `element_token`, `role`, `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`), or `"both"` for both (about twice the size). Set `query` to project the tree to matching rows plus their ancestor chain while preserving original indices; `total_element_count` reports the complete snapshot.++Size: a read is bounded by default (`max_elements` 250 actionable nodes). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the walker's own limits (≤5 000 nodes).-PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.+DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Linux yet.Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.The mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title); the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.-Optional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.+Optional `max_elements` / `max_depth` bound the AT-SPI walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250.TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.@@ -75,11 +79,11 @@ POPUP MENUS: a context menu / popover / combo list is an override-redirect windo| Parameter | Type | Default | Description || --- | --- | --- | --- |-| <span id="get_window_state--linux--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |+| <span id="get_window_state--linux--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). || <span id="get_window_state--linux--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action). || <span id="get_window_state--linux--max_depth"></span>`max_depth` | `integer` | | Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps. Minimum: 1. || <span id="get_window_state--linux--max_dimension"></span>`max_dimension` | `integer` | | Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted. Minimum: 1. |-| <span id="get_window_state--linux--max_elements"></span>`max_elements` | `integer` | | Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees. Minimum: 1. |+| <span id="get_window_state--linux--max_elements"></span>`max_elements` | `integer` | | Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. || <span id="get_window_state--linux--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior. Minimum: 0. || <span id="get_window_state--linux--pid"></span>`pid` | `integer` | required | Process ID that owns the window. || <span id="get_window_state--linux--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |@@ -90,17 +94,19 @@ POPUP MENUS: a context menu / popover / combo list is an override-redirect windo</Tab><Tab value="Windows">-Walk a running app's UIA tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `scroll`, etc.+Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:"markdown"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:"elements"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `"both"` for both (about twice the size).-INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.+Size: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).-PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of UIA patterns exposed as actions, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side.+DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.++INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.The UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about: the driver never picks a window implicitly.`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.-Set `query` to a case-insensitive substring to project BOTH `tree_markdown` and `structuredContent.elements` to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.+Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.@@ -108,7 +114,7 @@ The mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk eUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).-Optional `max_elements` / `max_depth` bound the UIA walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤5 000 elements, depth ≤25).+Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.CHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.@@ -118,11 +124,11 @@ Windows requires no special permissions.| Parameter | Type | Default | Description || --- | --- | --- | --- |-| <span id="get_window_state--windows--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). |+| <span id="get_window_state--windows--include_accessibility_tree"></span>`include_accessibility_tree` | `boolean` | | Default true: walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title): the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). || <span id="get_window_state--windows--include_screenshot"></span>`include_screenshot` | `boolean` | | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action). || <span id="get_window_state--windows--max_depth"></span>`max_depth` | `integer` | | Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees. Minimum: 1. || <span id="get_window_state--windows--max_dimension"></span>`max_dimension` | `integer` | | Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins. Minimum: 1. |-| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees. Minimum: 1. |+| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. || <span id="get_window_state--windows--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default. Minimum: 0. || <span id="get_window_state--windows--pid"></span>`pid` | `integer` | required | Process ID from `list_apps`. || <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |@@ -138,7 +144,11 @@ Windows requires no special permissions.| Parameter | Type | Default | Description || --- | --- | --- | --- || <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |+| <span id="get_window_state--full_output"></span>`full_output` | `boolean` | | Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`. |+| <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. || <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |+| <span id="get_window_state--tree_format"></span>`tree_format` | `"markdown" \| "elements" \| "both"` | | Which tree representation to return. Default "markdown": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. "elements": the structured `elements` array only (per-element element_token, frame, parent_index, value). "both": the two together, as before; roughly doubles the size. |+| <span id="get_window_state--verbose"></span>`verbose` | `boolean` | | Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context. |**Example arguments**diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.jsonindex 6a668a2623..80d7ca449f 100644--- a/libs/cua-driver/contract/manifest.json+++ b/libs/cua-driver/contract/manifest.json@@ -1652,6 +1652,10 @@"input_schema": {"additionalProperties": false,"properties": {+ "full_output": {+ "description": "Restore the previous full response (both representations, all\nmetadata, platform walk limits).",+ "type": "boolean"+ },"include_accessibility_tree": {"type": "boolean"},@@ -1689,12 +1693,29 @@"session": {"type": "string"},+ "since": {+ "description": "Return only what changed since this earlier `snapshot_id`.",+ "type": "string"+ },"timeout_ms": {"description": "Wall-clock budget for the accessibility walk in milliseconds\n(default 1000 on every platform). A walk that runs out returns a\npartial tree flagged `truncated` rather than failing.","maximum": 120000,"minimum": 100,"type": "integer"},+ "tree_format": {+ "description": "Tree representation to return: `markdown` (default), `elements`, or\n`both`.",+ "enum": [+ "markdown",+ "elements",+ "both"+ ],+ "type": "string"+ },+ "verbose": {+ "description": "Include `_note` and the full `background_input` report.",+ "type": "boolean"+ },"window_id": {"minimum": 1,"type": "integer"@@ -1919,6 +1940,18 @@"null"]},+ "since": {+ "type": [+ "string",+ "null"+ ]+ },+ "since_status": {+ "type": [+ "string",+ "null"+ ]+ },"snapshot_id": {"type": ["string",@@ -1932,6 +1965,18 @@"null"]},+ "tree_diff": {+ "type": [+ "string",+ "null"+ ]+ },+ "tree_format": {+ "type": [+ "string",+ "null"+ ]+ },"tree_markdown": {"type": ["string",@@ -1944,6 +1989,12 @@"null"]},+ "truncation_hint": {+ "type": [+ "string",+ "null"+ ]+ },"truncation_reason": {"type": ["string",diff --git a/libs/cua-driver/python/src/cua_driver/_native_contract.py b/libs/cua-driver/python/src/cua_driver/_native_contract.pyindex 9a3c21e3b0..d269f746ad 100644--- a/libs/cua-driver/python/src/cua_driver/_native_contract.py+++ b/libs/cua-driver/python/src/cua_driver/_native_contract.py@@ -4157,7 +4157,7 @@ class _UniffiFfiConverterTypeGetSessionStateInput(_UniffiConverterRustBuffer):@dataclassclass GetWindowStateInput:- def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int], timeout_ms:typing.Optional[int] = _DEFAULT):+ def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int], timeout_ms:typing.Optional[int] = _DEFAULT, tree_format:typing.Optional[str] = _DEFAULT, since:typing.Optional[str] = _DEFAULT, verbose:typing.Optional[bool] = _DEFAULT, full_output:typing.Optional[bool] = _DEFAULT):self.pid = pidself.window_id = window_idself.session = session@@ -4173,12 +4173,28 @@ class GetWindowStateInput:self.timeout_ms = Noneelse:self.timeout_ms = timeout_ms+ if tree_format is _DEFAULT:+ self.tree_format = None+ else:+ self.tree_format = tree_format+ if since is _DEFAULT:+ self.since = None+ else:+ self.since = since+ if verbose is _DEFAULT:+ self.verbose = None+ else:+ self.verbose = verbose+ if full_output is _DEFAULT:+ self.full_output = None+ else:+ self.full_output = full_outputdef __str__(self):- return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={}, timeout_ms={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension, self.timeout_ms)+ return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={}, timeout_ms={}, tree_format={}, since={}, verbose={}, full_output={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension, self.timeout_ms, self.tree_format, self.since, self.verbose, self.full_output)def __eq__(self, other):if self.pid != other.pid:return False@@ -4204,6 +4220,14 @@ class GetWindowStateInput:return Falseif self.timeout_ms != other.timeout_ms:return False+ if self.tree_format != other.tree_format:+ return False+ if self.since != other.since:+ return False+ if self.verbose != other.verbose:+ return False+ if self.full_output != other.full_output:+ return Falsereturn Trueclass _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):@@ -4222,6 +4246,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):max_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),max_image_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),timeout_ms=_UniffiFfiConverterOptionalUInt32.read(buf),+ tree_format=_UniffiFfiConverterOptionalString.read(buf),+ since=_UniffiFfiConverterOptionalString.read(buf),+ verbose=_UniffiFfiConverterOptionalBoolean.read(buf),+ full_output=_UniffiFfiConverterOptionalBoolean.read(buf),)@staticmethod@@ -4238,6 +4266,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalUInt32.check_lower(value.max_dimension)_UniffiFfiConverterOptionalUInt32.check_lower(value.max_image_dimension)_UniffiFfiConverterOptionalUInt32.check_lower(value.timeout_ms)+ _UniffiFfiConverterOptionalString.check_lower(value.tree_format)+ _UniffiFfiConverterOptionalString.check_lower(value.since)+ _UniffiFfiConverterOptionalBoolean.check_lower(value.verbose)+ _UniffiFfiConverterOptionalBoolean.check_lower(value.full_output)@staticmethoddef write(value, buf):@@ -4253,6 +4285,10 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalUInt32.write(value.max_dimension, buf)_UniffiFfiConverterOptionalUInt32.write(value.max_image_dimension, buf)_UniffiFfiConverterOptionalUInt32.write(value.timeout_ms, buf)+ _UniffiFfiConverterOptionalString.write(value.tree_format, buf)+ _UniffiFfiConverterOptionalString.write(value.since, buf)+ _UniffiFfiConverterOptionalBoolean.write(value.verbose, buf)+ _UniffiFfiConverterOptionalBoolean.write(value.full_output, buf)@dataclassclass HotkeyInput:@@ -8195,7 +8231,7 @@ class _UniffiFfiConverterSequenceTypeSnapshotImage(_UniffiConverterRustBuffer):@dataclassclass WindowStateOutput:- def __init__(self, *, pid:int, window_id:int, snapshot_id:typing.Optional[str], app_name:typing.Optional[str], window_title:typing.Optional[str], tree_markdown:typing.Optional[str], elements:typing.Optional[typing.List[WindowElement]], element_count:typing.Optional[int], total_element_count:typing.Optional[int], returned_element_count:typing.Optional[int], filtered_element_count:typing.Optional[int], elements_complete:typing.Optional[bool], degraded:typing.Optional[bool], degraded_reason:typing.Optional[str], truncated:typing.Optional[bool], truncation_reason:typing.Optional[str], screenshot_width:typing.Optional[int], screenshot_height:typing.Optional[int], screenshot_scale:typing.Optional[float], screenshot_mime_type:typing.Optional[str], screenshot_file_path:typing.Optional[str], screenshot_frame_valid:typing.Optional[bool], window_bounds:typing.Optional[WindowBounds], images:typing.List[SnapshotImage]):+ def __init__(self, *, pid:int, window_id:int, snapshot_id:typing.Optional[str], app_name:typing.Optional[str], window_title:typing.Optional[str], tree_markdown:typing.Optional[str], elements:typing.Optional[typing.List[WindowElement]], element_count:typing.Optional[int], total_element_count:typing.Optional[int], returned_element_count:typing.Optional[int], filtered_element_count:typing.Optional[int], elements_complete:typing.Optional[bool], degraded:typing.Optional[bool], degraded_reason:typing.Optional[str], truncated:typing.Optional[bool], truncation_reason:typing.Optional[str], truncation_hint:typing.Optional[str], tree_format:typing.Optional[str], since:typing.Optional[str], since_status:typing.Optional[str], tree_diff:typing.Optional[str], screenshot_width:typing.Optional[int], screenshot_height:typing.Optional[int], screenshot_scale:typing.Optional[float], screenshot_mime_type:typing.Optional[str], screenshot_file_path:typing.Optional[str], screenshot_frame_valid:typing.Optional[bool], window_bounds:typing.Optional[WindowBounds], images:typing.List[SnapshotImage]):self.pid = pidself.window_id = window_idself.snapshot_id = snapshot_id@@ -8212,6 +8248,11 @@ class WindowStateOutput:self.degraded_reason = degraded_reasonself.truncated = truncatedself.truncation_reason = truncation_reason+ self.truncation_hint = truncation_hint+ self.tree_format = tree_format+ self.since = since+ self.since_status = since_status+ self.tree_diff = tree_diffself.screenshot_width = screenshot_widthself.screenshot_height = screenshot_heightself.screenshot_scale = screenshot_scale@@ -8225,7 +8266,7 @@ class WindowStateOutput:def __str__(self):- return "WindowStateOutput(pid={}, window_id={}, snapshot_id={}, app_name={}, window_title={}, tree_markdown={}, elements={}, element_count={}, total_element_count={}, returned_element_count={}, filtered_element_count={}, elements_complete={}, degraded={}, degraded_reason={}, truncated={}, truncation_reason={}, screenshot_width={}, screenshot_height={}, screenshot_scale={}, screenshot_mime_type={}, screenshot_file_path={}, screenshot_frame_valid={}, window_bounds={}, images={})".format(self.pid, self.window_id, self.snapshot_id, self.app_name, self.window_title, self.tree_markdown, self.elements, self.element_count, self.total_element_count, self.returned_element_count, self.filtered_element_count, self.elements_complete, self.degraded, self.degraded_reason, self.truncated, self.truncation_reason, self.screenshot_width, self.screenshot_height, self.screenshot_scale, self.screenshot_mime_type, self.screenshot_file_path, self.screenshot_frame_valid, self.window_bounds, self.images)+ return "WindowStateOutput(pid={}, window_id={}, snapshot_id={}, app_name={}, window_title={}, tree_markdown={}, elements={}, element_count={}, total_element_count={}, returned_element_count={}, filtered_element_count={}, elements_complete={}, degraded={}, degraded_reason={}, truncated={}, truncation_reason={}, truncation_hint={}, tree_format={}, since={}, since_status={}, tree_diff={}, screenshot_width={}, screenshot_height={}, screenshot_scale={}, screenshot_mime_type={}, screenshot_file_path={}, screenshot_frame_valid={}, window_bounds={}, images={})".format(self.pid, self.window_id, self.snapshot_id, self.app_name, self.window_title, self.tree_markdown, self.elements, self.element_count, self.total_element_count, self.returned_element_count, self.filtered_element_count, self.elements_complete, self.degraded, self.degraded_reason, self.truncated, self.truncation_reason, self.truncation_hint, self.tree_format, self.since, self.since_status, self.tree_diff, self.screenshot_width, self.screenshot_height, self.screenshot_scale, self.screenshot_mime_type, self.screenshot_file_path, self.screenshot_frame_valid, self.window_bounds, self.images)def __eq__(self, other):if self.pid != other.pid:return False@@ -8259,6 +8300,16 @@ class WindowStateOutput:return Falseif self.truncation_reason != other.truncation_reason:return False+ if self.truncation_hint != other.truncation_hint:+ return False+ if self.tree_format != other.tree_format:+ return False+ if self.since != other.since:+ return False+ if self.since_status != other.since_status:+ return False+ if self.tree_diff != other.tree_diff:+ return Falseif self.screenshot_width != other.screenshot_width:return Falseif self.screenshot_height != other.screenshot_height:@@ -8297,6 +8348,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):degraded_reason=_UniffiFfiConverterOptionalString.read(buf),truncated=_UniffiFfiConverterOptionalBoolean.read(buf),truncation_reason=_UniffiFfiConverterOptionalString.read(buf),+ truncation_hint=_UniffiFfiConverterOptionalString.read(buf),+ tree_format=_UniffiFfiConverterOptionalString.read(buf),+ since=_UniffiFfiConverterOptionalString.read(buf),+ since_status=_UniffiFfiConverterOptionalString.read(buf),+ tree_diff=_UniffiFfiConverterOptionalString.read(buf),screenshot_width=_UniffiFfiConverterOptionalUInt32.read(buf),screenshot_height=_UniffiFfiConverterOptionalUInt32.read(buf),screenshot_scale=_UniffiFfiConverterOptionalFloat64.read(buf),@@ -8325,6 +8381,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalString.check_lower(value.degraded_reason)_UniffiFfiConverterOptionalBoolean.check_lower(value.truncated)_UniffiFfiConverterOptionalString.check_lower(value.truncation_reason)+ _UniffiFfiConverterOptionalString.check_lower(value.truncation_hint)+ _UniffiFfiConverterOptionalString.check_lower(value.tree_format)+ _UniffiFfiConverterOptionalString.check_lower(value.since)+ _UniffiFfiConverterOptionalString.check_lower(value.since_status)+ _UniffiFfiConverterOptionalString.check_lower(value.tree_diff)_UniffiFfiConverterOptionalUInt32.check_lower(value.screenshot_width)_UniffiFfiConverterOptionalUInt32.check_lower(value.screenshot_height)_UniffiFfiConverterOptionalFloat64.check_lower(value.screenshot_scale)@@ -8352,6 +8413,11 @@ class _UniffiFfiConverterTypeWindowStateOutput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalString.write(value.degraded_reason, buf)_UniffiFfiConverterOptionalBoolean.write(value.truncated, buf)_UniffiFfiConverterOptionalString.write(value.truncation_reason, buf)+ _UniffiFfiConverterOptionalString.write(value.truncation_hint, buf)+ _UniffiFfiConverterOptionalString.write(value.tree_format, buf)+ _UniffiFfiConverterOptionalString.write(value.since, buf)+ _UniffiFfiConverterOptionalString.write(value.since_status, buf)+ _UniffiFfiConverterOptionalString.write(value.tree_diff, buf)_UniffiFfiConverterOptionalUInt32.write(value.screenshot_width, buf)_UniffiFfiConverterOptionalUInt32.write(value.screenshot_height, buf)_UniffiFfiConverterOptionalFloat64.write(value.screenshot_scale, buf)diff --git a/libs/cua-driver/python/tests/test_remote_channel.py b/libs/cua-driver/python/tests/test_remote_channel.pyindex d07f2d7a00..aed628981f 100644--- a/libs/cua-driver/python/tests/test_remote_channel.py+++ b/libs/cua-driver/python/tests/test_remote_channel.py@@ -346,6 +346,8 @@ class NativeWindowTests(unittest.IsolatedAsyncioTestCase):"max_depth": 3,"max_dimension": 800,"max_image_dimension": 1600,+ # The typed SDK keeps the full response unless told otherwise.+ "full_output": True,},)diff --git a/libs/cua-driver/rust/CHANGELOG.md b/libs/cua-driver/rust/CHANGELOG.mdindex 98a9eafd0f..bb529c4e5f 100644--- a/libs/cua-driver/rust/CHANGELOG.md+++ b/libs/cua-driver/rust/CHANGELOG.md@@ -2,6 +2,10 @@## Unreleased+### Features++* **cua-driver:** `get_window_state` returns one tree representation by default (compact markdown; `tree_format` selects `elements` or `both`), walks at most 250 nodes unless `max_elements` says otherwise and says when it truncated, and omits `_note` / `background_input` unless `verbose:true`. `full_output:true` restores the previous full response. New `since:<snapshot_id>` returns only added, changed and removed rows (CUA-1192, CUA-1193). The typed SDKs keep the full response unless a shape is requested.+### Bug Fixes* **cua-driver:** deliver X11 key-down before the tap delay and finish background keyboard delivery before closing the input connection.diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.mdindex 56ab893861..025f0b2ea7 100644--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md@@ -77,7 +77,7 @@ sentence naming the call that targets each one(`get_window_state(pid, window_id=<that id>)`). When one overlaps the windowthe screenshot is taken from the screen (`screenshot_composited:true`) so themenu or dialog is visible; the window's own drawable never shows them. The-payload states its frame (`coordinate_frame:"window"`, `frame_note`): `x`/`y`+payload states its frame (`coordinate_frame:"window"`, plus `frame_note` with `verbose:true`): `x`/`y`of the pointer tools are pixels of THIS screenshot. Closed menus are listedwith a `description` ("closed menu with N items…") and are not walked;click the menu (a real press) and read the popup by its window_id.@@ -212,6 +212,12 @@ doctor` now probes `org.a11y.Bus` for real (not just "is there a bus?")daemon as root against a user session is the Linux analogue of the Windows"Session 0" isolation problem.+`get_window_state` returns one compact markdown tree capped at 250 actionable+nodes by default (`full_output:true` restores 5 000 nodes and both forms), and+`since:<snapshot_id>` returns only added/changed/removed rows. The focused+element is not reported on Linux yet, so an unchanged read says only+`no change since …`.+An empty AT-SPI walk is now surfaced honestly: `get_window_state` sets`degraded: true` + a `degraded_reason` (instead of a bare `elements: []`) so acaller can tell "this window genuinely has no controls" apart from "the a11ydiff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex a8cf28bdb9..9fc884e7b9 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -122,6 +122,11 @@ For authorized foreground input, use the Cua action'sWhen a cua-driver call surprises you, diagnose cua-driver first:+- **Large tree or repeated reads?** `get_window_state` returns one compact+ markdown tree capped at 250 nodes; pass `since:<snapshot_id>` for a diff+ (`no change since …; focused element is …` names the focused row on macOS)+ and `tree_format:"elements"` for structured rows. `full_output:true` restores+ the old payload (both forms, `_note`, `background_input`, 2 000 nodes).- **Empty `tree_markdown`?** `get_window_state` returns **both** theAX tree and a screenshot by default: there's nothing to configure andno capture mode to pick. An empty tree means the surface isn't AX (adiff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex 05cb1225a0..4b78f68900 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -67,7 +67,7 @@ Check the installed version and advertised schema before using unfamiliar parame1. Select the exact target on each action. A session is lifecycle metadata, not capture scope or permission authority.2. Observe before input and verify the outcome at checkpoints, not after every action. `effect:"unverifiable"` and a successful exit are not task success; never replay a partial, canceled, or unknown action blindly.-3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`.+3. Use returned tokens, never invented indices. A fresh snapshot replaces prior element handles and lists them in `invalidated_snapshot_ids`; act with `element_token`. Reads are bounded and return one tree form by default; follow up with `since:<snapshot_id>` instead of re-reading a large window.4. Keep background window actions non-interfering. Foreground delivery and desktop input require authorization for visible control; an unavailable route is not permission to escalate.5. Never infer pixels from a missing image, a different window, or an unaccounted-for resized preview. Capture failure and an empty accessibility tree are different failures.6. Keep one controller for a shared desktop. Distinct sessions/cursors do not isolate focus, keyboard input, application state, or snapshot caches.@@ -79,7 +79,7 @@ Check the installed version and advertised schema before using unfamiliar parame| --------------------------------------------------------- | ------------------------------------------------------------------------------------- || Missing binary, mismatched daemon, unknown tool/field | [Runtime preflight](RUNTIME.md#preflight-and-transport) || Stale token or ambiguous window | Refresh `list_windows` / `get_window_state`; choose the intended live target |-| Large or sparse tree | [Bounded observation](WORKFLOW.md#observe) |+| Large, truncated, or sparse tree; repeated reads | [Bounded observation and `since` diffs](WORKFLOW.md#observe) || `surface_identity_unproven` or screenshot permission wait | [Wayland capture recovery](LINUX.md#capture-recovery) || `background_unavailable` | Verify current state; ask before foreground/desktop control if not already authorized || Text did not visibly change | Reobserve before retrying; [text and value semantics](WORKFLOW.md#act-once) |diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.mdindex 10bc995f2e..3f1a1012ed 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md@@ -376,6 +376,11 @@ When a cua-driver surprises you, diagnose cua-driver first:click in.- **`Invalid window handle (0x80070578)`?** The HWND you passed isstale (window closed, recreated). Re-resolve via `list_windows`.+- **Large tree or repeated reads?** `get_window_state` returns one compact+ markdown tree capped at 250 nodes (`full_output:true` restores 5 000 nodes+ and both forms). Pass `since:<snapshot_id>` for added/changed/removed rows;+ the focused element is not reported on Windows yet, so an unchanged read+ says only `no change since …`.- **Empty `tree_markdown` / sparse UIA tree?** Some apps populatetheir UIA tree lazily on first call; retry `get_window_state`once. If still empty, the app has no UIA provider: fall back to andiff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.mdindex 80ea3c4536..0086d3e6f7 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md@@ -37,10 +37,12 @@ These objects are argument fragments for tools advertising `target`, not standal`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Start with `query` (or `max_elements`) instead of a full read, and verify at checkpoints rather than after every action. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.-Prefer `structuredContent.elements` in MCP (the CLI prints structured fields directly) over parsing `tree_markdown`. Rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.+By default the tree comes back **once**, as compact `tree_markdown` (`tree_format:"markdown"`). A row `[N]` is addressed with `element_token` `<snapshot_id>:N`; the `snapshot_id` is in the response header and in `structuredContent`. Use `tree_format:"elements"` for the structured `elements` array when you need explicit tokens, frames, parents, or values, or `"both"` for the old double payload (about twice the size). `full_output:true` restores the previous full response, including the platform's walk limits. In `elements`, rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.- Use `query` to project matching rows plus ancestors without renumbering their indices.-- Use `max_elements` / `max_depth` to bound the walk, and compare returned/total counts. Truncation does not prove absence.+- A read walks at most 250 nodes by default. When it stops there the response says `Tree truncated at max_elements=…` (`truncated:true`, `truncation_hint`). Pass a larger `max_elements`, or narrow with `query` / `max_depth`. Truncation does not prove absence.+- After the first read, pass `since:<snapshot_id>` to get only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.+- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.- Use `include_screenshot:false` only when tree-only observation is enough; it cannot ground a pixel action.- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. Check the installed schema first.- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rsindex ee76c55116..25fb314eb5 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs@@ -104,6 +104,32 @@ pub struct GetWindowStateInput {#[schemars(schema_with = "timeout_ms_schema")]#[uniffi(default = None)]pub timeout_ms: Option<u32>,+ /// Tree representation to return: `markdown` (default), `elements`, or+ /// `both`.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "tree_format_schema")]+ #[uniffi(default = None)]+ pub tree_format: Option<String>,+ /// Return only what changed since this earlier `snapshot_id`.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "string_schema")]+ #[uniffi(default = None)]+ pub since: Option<String>,+ /// Include `_note` and the full `background_input` report.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "bool_schema")]+ #[uniffi(default = None)]+ pub verbose: Option<bool>,+ /// Restore the previous full response (both representations, all+ /// metadata, platform walk limits).+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "bool_schema")]+ #[uniffi(default = None)]+ pub full_output: Option<bool>,+}++fn tree_format_schema(_: &mut schemars::SchemaGenerator) -> schemars::Schema {+ schemars::json_schema!({"type":"string", "enum":["markdown","elements","both"]})}/// Bounds of the accessibility-walk budget, shared with every live backend@@ -303,6 +329,24 @@ pub struct WindowStateOutput {pub truncated: Option<bool>,#[serde(default, skip_serializing_if = "Option::is_none")]pub truncation_reason: Option<String>,+ /// Plain-language line saying the tree was cut at `max_elements` and how+ /// to read more.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub truncation_hint: Option<String>,+ /// Which representation this response carries.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub tree_format: Option<String>,+ /// The `since` snapshot_id the caller asked to diff against.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub since: Option<String>,+ /// `diff`, `no_change`, or the reason a full read was returned instead+ /// (`unknown_snapshot`, `other_window`, `view_changed`, `too_much_changed`,+ /// `diff_too_large`, `no_snapshot`).+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub since_status: Option<String>,+ /// Added/changed/removed rows against `since`, one per line.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub tree_diff: Option<String>,#[serde(default, skip_serializing_if = "Option::is_none")]pub screenshot_width: Option<u32>,#[serde(default, skip_serializing_if = "Option::is_none")]diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex 33217a6c7b..09d691965e 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -113,6 +113,7 @@ pub mod video_ffmpeg;pub mod walk_budget;pub mod window_inspection;pub mod window_observation;+pub mod window_state_view;pub mod window_target;pub use cua_driver_contract::{CaptureScope, EscalationReason, TOOL_INVOCATION_FAILED_CODE};diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rsindex c78c09135b..324832362f 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs@@ -217,10 +217,15 @@ pub fn truncation_note(Some(other) => other.to_owned(),None => "the walk stopped early".to_owned(),};+ let remedy = if reason == Some("node_budget") {+ "retry with a larger max_elements"+ } else {+ "retry with a larger timeout_ms (e.g. 5000)"+ };format!("⚠️ PARTIAL TREE: {why} after {visited} node(s) ({pending} discovered but not visited). \Every element listed is real; elements after the cut are missing. If the element you \- need is absent, retry with a larger timeout_ms (e.g. 5000) or narrow with query / max_depth."+ need is absent, {remedy} or narrow with query / max_depth.")}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rsnew file mode 100644index 0000000000..330b427a41--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs@@ -0,0 +1,1074 @@+//! Response shaping for `get_window_state`, shared by every platform backend.+//!+//! A backend walks its accessibility API and builds the complete legacy+//! payload (structured `elements`, `tree_markdown`, a text block with the same+//! markdown, plus metadata). [`apply`] then reduces that payload to what a+//! model needs:+//!+//! * ONE tree representation (`tree_format`, default compact markdown),+//! * trimmed metadata (`_note`, `background_input`) unless `verbose`,+//! * a stated truncation line when the walk hit the `max_elements` budget,+//! * with `since: <snapshot_id>`, only the rows that changed against that+//! snapshot, falling back to a full read when the snapshot is unknown.+//!+//! `full_output: true` restores the previous payload byte for byte (both+//! representations, all metadata, the platform walk caps).+//!+//! The diff works on the rendered markdown rows, which every backend emits in+//! the same `- [N] Role "label" ...` shape, so it also sees display-only rows+//! (static text) that the structured `elements` array omits.++use crate::protocol::{Content, ToolResult};+use serde_json::{json, Value};+use std::collections::VecDeque;+use std::sync::{Mutex, OnceLock};+use std::time::{Duration, Instant};++/// Default accessibility-node budget for a model-facing read. The platform+/// walkers default to 2 000 (macOS) or 5 000 (Windows, Linux) nodes, which+/// renders 40-100K characters for an ordinary window.+pub const DEFAULT_MAX_ELEMENTS: usize = 250;++/// Snapshots remembered per (pid, window) for `since`.+const STORE_PER_WINDOW: usize = 8;+/// Snapshots remembered across all windows.+const STORE_TOTAL: usize = 32;+/// A remembered snapshot stops being a valid `since` baseline after this long.+const STORE_TTL: Duration = Duration::from_secs(15 * 60);+/// Largest old-rows x new-rows product the line diff will compute.+const DIFF_CELL_LIMIT: usize = 6_000_000;+/// A diff larger than this share (percent) of the full markdown is not worth+/// it: return the full read instead.+const DIFF_MAX_PERCENT_OF_FULL: usize = 60;++#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum TreeFormat {+ /// Compact markdown only (default).+ Markdown,+ /// Structured `elements` only.+ Elements,+ /// Both, as before this change.+ Both,+}++impl TreeFormat {+ pub fn as_str(self) -> &'static str {+ match self {+ TreeFormat::Markdown => "markdown",+ TreeFormat::Elements => "elements",+ TreeFormat::Both => "both",+ }+ }+}++/// Caller-controlled shape of the response.+#[derive(Clone, Debug)]+pub struct ViewOptions {+ pub tree_format: TreeFormat,+ pub verbose: bool,+ pub full_output: bool,+ pub since: Option<String>,+}++/// JSON schema fragments for the shared arguments, so every backend declares+/// the same properties.+pub fn schema_properties() -> Value {+ json!({+ "tree_format": {+ "type": "string",+ "enum": ["markdown", "elements", "both"],+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size."+ },+ "since": {+ "type": "string",+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise."+ },+ "verbose": {+ "type": "boolean",+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context."+ },+ "full_output": {+ "type": "boolean",+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`."+ }+ })+}++/// Add the shared `tree_format` / `since` / `verbose` / `full_output`+/// properties to a backend's `get_window_state` input schema.+pub fn extend_input_schema(mut schema: Value) -> Value {+ if let (Some(props), Some(extra)) = (+ schema.get_mut("properties").and_then(Value::as_object_mut),+ schema_properties().as_object().cloned(),+ ) {+ props.extend(extra);+ }+ schema+}++impl ViewOptions {+ pub fn from_args(args: &Value) -> Result<Self, ToolResult> {+ let full_output = args.get("full_output").and_then(Value::as_bool) == Some(true);+ let tree_format = match args.get("tree_format") {+ None | Some(Value::Null) => {+ if full_output {+ TreeFormat::Both+ } else {+ TreeFormat::Markdown+ }+ }+ Some(Value::String(s)) => match s.as_str() {+ "markdown" => TreeFormat::Markdown,+ "elements" => TreeFormat::Elements,+ "both" => TreeFormat::Both,+ other => {+ return Err(ToolResult::error(format!(+ "tree_format must be one of \"markdown\", \"elements\", \"both\"; got \"{other}\"."+ )))+ }+ },+ Some(_) => {+ return Err(ToolResult::error(+ "tree_format must be one of \"markdown\", \"elements\", \"both\".",+ ))+ }+ };+ let since = match args.get("since") {+ None | Some(Value::Null) => None,+ Some(Value::String(s)) if !s.trim().is_empty() => Some(s.trim().to_owned()),+ Some(_) => {+ return Err(ToolResult::error(+ "since must be a snapshot_id string from an earlier get_window_state.",+ ))+ }+ };+ Ok(Self {+ tree_format,+ verbose: args.get("verbose").and_then(Value::as_bool) == Some(true),+ full_output,+ since: if full_output { None } else { since },+ })+ }++ /// Node budget for the walk. An explicit `max_elements` always wins;+ /// otherwise a model-facing read gets [`DEFAULT_MAX_ELEMENTS`], while+ /// `full_output` and internal observation reads keep the platform default.+ pub fn max_elements(+ &self,+ args: &Value,+ platform_default: usize,+ observation_only: bool,+ ) -> usize {+ if let Some(n) = args.get("max_elements").and_then(Value::as_u64) {+ return n.max(1) as usize;+ }+ if self.full_output || observation_only {+ platform_default+ } else {+ DEFAULT_MAX_ELEMENTS.min(platform_default)+ }+ }+}++/// What a `since` read resolved to.+#[derive(Clone, Debug, PartialEq, Eq)]+enum SinceOutcome {+ Diff(DiffReport),+ Fallback(&'static str),+}++/// Inputs that identify the view a snapshot was read with. A diff is only+/// meaningful between two reads of the same view.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct ViewKey {+ pub query: Option<String>,+ pub max_elements: usize,+ pub max_depth: Option<usize>,+}++pub struct ViewContext<'a> {+ pub pid: i64,+ pub window_id: u64,+ pub key: ViewKey,+ /// Describes the focused element, called only for a `no change` answer.+ pub focus_probe: Option<&'a dyn Fn() -> Option<String>>,+}++/// Shape a finished `get_window_state` payload in place.+pub fn apply(+ opts: &ViewOptions,+ ctx: &ViewContext<'_>,+ content: &mut [Content],+ structured: &mut Value,+) {+ if opts.full_output {+ return;+ }+ trim_metadata(opts, structured);+ let Some(md) = structured+ .get("tree_markdown")+ .and_then(Value::as_str)+ .map(str::to_owned)+ else {+ return;+ };+ let snapshot_id = structured+ .get("snapshot_id")+ .and_then(Value::as_str)+ .map(str::to_owned);++ let outcome = opts+ .since+ .as_deref()+ .map(|since| resolve_since(since, snapshot_id.as_deref(), &md, ctx));+ if let Some(sid) = &snapshot_id {+ remember(sid, ctx, &md);+ }++ // Header text added to the first line of the tree block.+ let mut header_extra = String::new();+ if let Some(sid) = &snapshot_id {+ header_extra.push_str(&format!(" snapshot_id={sid}"));+ }+ let truncation = truncation_line(structured, &ctx.key);+ if let Some(line) = &truncation {+ structured["truncation_hint"] = json!(line);+ }++ let body;+ match outcome {+ Some(SinceOutcome::Diff(report)) => {+ let since = opts.since.as_deref().unwrap_or_default();+ let focus = if report.is_empty() {+ ctx.focus_probe.and_then(|probe| probe())+ } else {+ None+ };+ let text = report.render(since, snapshot_id.as_deref(), focus.as_deref());+ structured["since"] = json!(since);+ structured["since_status"] = json!(if report.is_empty() {+ "no_change"+ } else {+ "diff"+ });+ structured["tree_diff"] = json!(text);+ structured["diff_counts"] = json!({+ "added": report.added.len(),+ "changed": report.changed.len(),+ "removed": report.removed.len(),+ });+ if let Some(focus) = focus {+ structured["focused_element"] = json!(focus);+ }+ let keep: std::collections::HashSet<u64> = report+ .added+ .iter()+ .chain(report.changed.iter())+ .filter_map(|row| row.index)+ .collect();+ retain_elements(structured, opts.tree_format, Some(&keep));+ if let Some(obj) = structured.as_object_mut() {+ obj.remove("tree_markdown");+ }+ body = text;+ }+ other => {+ if let Some(SinceOutcome::Fallback(reason)) = &other {+ let since = opts.since.as_deref().unwrap_or_default();+ structured["since"] = json!(since);+ structured["since_status"] = json!(reason);+ header_extra.push_str(&format!(+ "\nsince={since}: {}; full read follows.",+ fallback_text(reason)+ ));+ }+ match opts.tree_format {+ TreeFormat::Markdown => {+ body = md.clone();+ retain_elements(structured, TreeFormat::Markdown, None);+ }+ TreeFormat::Elements => {+ body = "(tree is in structuredContent.elements; pass tree_format \"markdown\" for the compact text rendering)".to_owned();+ if let Some(obj) = structured.as_object_mut() {+ obj.remove("tree_markdown");+ }+ }+ TreeFormat::Both => {+ body = md.clone();+ }+ }+ if opts.tree_format == TreeFormat::Markdown && snapshot_id.is_some() {+ header_extra+ .push_str("\nelement_token for row [N] = <snapshot_id>:N (e.g. s0000002a:N)");+ }+ }+ }+ structured["tree_format"] = json!(opts.tree_format.as_str());++ if let Some(line) = truncation {+ // Backends that already print the long PARTIAL TREE note keep it.+ let already_noted = content.iter().any(|c| match c {+ Content::Text { text, .. } => text.contains("PARTIAL TREE"),+ _ => false,+ });+ if !already_noted {+ header_extra.push('\n');+ header_extra.push_str(&line);+ }+ }++ rewrite_tree_block(content, &md, &header_extra, &body);+}++fn fallback_text(reason: &str) -> &'static str {+ match reason {+ "unknown_snapshot" => "unknown or expired snapshot_id",+ "other_window" => "that snapshot belongs to a different window",+ "view_changed" => "query, max_elements or max_depth differ from that snapshot's read",+ "too_much_changed" => "most of the tree changed",+ "diff_too_large" => "the trees are too large to compare",+ "no_snapshot" => "this read produced no snapshot to compare against",+ _ => "cannot diff",+ }+}++/// Replace the markdown inside the platform's text block with `body`, and add+/// `header_extra` to the end of that block's first line.+fn rewrite_tree_block(content: &mut [Content], md: &str, header_extra: &str, body: &str) {+ if md.is_empty() {+ return;+ }+ for part in content.iter_mut() {+ let Content::Text { text, .. } = part else {+ continue;+ };+ let Some(pos) = text.find(md) else { continue };+ let prefix = &text[..pos];+ let suffix = &text[pos + md.len()..];+ let mut out = String::with_capacity(text.len());+ match prefix.find('\n') {+ Some(nl) => {+ out.push_str(&prefix[..nl]);+ out.push_str(header_extra);+ out.push_str(&prefix[nl..]);+ }+ None => {+ out.push_str(prefix);+ out.push_str(header_extra);+ out.push('\n');+ }+ }+ out.push_str(body);+ out.push_str(suffix);+ *text = out;+ return;+ }+}++fn trim_metadata(opts: &ViewOptions, structured: &mut Value) {+ if opts.verbose {+ return;+ }+ let degraded = structured.get("degraded").and_then(Value::as_bool) == Some(true);+ if let Some(obj) = structured.as_object_mut() {+ obj.remove("_note");+ // Linux restates the pixel-coordinate convention on every read; the+ // tool description already carries it.+ obj.remove("frame_note");+ if !degraded {+ obj.remove("background_input");+ }+ }+}++/// Keep `elements` per the requested format. `keep` limits them to the given+/// element indices (diff reads).+fn retain_elements(+ structured: &mut Value,+ format: TreeFormat,+ keep: Option<&std::collections::HashSet<u64>>,+) {+ let Some(obj) = structured.as_object_mut() else {+ return;+ };+ if format == TreeFormat::Markdown {+ obj.remove("elements");+ return;+ }+ if let (Some(keep), Some(Value::Array(elements))) = (keep, obj.get_mut("elements")) {+ elements.retain(|e| {+ e.get("element_index")+ .and_then(Value::as_u64)+ .is_some_and(|i| keep.contains(&i))+ });+ }+}++fn truncation_line(structured: &Value, key: &ViewKey) -> Option<String> {+ if structured.get("truncated").and_then(Value::as_bool) != Some(true) {+ return None;+ }+ let reason = structured+ .get("truncation_reason")+ .and_then(Value::as_str)+ .unwrap_or("");+ if reason != "node_budget" {+ return None;+ }+ let pending = structured+ .get("nodes_pending")+ .and_then(Value::as_u64)+ .unwrap_or(0);+ Some(format!(+ "Tree truncated at max_elements={} ({pending} more node(s) not shown). Pass a larger max_elements, or query/max_depth to reach the rest.",+ key.max_elements+ ))+}++// ── snapshot memory for `since` ──────────────────────────────────────────++#[derive(Clone)]+struct StoredSnapshot {+ snapshot_id: String,+ pid: i64,+ window_id: u64,+ key: ViewKey,+ markdown: String,+ at: Instant,+}++fn store() -> &'static Mutex<VecDeque<StoredSnapshot>> {+ static STORE: OnceLock<Mutex<VecDeque<StoredSnapshot>>> = OnceLock::new();+ STORE.get_or_init(|| Mutex::new(VecDeque::new()))+}++fn remember(snapshot_id: &str, ctx: &ViewContext<'_>, md: &str) {+ let mut guard = store().lock().unwrap_or_else(|e| e.into_inner());+ guard.retain(|s| s.snapshot_id != snapshot_id);+ guard.push_back(StoredSnapshot {+ snapshot_id: snapshot_id.to_owned(),+ pid: ctx.pid,+ window_id: ctx.window_id,+ key: ctx.key.clone(),+ markdown: md.to_owned(),+ at: Instant::now(),+ });+ let same_window = |s: &StoredSnapshot| s.pid == ctx.pid && s.window_id == ctx.window_id;+ while guard.iter().filter(|s| same_window(s)).count() > STORE_PER_WINDOW {+ if let Some(pos) = guard.iter().position(same_window) {+ guard.remove(pos);+ }+ }+ while guard.len() > STORE_TOTAL {+ guard.pop_front();+ }+}++fn resolve_since(+ since: &str,+ current_id: Option<&str>,+ md: &str,+ ctx: &ViewContext<'_>,+) -> SinceOutcome {+ if current_id.is_none() {+ return SinceOutcome::Fallback("no_snapshot");+ }+ let found = {+ let mut guard = store().lock().unwrap_or_else(|e| e.into_inner());+ guard.retain(|s| s.at.elapsed() <= STORE_TTL);+ guard.iter().find(|s| s.snapshot_id == since).cloned()+ };+ let Some(old) = found else {+ return SinceOutcome::Fallback("unknown_snapshot");+ };+ if old.pid != ctx.pid || old.window_id != ctx.window_id {+ return SinceOutcome::Fallback("other_window");+ }+ if old.key != ctx.key {+ return SinceOutcome::Fallback("view_changed");+ }+ match diff_markdown(&old.markdown, md) {+ Err(DiffError::TooLarge) => SinceOutcome::Fallback("diff_too_large"),+ Ok(report) => {+ if !report.is_empty() && report.render_len() * 100 > md.len() * DIFF_MAX_PERCENT_OF_FULL+ {+ SinceOutcome::Fallback("too_much_changed")+ } else {+ SinceOutcome::Diff(report)+ }+ }+ }+}++// ── markdown row diff ────────────────────────────────────────────────────++#[derive(Clone, Debug, PartialEq, Eq)]+struct Row {+ indent: usize,+ index: Option<u64>,+ /// Row text without indentation and without the `[N]` index.+ key: String,+ /// Role plus identifier or label; pairs a removed row with its changed+ /// replacement.+ identity: String,+ /// Row text as rendered, index included, without indentation.+ text: String,+}++#[derive(Clone, Debug, PartialEq, Eq, Default)]+pub struct DiffReport {+ added: Vec<Row>,+ changed: Vec<Row>,+ removed: Vec<Row>,+ reindexed: Vec<(u64, u64, i64)>,+}++#[derive(Debug, PartialEq, Eq)]+enum DiffError {+ TooLarge,+}++fn parse_rows(md: &str) -> Vec<Row> {+ let mut rows: Vec<Row> = Vec::new();+ for line in md.lines() {+ let trimmed = line.trim_start();+ if let Some(rest) = trimmed.strip_prefix("- ") {+ let indent = line.len() - trimmed.len();+ let (index, after) = match rest.strip_prefix('[') {+ Some(r) => match r.split_once(']') {+ Some((n, tail)) if n.parse::<u64>().is_ok() => {+ (n.parse::<u64>().ok(), tail.trim_start())+ }+ _ => (None, rest),+ },+ None => (None, rest),+ };+ rows.push(Row {+ indent,+ index,+ identity: identity_of(indent, after),+ key: after.to_owned(),+ text: rest.to_owned(),+ });+ } else if let Some(last) = rows.last_mut() {+ if !line.trim().is_empty() {+ // Continuation of a multi-line row (an action name with a+ // newline in it).+ last.key.push('\n');+ last.key.push_str(line);+ last.text.push('\n');+ last.text.push_str(line);+ }+ }+ }+ rows+}++fn identity_of(indent: usize, after_index: &str) -> String {+ let role = after_index.split_whitespace().next().unwrap_or("");+ let id = after_index.find("id=").map(|p| {+ let tail = &after_index[p + 3..];+ let end = tail+ .find(|c: char| c.is_whitespace() || c == ']')+ .unwrap_or(tail.len());+ &tail[..end]+ });+ // A quoted string right after the role is the title; after `=` it is the+ // value, which changes and so cannot identify the row.+ let label = after_index+ .split_once(char::is_whitespace)+ .map(|(_, tail)| tail.trim_start())+ .and_then(|tail| tail.strip_prefix('"'))+ .and_then(|tail| tail.split('"').next());+ format!("{indent}|{role}|{}", id.or(label).unwrap_or(""))+}++fn diff_markdown(old_md: &str, new_md: &str) -> Result<DiffReport, DiffError> {+ let old = parse_rows(old_md);+ let new = parse_rows(new_md);+ let ops = diff_ops(&old, &new)?;++ let mut report = DiffReport::default();+ let mut dels: Vec<usize> = Vec::new();+ let mut inss: Vec<usize> = Vec::new();+ // Unchanged rows whose index moved, grouped into runs of equal delta.+ let mut run: Option<(u64, u64, i64)> = None;++ let flush_hunk = |dels: &mut Vec<usize>, inss: &mut Vec<usize>, report: &mut DiffReport| {+ let mut used = vec![false; dels.len()];+ for &i in inss.iter() {+ let pair = dels+ .iter()+ .enumerate()+ .find(|(d, &o)| !used[*d] && old[o].identity == new[i].identity);+ match pair {+ Some((d, _)) => {+ used[d] = true;+ report.changed.push(new[i].clone());+ }+ None => report.added.push(new[i].clone()),+ }+ }+ for (d, &o) in dels.iter().enumerate() {+ if !used[d] {+ report.removed.push(old[o].clone());+ }+ }+ dels.clear();+ inss.clear();+ };++ for op in ops {+ match op {+ Op::Del(o) => dels.push(o),+ Op::Ins(n) => inss.push(n),+ Op::Equal(o, n) => {+ flush_hunk(&mut dels, &mut inss, &mut report);+ if let (Some(oi), Some(ni)) = (old[o].index, new[n].index) {+ let delta = ni as i64 - oi as i64;+ match (&mut run, delta) {+ (Some((_, last_old, d)), delta2) if *d == delta2 && delta2 != 0 => {+ *last_old = oi;+ }+ _ => {+ if let Some(done) = run.take() {+ report.reindexed.push(done);+ }+ if delta != 0 {+ run = Some((oi, oi, delta));+ }+ }+ }+ }+ }+ }+ }+ flush_hunk(&mut dels, &mut inss, &mut report);+ if let Some(done) = run.take() {+ report.reindexed.push(done);+ }+ Ok(report)+}++enum Op {+ Equal(usize, usize),+ Del(usize),+ Ins(usize),+}++fn diff_ops(old: &[Row], new: &[Row]) -> Result<Vec<Op>, DiffError> {+ let mut prefix = 0;+ while prefix < old.len()+ && prefix < new.len()+ && old[prefix].key == new[prefix].key+ && old[prefix].indent == new[prefix].indent+ {+ prefix += 1;+ }+ let mut suffix = 0;+ while suffix < old.len() - prefix+ && suffix < new.len() - prefix+ && old[old.len() - 1 - suffix].key == new[new.len() - 1 - suffix].key+ && old[old.len() - 1 - suffix].indent == new[new.len() - 1 - suffix].indent+ {+ suffix += 1;+ }+ let a = &old[prefix..old.len() - suffix];+ let b = &new[prefix..new.len() - suffix];+ let (n, m) = (a.len(), b.len());+ if n.saturating_mul(m) > DIFF_CELL_LIMIT {+ return Err(DiffError::TooLarge);+ }+ let same = |x: &Row, y: &Row| x.key == y.key && x.indent == y.indent;++ let mut ops: Vec<Op> = (0..prefix).map(|i| Op::Equal(i, i)).collect();+ // dp[i][j] = LCS length of a[i..] and b[j..].+ let width = m + 1;+ let mut dp = vec![0u32; (n + 1) * width];+ for i in (0..n).rev() {+ for j in (0..m).rev() {+ dp[i * width + j] = if same(&a[i], &b[j]) {+ dp[(i + 1) * width + j + 1] + 1+ } else {+ dp[(i + 1) * width + j].max(dp[i * width + j + 1])+ };+ }+ }+ let (mut i, mut j) = (0, 0);+ while i < n && j < m {+ if same(&a[i], &b[j]) {+ ops.push(Op::Equal(prefix + i, prefix + j));+ i += 1;+ j += 1;+ } else if dp[(i + 1) * width + j] >= dp[i * width + j + 1] {+ ops.push(Op::Del(prefix + i));+ i += 1;+ } else {+ ops.push(Op::Ins(prefix + j));+ j += 1;+ }+ }+ while i < n {+ ops.push(Op::Del(prefix + i));+ i += 1;+ }+ while j < m {+ ops.push(Op::Ins(prefix + j));+ j += 1;+ }+ for k in 0..suffix {+ ops.push(Op::Equal(old.len() - suffix + k, new.len() - suffix + k));+ }+ Ok(ops)+}++impl DiffReport {+ pub fn is_empty(&self) -> bool {+ self.added.is_empty() && self.changed.is_empty() && self.removed.is_empty()+ }++ fn render_len(&self) -> usize {+ self.added+ .iter()+ .chain(&self.changed)+ .chain(&self.removed)+ .map(|r| r.text.len() + 3)+ .sum()+ }++ fn render(&self, since: &str, snapshot_id: Option<&str>, focus: Option<&str>) -> String {+ let new_id = snapshot_id.unwrap_or("the new snapshot");+ if self.is_empty() {+ let mut out = format!("no change since {since}");+ if let Some(focus) = focus {+ out.push_str(&format!("; focused element is {focus}"));+ }+ out.push_str(&format!(+ ". Rows and indices are as before; use snapshot_id {new_id} in element_tokens ({new_id}:N)."+ ));+ return out;+ }+ let mut out = format!(+ "since {since}: {} added, {} changed, {} removed. Use element_tokens {new_id}:N for + and ~ rows and unchanged rows; - ids are from {since}.\n",+ self.added.len(),+ self.changed.len(),+ self.removed.len()+ );+ for row in &self.added {+ out.push_str(&format!("+ {}\n", row.text));+ }+ for row in &self.changed {+ out.push_str(&format!("~ {}\n", row.text));+ }+ for row in &self.removed {+ out.push_str(&format!("- {}\n", row.text));+ }+ const MAX_RUNS: usize = 20;+ for (first_old, last_old, delta) in self.reindexed.iter().take(MAX_RUNS) {+ out.push_str(&format!(+ "reindexed: [{first_old}-{last_old}] -> [{}-{}]\n",+ *first_old as i64 + delta,+ *last_old as i64 + delta+ ));+ }+ if self.reindexed.len() > MAX_RUNS {+ out.push_str("reindexed: more shifted ranges not listed; pass full_output or omit since for a fresh index.\n");+ }+ out.truncate(out.trim_end().len());+ out+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn opts(args: Value) -> ViewOptions {+ ViewOptions::from_args(&args).unwrap()+ }++ const OLD: &str = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Cancel\" [id=cancel actions=[press]]";++ #[test]+ fn identical_trees_report_no_change() {+ let report = diff_markdown(OLD, OLD).unwrap();+ assert!(report.is_empty());+ let text = report.render(+ "s00000001",+ Some("s00000002"),+ Some("[1] AXTextField \"Name\""),+ );+ assert!(text.starts_with("no change since s00000001; focused element is [1]"));+ assert!(text.contains("s00000002:N"));+ }++ #[test]+ fn value_edit_is_a_change_and_removal_and_addition_are_reported() {+ let new = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"ab\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Help\" [id=help actions=[press]]";+ let report = diff_markdown(OLD, new).unwrap();+ assert_eq!(report.changed.len(), 1);+ assert_eq!(report.changed[0].index, Some(1));+ assert_eq!(report.added.len(), 1);+ assert_eq!(report.removed.len(), 1);+ assert_eq!(report.removed[0].index, Some(3));+ let text = report.render("sA", Some("sB"), None);+ assert!(text.contains("~ [1] AXTextField \"Name\" = \"ab\""));+ assert!(text.contains("+ [3] AXButton \"Help\""));+ assert!(text.contains("- [3] AXButton \"Cancel\""));+ }++ #[test]+ fn inserted_row_reports_reindexed_range() {+ let new = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXCheckBox \"New\" [id=new]\n - [3] AXButton \"Save\" [id=save actions=[press]]\n - [4] AXButton \"Cancel\" [id=cancel actions=[press]]";+ let report = diff_markdown(OLD, new).unwrap();+ assert_eq!(report.added.len(), 1);+ assert!(report.removed.is_empty());+ assert_eq!(report.reindexed, vec![(2, 3, 1)]);+ assert!(report+ .render("sA", Some("sB"), None)+ .contains("reindexed: [2-3] -> [3-4]"));+ }++ #[test]+ fn display_only_text_changes_are_seen() {+ let new = OLD.replace("hello", "goodbye");+ let report = diff_markdown(OLD, &new).unwrap();+ assert_eq!(report.changed.len(), 1);+ assert_eq!(report.changed[0].index, None);+ }++ #[test]+ fn multiline_action_rows_stay_one_row() {+ let md = "- [0] AXWindow\n - [1] AXButton \"x\" [actions=[press,name:move\ntarget:0x0\nselector:(null)]]\n - [2] AXButton \"y\"";+ assert_eq!(parse_rows(md).len(), 3);+ assert!(diff_markdown(md, md).unwrap().is_empty());+ }++ #[test]+ fn options_parse_and_defaults() {+ let o = opts(json!({}));+ assert_eq!(o.tree_format, TreeFormat::Markdown);+ assert!(!o.verbose && !o.full_output && o.since.is_none());+ assert_eq!(+ opts(json!({"full_output": true})).tree_format,+ TreeFormat::Both+ );+ assert_eq!(+ opts(json!({"tree_format": "elements"})).tree_format,+ TreeFormat::Elements+ );+ assert!(ViewOptions::from_args(&json!({"tree_format": "xml"})).is_err());+ assert!(ViewOptions::from_args(&json!({"since": 3})).is_err());+ // full_output is a full read, never a diff.+ assert!(opts(json!({"full_output": true, "since": "s00000001"}))+ .since+ .is_none());+ }++ #[test]+ fn max_elements_precedence() {+ let o = opts(json!({}));+ assert_eq!(+ o.max_elements(&json!({}), 2000, false),+ DEFAULT_MAX_ELEMENTS+ );+ assert_eq!(+ o.max_elements(&json!({"max_elements": 900}), 2000, false),+ 900+ );+ assert_eq!(o.max_elements(&json!({}), 2000, true), 2000);+ let full = opts(json!({"full_output": true}));+ assert_eq!(full.max_elements(&json!({}), 5000, false), 5000);+ }++ fn payload(sid: &str, md: &str) -> (Vec<Content>, Value) {+ let content = vec![+ Content::image_png("AAAA".into()),+ Content::text(format!("window_id=7 pid=9 size=10x10 elements=4\n\n{md}")),+ ];+ let structured = json!({+ "window_id": 7, "pid": 9, "snapshot_id": sid,+ "tree_markdown": md,+ "elements": [{"element_index": 1}, {"element_index": 2}],+ "_note": "n", "background_input": {"routes": []},+ "element_count": 4+ });+ (content, structured)+ }++ fn ctx(window_id: u64) -> ViewContext<'static> {+ ViewContext {+ pid: 9,+ window_id,+ key: ViewKey {+ query: None,+ max_elements: 250,+ max_depth: None,+ },+ focus_probe: None,+ }+ }++ fn text_of(content: &[Content]) -> String {+ content+ .iter()+ .find_map(|c| match c {+ Content::Text { text, .. } => Some(text.clone()),+ _ => None,+ })+ .unwrap()+ }++ #[test]+ fn default_read_is_markdown_only_without_noise() {+ let (mut content, mut s) = payload("s000000a1", OLD);+ apply(&opts(json!({})), &ctx(7001), &mut content, &mut s);+ assert!(s.get("elements").is_none());+ assert!(s.get("_note").is_none());+ assert!(s.get("background_input").is_none());+ assert_eq!(s["tree_markdown"], OLD);+ assert_eq!(s["tree_format"], "markdown");+ let text = text_of(&content);+ assert!(text.starts_with("window_id=7 pid=9 size=10x10 elements=4 snapshot_id=s000000a1\n"));+ assert!(text.contains("<snapshot_id>:N"));+ assert!(text.ends_with(OLD));+ }++ #[test]+ fn full_output_is_untouched() {+ let (mut content, mut s) = payload("s000000a2", OLD);+ let before = s.clone();+ let text_before = text_of(&content);+ apply(+ &opts(json!({"full_output": true})),+ &ctx(7002),+ &mut content,+ &mut s,+ );+ assert_eq!(s, before);+ assert_eq!(text_of(&content), text_before);+ assert_eq!(s["elements"], before["elements"]);+ assert_eq!(s["_note"], before["_note"]);+ assert_eq!(s["background_input"], before["background_input"]);+ assert_eq!(s["tree_markdown"], OLD);+ }++ #[test]+ fn elements_format_drops_markdown() {+ let (mut content, mut s) = payload("s000000a3", OLD);+ apply(+ &opts(json!({"tree_format": "elements"})),+ &ctx(7003),+ &mut content,+ &mut s,+ );+ assert!(s.get("tree_markdown").is_none());+ assert_eq!(s["elements"].as_array().unwrap().len(), 2);+ assert!(!text_of(&content).contains("AXButton"));+ }++ #[test]+ fn since_returns_diff_then_falls_back() {+ let window = 7004;+ let (mut c1, mut s1) = payload("s000000b1", OLD);+ apply(&opts(json!({})), &ctx(window), &mut c1, &mut s1);++ // Unchanged tree.+ let (mut c2, mut s2) = payload("s000000b2", OLD);+ apply(+ &opts(json!({"since": "s000000b1"})),+ &ctx(window),+ &mut c2,+ &mut s2,+ );+ assert_eq!(s2["since_status"], "no_change");+ assert!(s2.get("tree_markdown").is_none());+ assert!(text_of(&c2).contains("no change since s000000b1"));++ // One edit, chained against the previous snapshot.+ let edited = OLD.replace("\"a\"", "\"ab\"");+ let (mut c3, mut s3) = payload("s000000b3", &edited);+ apply(+ &opts(json!({"since": "s000000b2"})),+ &ctx(window),+ &mut c3,+ &mut s3,+ );+ assert_eq!(s3["since_status"], "diff");+ assert_eq!(s3["diff_counts"]["changed"], 1);+ assert!(text_of(&c3).contains("~ [1] AXTextField"));+ assert!(!text_of(&c3).contains("AXButton \"Save\""));+ assert!(s3.get("elements").is_none());++ // Unknown snapshot falls back to the full read.+ let (mut c4, mut s4) = payload("s000000b4", &edited);+ apply(+ &opts(json!({"since": "s0000dead"})),+ &ctx(window),+ &mut c4,+ &mut s4,+ );+ assert_eq!(s4["since_status"], "unknown_snapshot");+ assert_eq!(s4["tree_markdown"], edited);+ assert!(text_of(&c4).contains("unknown or expired snapshot_id; full read follows"));++ // Another window's snapshot is not a baseline.+ let (mut c5, mut s5) = payload("s000000b5", &edited);+ apply(+ &opts(json!({"since": "s000000b4"})),+ &ctx(window + 1),+ &mut c5,+ &mut s5,+ );+ assert_eq!(s5["since_status"], "other_window");+ }++ #[test]+ fn a_rewritten_tree_falls_back_to_full() {+ let window = 7005;+ let (mut c1, mut s1) = payload("s000000c1", OLD);+ apply(&opts(json!({})), &ctx(window), &mut c1, &mut s1);+ let other = "- [0] AXWindow \"Other\"\n - [1] AXList \"x\"\n - [2] AXButton \"Q\"\n - [3] AXButton \"R\"\n - [4] AXButton \"S\"";+ let (mut c2, mut s2) = payload("s000000c2", other);+ apply(+ &opts(json!({"since": "s000000c1"})),+ &ctx(window),+ &mut c2,+ &mut s2,+ );+ assert_eq!(s2["since_status"], "too_much_changed");+ assert_eq!(s2["tree_markdown"], other);+ }++ #[test]+ fn node_budget_truncation_is_stated() {+ let (mut content, mut s) = payload("s000000d1", OLD);+ s["truncated"] = json!(true);+ s["truncation_reason"] = json!("node_budget");+ s["nodes_pending"] = json!(40);+ apply(&opts(json!({})), &ctx(7006), &mut content, &mut s);+ let text = text_of(&content);+ assert!(text.contains("Tree truncated at max_elements=250 (40 more node(s) not shown)"));+ assert!(s["truncation_hint"]+ .as_str()+ .unwrap()+ .contains("larger max_elements"));+ }++ #[test]+ fn degraded_snapshots_keep_background_input() {+ let (mut content, mut s) = payload("s000000e1", OLD);+ s["degraded"] = json!(true);+ apply(&opts(json!({})), &ctx(7007), &mut content, &mut s);+ assert!(s.get("background_input").is_some());+ let (mut content, mut s) = payload("s000000e2", OLD);+ apply(+ &opts(json!({"verbose": true})),+ &ctx(7008),+ &mut content,+ &mut s,+ );+ assert!(s.get("background_input").is_some());+ assert!(s.get("_note").is_some());+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rsindex ccadcd58a0..77925cc654 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs@@ -861,6 +861,10 @@ fn sdk_window_input(fixture: &Fixture) -> GetWindowStateInput {max_depth: None,max_dimension: None,max_image_dimension: None,+ tree_format: None,+ since: None,+ verbose: None,+ full_output: None,}}diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rsindex 1aee34b717..11e7205cfc 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs@@ -755,8 +755,15 @@ macro_rules! define_native_window_methods {pub async fn get_window_state(&self,- input: GetWindowStateInput,+ mut input: GetWindowStateInput,) -> Result<WindowStateOutput, DriverError> {+ // The typed snapshot is read by programs, not budgeted+ // against a model context: keep the full response (both tree+ // representations, platform walk limits) unless the caller+ // picks a leaner shape.+ if input.tree_format.is_none() && input.full_output.is_none() {+ input.full_output = Some(true);+ }self.invoke_typed(GetWindowStateInput::TOOL_NAME, input).await?.window_state_success()diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rsindex c7c2eeb045..7e2f7878ad 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/tests/native_windows.rs@@ -285,7 +285,26 @@ async fn typed_snapshot_request_preserves_target_and_session() {assert_eq!(request["name"], "get_window_state");assert_eq!(request["args"],- json!({"pid":42,"window_id":73,"session":"run-1","include_screenshot":true,"max_elements":10})+ // The typed SDK keeps the full response unless the caller picks a+ // leaner shape.+ json!({"pid":42,"window_id":73,"session":"run-1","include_screenshot":true,"max_elements":10,"full_output":true})+ );+}++#[tokio::test]+#[cfg(unix)]+async fn typed_snapshot_request_forwards_an_explicit_tree_shape() {+ let (_directory, socket, server) = serve_once(json!({"ok":true,"result":snapshot_envelope()}));+ let driver = CuaDriver::connect(Some(socket)).unwrap();+ let input: GetWindowStateInput = serde_json::from_value(json!({+ "pid":42,"window_id":73,"tree_format":"markdown","since":"s0000002a"+ }))+ .unwrap();+ driver.get_window_state(input).await.unwrap();+ let request = server.join().unwrap();+ assert_eq!(+ request["args"],+ json!({"pid":42,"window_id":73,"tree_format":"markdown","since":"s0000002a"}));}diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rsindex 8b03458596..ef3fa70c06 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs@@ -111,6 +111,7 @@ impl Driver for CliDriver {}};+ let args = crate::driver::with_full_window_state(tool, args);if let Some(mut stdin) = child.stdin.take() {let _ = writeln!(stdin, "{}", serde_json::to_string(&args).unwrap());}diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rsindex f0c21717e9..1848db36e0 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs@@ -18,3 +18,21 @@ pub trait Driver {pub trait BehaviorRecording {fn start_behavior_recording(&mut self);}++/// Harness scenarios read both `elements` and `tree_markdown` from+/// `get_window_state`. The tool's model-facing default is now one compact+/// markdown tree, so request the full response unless the scenario chose a+/// shape itself (`tree_format`, `full_output`, or `since`).+pub fn with_full_window_state(tool: &str, mut args: Value) -> Value {+ if tool == "get_window_state" {+ if let Some(map) = args.as_object_mut() {+ if !map.contains_key("tree_format")+ && !map.contains_key("full_output")+ && !map.contains_key("since")+ {+ map.insert("full_output".into(), Value::Bool(true));+ }+ }+ }+ args+}diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rsindex e1fbcb4994..f01eda3527 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs@@ -487,6 +487,7 @@ impl McpDriver {let id = self.next_id;self.next_id += 1;let timeout = call_timeout(self.recording_started, tool, cfg!(target_os = "macos"));+ let args = crate::driver::with_full_window_state(tool, args);self.send(serde_json::json!({"jsonrpc": "2.0", "id": id, "method": "tools/call","params": { "name": tool, "arguments": args }diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rsindex 2bf67c3539..4c2faff25e 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs@@ -4608,6 +4608,8 @@ const CLI_EXAMPLES: &[(&str, &[(&str, &str)])] = &[("cua-driver call list_apps", "Call a tool with no arguments"),("cua-driver call click '{\"pid\":844,\"x\":100,\"y\":200}'", "Click at window coordinates"),("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725}' --screenshot-out-file state.png", "Save the screenshot from a tool response"),+ ("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"since\":\"s0000002a\"}'", "Read only what changed since an earlier snapshot_id"),+ ("cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"tree_format\":\"elements\"}'", "Return the structured elements instead of the markdown tree"),]),("serve", &[("cua-driver serve", "Run the daemon in the foreground"),diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex ca0dec422a..2f971479b1 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -1005,6 +1005,9 @@ pub struct GetWindowStateTool {const COLD_START_WALK_TIMEOUT_MS: u64 = 2_000;+/// The AT-SPI walker's own node budget when the caller gives none.+const LINUX_DEFAULT_MAX_ELEMENTS: usize = 5_000;+fn linux_snapshot_timeout_ms(timeout: Option<&Value>, has_prior_snapshot: bool) -> u64 {cua_driver_core::tool_schema::resolve_timeout_ms_with_first_snapshot_grace(timeout,@@ -1020,24 +1023,31 @@ impl Tool for GetWindowStateTool {fn def(&self) -> &ToolDef {GWS_DEF.get_or_init(|| ToolDef {name: "get_window_state".into(),- description: "Walk a running app's AT-SPI tree and return BOTH a \- structured `elements` array (preferred) AND a Markdown rendering of \- the same tree (back-compat). Every actionable element is tagged \- with [element_index N] in the markdown and as `element_index` in \- the structured array; pass each element's `element_token` to \- `click`, `type_text`, `set_value`, etc.\n\n\- PREFERRED CONSUMERS read `structuredContent.elements` (one entry \- per indexed row with `element_index`, `role`, `label`, `value`, \- `enabled`, `selected`, `actions` (names of AT-SPI actions exposed \- by the element, omitted when empty), \- `frame: {x,y,w,h}` when AT-SPI reports usable bounds, \- `parent_index`, `depth`). The markdown `tree_markdown` stays \- available and unchanged in shape for existing text-parsing \- callers — but new fields will only be added to the structured \- side. Set `query` to project BOTH representations to matching \- rows plus their ancestor chain while preserving original indices. \- `total_element_count` reports the complete snapshot and \- `returned_element_count` reports the projection.\n\n\+ description: "Walk a running app's AT-SPI tree and return it ONCE, as \+ compact Markdown by default (`tree_format:\"markdown\"`). Every actionable \+ element is tagged `[N]` (an element_index); its element_token is \+ `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and \+ returned in structuredContent. Pass `tree_format:\"elements\"` for the \+ structured `elements` array instead (explicit `element_token`, `role`, \+ `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when \+ AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for \+ both (about twice the size). Set `query` to project the tree to matching rows \+ plus their ancestor chain while preserving original indices; \+ `total_element_count` reports the complete snapshot.\n\n\+ Size: a read is bounded by default (`max_elements` 250 actionable nodes). \+ When the walk stops at that budget the response says `Tree truncated at \+ max_elements=…` and `truncated: true`; pass a larger `max_elements`, or \+ `query` / `max_depth`, to reach the rest. `_note` is omitted unless \+ `verbose:true`. `full_output:true` restores the previous full response: both \+ representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\n\+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \+ back to a full read and `since_status` says why. The focused element is not \+ reported on Linux yet.\n\n\Always returns BOTH the element tree AND a screenshot — ground on \both and cross-check (the tree lies on some surfaces). Choose the \modality at ACTION time: an element ax action \@@ -1055,11 +1065,9 @@ impl Tool for GetWindowStateTool {`include_screenshot:false` is an error. Optional `max_dimension` \caps the returned screenshot's long edge in pixels for a cheap \thumbnail.\n\n\- Optional `max_elements` / `max_depth` bound the AT-SPI walk to \- mitigate context-window blow-up on Electron / large web apps \- that produce 10k+ element trees. When applied, BOTH \- the markdown and the structured elements are truncated \- identically. Omit both for current default behaviour.\n\n\+ Optional `max_elements` / `max_depth` bound the AT-SPI walk \+ (Electron / large web apps produce 10k+ element trees). Markdown and \+ structured elements are truncated identically. Default: `max_elements` 250.\n\n\TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI \walk. When omitted, a window's first snapshot gets a 2000 ms \cold-start budget so Chromium/Electron can finish publishing its \@@ -1085,24 +1093,24 @@ impl Tool for GetWindowStateTool {popup's own AT-SPI toplevel so its menu items get element indices \(then click them by element_token). Omitting window_id while a popup \of this pid is open walks that popup.".into(),- input_schema: json!({"type":"object","required":["pid"],"properties":{+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer","description":"Process ID that owns the window."},"window_id":{"type":"integer","description":"Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window."},"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),"include_accessibility_tree":{"type":"boolean",- "description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},+ "description":"Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},- "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."},+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted."},"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior."}- },"additionalProperties":false}),+ },"additionalProperties":false})),read_only: true, destructive: false, idempotent: false, open_world: false,})}@@ -1184,6 +1192,10 @@ impl Tool for GetWindowStateTool {// time: an element ax action (element_token) or element px action (x,y).// We don't even read the arg; it stays in the schema only so old callers// don't trip additionalProperties:false.+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {+ Ok(view) => view,+ Err(refusal) => return refusal,+ };let query = args.opt_str("query");let session_id = args.opt_str("_session_id");// include_screenshot (default true) — the perf opt-out. The tree+screenshot@@ -1210,10 +1222,15 @@ impl Tool for GetWindowStateTool {});// Optional caps — when omitted, the AT-SPI walker uses its built-in// defaults (#22865).- let max_elements = args- .get("max_elements")- .and_then(|v| v.as_u64())- .map(|v| v.max(1) as usize);+ let max_elements = Some(+ view.max_elements(+ &args,+ LINUX_DEFAULT_MAX_ELEMENTS,+ args.get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true),+ ),+ );let max_depth = args.get("max_depth").and_then(|v| v.as_u64())@@ -1752,6 +1769,24 @@ impl Tool for GetWindowStateTool {.with_structured(structured);}+ if !observation_only {+ cua_driver_core::window_state_view::apply(+ &view,+ &cua_driver_core::window_state_view::ViewContext {+ pid: i64::from(pid),+ window_id: xid,+ key: cua_driver_core::window_state_view::ViewKey {+ query: query.clone(),+ max_elements: max_elements.unwrap_or(LINUX_DEFAULT_MAX_ELEMENTS),+ max_depth,+ },+ focus_probe: None,+ },+ &mut content,+ &mut structured,+ );+ }+ToolResult {content,is_error: None,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex 2d1f2e4687..e69b4d46ad 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -27,14 +27,16 @@ const AX_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_sefn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "get_window_state".into(),- description: "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\+ description: "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\\n\- START NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\+ START NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\+ \n\+ DIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\\n\A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\\n\Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),- input_schema: serde_json::json!({+ input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({"type": "object","required": ["pid", "window_id"],"properties": {@@ -58,7 +60,7 @@ fn def() -> &'static ToolDef {"max_elements": {"type": "integer","minimum": 1,- "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000."+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut."},"max_depth": {"type": "integer",@@ -78,7 +80,7 @@ fn def() -> &'static ToolDef {}},"additionalProperties": false- }),+ })),read_only: true,destructive: false,idempotent: false,@@ -157,6 +159,10 @@ impl Tool for GetWindowStateTool {}}+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {+ Ok(view) => view,+ Err(refusal) => return refusal,+ };let query = args.opt_str("query");let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {// Expand ~ prefix.@@ -228,11 +234,11 @@ impl Tool for GetWindowStateTool {// the AX walker (#22865). minimum:1 keyed in the schema, but defend// against 0 here as well so a misbehaving client can't disable the// walk entirely.- let max_elements = args- .get("max_elements")- .and_then(|v| v.as_u64())- .map(|v| v.max(1) as usize)- .unwrap_or(crate::ax::tree::DEFAULT_MAX_ELEMENTS);+ let max_elements = view.max_elements(+ &args,+ crate::ax::tree::DEFAULT_MAX_ELEMENTS,+ observation_only,+ );let max_depth = args.get("max_depth").and_then(|v| v.as_u64())@@ -720,6 +726,27 @@ impl Tool for GetWindowStateTool {cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::MayBeIncomplete,),);+ if !observation_only {+ let focus_probe = || focused_element_description(pid, tree_result.as_ref());+ cua_driver_core::window_state_view::apply(+ &view,+ &cua_driver_core::window_state_view::ViewContext {+ pid: i64::from(pid),+ window_id: u64::from(window_id),+ key: cua_driver_core::window_state_view::ViewKey {+ query: query.clone(),+ max_elements,+ max_depth: args+ .get("max_depth")+ .and_then(|v| v.as_u64())+ .map(|v| v as usize),+ },+ focus_probe: Some(&focus_probe),+ },+ &mut content,+ &mut structured,+ );+ }ToolResult {content,is_error: None,@@ -729,6 +756,49 @@ impl Tool for GetWindowStateTool {}}+/// Describe the app's focused UI element as `[N] Role "title"` when it is one+/// of the indexed rows of this walk. Called only for a `since` read that found+/// no change.+fn focused_element_description(+ pid: i32,+ tree: Option<&crate::ax::tree::TreeWalkResult>,+) -> Option<String> {+ use crate::ax::bindings::{copy_element_attr, AXUIElementCreateApplication};+ use core_foundation::base::{CFEqual, CFRelease, CFTypeRef};+ let tree = tree?;+ unsafe {+ let app = AXUIElementCreateApplication(pid);+ if app.is_null() {+ return None;+ }+ let focused = copy_element_attr(app, "AXFocusedUIElement");+ CFRelease(app as CFTypeRef);+ let focused = focused?;+ let found = tree.nodes.iter().find(|node| {+ node.element_index.is_some()+ && node.element_ptr != 0+ && CFEqual(node.element_ptr as CFTypeRef, focused as CFTypeRef) != 0+ });+ let described = found.map(|node| {+ let label = node+ .title+ .as_deref()+ .or(node.description.as_deref())+ .or(node.identifier.as_deref());+ match label {+ Some(label) => format!(+ "[{}] {} \"{label}\"",+ node.element_index.unwrap_or_default(),+ node.role+ ),+ None => format!("[{}] {}", node.element_index.unwrap_or_default(), node.role),+ }+ });+ CFRelease(focused as CFTypeRef);+ described+ }+}+/// Turn an unresolvable window scope into a structured refusal, or `None` when/// the scope is one the caller can still be served (issue #2237).///@@ -1150,6 +1220,31 @@ mod window_scope_contract_tests {"description must document the both-false error");}++ #[test]+ fn schema_advertises_single_tree_and_diff_controls() {+ let d = def();+ let props = &d.input_schema["properties"];+ assert_eq!(+ props["tree_format"]["enum"],+ serde_json::json!(["markdown", "elements", "both"])+ );+ for name in ["since", "verbose", "full_output"] {+ assert!(props.get(name).is_some(), "schema must advertise {name}");+ }+ assert_eq!(d.input_schema["additionalProperties"], false);+ for needle in [+ "at most 250 nodes",+ "since:",+ "full_output:true",+ "tree_format",+ ] {+ assert!(+ d.description.contains(needle),+ "description must mention {needle}"+ );+ }+ }}#[cfg(test)]diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 5bb612937c..b5152da090 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -1164,30 +1164,39 @@ impl Tool for GetWindowStateTool {// Windows-specific adaptations (UIA instead of AX; no SkyLight// Space validation; no per-call `javascript` field — that's a// macOS AppleScript hook).- description: "Walk a running app's UIA tree and return BOTH a structured \- `elements` array (preferred) AND a Markdown rendering of the same tree \- (back-compat). Every actionable element is tagged with [element_index N] \- in the markdown and as `element_index` in the structured array; pass each \- element's `element_token` to `click`, `type_text`, `scroll`, etc.\n\n\+ description: "Walk a running app's UIA tree and return it ONCE, as compact \+ Markdown by default (`tree_format:\"markdown\"`). Every actionable element is \+ tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where \+ `snapshot_id` is printed in the response header and returned in structuredContent. \+ Pass `tree_format:\"elements\"` for the structured `elements` array instead \+ (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), \+ or `\"both\"` for both (about twice the size).\n\n\+ Size: a read is bounded by default (`max_elements` 250). When the walk stops at \+ that budget the response says `Tree truncated at max_elements=…` and `truncated: \+ true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. \+ `_note` is omitted unless `verbose:true`. `full_output:true` restores the \+ previous full response: both representations, all metadata, and the platform's \+ walk limits (≤5 000 elements, depth ≤25).\n\n\+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \+ back to a full read and `since_status` says why. The focused element is not \+ reported on Windows yet.\n\n\INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \any element action against that window. The next snapshot of the same \(pid, window_id) replaces this one, stales its element tokens, and lists the \replaced ids in `invalidated_snapshot_ids`.\n\n\- PREFERRED CONSUMERS read `structuredContent.elements` (one entry per \- indexed row with `element_index`, `role`, `label`, `value`, `enabled`, \- `selected`, `actions` (names of UIA patterns exposed as actions, \- omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \- `tree_markdown` stays available \- and unchanged in shape for existing text-parsing callers — but new \- fields will only be added to the structured side.\n\n\The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \window bounds reported come from the same `window_id`. This is the source of \truth for which window the caller intends to reason about — the driver never \picks a window implicitly.\n\n\`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. \The driver does not auto-fall-back to a different window.\n\n\- Set `query` to a case-insensitive substring to project BOTH `tree_markdown` \- and `structuredContent.elements` to matching rows plus their ancestor chain. \+ Set `query` to a case-insensitive substring to project the tree to matching \+ rows plus their ancestor chain. \Original element indices are preserved. `total_element_count` reports the \complete snapshot; `returned_element_count` reports the projection.\n\n\Always returns BOTH the element tree AND a screenshot — ground on both \@@ -1206,11 +1215,9 @@ impl Tool for GetWindowStateTool {Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a \single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of \timing out at 4s with per-property RPCs).\n\n\- Optional `max_elements` / `max_depth` bound the UIA walk to mitigate \- context-window blow-up on Electron / large web apps that produce 10k+ \- element trees. When applied, BOTH the markdown and the structured \- elements are truncated identically. Omit both for current default behaviour \- (≤5 000 elements, depth ≤25).\n\n\+ Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \+ apps produce 10k+ element trees). Markdown and structured elements are \+ truncated identically. Default: `max_elements` 250, depth ≤25.\n\n\CHROMIUM COVERAGE: a browser-owned permission bubble can be \composited outside the requested native window. Chromium-family \snapshots therefore describe this limit in structuredContent.capture_coverage. \@@ -1220,21 +1227,21 @@ impl Tool for GetWindowStateTool {This is separate from page JavaScript dialogs, which remain on \browser_dialog.\n\n\Windows requires no special permissions.".into(),- input_schema: json!({"type":"object","required":["pid","window_id"],"properties":{+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid","window_id"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer","description":"Process ID from `list_apps`."},"window_id":{"type":"integer","description":"HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array."},"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),- "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},+ "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},- "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."},+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default."},"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins."}- },"additionalProperties":false}),+ },"additionalProperties":false})),// Swift annotation: idempotent: false (each call is a fresh snapshot).read_only: true, destructive: false, idempotent: false, open_world: false,})@@ -1333,17 +1340,23 @@ impl Tool for GetWindowStateTool {// time: an element ax action (element_index) or an element px action (x,y).// We don't read the arg; it stays in the schema only so old callers don't// trip additionalProperties:false.+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {+ Ok(view) => view,+ Err(refusal) => return refusal,+ };let query = args.opt_str("query");let session_id = args.opt_str("_session_id");let screenshot_out_file = args.opt_str("screenshot_out_file");// Optional caps — when omitted, fall back to the walker's built-in// defaults (#22865). minimum:1 enforced in the schema, but defend// against 0 here too.- let max_elements = args- .get("max_elements")- .and_then(|v| v.as_u64())- .map(|v| v.max(1) as usize)- .unwrap_or(crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS);+ let max_elements = view.max_elements(+ &args,+ crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,+ args.get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true),+ );let max_depth = args.get("max_depth").and_then(|v| v.as_u64())@@ -1748,6 +1761,27 @@ impl Tool for GetWindowStateTool {.with_structured(structured);}+ if !observation_only {+ cua_driver_core::window_state_view::apply(+ &view,+ &cua_driver_core::window_state_view::ViewContext {+ pid: i64::from(pid),+ window_id: hwnd,+ key: cua_driver_core::window_state_view::ViewKey {+ query: query.clone(),+ max_elements,+ max_depth: args+ .get("max_depth")+ .and_then(|v| v.as_u64())+ .map(|v| v as usize),+ },+ focus_probe: None,+ },+ &mut content,+ &mut structured,+ );+ }+ToolResult {content,is_error: None,diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.tsindex c63b6e3051..4a32a7a606 100644--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts@@ -2852,7 +2852,25 @@ export type GetWindowStateInput = {* (default 1000 on every platform). A walk that runs out returns a* partial tree flagged `truncated` rather than failing.*/- timeoutMs?: number+ timeoutMs?: number,+ /**+ * Tree representation to return: `markdown` (default), `elements`, or+ * `both`.+ */+ treeFormat?: string,+ /**+ * Return only what changed since this earlier `snapshot_id`.+ */+ since?: string,+ /**+ * Include `_note` and the full `background_input` report.+ */+ verbose?: boolean,+ /**+ * Restore the previous full response (both representations, all+ * metadata, platform walk limits).+ */+ fullOutput?: boolean}/**@@ -2860,7 +2878,11 @@ export type GetWindowStateInput = {*/export const GetWindowStateInput = (() => {const defaults = () => ({- timeoutMs: undefined+ timeoutMs: undefined,+ treeFormat: undefined,+ since: undefined,+ verbose: undefined,+ fullOutput: undefined});const create = (() => {return uniffiCreateRecord<GetWindowStateInput, ReturnType<typeof defaults>>(defaults);@@ -2888,7 +2910,11 @@ const FfiConverterTypeGetWindowStateInput = (() => {maxDepth: FfiConverterOptionalUInt32.read(from),maxDimension: FfiConverterOptionalUInt32.read(from),maxImageDimension: FfiConverterOptionalUInt32.read(from),- timeoutMs: FfiConverterOptionalUInt32.read(from)+ timeoutMs: FfiConverterOptionalUInt32.read(from),+ treeFormat: FfiConverterOptionalString.read(from),+ since: FfiConverterOptionalString.read(from),+ verbose: FfiConverterOptionalBoolean.read(from),+ fullOutput: FfiConverterOptionalBoolean.read(from)};}write(value: TypeName, into: RustBuffer): void {@@ -2904,6 +2930,10 @@ const FfiConverterTypeGetWindowStateInput = (() => {FfiConverterOptionalUInt32.write(value.maxDimension, into);FfiConverterOptionalUInt32.write(value.maxImageDimension, into);FfiConverterOptionalUInt32.write(value.timeoutMs, into);+ FfiConverterOptionalString.write(value.treeFormat, into);+ FfiConverterOptionalString.write(value.since, into);+ FfiConverterOptionalBoolean.write(value.verbose, into);+ FfiConverterOptionalBoolean.write(value.fullOutput, into);}allocationSize(value: TypeName): number {return FfiConverterUInt32.allocationSize(value.pid) +@@ -2917,7 +2947,11 @@ const FfiConverterTypeGetWindowStateInput = (() => {FfiConverterOptionalUInt32.allocationSize(value.maxDepth) +FfiConverterOptionalUInt32.allocationSize(value.maxDimension) +FfiConverterOptionalUInt32.allocationSize(value.maxImageDimension) +- FfiConverterOptionalUInt32.allocationSize(value.timeoutMs);+ FfiConverterOptionalUInt32.allocationSize(value.timeoutMs) ++ FfiConverterOptionalString.allocationSize(value.treeFormat) ++ FfiConverterOptionalString.allocationSize(value.since) ++ FfiConverterOptionalBoolean.allocationSize(value.verbose) ++ FfiConverterOptionalBoolean.allocationSize(value.fullOutput);}};@@ -6275,6 +6309,29 @@ export type WindowStateOutput = {degradedReason?: string,truncated?: boolean,truncationReason?: string,+ /**+ * Plain-language line saying the tree was cut at `max_elements` and how+ * to read more.+ */+ truncationHint?: string,+ /**+ * Which representation this response carries.+ */+ treeFormat?: string,+ /**+ * The `since` snapshot_id the caller asked to diff against.+ */+ since?: string,+ /**+ * `diff`, `no_change`, or the reason a full read was returned instead+ * (`unknown_snapshot`, `other_window`, `view_changed`, `too_much_changed`,+ * `diff_too_large`, `no_snapshot`).+ */+ sinceStatus?: string,+ /**+ * Added/changed/removed rows against `since`, one per line.+ */+ treeDiff?: string,screenshotWidth?: number,screenshotHeight?: number,screenshotScale?: number,@@ -6325,6 +6382,11 @@ const FfiConverterTypeWindowStateOutput = (() => {degradedReason: FfiConverterOptionalString.read(from),truncated: FfiConverterOptionalBoolean.read(from),truncationReason: FfiConverterOptionalString.read(from),+ truncationHint: FfiConverterOptionalString.read(from),+ treeFormat: FfiConverterOptionalString.read(from),+ since: FfiConverterOptionalString.read(from),+ sinceStatus: FfiConverterOptionalString.read(from),+ treeDiff: FfiConverterOptionalString.read(from),screenshotWidth: FfiConverterOptionalUInt32.read(from),screenshotHeight: FfiConverterOptionalUInt32.read(from),screenshotScale: FfiConverterOptionalFloat64.read(from),@@ -6352,6 +6414,11 @@ const FfiConverterTypeWindowStateOutput = (() => {FfiConverterOptionalString.write(value.degradedReason, into);FfiConverterOptionalBoolean.write(value.truncated, into);FfiConverterOptionalString.write(value.truncationReason, into);+ FfiConverterOptionalString.write(value.truncationHint, into);+ FfiConverterOptionalString.write(value.treeFormat, into);+ FfiConverterOptionalString.write(value.since, into);+ FfiConverterOptionalString.write(value.sinceStatus, into);+ FfiConverterOptionalString.write(value.treeDiff, into);FfiConverterOptionalUInt32.write(value.screenshotWidth, into);FfiConverterOptionalUInt32.write(value.screenshotHeight, into);FfiConverterOptionalFloat64.write(value.screenshotScale, into);@@ -6378,6 +6445,11 @@ const FfiConverterTypeWindowStateOutput = (() => {FfiConverterOptionalString.allocationSize(value.degradedReason) +FfiConverterOptionalBoolean.allocationSize(value.truncated) +FfiConverterOptionalString.allocationSize(value.truncationReason) ++ FfiConverterOptionalString.allocationSize(value.truncationHint) ++ FfiConverterOptionalString.allocationSize(value.treeFormat) ++ FfiConverterOptionalString.allocationSize(value.since) ++ FfiConverterOptionalString.allocationSize(value.sinceStatus) ++ FfiConverterOptionalString.allocationSize(value.treeDiff) +FfiConverterOptionalUInt32.allocationSize(value.screenshotWidth) +FfiConverterOptionalUInt32.allocationSize(value.screenshotHeight) +FfiConverterOptionalFloat64.allocationSize(value.screenshotScale) +diff --git a/libs/cua-driver/typescript/test/native-loader.test.mjs b/libs/cua-driver/typescript/test/native-loader.test.mjsindex 71b315e075..37a0d9f049 100644--- a/libs/cua-driver/typescript/test/native-loader.test.mjs+++ b/libs/cua-driver/typescript/test/native-loader.test.mjs@@ -266,6 +266,7 @@ test(pid: 42, window_id: 123, session: "node-run", query: "Save",include_screenshot: true, include_accessibility_tree: true,max_elements: 10, max_depth: 3, max_dimension: 800, max_image_dimension: 1600,+ full_output: true,})assert.deepEqual(requests[5].args, {target: { kind: "window", pid: 42, window_id: 123 },diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.jsonindex e1bd320e28..53d48a7aa1 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json@@ -43,7 +43,7 @@"read_only": true},{- "description": "Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `set_value`, etc.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.\n\nTIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.\n\nSCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\nPOPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_token). Omitting window_id while a popup of this pid is open walks that popup.",+ "description": "Walk a running app's AT-SPI tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `role`, `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for both (about twice the size). Set `query` to project the tree to matching rows plus their ancestor chain while preserving original indices; `total_element_count` reports the complete snapshot.\n\nSize: a read is bounded by default (`max_elements` 250 actionable nodes). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Linux yet.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250.\n\nTIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.\n\nSCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\nPOPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_token). Omitting window_id while a popup of this pid is open walks that popup.","destructive": false,"idempotent": false,"input_schema": {@@ -57,8 +57,12 @@],"type": "string"},+ "full_output": {+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",+ "type": "boolean"+ },"include_accessibility_tree": {- "description": "Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",+ "description": "Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).","type": "boolean"},"include_screenshot": {@@ -76,7 +80,7 @@"type": "integer"},"max_elements": {- "description": "Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees.",+ "description": "Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window.","minimum": 1,"type": "integer"},@@ -101,6 +105,10 @@"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type": "string"},+ "since": {+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",+ "type": "string"+ },"timeout_ms": {"default": 1000,"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",@@ -108,6 +116,19 @@"minimum": 100,"type": "integer"},+ "tree_format": {+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",+ "enum": [+ "markdown",+ "elements",+ "both"+ ],+ "type": "string"+ },+ "verbose": {+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",+ "type": "boolean"+ },"window_id": {"description": "Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window.","type": "integer"diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 68dbcffc03..e19656720a 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -43,7 +43,7 @@"read_only": true},{- "description": "Snapshot a window: structured `elements` (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`), the same tree as Markdown in `tree_markdown`, and a screenshot. Pass an element's `element_token` to click, type_text, press_key, etc.\n\nSTART NARROW: `query` (case-insensitive substring) returns only matching rows plus their ancestors, and `max_elements` / `max_depth` bound the walk, so large Electron or web trees do not flood context. Indices and tokens stay valid for the whole snapshot. A full read can be 20x larger than a targeted one.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS: `since:<snapshot_id>` from an earlier read of the same window returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only; `include_accessibility_tree:false` returns only the screenshot and window metadata. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.","destructive": false,"idempotent": false,"input_schema": {@@ -57,6 +57,10 @@],"type": "string"},+ "full_output": {+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",+ "type": "boolean"+ },"include_accessibility_tree": {"description": "Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error.","type": "boolean"@@ -76,7 +80,7 @@"type": "integer"},"max_elements": {- "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 2000.",+ "description": "Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut.","minimum": 1,"type": "integer"},@@ -101,6 +105,10 @@"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it.","type": "string"},+ "since": {+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",+ "type": "string"+ },"timeout_ms": {"default": 1000,"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",@@ -108,6 +116,19 @@"minimum": 100,"type": "integer"},+ "tree_format": {+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",+ "enum": [+ "markdown",+ "elements",+ "both"+ ],+ "type": "string"+ },+ "verbose": {+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",+ "type": "boolean"+ },"window_id": {"description": "Target window ID from list_windows.","type": "integer"diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.jsonindex bbf00bc656..c49a8a33de 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json@@ -43,7 +43,7 @@"read_only": true},{- "description": "Walk a running app's UIA tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to `click`, `type_text`, `scroll`, etc.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of UIA patterns exposed as actions, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project BOTH `tree_markdown` and `structuredContent.elements` to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤5 000 elements, depth ≤25).\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.","destructive": false,"idempotent": false,"input_schema": {@@ -57,8 +57,12 @@],"type": "string"},+ "full_output": {+ "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",+ "type": "boolean"+ },"include_accessibility_tree": {- "description": "Default true — walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",+ "description": "Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).","type": "boolean"},"include_screenshot": {@@ -76,7 +80,7 @@"type": "integer"},"max_elements": {- "description": "Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees.",+ "description": "Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window.","minimum": 1,"type": "integer"},@@ -101,6 +105,10 @@"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type": "string"},+ "since": {+ "description": "A `snapshot_id` from an earlier get_window_state of the same window. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.",+ "type": "string"+ },"timeout_ms": {"default": 1000,"description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",@@ -108,6 +116,19 @@"minimum": 100,"type": "integer"},+ "tree_format": {+ "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `<snapshot_id>:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.",+ "enum": [+ "markdown",+ "elements",+ "both"+ ],+ "type": "string"+ },+ "verbose": {+ "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.",+ "type": "boolean"+ },"window_id": {"description": "HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array.","type": "integer"diff --git a/scripts/docs-generators/cli-specs/cua-driver.json b/scripts/docs-generators/cli-specs/cua-driver.jsonindex d812e0025e..282830899f 100644--- a/scripts/docs-generators/cli-specs/cua-driver.json+++ b/scripts/docs-generators/cli-specs/cua-driver.json@@ -147,6 +147,14 @@{"command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725}' --screenshot-out-file state.png","description": "Save the screenshot from a tool response"+ },+ {+ "command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"since\":\"s0000002a\"}'",+ "description": "Read only what changed since an earlier snapshot_id"+ },+ {+ "command": "cua-driver call get_window_state '{\"pid\":844,\"window_id\":10725,\"tree_format\":\"elements\"}'",+ "description": "Return the structured elements instead of the markdown tree"}],"flags": [],