Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): add display_only to get_window_state for live previews (#4881)
Francesco Bonacci4cbd0966c505
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdxindex 2579763216..fc8d796023 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx@@ -34,7 +34,7 @@ START NARROW: a read walks at most 250 nodes by default and says `Tree truncatedDIFF READS (use them for every re-read after acting): `since:"latest"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.-A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.+A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.@@ -100,7 +100,7 @@ Size: a read is bounded by default (`max_elements` 250). When the walk stops atDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.-INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.+INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.The UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about: the driver never picks a window implicitly.@@ -144,6 +144,7 @@ Windows requires no special permissions.| Parameter | Type | Default | Description || --- | --- | --- | --- || <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |+| <span id="get_window_state--display_only"></span>`display_only` | `boolean` | | Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions. || <span id="get_window_state--full_output"></span>`full_output` | `boolean` | | Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`. || <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window, or "latest" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. || <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex eeea1110cb..6fbdc88fa3 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -40,7 +40,7 @@ The MCP server instructions carry the same rules, so they apply even when this f1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`. Widen only if the target is missing.2. Act by `element_token`: row `[N]` of a read is `<snapshot_id>:N` (for example `"s0000002a:11"`, not `"11"`). Use pixels only for surfaces missing from the tree.3. Act through `run_actions` with `observe:true`, even for a single step. It runs the steps, stops at the first failure, and returns only what changed since your last read, so you do not need a separate read after acting; see [Workflow](WORKFLOW.md#batch-known-actions).-4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (on macOS it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.+4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.5. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.## Actdiff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.mdindex c75b6cc0c1..5d1e613648 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md@@ -44,7 +44,7 @@ By default the tree comes back **once**, as compact `tree_markdown` (`tree_forma- After the first read, pass `since:"latest"` (or `since:<snapshot_id>`) to get only what changed. `latest` is this window's most recent read with the same `query`, `max_elements` and `max_depth`. You get: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.- `include_screenshot:false` returns the tree only. On macOS the window keeps the pixel frame of its last screenshot while its size is unchanged, so `x,y` read off that screenshot still work; elsewhere, and after a resize, take a new screenshot before a pixel action.-- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. On macOS it keeps the current snapshot's rows and tokens valid. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.+- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. It keeps the current snapshot's rows and tokens valid. `display_only:true` with it is for live previews: it also leaves your pixel frame alone, and its image cannot ground actions. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.- Row numbers belong to one read. A narrower re-read (smaller `max_elements`, `max_depth`, another `query`) can stop before a row you saw earlier, and `<new snapshot_id>:N` for that row is then out of range. Use rows printed in the read whose `snapshot_id` you use.- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.- Use `screenshot_out_file` to save a PNG instead of inlining it, then actually read the image. Use an absolute, run-scoped output path.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rsindex af06ebea1a..554206e907 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs@@ -90,11 +90,56 @@ pub fn schema_properties() -> Value {"full_output": {"type": "boolean","description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`."+ },+ "display_only": {+ "type": "boolean",+ "description": DISPLAY_ONLY_DESCRIPTION}})}-/// Add the shared `tree_format` / `since` / `verbose` / `full_output`+// `display_only` is a public, side-effect-free capture for live previews+// (picture-in-picture) that poll a window several times a second. It maps+// onto each backend's internal observation-only read: no snapshot is+// published, refreshed or removed, and no action capture is registered, so+// the agent's element tokens and pixel frame for the window stay valid.+//+// Hosts detect it by the property in `get_window_state`'s input schema+// (T3 Code checks `"display_only" in input_schema.properties`), so the+// description, refusal and response fields match T3 Code's Linux patch+// word for word.+const DISPLAY_ONLY_DESCRIPTION: &str = "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.";++/// Refusal when `display_only` is asked for together with a tree walk.+pub const DISPLAY_ONLY_NEEDS_NO_TREE: &str =+ "display_only requires include_accessibility_tree:false";++/// `frame_note` on a display-only response.+pub const DISPLAY_ONLY_FRAME_NOTE: &str = "Display-only preview; not registered for input actions. Obtain a normal get_window_state screenshot before acting.";++/// Read `display_only`. A display-only read must skip the tree walk, because+/// a walk is what produces element rows and tokens.+pub fn display_only(args: &Value) -> Result<bool, ToolResult> {+ if args.get("display_only").and_then(Value::as_bool) != Some(true) {+ return Ok(false);+ }+ if args+ .get("include_accessibility_tree")+ .and_then(Value::as_bool)+ != Some(false)+ {+ return Err(ToolResult::error(DISPLAY_ONLY_NEEDS_NO_TREE));+ }+ Ok(true)+}++/// Mark a display-only response.+pub fn mark_display_only(structured: &mut Value) {+ structured["display_only"] = json!(true);+ structured["frame_note"] = json!(DISPLAY_ONLY_FRAME_NOTE);+}++/// Add the shared `tree_format` / `since` / `verbose` / `full_output` / `display_only`/// properties to a backend's `get_window_state` input schema.pub fn extend_input_schema(mut schema: Value) -> Value {if let (Some(props), Some(extra)) = (@@ -932,6 +977,43 @@ mod tests {ViewOptions::from_args(&args).unwrap()}+ #[test]+ fn display_only_is_advertised_in_every_backend_schema() {+ let schema = extend_input_schema(json!({"type": "object", "properties": {}}));+ assert_eq!(schema["properties"]["display_only"]["type"], "boolean");+ }++ #[test]+ fn display_only_requires_a_screenshot_only_read() {+ assert_eq!(display_only(&json!({})).ok(), Some(false));+ assert_eq!(+ display_only(&json!({"display_only": false})).ok(),+ Some(false)+ );+ assert_eq!(+ display_only(&json!({"display_only": true, "include_accessibility_tree": false})).ok(),+ Some(true)+ );+ for args in [+ json!({"display_only": true}),+ json!({"display_only": true, "include_accessibility_tree": true}),+ ] {+ let refusal = display_only(&args).unwrap_err();+ assert_eq!(+ serde_json::to_value(refusal).unwrap(),+ serde_json::to_value(ToolResult::error(DISPLAY_ONLY_NEEDS_NO_TREE)).unwrap()+ );+ }+ }++ #[test]+ fn display_only_response_carries_the_frame_note() {+ let mut structured = json!({"screenshot_width": 800});+ mark_display_only(&mut structured);+ assert_eq!(structured["display_only"], true);+ assert_eq!(structured["frame_note"], DISPLAY_ONLY_FRAME_NOTE);+ }+const OLD: &str = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Cancel\" [id=cancel actions=[press]]";#[test]diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex de4e06c3cb..ed78dd43de 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -1124,6 +1124,10 @@ impl Tool for GetWindowStateTool {}async fn invoke(&self, args: Value) -> ToolResult {+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {+ Ok(display_only) => display_only,+ Err(refusal) => return refusal,+ };use cua_driver_core::tool_args::ArgsExt;let pid = match args.require_u32("pid") {Ok(v) => v,@@ -1234,9 +1238,11 @@ impl Tool for GetWindowStateTool {view.max_elements(&args,LINUX_DEFAULT_MAX_ELEMENTS,- args.get("_observation_only")- .and_then(|value| value.as_bool())- == Some(true),+ display_only+ || args+ .get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true),),);let max_depth = args@@ -1313,10 +1319,13 @@ impl Tool for GetWindowStateTool {screenshot_out_file to force a capture.",);}- let observation_only = args- .get("_observation_only")- .and_then(|value| value.as_bool())- == Some(true);+ // `display_only` is the public form of the internal observation-only+ // read: pixels for a preview, with no snapshot or capture change.+ let observation_only = display_only+ || args+ .get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true);let state = self.state.clone();let state_for_capture = state.clone();let query_for_walk = query.clone();@@ -1656,17 +1665,42 @@ impl Tool for GetWindowStateTool {}}- if !observation_only && !published_snapshot && screenshot_scale.is_some() {- if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(+ // A screenshot-only read adds an image but no element rows: it+ // refreshes the current snapshot's pixel frame in place and+ // keeps its tokens valid. Only a window with no element+ // snapshot yet gets a new capture-only one.+ let mut refreshed = None;+ if let (false, false, Some(scale)) =+ (observation_only, published_snapshot, screenshot_scale)+ {+ refreshed = state.snapshots.refresh_screenshot_for_session(pid as i32,xid,- crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),session_id.as_deref(),- screenshot_scale,- ) {- invalidated.extend(replaced);+ scale,+ None,+ );+ if refreshed.is_none() {+ if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(+ pid as i32,+ xid,+ crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),+ session_id.as_deref(),+ screenshot_scale,+ ) {+ invalidated.extend(replaced);+ }}}+ if let Some(id) = refreshed {+ let sid = cua_driver_core::element_token::format_snapshot_id(id);+ content.push(cua_driver_core::protocol::Content::text(format!(+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \+ ({sid}:N) stay valid."+ )));+ structured["snapshot_id"] = json!(sid);+ structured["screenshot_refreshed_snapshot"] = json!(true);+ }if !invalidated.is_empty() {let ids: Vec<String> = invalidated.into_iter()@@ -1759,6 +1793,9 @@ impl Tool for GetWindowStateTool {window_bounds is where it sits on the screen. Pass scope:\"desktop\" only \for get_desktop_state pixels.");+ if display_only {+ cua_driver_core::window_state_view::mark_display_only(&mut structured);+ }// The capture-only path (include_accessibility_tree:false) leaves// `content` empty when the screenshot was also unavailable — most@@ -14647,6 +14684,8 @@ mod pid_window_target_tests;#[cfg(test)]mod session_cursor_target_tests;+#[cfg(test)]+mod display_only_tests;#[cfg(test)]mod window_capture_dimension_tests;diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rsnew file mode 100644index 0000000000..4876c86c8a--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs@@ -0,0 +1,35 @@+use super::{GetWindowStateTool, ToolState};+use cua_driver_core::tool::Tool;++// T3 Code feature-detects the preview mode from this schema property.+#[test]+fn display_only_is_advertised() {+ let tool = GetWindowStateTool {+ state: ToolState::new(),+ };+ assert_eq!(+ tool.def().input_schema["properties"]["display_only"]["type"],+ "boolean"+ );+}++#[tokio::test]+async fn display_only_refuses_accessibility_snapshot_requests() {+ let tool = GetWindowStateTool {+ state: ToolState::new(),+ };+ for tree in [None, Some(true)] {+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});+ if let Some(tree) = tree {+ args["include_accessibility_tree"] = serde_json::json!(tree);+ }+ let result = tool.invoke(args).await;+ assert_eq!(+ serde_json::to_value(result).unwrap(),+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(+ "display_only requires include_accessibility_tree:false"+ ))+ .unwrap()+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex 2f120cd150..1ce49017f3 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -33,7 +33,7 @@ fn def() -> &'static ToolDef {\n\DIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\\n\- A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\+ A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\\n\Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({@@ -121,6 +121,10 @@ impl Tool for GetWindowStateTool {}async fn invoke(&self, args: Value) -> ToolResult {+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {+ Ok(display_only) => display_only,+ Err(refusal) => return refusal,+ };use cua_driver_core::tool_args::ArgsExt;let pid = match args.require_i32("pid") {Ok(v) => v,@@ -226,10 +230,13 @@ impl Tool for GetWindowStateTool {// Internal direct-tool mode used by verify_state. Registry ingress// strips underscore-prefixed arguments before public dispatch; only// a trusted direct in-process invocation can enable this mode.- let observation_only = args- .get("_observation_only")- .and_then(|value| value.as_bool())- == Some(true);+ // `display_only` is the public form of the same mode: pixels for a+ // preview, with no snapshot or capture change.+ let observation_only = display_only+ || args+ .get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true);// Optional caps — when omitted, fall back to the defaults baked into// the AX walker (#22865). minimum:1 keyed in the schema, but defend// against 0 here as well so a misbehaving client can't disable the@@ -701,7 +708,9 @@ impl Tool for GetWindowStateTool {// background mutation, reported per route so an agent can choose// before acting. Every action still revalidates — this is advisory,// not a promise. Old consumers ignore the extra field.- {+ // A display-only frame cannot ground input, so skip the input probe:+ // previews poll several times a second.+ if !display_only {let capture_available = screenshot_dims.is_some();let report = tokio::task::spawn_blocking(move || {let facts = crate::ax::exact_target::gather_background_facts(pid, window_id, None);@@ -763,6 +772,9 @@ impl Tool for GetWindowStateTool {cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::MayBeIncomplete,),);+ if display_only {+ cua_driver_core::window_state_view::mark_display_only(&mut structured);+ }if !observation_only {let focus_probe = || focused_element_description(pid, tree_result.as_ref());cua_driver_core::window_state_view::apply(@@ -1290,6 +1302,35 @@ mod tests {use crate::ax::tree::AXNode;use serde_json::json;+ // T3 Code feature-detects the preview mode from this schema property.+ #[test]+ fn display_only_is_advertised() {+ let tool = GetWindowStateTool::new(Arc::new(ToolState::new(false, false, None)));+ assert_eq!(+ tool.def().input_schema["properties"]["display_only"]["type"],+ "boolean"+ );+ }++ #[tokio::test]+ async fn display_only_refuses_accessibility_snapshot_requests() {+ let tool = GetWindowStateTool::new(Arc::new(ToolState::new(false, false, None)));+ for tree in [None, Some(true)] {+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});+ if let Some(tree) = tree {+ args["include_accessibility_tree"] = serde_json::json!(tree);+ }+ let result = tool.invoke(args).await;+ assert_eq!(+ serde_json::to_value(result).unwrap(),+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(+ "display_only requires include_accessibility_tree:false"+ ))+ .unwrap()+ );+ }+ }+fn node(idx: Option<usize>,role: &str,diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 6565815d8b..0173684024 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -1199,7 +1199,9 @@ impl Tool for GetWindowStateTool {INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \any element action against that window. The next snapshot of the same \(pid, window_id) replaces this one, stales its element tokens, and lists the \- replaced ids in `invalidated_snapshot_ids`.\n\n\+ replaced ids in `invalidated_snapshot_ids`. A screenshot-only read \+ (`include_accessibility_tree:false`) keeps the current snapshot's rows and \+ tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\n\The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \window bounds reported come from the same `window_id`. This is the source of \truth for which window the caller intends to reason about — the driver never \@@ -1259,6 +1261,10 @@ impl Tool for GetWindowStateTool {}async fn invoke(&self, args: Value) -> ToolResult {+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {+ Ok(display_only) => display_only,+ Err(refusal) => return refusal,+ };// Swift error wording 1:1.let pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(v) => v as u32,@@ -1358,15 +1364,20 @@ impl Tool for GetWindowStateTool {let query = args.opt_str("query");let session_id = args.opt_str("_session_id");let screenshot_out_file = args.opt_str("screenshot_out_file");+ // `display_only` is the public form of the internal observation-only+ // read: pixels for a preview, with no snapshot or capture change.+ let observation_only = display_only+ || args+ .get("_observation_only")+ .and_then(|value| value.as_bool())+ == Some(true);// Optional caps — when omitted, fall back to the walker's built-in// defaults (#22865). minimum:1 enforced in the schema, but defend// against 0 here too.let max_elements = view.max_elements(&args,crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,- args.get("_observation_only")- .and_then(|value| value.as_bool())- == Some(true),+ observation_only,);let max_depth = args.get("max_depth")@@ -1383,10 +1394,6 @@ impl Tool for GetWindowStateTool {// `screenshot_out_file` the bytes go to disk and the path is surfaced// instead of embedding base64; otherwise the base64 PNG is embedded.let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());- let observation_only = args- .get("_observation_only")- .and_then(|value| value.as_bool())- == Some(true);// `include_accessibility_tree` (default true) mirrors include_screenshot:// set false to SKIP the UIA walk and return just the screenshot + window// metadata (the capture-only / preview path).@@ -1638,21 +1645,46 @@ impl Tool for GetWindowStateTool {}}- if !observation_only && !published_snapshot && screenshot_scale.is_some() {- let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(- &[],- crate::uia::snapshot::SnapshotKind::Uia,- );- if let Some((_, replaced)) = state.snapshots.publish_for_session(+ // A screenshot-only read adds an image but no element rows: it+ // refreshes the current snapshot's pixel frame in place and+ // keeps its tokens valid. Only a window with no element+ // snapshot yet gets a new, empty one.+ let mut refreshed = None;+ if let (false, false, Some(scale)) =+ (observation_only, published_snapshot, screenshot_scale)+ {+ refreshed = state.snapshots.refresh_screenshot_for_session(pid as i32,hwnd,- payload,session_id.as_deref(),- screenshot_scale,- ) {- invalidated.extend(replaced);+ scale,+ None,+ );+ if refreshed.is_none() {+ let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(+ &[],+ crate::uia::snapshot::SnapshotKind::Uia,+ );+ if let Some((_, replaced)) = state.snapshots.publish_for_session(+ pid as i32,+ hwnd,+ payload,+ session_id.as_deref(),+ screenshot_scale,+ ) {+ invalidated.extend(replaced);+ }}}+ if let Some(id) = refreshed {+ let sid = cua_driver_core::element_token::format_snapshot_id(id);+ content.push(cua_driver_core::protocol::Content::text(format!(+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \+ ({sid}:N) stay valid."+ )));+ structured["snapshot_id"] = json!(sid);+ structured["screenshot_refreshed_snapshot"] = json!(true);+ }if !invalidated.is_empty() {let ids: Vec<String> = invalidated.into_iter()@@ -1772,6 +1804,9 @@ impl Tool for GetWindowStateTool {.with_structured(structured);}+ if display_only {+ cua_driver_core::window_state_view::mark_display_only(&mut structured);+ }if !observation_only {cua_driver_core::window_state_view::apply(&view,@@ -10490,5 +10525,7 @@ mod desktop_scope_tests;#[cfg(test)]mod pid_window_target_tests;+#[cfg(test)]+mod display_only_tests;#[cfg(test)]mod value_write_readback_tests;diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rsnew file mode 100644index 0000000000..e39fda62c1--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs@@ -0,0 +1,35 @@+use super::{GetWindowStateTool, ToolState};+use cua_driver_core::tool::Tool;++// T3 Code feature-detects the preview mode from this schema property.+#[test]+fn display_only_is_advertised() {+ let tool = GetWindowStateTool {+ state: ToolState::new(None),+ };+ assert_eq!(+ tool.def().input_schema["properties"]["display_only"]["type"],+ "boolean"+ );+}++#[tokio::test]+async fn display_only_refuses_accessibility_snapshot_requests() {+ let tool = GetWindowStateTool {+ state: ToolState::new(None),+ };+ for tree in [None, Some(true)] {+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});+ if let Some(tree) = tree {+ args["include_accessibility_tree"] = serde_json::json!(tree);+ }+ let result = tool.invoke(args).await;+ assert_eq!(+ serde_json::to_value(result).unwrap(),+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(+ "display_only requires include_accessibility_tree:false"+ ))+ .unwrap()+ );+ }+}diff --git a/scripts/ci/cua-driver-live-smoke/smoke.py b/scripts/ci/cua-driver-live-smoke/smoke.pyindex 9e7b7d34bd..9a2ebfa099 100644--- a/scripts/ci/cua-driver-live-smoke/smoke.py+++ b/scripts/ci/cua-driver-live-smoke/smoke.py@@ -8,7 +8,9 @@ fixture window (GTK3 on Linux, WinForms on Windows) and checks:* the act-by-element_token loop (snapshot, click a token, fresh snapshot),* `since` (a diff against an earlier snapshot, and `no change`),* run_actions (one batch with an end-of-batch observation),- * set_value on a drop-down combo box and on an editable combo box.+ * set_value on a drop-down combo box and on an editable combo box,+ * display_only previews (and plain screenshot-only reads) keep the agent's+ element tokens valid.Also checked: an element_token alone (no pid) acts and a stale one isrefused as `stale_element_token`. Those are required: each is `pass` or@@ -402,6 +404,9 @@ def run_checks(smoke):# 7. element_token alone, no pid: the token names its pid.token_without_pid_check(smoke)+ # 7a. display_only previews leave the agent's snapshot alone.+ display_only_check(smoke)+# 7b. Lean-read ergonomics from the v035 bench: since:"latest", a bare row# number as element_token, run_actions observe:true, zoom x/y/width/height.lean_read_ergonomics_check(smoke)@@ -480,6 +485,73 @@ def token_without_pid_check(smoke):)+DISPLAY_ONLY_READS = 8+++def display_only_check(smoke):+ """A preview polling display_only reads (T3 Code's picture-in-picture)+ must not stale the agent's element tokens; neither may a plain+ screenshot-only read."""+ check = f"tokens stay valid after {DISPLAY_ONLY_READS} display_only reads"+ problems = []+ listed = smoke.mcp.request("tools/list", {})+ tools = (listed.get("result") or {}).get("tools") or []+ schema = next(+ (t.get("inputSchema") or {} for t in tools if t.get("name") == "get_window_state"), {}+ )+ if "display_only" not in (schema.get("properties") or {}):+ problems.append("get_window_state does not advertise display_only")++ state = smoke.read()+ snapshot = state.structured.get("snapshot_id")+ before = count_value(tree(state))+ increment = find_row(tree(state), BUTTON, "Increment")+ preview = {"include_accessibility_tree": False, "include_screenshot": True,+ "max_dimension": 480, "display_only": True}+ for n in range(DISPLAY_ONLY_READS):+ frame = smoke.read(**preview)+ f = frame.structured+ bad = []+ if frame.is_error:+ bad.append(frame.first_line())+ if f.get("display_only") is not True or not f.get("frame_note"):+ bad.append("no display_only/frame_note")+ if not isinstance(f.get("screenshot_width"), int) or not isinstance(f.get("screenshot_height"), int):+ bad.append("no screenshot_width/height")+ for key in ("snapshot_id", "invalidated_snapshot_ids", "capture_id"):+ if key in f:+ bad.append(f"carries {key}={f[key]!r}")+ if bad:+ problems.append(f"display_only read {n + 1}: " + "; ".join(bad))+ break+ time.sleep(0.25)+ refused = smoke.read(display_only=True)+ if not refused.is_error or "display_only requires include_accessibility_tree:false" not in refused.text:+ problems.append(f"display_only with a tree walk was not refused: {refused.first_line()}")++ plain = smoke.read(include_accessibility_tree=False)+ if plain.structured.get("invalidated_snapshot_ids"):+ problems.append(+ f"a plain screenshot-only read invalidated {plain.structured['invalidated_snapshot_ids']}"+ )++ args = {"pid": smoke.pid, "element_token": token(state, increment)}+ click = smoke.call("click", args)+ after = smoke.wait_for_text(f"Count: {(before or 0) + 1}")+ if click.is_error:+ problems.append(f"click with the pre-preview token {args['element_token']} -> {click.first_line()}")+ elif f"Count: {(before or 0) + 1}" not in tree(after):+ problems.append(f"click ok but the count did not move from {before}")+ smoke.record(+ check,+ "fail" if problems else "pass",+ "; ".join(problems)+ or f"schema advertises display_only; {DISPLAY_ONLY_READS} display_only reads of "+ f"snapshot {snapshot} carried no snapshot_id/capture_id; a plain screenshot-only "+ f"read invalidated nothing; click {args['element_token']} -> Count: {(before or 0) + 1}",+ )++def lean_read_ergonomics_check(smoke):check = "since:latest, bare row token, run_actions observe:true, zoom x/y/width/height, scroll dy"problems = []diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.jsonindex ea5beeda3d..02a67e99a0 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json@@ -57,6 +57,10 @@],"type": "string"},+ "display_only": {+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",+ "type": "boolean"+ },"full_output": {"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.","type": "boolean"diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 9a6d4955a3..e44a85b45e 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -43,7 +43,7 @@"read_only": true},{- "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.","destructive": false,"idempotent": false,"input_schema": {@@ -57,6 +57,10 @@],"type": "string"},+ "display_only": {+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",+ "type": "boolean"+ },"full_output": {"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.","type": "boolean"diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.jsonindex 1599f4b114..34d41062a6 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json@@ -43,7 +43,7 @@"read_only": true},{- "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.","destructive": false,"idempotent": false,"input_schema": {@@ -57,6 +57,10 @@],"type": "string"},+ "display_only": {+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",+ "type": "boolean"+ },"full_output": {"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.","type": "boolean"