Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): add display_only to get_window_state for live previews (#4881)

Francesco Bonacci4cbd0966c505
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
index 2579763216..fc8d796023 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
@@ -34,7 +34,7 @@ START NARROW: a read walks at most 250 nodes by default and says `Tree truncated
DIFF READS (use them for every re-read after acting): `since:"latest"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.
-A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
+A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.
@@ -100,7 +100,7 @@ Size: a read is bounded by default (`max_elements` 250). When the walk stops at
DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.
-INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.
+INVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.
The UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about: the driver never picks a window implicitly.
@@ -144,6 +144,7 @@ Windows requires no special permissions.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="get_window_state--capture_mode"></span>`capture_mode` | `"ax" \| "vision"` | | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. |
+| <span id="get_window_state--display_only"></span>`display_only` | `boolean` | | Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions. |
| <span id="get_window_state--full_output"></span>`full_output` | `boolean` | | Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`. |
| <span id="get_window_state--since"></span>`since` | `string` | | A `snapshot_id` from an earlier get_window_state of the same window, or "latest" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise. |
| <span id="get_window_state--timeout_ms"></span>`timeout_ms` | `integer` | `1000` | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. Range: 100 to 120000. |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index eeea1110cb..6fbdc88fa3 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -40,7 +40,7 @@ The MCP server instructions carry the same rules, so they apply even when this f
1. Start with a targeted read: `get_window_state({pid, window_id, query:"Save"})` returns about 2K chars where a full snapshot is about 42K. Bound large trees with `max_elements` / `max_depth`. Widen only if the target is missing.
2. Act by `element_token`: row `[N]` of a read is `<snapshot_id>:N` (for example `"s0000002a:11"`, not `"11"`). Use pixels only for surfaces missing from the tree.
3. Act through `run_actions` with `observe:true`, even for a single step. It runs the steps, stops at the first failure, and returns only what changed since your last read, so you do not need a separate read after acting; see [Workflow](WORKFLOW.md#batch-known-actions).
-4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (on macOS it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.
+4. To re-read a window, pass `since:"latest"`: you get the changed rows, not the whole tree. To just look, use `include_accessibility_tree:false` where advertised (it keeps the current tokens valid). Do not use `max_elements:1` or a junk `query` to get a picture.
5. Verify at checkpoints (after a meaningful state change, before finishing), not after every action. One `verify_state` or one targeted `get_window_state` is usually enough.
## Act
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
index c75b6cc0c1..5d1e613648 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
@@ -44,7 +44,7 @@ By default the tree comes back **once**, as compact `tree_markdown` (`tree_forma
- After the first read, pass `since:"latest"` (or `since:<snapshot_id>`) to get only what changed. `latest` is this window's most recent read with the same `query`, `max_elements` and `max_depth`. You get: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), or `no change since …; focused element is …` (focus is reported on macOS only). The response carries a new `snapshot_id`: use it in tokens (`<new id>:N`). Rows not listed keep their `[N]` unless a `reindexed:` line says otherwise. An unknown, expired, other-window, or differently-scoped `since` returns a full read, and `since_status` says why. A diff reads the rendered tree rows only; it does not prove the screenshot is unchanged.
- `_note`, `background_input`, and (Linux) `frame_note` are omitted unless `verbose:true`; a degraded snapshot keeps `background_input`.
- `include_screenshot:false` returns the tree only. On macOS the window keeps the pixel frame of its last screenshot while its size is unchanged, so `x,y` read off that screenshot still work; elsewhere, and after a resize, take a new screenshot before a pixel action.
-- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. On macOS it keeps the current snapshot's rows and tokens valid. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.
+- Where advertised, `include_accessibility_tree:false` requests capture without a tree walk. It keeps the current snapshot's rows and tokens valid. `display_only:true` with it is for live previews: it also leaves your pixel frame alone, and its image cannot ground actions. Use it when you only need to look; do not use `max_elements:1` or a `query` that matches nothing for that, because those replace the snapshot with an almost empty one.
- Row numbers belong to one read. A narrower re-read (smaller `max_elements`, `max_depth`, another `query`) can stop before a row you saw earlier, and `<new snapshot_id>:N` for that row is then out of range. Use rows printed in the read whose `snapshot_id` you use.
- `capture_mode` is deprecated and ignored. Do not change configuration to repair a sparse tree.
- Use `screenshot_out_file` to save a PNG instead of inlining it, then actually read the image. Use an absolute, run-scoped output path.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
index af06ebea1a..554206e907 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_state_view.rs
@@ -90,11 +90,56 @@ pub fn schema_properties() -> Value {
"full_output": {
"type": "boolean",
"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`."
+ },
+ "display_only": {
+ "type": "boolean",
+ "description": DISPLAY_ONLY_DESCRIPTION
}
})
}
-/// Add the shared `tree_format` / `since` / `verbose` / `full_output`
+// `display_only` is a public, side-effect-free capture for live previews
+// (picture-in-picture) that poll a window several times a second. It maps
+// onto each backend's internal observation-only read: no snapshot is
+// published, refreshed or removed, and no action capture is registered, so
+// the agent's element tokens and pixel frame for the window stay valid.
+//
+// Hosts detect it by the property in `get_window_state`'s input schema
+// (T3 Code checks `"display_only" in input_schema.properties`), so the
+// description, refusal and response fields match T3 Code's Linux patch
+// word for word.
+const DISPLAY_ONLY_DESCRIPTION: &str = "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.";
+
+/// Refusal when `display_only` is asked for together with a tree walk.
+pub const DISPLAY_ONLY_NEEDS_NO_TREE: &str =
+ "display_only requires include_accessibility_tree:false";
+
+/// `frame_note` on a display-only response.
+pub const DISPLAY_ONLY_FRAME_NOTE: &str = "Display-only preview; not registered for input actions. Obtain a normal get_window_state screenshot before acting.";
+
+/// Read `display_only`. A display-only read must skip the tree walk, because
+/// a walk is what produces element rows and tokens.
+pub fn display_only(args: &Value) -> Result<bool, ToolResult> {
+ if args.get("display_only").and_then(Value::as_bool) != Some(true) {
+ return Ok(false);
+ }
+ if args
+ .get("include_accessibility_tree")
+ .and_then(Value::as_bool)
+ != Some(false)
+ {
+ return Err(ToolResult::error(DISPLAY_ONLY_NEEDS_NO_TREE));
+ }
+ Ok(true)
+}
+
+/// Mark a display-only response.
+pub fn mark_display_only(structured: &mut Value) {
+ structured["display_only"] = json!(true);
+ structured["frame_note"] = json!(DISPLAY_ONLY_FRAME_NOTE);
+}
+
+/// Add the shared `tree_format` / `since` / `verbose` / `full_output` / `display_only`
/// properties to a backend's `get_window_state` input schema.
pub fn extend_input_schema(mut schema: Value) -> Value {
if let (Some(props), Some(extra)) = (
@@ -932,6 +977,43 @@ mod tests {
ViewOptions::from_args(&args).unwrap()
}
+ #[test]
+ fn display_only_is_advertised_in_every_backend_schema() {
+ let schema = extend_input_schema(json!({"type": "object", "properties": {}}));
+ assert_eq!(schema["properties"]["display_only"]["type"], "boolean");
+ }
+
+ #[test]
+ fn display_only_requires_a_screenshot_only_read() {
+ assert_eq!(display_only(&json!({})).ok(), Some(false));
+ assert_eq!(
+ display_only(&json!({"display_only": false})).ok(),
+ Some(false)
+ );
+ assert_eq!(
+ display_only(&json!({"display_only": true, "include_accessibility_tree": false})).ok(),
+ Some(true)
+ );
+ for args in [
+ json!({"display_only": true}),
+ json!({"display_only": true, "include_accessibility_tree": true}),
+ ] {
+ let refusal = display_only(&args).unwrap_err();
+ assert_eq!(
+ serde_json::to_value(refusal).unwrap(),
+ serde_json::to_value(ToolResult::error(DISPLAY_ONLY_NEEDS_NO_TREE)).unwrap()
+ );
+ }
+ }
+
+ #[test]
+ fn display_only_response_carries_the_frame_note() {
+ let mut structured = json!({"screenshot_width": 800});
+ mark_display_only(&mut structured);
+ assert_eq!(structured["display_only"], true);
+ assert_eq!(structured["frame_note"], DISPLAY_ONLY_FRAME_NOTE);
+ }
+
const OLD: &str = "- [0] AXWindow \"Doc\"\n - [1] AXTextField \"Name\" = \"a\" [id=name]\n - AXStaticText = \"hello\"\n - [2] AXButton \"Save\" [id=save actions=[press]]\n - [3] AXButton \"Cancel\" [id=cancel actions=[press]]";
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index de4e06c3cb..ed78dd43de 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1124,6 +1124,10 @@ impl Tool for GetWindowStateTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {
+ Ok(display_only) => display_only,
+ Err(refusal) => return refusal,
+ };
use cua_driver_core::tool_args::ArgsExt;
let pid = match args.require_u32("pid") {
Ok(v) => v,
@@ -1234,9 +1238,11 @@ impl Tool for GetWindowStateTool {
view.max_elements(
&args,
LINUX_DEFAULT_MAX_ELEMENTS,
- args.get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true),
+ display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true),
),
);
let max_depth = args
@@ -1313,10 +1319,13 @@ impl Tool for GetWindowStateTool {
screenshot_out_file to force a capture.",
);
}
- let observation_only = args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true);
+ // `display_only` is the public form of the internal observation-only
+ // read: pixels for a preview, with no snapshot or capture change.
+ let observation_only = display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true);
let state = self.state.clone();
let state_for_capture = state.clone();
let query_for_walk = query.clone();
@@ -1656,17 +1665,42 @@ impl Tool for GetWindowStateTool {
}
}
- if !observation_only && !published_snapshot && screenshot_scale.is_some() {
- if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(
+ // A screenshot-only read adds an image but no element rows: it
+ // refreshes the current snapshot's pixel frame in place and
+ // keeps its tokens valid. Only a window with no element
+ // snapshot yet gets a new capture-only one.
+ let mut refreshed = None;
+ if let (false, false, Some(scale)) =
+ (observation_only, published_snapshot, screenshot_scale)
+ {
+ refreshed = state.snapshots.refresh_screenshot_for_session(
pid as i32,
xid,
- crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
session_id.as_deref(),
- screenshot_scale,
- ) {
- invalidated.extend(replaced);
+ scale,
+ None,
+ );
+ if refreshed.is_none() {
+ if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(
+ pid as i32,
+ xid,
+ crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
+ session_id.as_deref(),
+ screenshot_scale,
+ ) {
+ invalidated.extend(replaced);
+ }
}
}
+ if let Some(id) = refreshed {
+ let sid = cua_driver_core::element_token::format_snapshot_id(id);
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
+ ({sid}:N) stay valid."
+ )));
+ structured["snapshot_id"] = json!(sid);
+ structured["screenshot_refreshed_snapshot"] = json!(true);
+ }
if !invalidated.is_empty() {
let ids: Vec<String> = invalidated
.into_iter()
@@ -1759,6 +1793,9 @@ impl Tool for GetWindowStateTool {
window_bounds is where it sits on the screen. Pass scope:\"desktop\" only \
for get_desktop_state pixels."
);
+ if display_only {
+ cua_driver_core::window_state_view::mark_display_only(&mut structured);
+ }
// The capture-only path (include_accessibility_tree:false) leaves
// `content` empty when the screenshot was also unavailable — most
@@ -14647,6 +14684,8 @@ mod pid_window_target_tests;
#[cfg(test)]
mod session_cursor_target_tests;
+#[cfg(test)]
+mod display_only_tests;
#[cfg(test)]
mod window_capture_dimension_tests;
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs
new file mode 100644
index 0000000000..4876c86c8a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs
@@ -0,0 +1,35 @@
+use super::{GetWindowStateTool, ToolState};
+use cua_driver_core::tool::Tool;
+
+// T3 Code feature-detects the preview mode from this schema property.
+#[test]
+fn display_only_is_advertised() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(),
+ };
+ assert_eq!(
+ tool.def().input_schema["properties"]["display_only"]["type"],
+ "boolean"
+ );
+}
+
+#[tokio::test]
+async fn display_only_refuses_accessibility_snapshot_requests() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(),
+ };
+ for tree in [None, Some(true)] {
+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
+ if let Some(tree) = tree {
+ args["include_accessibility_tree"] = serde_json::json!(tree);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ serde_json::to_value(result).unwrap(),
+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
+ "display_only requires include_accessibility_tree:false"
+ ))
+ .unwrap()
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index 2f120cd150..1ce49017f3 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -33,7 +33,7 @@ fn def() -> &'static ToolDef {
\n\
DIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\
\n\
- A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
+ A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
\n\
Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({
@@ -121,6 +121,10 @@ impl Tool for GetWindowStateTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {
+ Ok(display_only) => display_only,
+ Err(refusal) => return refusal,
+ };
use cua_driver_core::tool_args::ArgsExt;
let pid = match args.require_i32("pid") {
Ok(v) => v,
@@ -226,10 +230,13 @@ impl Tool for GetWindowStateTool {
// Internal direct-tool mode used by verify_state. Registry ingress
// strips underscore-prefixed arguments before public dispatch; only
// a trusted direct in-process invocation can enable this mode.
- let observation_only = args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true);
+ // `display_only` is the public form of the same mode: pixels for a
+ // preview, with no snapshot or capture change.
+ let observation_only = display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true);
// Optional caps — when omitted, fall back to the defaults baked into
// the AX walker (#22865). minimum:1 keyed in the schema, but defend
// against 0 here as well so a misbehaving client can't disable the
@@ -701,7 +708,9 @@ impl Tool for GetWindowStateTool {
// background mutation, reported per route so an agent can choose
// before acting. Every action still revalidates — this is advisory,
// not a promise. Old consumers ignore the extra field.
- {
+ // A display-only frame cannot ground input, so skip the input probe:
+ // previews poll several times a second.
+ if !display_only {
let capture_available = screenshot_dims.is_some();
let report = tokio::task::spawn_blocking(move || {
let facts = crate::ax::exact_target::gather_background_facts(pid, window_id, None);
@@ -763,6 +772,9 @@ impl Tool for GetWindowStateTool {
cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::MayBeIncomplete,
),
);
+ if display_only {
+ cua_driver_core::window_state_view::mark_display_only(&mut structured);
+ }
if !observation_only {
let focus_probe = || focused_element_description(pid, tree_result.as_ref());
cua_driver_core::window_state_view::apply(
@@ -1290,6 +1302,35 @@ mod tests {
use crate::ax::tree::AXNode;
use serde_json::json;
+ // T3 Code feature-detects the preview mode from this schema property.
+ #[test]
+ fn display_only_is_advertised() {
+ let tool = GetWindowStateTool::new(Arc::new(ToolState::new(false, false, None)));
+ assert_eq!(
+ tool.def().input_schema["properties"]["display_only"]["type"],
+ "boolean"
+ );
+ }
+
+ #[tokio::test]
+ async fn display_only_refuses_accessibility_snapshot_requests() {
+ let tool = GetWindowStateTool::new(Arc::new(ToolState::new(false, false, None)));
+ for tree in [None, Some(true)] {
+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
+ if let Some(tree) = tree {
+ args["include_accessibility_tree"] = serde_json::json!(tree);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ serde_json::to_value(result).unwrap(),
+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
+ "display_only requires include_accessibility_tree:false"
+ ))
+ .unwrap()
+ );
+ }
+ }
+
fn node(
idx: Option<usize>,
role: &str,
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 6565815d8b..0173684024 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -1199,7 +1199,9 @@ impl Tool for GetWindowStateTool {
INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \
any element action against that window. The next snapshot of the same \
(pid, window_id) replaces this one, stales its element tokens, and lists the \
- replaced ids in `invalidated_snapshot_ids`.\n\n\
+ replaced ids in `invalidated_snapshot_ids`. A screenshot-only read \
+ (`include_accessibility_tree:false`) keeps the current snapshot's rows and \
+ tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\n\
The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \
window bounds reported come from the same `window_id`. This is the source of \
truth for which window the caller intends to reason about — the driver never \
@@ -1259,6 +1261,10 @@ impl Tool for GetWindowStateTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {
+ Ok(display_only) => display_only,
+ Err(refusal) => return refusal,
+ };
// Swift error wording 1:1.
let pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(v) => v as u32,
@@ -1358,15 +1364,20 @@ impl Tool for GetWindowStateTool {
let query = args.opt_str("query");
let session_id = args.opt_str("_session_id");
let screenshot_out_file = args.opt_str("screenshot_out_file");
+ // `display_only` is the public form of the internal observation-only
+ // read: pixels for a preview, with no snapshot or capture change.
+ let observation_only = display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true);
// Optional caps — when omitted, fall back to the walker's built-in
// defaults (#22865). minimum:1 enforced in the schema, but defend
// against 0 here too.
let max_elements = view.max_elements(
&args,
crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,
- args.get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true),
+ observation_only,
);
let max_depth = args
.get("max_depth")
@@ -1383,10 +1394,6 @@ impl Tool for GetWindowStateTool {
// `screenshot_out_file` the bytes go to disk and the path is surfaced
// instead of embedding base64; otherwise the base64 PNG is embedded.
let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
- let observation_only = args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true);
// `include_accessibility_tree` (default true) mirrors include_screenshot:
// set false to SKIP the UIA walk and return just the screenshot + window
// metadata (the capture-only / preview path).
@@ -1638,21 +1645,46 @@ impl Tool for GetWindowStateTool {
}
}
- if !observation_only && !published_snapshot && screenshot_scale.is_some() {
- let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
- &[],
- crate::uia::snapshot::SnapshotKind::Uia,
- );
- if let Some((_, replaced)) = state.snapshots.publish_for_session(
+ // A screenshot-only read adds an image but no element rows: it
+ // refreshes the current snapshot's pixel frame in place and
+ // keeps its tokens valid. Only a window with no element
+ // snapshot yet gets a new, empty one.
+ let mut refreshed = None;
+ if let (false, false, Some(scale)) =
+ (observation_only, published_snapshot, screenshot_scale)
+ {
+ refreshed = state.snapshots.refresh_screenshot_for_session(
pid as i32,
hwnd,
- payload,
session_id.as_deref(),
- screenshot_scale,
- ) {
- invalidated.extend(replaced);
+ scale,
+ None,
+ );
+ if refreshed.is_none() {
+ let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
+ &[],
+ crate::uia::snapshot::SnapshotKind::Uia,
+ );
+ if let Some((_, replaced)) = state.snapshots.publish_for_session(
+ pid as i32,
+ hwnd,
+ payload,
+ session_id.as_deref(),
+ screenshot_scale,
+ ) {
+ invalidated.extend(replaced);
+ }
}
}
+ if let Some(id) = refreshed {
+ let sid = cua_driver_core::element_token::format_snapshot_id(id);
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
+ ({sid}:N) stay valid."
+ )));
+ structured["snapshot_id"] = json!(sid);
+ structured["screenshot_refreshed_snapshot"] = json!(true);
+ }
if !invalidated.is_empty() {
let ids: Vec<String> = invalidated
.into_iter()
@@ -1772,6 +1804,9 @@ impl Tool for GetWindowStateTool {
.with_structured(structured);
}
+ if display_only {
+ cua_driver_core::window_state_view::mark_display_only(&mut structured);
+ }
if !observation_only {
cua_driver_core::window_state_view::apply(
&view,
@@ -10490,5 +10525,7 @@ mod desktop_scope_tests;
#[cfg(test)]
mod pid_window_target_tests;
+#[cfg(test)]
+mod display_only_tests;
#[cfg(test)]
mod value_write_readback_tests;
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs
new file mode 100644
index 0000000000..e39fda62c1
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs
@@ -0,0 +1,35 @@
+use super::{GetWindowStateTool, ToolState};
+use cua_driver_core::tool::Tool;
+
+// T3 Code feature-detects the preview mode from this schema property.
+#[test]
+fn display_only_is_advertised() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(None),
+ };
+ assert_eq!(
+ tool.def().input_schema["properties"]["display_only"]["type"],
+ "boolean"
+ );
+}
+
+#[tokio::test]
+async fn display_only_refuses_accessibility_snapshot_requests() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(None),
+ };
+ for tree in [None, Some(true)] {
+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
+ if let Some(tree) = tree {
+ args["include_accessibility_tree"] = serde_json::json!(tree);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ serde_json::to_value(result).unwrap(),
+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
+ "display_only requires include_accessibility_tree:false"
+ ))
+ .unwrap()
+ );
+ }
+}
diff --git a/scripts/ci/cua-driver-live-smoke/smoke.py b/scripts/ci/cua-driver-live-smoke/smoke.py
index 9e7b7d34bd..9a2ebfa099 100644
--- a/scripts/ci/cua-driver-live-smoke/smoke.py
+++ b/scripts/ci/cua-driver-live-smoke/smoke.py
@@ -8,7 +8,9 @@ fixture window (GTK3 on Linux, WinForms on Windows) and checks:
* the act-by-element_token loop (snapshot, click a token, fresh snapshot),
* `since` (a diff against an earlier snapshot, and `no change`),
* run_actions (one batch with an end-of-batch observation),
- * set_value on a drop-down combo box and on an editable combo box.
+ * set_value on a drop-down combo box and on an editable combo box,
+ * display_only previews (and plain screenshot-only reads) keep the agent's
+ element tokens valid.
Also checked: an element_token alone (no pid) acts and a stale one is
refused as `stale_element_token`. Those are required: each is `pass` or
@@ -402,6 +404,9 @@ def run_checks(smoke):
# 7. element_token alone, no pid: the token names its pid.
token_without_pid_check(smoke)
+ # 7a. display_only previews leave the agent's snapshot alone.
+ display_only_check(smoke)
+
# 7b. Lean-read ergonomics from the v035 bench: since:"latest", a bare row
# number as element_token, run_actions observe:true, zoom x/y/width/height.
lean_read_ergonomics_check(smoke)
@@ -480,6 +485,73 @@ def token_without_pid_check(smoke):
)
+DISPLAY_ONLY_READS = 8
+
+
+def display_only_check(smoke):
+ """A preview polling display_only reads (T3 Code's picture-in-picture)
+ must not stale the agent's element tokens; neither may a plain
+ screenshot-only read."""
+ check = f"tokens stay valid after {DISPLAY_ONLY_READS} display_only reads"
+ problems = []
+ listed = smoke.mcp.request("tools/list", {})
+ tools = (listed.get("result") or {}).get("tools") or []
+ schema = next(
+ (t.get("inputSchema") or {} for t in tools if t.get("name") == "get_window_state"), {}
+ )
+ if "display_only" not in (schema.get("properties") or {}):
+ problems.append("get_window_state does not advertise display_only")
+
+ state = smoke.read()
+ snapshot = state.structured.get("snapshot_id")
+ before = count_value(tree(state))
+ increment = find_row(tree(state), BUTTON, "Increment")
+ preview = {"include_accessibility_tree": False, "include_screenshot": True,
+ "max_dimension": 480, "display_only": True}
+ for n in range(DISPLAY_ONLY_READS):
+ frame = smoke.read(**preview)
+ f = frame.structured
+ bad = []
+ if frame.is_error:
+ bad.append(frame.first_line())
+ if f.get("display_only") is not True or not f.get("frame_note"):
+ bad.append("no display_only/frame_note")
+ if not isinstance(f.get("screenshot_width"), int) or not isinstance(f.get("screenshot_height"), int):
+ bad.append("no screenshot_width/height")
+ for key in ("snapshot_id", "invalidated_snapshot_ids", "capture_id"):
+ if key in f:
+ bad.append(f"carries {key}={f[key]!r}")
+ if bad:
+ problems.append(f"display_only read {n + 1}: " + "; ".join(bad))
+ break
+ time.sleep(0.25)
+ refused = smoke.read(display_only=True)
+ if not refused.is_error or "display_only requires include_accessibility_tree:false" not in refused.text:
+ problems.append(f"display_only with a tree walk was not refused: {refused.first_line()}")
+
+ plain = smoke.read(include_accessibility_tree=False)
+ if plain.structured.get("invalidated_snapshot_ids"):
+ problems.append(
+ f"a plain screenshot-only read invalidated {plain.structured['invalidated_snapshot_ids']}"
+ )
+
+ args = {"pid": smoke.pid, "element_token": token(state, increment)}
+ click = smoke.call("click", args)
+ after = smoke.wait_for_text(f"Count: {(before or 0) + 1}")
+ if click.is_error:
+ problems.append(f"click with the pre-preview token {args['element_token']} -> {click.first_line()}")
+ elif f"Count: {(before or 0) + 1}" not in tree(after):
+ problems.append(f"click ok but the count did not move from {before}")
+ smoke.record(
+ check,
+ "fail" if problems else "pass",
+ "; ".join(problems)
+ or f"schema advertises display_only; {DISPLAY_ONLY_READS} display_only reads of "
+ f"snapshot {snapshot} carried no snapshot_id/capture_id; a plain screenshot-only "
+ f"read invalidated nothing; click {args['element_token']} -> Count: {(before or 0) + 1}",
+ )
+
+
def lean_read_ergonomics_check(smoke):
check = "since:latest, bare row token, run_actions observe:true, zoom x/y/width/height, scroll dy"
problems = []
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
index ea5beeda3d..02a67e99a0 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
@@ -57,6 +57,10 @@
],
"type": "string"
},
+ "display_only": {
+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",
+ "type": "boolean"
+ },
"full_output": {
"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
"type": "boolean"
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 9a6d4955a3..e44a85b45e 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look). The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -57,6 +57,10 @@
],
"type": "string"
},
+ "display_only": {
+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",
+ "type": "boolean"
+ },
"full_output": {
"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
"type": "boolean"
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index 1599f4b114..34d41062a6 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -43,7 +43,7 @@
"read_only": true
},
{
- "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -57,6 +57,10 @@
],
"type": "string"
},
+ "display_only": {
+ "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.",
+ "type": "boolean"
+ },
"full_output": {
"description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.",
"type": "boolean"