Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): report Windows tree completeness
injaneityd0eebf0f4b52
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdxindex 4204f5615f..6a9058dd3e 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx@@ -492,7 +492,7 @@ The UIA tree walked is the window's tree (HWND-scoped); the screenshot and windo`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.-Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.+Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the unprojected walked snapshot; `returned_element_count` reports the projection. `elements_complete` describes the underlying walk and is unaffected by query projection.Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.@@ -500,7 +500,7 @@ The mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk eUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).-Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.+Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. `elements_complete` is false when a bound skips a node or subtree; ending exactly at a bound remains complete. Default: `max_elements` 250, depth ≤25.CHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.@@ -517,7 +517,7 @@ Windows requires no special permissions.| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. || <span id="get_window_state--windows--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default. Minimum: 0. || <span id="get_window_state--windows--pid"></span>`pid` | `integer` | required | Process ID from `list_apps`. |-| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |+| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk. || <span id="get_window_state--windows--screenshot_out_file"></span>`screenshot_out_file` | `string` | | When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead. || <span id="get_window_state--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. || <span id="get_window_state--windows--window_id"></span>`window_id` | `integer` | required | HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array. |diff --git a/libs/cua-driver/docs/tool-output-format.md b/libs/cua-driver/docs/tool-output-format.mdindex 0ae00df560..f1c4106051 100644--- a/libs/cua-driver/docs/tool-output-format.md+++ b/libs/cua-driver/docs/tool-output-format.md@@ -35,3 +35,16 @@ Observation tools retain their typed tool-specific structured payloads.records in `structuredContent` and can attach a PNG as image content. Amultimodal harness interprets the image; Cua Driver does not OCR it or assigntask meaning.++On Windows, `get_window_state.elements_complete` is true only when the+unprojected UI Automation walk visited every exposed node without a traversal+bound, timeout, or enumeration failure skipping a node or subtree. Reaching+`max_elements` or `max_depth` exactly at the end of a tree remains complete.+`query` projects the returned Markdown and structured element rows, but does+not make the underlying snapshot incomplete; compare `total_element_count`+with `returned_element_count` to measure that projection. Partial MSAA trees,+unusable UIA trees, and process-root fallbacks remain incomplete because they+cannot prove the exact target window's full UIA tree.+Internal `verify_state` reads also remain incomplete while the Windows+structured projection omits display-only UIA nodes. A complete native walk+does not make that projection exhaustive.diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rsindex c6a0740a38..ce915e57da 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs@@ -339,7 +339,81 @@ fn harness_wpf_query_projects_structured_elements() {returned < total,"query did not compact {returned}/{total} elements");+ assert_eq!(+ response.structured()["elements_complete"].as_bool(),+ Some(true),+ "query projection must preserve the complete underlying UIA snapshot"+ );let _ = element_token_by_id(&response, "btn-increment");++ let bounded = driver.call(+ "get_window_state",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "max_elements": 1,+ "include_screenshot": false+ }),+ );+ assert!(+ !bounded.is_error(),+ "bounded snapshot failed: {}",+ bounded.text()+ );+ assert_eq!(+ bounded.structured()["elements_complete"].as_bool(),+ Some(false),+ "a bound that skips the WPF root's descendants must report an incomplete snapshot"+ );++ let shallow = driver.call(+ "get_window_state",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "max_depth": 1,+ "include_screenshot": false+ }),+ );+ assert!(+ !shallow.is_error(),+ "depth-bounded snapshot failed: {}",+ shallow.text()+ );+ assert_eq!(+ shallow.structured()["elements_complete"].as_bool(),+ Some(false),+ "a depth bound that skips WPF descendants must report an incomplete snapshot"+ );++ let verified = driver.call(+ "verify_state",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "expect": [{"element": {+ "selector": {"label_contains": "__cua_missing_completeness_probe__"},+ "exists": true+ }}],+ "timeout_ms": 0,+ "stable_samples": 1,+ "include_screenshot": false+ }),+ );+ assert!(+ !verified.is_error(),+ "verification failed: {}",+ verified.text()+ );+ assert_eq!(+ verified.structured()["status"],+ "unknown",+ "a structured projection that omits display-only nodes cannot prove absence"+ );+ assert_eq!(+ verified.structured()["predicates"][0]["unknown_reason"],+ "observation_unavailable"+ );Observation::delivered(vec![OracleKind::AxState], Evidence::default())},);diff --git a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rsindex 36339073be..0c18576156 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs@@ -93,6 +93,7 @@ unsafe fn walk_unsafe(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {"- Window <SAL/VCL — MSAA fallback failed (AccessibleObjectFromWindow hr={hr:?})>\n"),nodes: Vec::new(),+ elements_complete: false,};}let root: IAccessible = IAccessible::from_raw(raw_root);@@ -107,6 +108,7 @@ unsafe fn walk_unsafe(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {UiaTreeResult {tree_markdown,nodes,+ elements_complete: false,}}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex e29e1f6bed..65b6b507c8 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -1225,7 +1225,9 @@ impl Tool for GetWindowStateTool {Set `query` to a case-insensitive substring to project the tree to matching \rows plus their ancestor chain. \Original element indices are preserved. `total_element_count` reports the \- complete snapshot; `returned_element_count` reports the projection.\n\n\+ unprojected walked snapshot; `returned_element_count` reports the projection. \+ `elements_complete` describes the underlying walk and is unaffected by query \+ projection.\n\n\Always returns BOTH the element tree AND a screenshot — ground on both \and cross-check (the tree lies on some surfaces). Choose the modality at \ACTION time: an element ax action (element_token → \@@ -1244,7 +1246,9 @@ impl Tool for GetWindowStateTool {timing out at 4s with per-property RPCs).\n\n\Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \apps produce 10k+ element trees). Markdown and structured elements are \- truncated identically. Default: `max_elements` 250, depth ≤25.\n\n\+ truncated identically. `elements_complete` is false when a bound skips a node \+ or subtree; ending exactly at a bound remains complete. Default: \+ `max_elements` 250, depth ≤25.\n\n\CHROMIUM COVERAGE: a browser-owned permission bubble can be \composited outside the requested native window. Chromium-family \snapshots therefore describe this limit in structuredContent.capture_coverage. \@@ -1262,7 +1266,7 @@ impl Tool for GetWindowStateTool {"include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},- "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk."},"max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),@@ -1464,6 +1468,7 @@ impl Tool for GetWindowStateTool {delivery_mode:\"foreground\" for a dialog's default accelerator.)"),nodes: Vec::new(),+ elements_complete: false,};let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(&[],@@ -1556,10 +1561,11 @@ impl Tool for GetWindowStateTool {header + &tr.tree_markdown,));structured["element_count"] = json!(count);- // UIA currently does not expose whether a bounded walk- // exhausted every subtree. Keep negative existence- // conservative until that proof is available.- structured["elements_complete"] = json!(false);+ // Verification consumes only structured rows, which still+ // omit non-actionable UIA nodes on Windows. A complete+ // native walk cannot prove absence in that projection.+ structured["elements_complete"] =+ json!(tr.elements_complete && !observation_only);structured["tree_markdown"] = json!(tr.tree_markdown);walk.apply(&mut structured);diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rsindex 7536aed789..04b9f0f2e1 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs@@ -16,7 +16,7 @@ use windows::Win32::System::Com::{};use windows::Win32::UI::Accessibility::{CUIAutomation, IUIAutomation, IUIAutomationCacheRequest, IUIAutomationElement,- IUIAutomationExpandCollapsePattern, IUIAutomationInvokePattern,+ IUIAutomationElementArray, IUIAutomationExpandCollapsePattern, IUIAutomationInvokePattern,IUIAutomationSelectionItemPattern, IUIAutomationTogglePattern, ToggleState_Off, ToggleState_On,TreeScope, TreeScope_Children, TreeScope_Element, TreeScope_Subtree,UIA_AutomationIdPropertyId, UIA_BoundingRectanglePropertyId, UIA_ControlTypePropertyId,@@ -40,14 +40,6 @@ pub const DEFAULT_MAX_DEPTH: usize = 25;/// Default cap; callers can override via [`walk_tree_bounded`].pub const DEFAULT_MAX_TOTAL_ELEMENTS: usize = 5000;-// Historical aliases — referenced by the thin `walk_cached` shim that-// keeps the pre-#22865 call signature compiling. `walk_tree_bounded`-// reads from the caller-supplied caps instead.-#[allow(dead_code)]-const MAX_DEPTH: usize = DEFAULT_MAX_DEPTH;-#[allow(dead_code)]-const MAX_TOTAL_ELEMENTS: usize = DEFAULT_MAX_TOTAL_ELEMENTS;-/// A single node in the accessibility tree.////// Same shape for the UIA primary path AND the MSAA fallback (used for@@ -98,6 +90,43 @@ pub struct UiaNode {pub struct UiaTreeResult {pub tree_markdown: String,pub nodes: Vec<UiaNode>,+ /// True when the unprojected native walk visited every node exposed by+ /// the selected root without a bound or enumeration failure skipping a+ /// node/subtree. A `query` only projects the response and does not change+ /// this underlying snapshot property.+ pub elements_complete: bool,+}++#[derive(Debug)]+struct WalkState<'a> {+ budget: &'a mut WalkBudget,+ max_depth: usize,+ complete: bool,+}++impl<'a> WalkState<'a> {+ fn new(budget: &'a mut WalkBudget, max_depth: usize) -> Self {+ Self {+ budget,+ max_depth,+ complete: true,+ }+ }++ /// Admit one node to the walk. Reaching a bound exactly is not itself+ /// truncation; completeness changes only when a node beyond that bound is+ /// actually encountered and skipped.+ fn enter(&mut self, depth: usize) -> bool {+ if depth > self.max_depth || !self.budget.admit() {+ self.complete = false;+ return false;+ }+ true+ }++ fn mark_incomplete(&mut self) {+ self.complete = false;+ }}/// Walk the UIA tree for the window with the given HWND.@@ -273,28 +302,31 @@ unsafe fn walk_tree_unsafe(return UiaTreeResult {tree_markdown: format!("UIA init failed: {e}"),nodes: Vec::new(),+ elements_complete: false,}}};// Build a cache request that fetches everything we need in ONE bulk RPC.- let cache_req = match node_cache_request(&automation) {+ let (cache_req, mut cache_request_complete) = match node_cache_request(&automation) {Ok(r) => r,Err(e) => {return UiaTreeResult {tree_markdown: format!("CreateCacheRequest failed: {e}"),nodes: Vec::new(),+ elements_complete: false,}}};// Fetch entire subtree in one call.- let _ = cache_req.SetTreeScope(TreeScope_Subtree);+ cache_request_complete &= cache_req.SetTreeScope(TreeScope_Subtree).is_ok();// Apply control-view filter (same as ControlViewWalker).- if let Ok(ctrl_cond) = automation.ControlViewCondition() {- let _ = cache_req.SetTreeFilter(&ctrl_cond);- }+ cache_request_complete &= automation+ .ControlViewCondition()+ .and_then(|condition| cache_req.SetTreeFilter(&condition))+ .is_ok();let hwnd_win = windows::Win32::Foundation::HWND(hwnd as *mut _);@@ -338,6 +370,7 @@ unsafe fn walk_tree_unsafe(return UiaTreeResult {tree_markdown: format!("ElementFromHandle failed: {e}"),nodes: Vec::new(),+ elements_complete: false,}}};@@ -362,6 +395,7 @@ unsafe fn walk_tree_unsafe("BuildUpdatedCache failed after {attempt} attempts: {e}"),nodes: Vec::new(),+ elements_complete: false,};}std::thread::sleep(std::time::Duration::from_millis(40));@@ -373,6 +407,10 @@ unsafe fn walk_tree_unsafe(let mut nodes: Vec<UiaNode> = Vec::new();let mut lines: Vec<(usize, String)> = Vec::new();let mut counter = 0usize;+ let mut walk_state = WalkState::new(budget, max_depth);+ if !cache_request_complete {+ walk_state.mark_incomplete();+ }walk_cached_bounded(&root_elem,@@ -382,8 +420,7 @@ unsafe fn walk_tree_unsafe(&mut nodes,&mut lines,&mut counter,- budget,- max_depth,+ &mut walk_state,);// Fallback for CoreWindow-class apps (Calculator, Settings, older UWPs).@@ -399,37 +436,15 @@ unsafe fn walk_tree_unsafe(//// Stage the fallback walk into fresh accumulators and only swap them in// if the fallback actually finds actionable elements. Otherwise the- // wrapper-only node from the primary walk stays the result — better than- // erasing it AND leaving the consumed `MAX_TOTAL_ELEMENTS` budget intact- // for the fallback (which would then truncate large trees prematurely).- if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 && !budget.expired() {- // Skip the desktop-root walk-by-pid fallback for VCL / SAL- // targets (LibreOffice, OpenOffice). The fallback does its own- // `BuildUpdatedCache(TreeScope.Subtree)` per matched top-level- // window — which is exactly the bulk-cache RPC shape that SAL- // hangs on. The primary two-call path (ElementFromHandle +- // BuildUpdatedCache on the dialog's own HWND) ALREADY dodged the- // hang on that one specific call, but the fallback re-introduces- // it. Returning the empty tree here lets the outer- // get_window_state timeout fire its structured diagnostic- // promptly instead of stalling 4 s on the fallback's hang.- //- // The diagnostic tells callers exactly how to drive the SAL- // dialog without the tree: pixel click off the screenshot- // get_window_state always returns, or press_key with- // delivery_mode:"foreground" for accelerator-style dismissal. That's enough for the- // common modal-dismissal case (Yes/No/Esc on a Confirmation),- // which is what SAL dialogs almost always need.- let is_sal = {- use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;- let mut buf = [0u16; 64];- let n = GetClassNameW(hwnd_win, &mut buf);- n > 0 && {- let class = String::from_utf16_lossy(&buf[..n as usize]);- class.starts_with("SAL")- }- };- if !is_sal {+ // wrapper-only node from the primary walk stays the result. Both paths+ // share the caller's time and node budget.+ if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {+ // An empty HWND wrapper cannot prove a complete window snapshot. The+ // fallback may discover controls in other windows of the process, so+ // it remains discovery evidence rather than an exact-window proof.+ walk_state.mark_incomplete();+ // SAL windows already returned through the MSAA path above.+ if !walk_state.budget.expired() {if let Some(target_pid) = pid_from_hwnd(hwnd_win) {let mut fallback_nodes: Vec<UiaNode> = Vec::new();let mut fallback_lines: Vec<(usize, String)> = Vec::new();@@ -447,45 +462,14 @@ unsafe fn walk_tree_unsafe(&mut fallback_nodes,&mut fallback_lines,&mut fallback_counter,- budget,- max_depth,+ &mut walk_state,);if fallback_nodes.iter().any(|n| n.element_index.is_some()) {nodes = fallback_nodes;lines = fallback_lines;- // counter isn't read after this point — it's only used- // by walk_cached's &mut params for element indexing- // inside that call. The budget spans both walks.}}- } else {- tracing::debug!(- target: "uia",- "SAL target hwnd 0x{hwnd:x} returned empty primary tree; \- skipping walk_root_by_pid fallback (known to re-hang on SAL Subtree fetch). \- Caller should use press_key/screenshot fallbacks per the get_window_state diagnostic."- );- // Return a tree_markdown that mirrors the get_window_state- // timeout diagnostic so callers get the same actionable- // fallback options even though the walk itself didn't hit- // the timeout_ms walk budget (because we skipped the hang-prone- // fallback). Without this the caller sees an empty tree- // and no error, which is less actionable.- let stub = format!(- "- Window <SAL class — empty primary UIA tree>\n\- (SAL providers don't expose modal-dialog children via \- ElementFromHandle, and the desktop-root fallback walk that \- would normally find them is known to hang on SAL Subtree \- BuildUpdatedCache. Use one of: \- (a) pixel `click(x, y)` off the screenshot `get_window_state` \- returns alongside this tree; \- (b) `press_key` with `delivery_mode:\"foreground\"` (Esc / Enter / Y / N).)\n"- );- return UiaTreeResult {- tree_markdown: stub,- nodes: Vec::new(),- };}}@@ -499,6 +483,7 @@ unsafe fn walk_tree_unsafe(UiaTreeResult {tree_markdown,nodes,+ elements_complete: walk_state.complete,}}@@ -532,13 +517,13 @@ unsafe fn walk_root_by_pid(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- budget: &mut WalkBudget,- max_depth: usize,+ walk_state: &mut WalkState<'_>,) {let root = match automation.GetRootElement() {Ok(r) => r,Err(e) => {tracing::debug!(target: "uia", "GetRootElement failed: {e}");+ walk_state.mark_incomplete();return;}};@@ -546,6 +531,7 @@ unsafe fn walk_root_by_pid(Ok(c) => c,Err(e) => {tracing::debug!(target: "uia", "CreateTrueCondition failed: {e}");+ walk_state.mark_incomplete();return;}};@@ -553,14 +539,26 @@ unsafe fn walk_root_by_pid(Ok(a) => a,Err(e) => {tracing::debug!(target: "uia", "root.FindAll(Children) failed: {e}");+ walk_state.mark_incomplete();+ return;+ }+ };+ let count = match kids.Length() {+ Ok(count) => count,+ Err(e) => {+ tracing::debug!(target: "uia", "root child count failed: {e}");+ walk_state.mark_incomplete();return;}};- let count = kids.Length().unwrap_or(0);for i in 0..count {let elem = match kids.GetElement(i) {Ok(e) => e,- Err(_) => continue,+ Err(e) => {+ tracing::debug!(target: "uia", "root child {i} fetch failed: {e}");+ walk_state.mark_incomplete();+ continue;+ }};// Read ProcessId without a cache — root.FindAll didn't use one.// VARIANT for VT_I4 (UIA's ProcessId type) puts the int at@@ -572,52 +570,35 @@ unsafe fn walk_root_by_pid(if raw.Anonymous.Anonymous.vt != 3/* VT_I4 */{+ walk_state.mark_incomplete();continue;}raw.Anonymous.Anonymous.Anonymous.lVal as u32}- Err(_) => continue,+ Err(e) => {+ tracing::debug!(target: "uia", "root child {i} ProcessId read failed: {e}");+ walk_state.mark_incomplete();+ continue;+ }};if pid != target_pid {continue;}// Match — pull a cached subtree from this element using the same- // cache_req shape as the primary path so walk_cached sees the same+ // cache_req shape as the primary path so walk_cached_bounded sees the same// properties + patterns.let cached = match elem.BuildUpdatedCache(cache_req) {Ok(e) => e,Err(e) => {tracing::debug!(target: "uia", "BuildUpdatedCache on pid={target_pid} match failed: {e}");+ walk_state.mark_incomplete();continue;}};- walk_cached_bounded(- &cached, 0, None, false, nodes, lines, counter, budget, max_depth,- );+ walk_cached_bounded(&cached, 0, None, false, nodes, lines, counter, walk_state);}}-#[allow(dead_code)]-unsafe fn walk_cached(- element: &IUIAutomationElement,- depth: usize,- nodes: &mut Vec<UiaNode>,- lines: &mut Vec<(usize, String)>,- counter: &mut usize,-) {- walk_cached_bounded(- element,- depth,- None,- false,- nodes,- lines,- counter,- &mut WalkBudget::nodes_only(MAX_TOTAL_ELEMENTS),- MAX_DEPTH,- );-}-#[allow(clippy::too_many_arguments)]unsafe fn walk_cached_bounded(element: &IUIAutomationElement,@@ -627,12 +608,9 @@ unsafe fn walk_cached_bounded(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- budget: &mut WalkBudget,- max_depth: usize,+ walk_state: &mut WalkState<'_>,) {- // Node and time budget: a refused node is counted as discovered but not- // visited, and the walk unwinds with the partial tree it has.- if depth > max_depth || !budget.admit() {+ if !walk_state.enter(depth) {return;}@@ -646,24 +624,49 @@ unsafe fn walk_cached_bounded(counter,);- // Recurse using cached children (no additional RPC).- if let Ok(children) = element.GetCachedChildren() {- let len = children.Length().unwrap_or(0);- for i in 0..len {- if let Ok(child) = children.GetElement(i) {- walk_cached_bounded(- &child,- depth + 1,- emitted_parent,- in_web_content || control_type.eq_ignore_ascii_case("Document"),- nodes,- lines,- counter,- budget,- max_depth,- );- }+ let children = match cached_children(element) {+ Ok(Some(children)) => children,+ Ok(None) => return,+ Err(_) => {+ walk_state.mark_incomplete();+ return;+ }+ };+ let len = match children.Length() {+ Ok(len) => len,+ Err(_) => {+ walk_state.mark_incomplete();+ return;}+ };+ for i in 0..len {+ match children.GetElement(i) {+ Ok(child) => walk_cached_bounded(+ &child,+ depth + 1,+ emitted_parent,+ in_web_content || control_type.eq_ignore_ascii_case("Document"),+ nodes,+ lines,+ counter,+ walk_state,+ ),+ Err(_) => walk_state.mark_incomplete(),+ }+ }+}++unsafe fn cached_children(+ element: &IUIAutomationElement,+) -> windows::core::Result<Option<IUIAutomationElementArray>> {+ let mut raw = std::ptr::null_mut();+ let result =+ (Interface::vtable(element).GetCachedChildren)(Interface::as_raw(element), &mut raw);+ result.ok()?;+ if raw.is_null() {+ Ok(None)+ } else {+ Ok(Some(IUIAutomationElementArray::from_raw(raw)))}}@@ -671,8 +674,9 @@ unsafe fn walk_cached_bounded(/// reads. Callers choose the tree scope and filter.unsafe fn node_cache_request(automation: &IUIAutomation,-) -> windows::core::Result<IUIAutomationCacheRequest> {+) -> windows::core::Result<(IUIAutomationCacheRequest, bool)> {let cache_req = automation.CreateCacheRequest()?;+ let mut complete = true;for prop in &[UIA_ControlTypePropertyId,UIA_NamePropertyId,@@ -685,7 +689,7 @@ unsafe fn node_cache_request(UIA_ToggleToggleStatePropertyId,UIA_SelectionItemIsSelectedPropertyId,] {- let _ = cache_req.AddProperty(*prop);+ complete &= cache_req.AddProperty(*prop).is_ok();}for pat in &[UIA_InvokePatternId,@@ -697,9 +701,9 @@ unsafe fn node_cache_request(UIA_TextPatternId,UIA_ScrollPatternId,] {- let _ = cache_req.AddPattern(*pat);+ complete &= cache_req.AddPattern(*pat).is_ok();}- Ok(cache_req)+ Ok((cache_req, complete))}/// Walk only a window's native (non-renderer) accessibility tree.@@ -722,7 +726,7 @@ pub fn walk_native_tree(hwnd: u64) -> Result<Vec<UiaNode>, String> {let automation: IUIAutomation =CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER).map_err(|e| format!("UIA init failed: {e}"))?;- let cache_req = node_cache_request(&automation)+ let (cache_req, _) = node_cache_request(&automation).map_err(|e| format!("CreateCacheRequest failed: {e}"))?;let _ = cache_req.SetTreeScope(TreeScope(TreeScope_Element.0 | TreeScope_Children.0));if let Ok(ctrl_cond) = automation.ControlViewCondition() {@@ -1180,3 +1184,135 @@ fn filter_tree(markdown: &str, query: &str) -> String {r.push('\n');r}++#[cfg(test)]+mod tests {+ use super::*;++ fn star(child_count: usize) -> Vec<Vec<usize>> {+ let mut children = vec![Vec::new(); child_count + 1];+ children[0] = (1..=child_count).collect();+ children+ }++ fn chain(len: usize) -> Vec<Vec<usize>> {+ let mut children = vec![Vec::new(); len];+ for i in 0..len.saturating_sub(1) {+ children[i] = vec![i + 1];+ }+ children+ }++ fn walk_synthetic_tree(+ children: &[Vec<usize>],+ node: usize,+ depth: usize,+ state: &mut WalkState<'_>,+ ) {+ if !state.enter(depth) {+ return;+ }+ for child in &children[node] {+ walk_synthetic_tree(children, *child, depth + 1, state);+ }+ }++ #[test]+ fn walk_state_is_complete_only_when_no_bound_skips_a_node() {+ struct Case {+ name: &'static str,+ children: Vec<Vec<usize>>,+ max_elements: usize,+ max_depth: usize,+ expected_visited: usize,+ expected_complete: bool,+ }++ let cases = [+ Case {+ name: "default bounds admit a 55-node tree",+ children: star(54),+ max_elements: DEFAULT_MAX_TOTAL_ELEMENTS,+ max_depth: DEFAULT_MAX_DEPTH,+ expected_visited: 55,+ expected_complete: true,+ },+ Case {+ name: "visiting exactly max_elements is still complete",+ children: star(54),+ max_elements: 55,+ max_depth: DEFAULT_MAX_DEPTH,+ expected_visited: 55,+ expected_complete: true,+ },+ Case {+ name: "skipping a node past max_elements is incomplete",+ children: star(54),+ max_elements: 54,+ max_depth: DEFAULT_MAX_DEPTH,+ expected_visited: 54,+ expected_complete: false,+ },+ Case {+ name: "a leaf at exactly max_depth is still complete",+ children: chain(3),+ max_elements: 10,+ max_depth: 2,+ expected_visited: 3,+ expected_complete: true,+ },+ Case {+ name: "skipping a descendant past max_depth is incomplete",+ children: chain(3),+ max_elements: 10,+ max_depth: 1,+ expected_visited: 2,+ expected_complete: false,+ },+ ];++ for case in cases {+ let mut budget = WalkBudget::nodes_only(case.max_elements);+ let mut state = WalkState::new(&mut budget, case.max_depth);+ walk_synthetic_tree(&case.children, 0, 0, &mut state);+ assert_eq!(+ state.budget.outcome().nodes_visited,+ case.expected_visited,+ "{}: visited",+ case.name+ );+ assert_eq!(+ state.complete, case.expected_complete,+ "{}: complete",+ case.name+ );+ }+ }++ #[test]+ fn walk_state_keeps_the_callers_budget_across_walks() {+ let mut budget = WalkBudget::nodes_only(2);+ assert!(budget.admit());+ let mut state = WalkState::new(&mut budget, DEFAULT_MAX_DEPTH);++ walk_synthetic_tree(&star(1), 0, 0, &mut state);++ assert!(!state.complete);+ let outcome = state.budget.outcome();+ assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (2, 1));+ assert_eq!(outcome.reason(), Some("node_budget"));+ }++ #[test]+ fn timed_out_walk_cannot_report_complete() {+ let mut budget = WalkBudget::new(100, 10);+ budget.stop_for_timeout();+ let mut state = WalkState::new(&mut budget, DEFAULT_MAX_DEPTH);++ walk_synthetic_tree(&star(0), 0, 0, &mut state);++ assert!(!state.complete);+ assert_eq!(state.budget.outcome().reason(), Some("timeout"));+ assert_eq!(state.budget.outcome().nodes_visited, 0);+ }+}diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.jsonindex 354a65854d..60205ec695 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json@@ -47,7 +47,7 @@"read_only": true},{- "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the unprojected walked snapshot; `returned_element_count` reports the projection. `elements_complete` describes the underlying walk and is unaffected by query projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. `elements_complete` is false when a bound skips a node or subtree; ending exactly at a bound remains complete. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.","destructive": false,"idempotent": false,"input_schema": {@@ -102,7 +102,7 @@"type": "integer"},"query": {- "description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.",+ "description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk.","type": "string"},"screenshot_out_file": {