Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): report Windows tree completeness

injaneityd0eebf0f4b52
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx
index 4204f5615f..6a9058dd3e 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/apps-windows-and-screen.mdx
@@ -492,7 +492,7 @@ The UIA tree walked is the window's tree (HWND-scoped); the screenshot and windo
`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.
-Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.
+Set `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the unprojected walked snapshot; `returned_element_count` reports the projection. `elements_complete` describes the underlying walk and is unaffected by query projection.
Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.
@@ -500,7 +500,7 @@ The mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk e
Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).
-Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.
+Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. `elements_complete` is false when a bound skips a node or subtree; ending exactly at a bound remains complete. Default: `max_elements` 250, depth ≤25.
CHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.
@@ -517,7 +517,7 @@ Windows requires no special permissions.
| <span id="get_window_state--windows--max_elements"></span>`max_elements` | `integer` | | Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window. Minimum: 1. |
| <span id="get_window_state--windows--max_image_dimension"></span>`max_image_dimension` | `integer` | | Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default. Minimum: 0. |
| <span id="get_window_state--windows--pid"></span>`pid` | `integer` | required | Process ID from `list_apps`. |
-| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. |
+| <span id="get_window_state--windows--query"></span>`query` | `string` | | Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk. |
| <span id="get_window_state--windows--screenshot_out_file"></span>`screenshot_out_file` | `string` | | When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead. |
| <span id="get_window_state--windows--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
| <span id="get_window_state--windows--window_id"></span>`window_id` | `integer` | required | HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array. |
diff --git a/libs/cua-driver/docs/tool-output-format.md b/libs/cua-driver/docs/tool-output-format.md
index 0ae00df560..f1c4106051 100644
--- a/libs/cua-driver/docs/tool-output-format.md
+++ b/libs/cua-driver/docs/tool-output-format.md
@@ -35,3 +35,16 @@ Observation tools retain their typed tool-specific structured payloads.
records in `structuredContent` and can attach a PNG as image content. A
multimodal harness interprets the image; Cua Driver does not OCR it or assign
task meaning.
+
+On Windows, `get_window_state.elements_complete` is true only when the
+unprojected UI Automation walk visited every exposed node without a traversal
+bound, timeout, or enumeration failure skipping a node or subtree. Reaching
+`max_elements` or `max_depth` exactly at the end of a tree remains complete.
+`query` projects the returned Markdown and structured element rows, but does
+not make the underlying snapshot incomplete; compare `total_element_count`
+with `returned_element_count` to measure that projection. Partial MSAA trees,
+unusable UIA trees, and process-root fallbacks remain incomplete because they
+cannot prove the exact target window's full UIA tree.
+Internal `verify_state` reads also remain incomplete while the Windows
+structured projection omits display-only UIA nodes. A complete native walk
+does not make that projection exhaustive.
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs
index c6a0740a38..ce915e57da 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_wpf_test.rs
@@ -339,7 +339,81 @@ fn harness_wpf_query_projects_structured_elements() {
returned < total,
"query did not compact {returned}/{total} elements"
);
+ assert_eq!(
+ response.structured()["elements_complete"].as_bool(),
+ Some(true),
+ "query projection must preserve the complete underlying UIA snapshot"
+ );
let _ = element_token_by_id(&response, "btn-increment");
+
+ let bounded = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "max_elements": 1,
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !bounded.is_error(),
+ "bounded snapshot failed: {}",
+ bounded.text()
+ );
+ assert_eq!(
+ bounded.structured()["elements_complete"].as_bool(),
+ Some(false),
+ "a bound that skips the WPF root's descendants must report an incomplete snapshot"
+ );
+
+ let shallow = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "max_depth": 1,
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !shallow.is_error(),
+ "depth-bounded snapshot failed: {}",
+ shallow.text()
+ );
+ assert_eq!(
+ shallow.structured()["elements_complete"].as_bool(),
+ Some(false),
+ "a depth bound that skips WPF descendants must report an incomplete snapshot"
+ );
+
+ let verified = driver.call(
+ "verify_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "expect": [{"element": {
+ "selector": {"label_contains": "__cua_missing_completeness_probe__"},
+ "exists": true
+ }}],
+ "timeout_ms": 0,
+ "stable_samples": 1,
+ "include_screenshot": false
+ }),
+ );
+ assert!(
+ !verified.is_error(),
+ "verification failed: {}",
+ verified.text()
+ );
+ assert_eq!(
+ verified.structured()["status"],
+ "unknown",
+ "a structured projection that omits display-only nodes cannot prove absence"
+ );
+ assert_eq!(
+ verified.structured()["predicates"][0]["unknown_reason"],
+ "observation_unavailable"
+ );
Observation::delivered(vec![OracleKind::AxState], Evidence::default())
},
);
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs
index 36339073be..0c18576156 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs
@@ -93,6 +93,7 @@ unsafe fn walk_unsafe(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {
"- Window <SAL/VCL — MSAA fallback failed (AccessibleObjectFromWindow hr={hr:?})>\n"
),
nodes: Vec::new(),
+ elements_complete: false,
};
}
let root: IAccessible = IAccessible::from_raw(raw_root);
@@ -107,6 +108,7 @@ unsafe fn walk_unsafe(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {
UiaTreeResult {
tree_markdown,
nodes,
+ elements_complete: false,
}
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index e29e1f6bed..65b6b507c8 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -1225,7 +1225,9 @@ impl Tool for GetWindowStateTool {
Set `query` to a case-insensitive substring to project the tree to matching \
rows plus their ancestor chain. \
Original element indices are preserved. `total_element_count` reports the \
- complete snapshot; `returned_element_count` reports the projection.\n\n\
+ unprojected walked snapshot; `returned_element_count` reports the projection. \
+ `elements_complete` describes the underlying walk and is unaffected by query \
+ projection.\n\n\
Always returns BOTH the element tree AND a screenshot — ground on both \
and cross-check (the tree lies on some surfaces). Choose the modality at \
ACTION time: an element ax action (element_token → \
@@ -1244,7 +1246,9 @@ impl Tool for GetWindowStateTool {
timing out at 4s with per-property RPCs).\n\n\
Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \
apps produce 10k+ element trees). Markdown and structured elements are \
- truncated identically. Default: `max_elements` 250, depth ≤25.\n\n\
+ truncated identically. `elements_complete` is false when a bound skips a node \
+ or subtree; ending exactly at a bound remains complete. Default: \
+ `max_elements` 250, depth ≤25.\n\n\
CHROMIUM COVERAGE: a browser-owned permission bubble can be \
composited outside the requested native window. Chromium-family \
snapshots therefore describe this limit in structuredContent.capture_coverage. \
@@ -1262,7 +1266,7 @@ impl Tool for GetWindowStateTool {
"include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
"include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},
- "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk."},
"max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},
"timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
@@ -1464,6 +1468,7 @@ impl Tool for GetWindowStateTool {
delivery_mode:\"foreground\" for a dialog's default accelerator.)"
),
nodes: Vec::new(),
+ elements_complete: false,
};
let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
&[],
@@ -1556,10 +1561,11 @@ impl Tool for GetWindowStateTool {
header + &tr.tree_markdown,
));
structured["element_count"] = json!(count);
- // UIA currently does not expose whether a bounded walk
- // exhausted every subtree. Keep negative existence
- // conservative until that proof is available.
- structured["elements_complete"] = json!(false);
+ // Verification consumes only structured rows, which still
+ // omit non-actionable UIA nodes on Windows. A complete
+ // native walk cannot prove absence in that projection.
+ structured["elements_complete"] =
+ json!(tr.elements_complete && !observation_only);
structured["tree_markdown"] = json!(tr.tree_markdown);
walk.apply(&mut structured);
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
index 7536aed789..04b9f0f2e1 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
@@ -16,7 +16,7 @@ use windows::Win32::System::Com::{
};
use windows::Win32::UI::Accessibility::{
CUIAutomation, IUIAutomation, IUIAutomationCacheRequest, IUIAutomationElement,
- IUIAutomationExpandCollapsePattern, IUIAutomationInvokePattern,
+ IUIAutomationElementArray, IUIAutomationExpandCollapsePattern, IUIAutomationInvokePattern,
IUIAutomationSelectionItemPattern, IUIAutomationTogglePattern, ToggleState_Off, ToggleState_On,
TreeScope, TreeScope_Children, TreeScope_Element, TreeScope_Subtree,
UIA_AutomationIdPropertyId, UIA_BoundingRectanglePropertyId, UIA_ControlTypePropertyId,
@@ -40,14 +40,6 @@ pub const DEFAULT_MAX_DEPTH: usize = 25;
/// Default cap; callers can override via [`walk_tree_bounded`].
pub const DEFAULT_MAX_TOTAL_ELEMENTS: usize = 5000;
-// Historical aliases — referenced by the thin `walk_cached` shim that
-// keeps the pre-#22865 call signature compiling. `walk_tree_bounded`
-// reads from the caller-supplied caps instead.
-#[allow(dead_code)]
-const MAX_DEPTH: usize = DEFAULT_MAX_DEPTH;
-#[allow(dead_code)]
-const MAX_TOTAL_ELEMENTS: usize = DEFAULT_MAX_TOTAL_ELEMENTS;
-
/// A single node in the accessibility tree.
///
/// Same shape for the UIA primary path AND the MSAA fallback (used for
@@ -98,6 +90,43 @@ pub struct UiaNode {
pub struct UiaTreeResult {
pub tree_markdown: String,
pub nodes: Vec<UiaNode>,
+ /// True when the unprojected native walk visited every node exposed by
+ /// the selected root without a bound or enumeration failure skipping a
+ /// node/subtree. A `query` only projects the response and does not change
+ /// this underlying snapshot property.
+ pub elements_complete: bool,
+}
+
+#[derive(Debug)]
+struct WalkState<'a> {
+ budget: &'a mut WalkBudget,
+ max_depth: usize,
+ complete: bool,
+}
+
+impl<'a> WalkState<'a> {
+ fn new(budget: &'a mut WalkBudget, max_depth: usize) -> Self {
+ Self {
+ budget,
+ max_depth,
+ complete: true,
+ }
+ }
+
+ /// Admit one node to the walk. Reaching a bound exactly is not itself
+ /// truncation; completeness changes only when a node beyond that bound is
+ /// actually encountered and skipped.
+ fn enter(&mut self, depth: usize) -> bool {
+ if depth > self.max_depth || !self.budget.admit() {
+ self.complete = false;
+ return false;
+ }
+ true
+ }
+
+ fn mark_incomplete(&mut self) {
+ self.complete = false;
+ }
}
/// Walk the UIA tree for the window with the given HWND.
@@ -273,28 +302,31 @@ unsafe fn walk_tree_unsafe(
return UiaTreeResult {
tree_markdown: format!("UIA init failed: {e}"),
nodes: Vec::new(),
+ elements_complete: false,
}
}
};
// Build a cache request that fetches everything we need in ONE bulk RPC.
- let cache_req = match node_cache_request(&automation) {
+ let (cache_req, mut cache_request_complete) = match node_cache_request(&automation) {
Ok(r) => r,
Err(e) => {
return UiaTreeResult {
tree_markdown: format!("CreateCacheRequest failed: {e}"),
nodes: Vec::new(),
+ elements_complete: false,
}
}
};
// Fetch entire subtree in one call.
- let _ = cache_req.SetTreeScope(TreeScope_Subtree);
+ cache_request_complete &= cache_req.SetTreeScope(TreeScope_Subtree).is_ok();
// Apply control-view filter (same as ControlViewWalker).
- if let Ok(ctrl_cond) = automation.ControlViewCondition() {
- let _ = cache_req.SetTreeFilter(&ctrl_cond);
- }
+ cache_request_complete &= automation
+ .ControlViewCondition()
+ .and_then(|condition| cache_req.SetTreeFilter(&condition))
+ .is_ok();
let hwnd_win = windows::Win32::Foundation::HWND(hwnd as *mut _);
@@ -338,6 +370,7 @@ unsafe fn walk_tree_unsafe(
return UiaTreeResult {
tree_markdown: format!("ElementFromHandle failed: {e}"),
nodes: Vec::new(),
+ elements_complete: false,
}
}
};
@@ -362,6 +395,7 @@ unsafe fn walk_tree_unsafe(
"BuildUpdatedCache failed after {attempt} attempts: {e}"
),
nodes: Vec::new(),
+ elements_complete: false,
};
}
std::thread::sleep(std::time::Duration::from_millis(40));
@@ -373,6 +407,10 @@ unsafe fn walk_tree_unsafe(
let mut nodes: Vec<UiaNode> = Vec::new();
let mut lines: Vec<(usize, String)> = Vec::new();
let mut counter = 0usize;
+ let mut walk_state = WalkState::new(budget, max_depth);
+ if !cache_request_complete {
+ walk_state.mark_incomplete();
+ }
walk_cached_bounded(
&root_elem,
@@ -382,8 +420,7 @@ unsafe fn walk_tree_unsafe(
&mut nodes,
&mut lines,
&mut counter,
- budget,
- max_depth,
+ &mut walk_state,
);
// Fallback for CoreWindow-class apps (Calculator, Settings, older UWPs).
@@ -399,37 +436,15 @@ unsafe fn walk_tree_unsafe(
//
// Stage the fallback walk into fresh accumulators and only swap them in
// if the fallback actually finds actionable elements. Otherwise the
- // wrapper-only node from the primary walk stays the result — better than
- // erasing it AND leaving the consumed `MAX_TOTAL_ELEMENTS` budget intact
- // for the fallback (which would then truncate large trees prematurely).
- if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 && !budget.expired() {
- // Skip the desktop-root walk-by-pid fallback for VCL / SAL
- // targets (LibreOffice, OpenOffice). The fallback does its own
- // `BuildUpdatedCache(TreeScope.Subtree)` per matched top-level
- // window — which is exactly the bulk-cache RPC shape that SAL
- // hangs on. The primary two-call path (ElementFromHandle +
- // BuildUpdatedCache on the dialog's own HWND) ALREADY dodged the
- // hang on that one specific call, but the fallback re-introduces
- // it. Returning the empty tree here lets the outer
- // get_window_state timeout fire its structured diagnostic
- // promptly instead of stalling 4 s on the fallback's hang.
- //
- // The diagnostic tells callers exactly how to drive the SAL
- // dialog without the tree: pixel click off the screenshot
- // get_window_state always returns, or press_key with
- // delivery_mode:"foreground" for accelerator-style dismissal. That's enough for the
- // common modal-dismissal case (Yes/No/Esc on a Confirmation),
- // which is what SAL dialogs almost always need.
- let is_sal = {
- use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
- let mut buf = [0u16; 64];
- let n = GetClassNameW(hwnd_win, &mut buf);
- n > 0 && {
- let class = String::from_utf16_lossy(&buf[..n as usize]);
- class.starts_with("SAL")
- }
- };
- if !is_sal {
+ // wrapper-only node from the primary walk stays the result. Both paths
+ // share the caller's time and node budget.
+ if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {
+ // An empty HWND wrapper cannot prove a complete window snapshot. The
+ // fallback may discover controls in other windows of the process, so
+ // it remains discovery evidence rather than an exact-window proof.
+ walk_state.mark_incomplete();
+ // SAL windows already returned through the MSAA path above.
+ if !walk_state.budget.expired() {
if let Some(target_pid) = pid_from_hwnd(hwnd_win) {
let mut fallback_nodes: Vec<UiaNode> = Vec::new();
let mut fallback_lines: Vec<(usize, String)> = Vec::new();
@@ -447,45 +462,14 @@ unsafe fn walk_tree_unsafe(
&mut fallback_nodes,
&mut fallback_lines,
&mut fallback_counter,
- budget,
- max_depth,
+ &mut walk_state,
);
if fallback_nodes.iter().any(|n| n.element_index.is_some()) {
nodes = fallback_nodes;
lines = fallback_lines;
- // counter isn't read after this point — it's only used
- // by walk_cached's &mut params for element indexing
- // inside that call. The budget spans both walks.
}
}
- } else {
- tracing::debug!(
- target: "uia",
- "SAL target hwnd 0x{hwnd:x} returned empty primary tree; \
- skipping walk_root_by_pid fallback (known to re-hang on SAL Subtree fetch). \
- Caller should use press_key/screenshot fallbacks per the get_window_state diagnostic."
- );
- // Return a tree_markdown that mirrors the get_window_state
- // timeout diagnostic so callers get the same actionable
- // fallback options even though the walk itself didn't hit
- // the timeout_ms walk budget (because we skipped the hang-prone
- // fallback). Without this the caller sees an empty tree
- // and no error, which is less actionable.
- let stub = format!(
- "- Window <SAL class — empty primary UIA tree>\n\
- (SAL providers don't expose modal-dialog children via \
- ElementFromHandle, and the desktop-root fallback walk that \
- would normally find them is known to hang on SAL Subtree \
- BuildUpdatedCache. Use one of: \
- (a) pixel `click(x, y)` off the screenshot `get_window_state` \
- returns alongside this tree; \
- (b) `press_key` with `delivery_mode:\"foreground\"` (Esc / Enter / Y / N).)\n"
- );
- return UiaTreeResult {
- tree_markdown: stub,
- nodes: Vec::new(),
- };
}
}
@@ -499,6 +483,7 @@ unsafe fn walk_tree_unsafe(
UiaTreeResult {
tree_markdown,
nodes,
+ elements_complete: walk_state.complete,
}
}
@@ -532,13 +517,13 @@ unsafe fn walk_root_by_pid(
nodes: &mut Vec<UiaNode>,
lines: &mut Vec<(usize, String)>,
counter: &mut usize,
- budget: &mut WalkBudget,
- max_depth: usize,
+ walk_state: &mut WalkState<'_>,
) {
let root = match automation.GetRootElement() {
Ok(r) => r,
Err(e) => {
tracing::debug!(target: "uia", "GetRootElement failed: {e}");
+ walk_state.mark_incomplete();
return;
}
};
@@ -546,6 +531,7 @@ unsafe fn walk_root_by_pid(
Ok(c) => c,
Err(e) => {
tracing::debug!(target: "uia", "CreateTrueCondition failed: {e}");
+ walk_state.mark_incomplete();
return;
}
};
@@ -553,14 +539,26 @@ unsafe fn walk_root_by_pid(
Ok(a) => a,
Err(e) => {
tracing::debug!(target: "uia", "root.FindAll(Children) failed: {e}");
+ walk_state.mark_incomplete();
+ return;
+ }
+ };
+ let count = match kids.Length() {
+ Ok(count) => count,
+ Err(e) => {
+ tracing::debug!(target: "uia", "root child count failed: {e}");
+ walk_state.mark_incomplete();
return;
}
};
- let count = kids.Length().unwrap_or(0);
for i in 0..count {
let elem = match kids.GetElement(i) {
Ok(e) => e,
- Err(_) => continue,
+ Err(e) => {
+ tracing::debug!(target: "uia", "root child {i} fetch failed: {e}");
+ walk_state.mark_incomplete();
+ continue;
+ }
};
// Read ProcessId without a cache — root.FindAll didn't use one.
// VARIANT for VT_I4 (UIA's ProcessId type) puts the int at
@@ -572,52 +570,35 @@ unsafe fn walk_root_by_pid(
if raw.Anonymous.Anonymous.vt != 3
/* VT_I4 */
{
+ walk_state.mark_incomplete();
continue;
}
raw.Anonymous.Anonymous.Anonymous.lVal as u32
}
- Err(_) => continue,
+ Err(e) => {
+ tracing::debug!(target: "uia", "root child {i} ProcessId read failed: {e}");
+ walk_state.mark_incomplete();
+ continue;
+ }
};
if pid != target_pid {
continue;
}
// Match — pull a cached subtree from this element using the same
- // cache_req shape as the primary path so walk_cached sees the same
+ // cache_req shape as the primary path so walk_cached_bounded sees the same
// properties + patterns.
let cached = match elem.BuildUpdatedCache(cache_req) {
Ok(e) => e,
Err(e) => {
tracing::debug!(target: "uia", "BuildUpdatedCache on pid={target_pid} match failed: {e}");
+ walk_state.mark_incomplete();
continue;
}
};
- walk_cached_bounded(
- &cached, 0, None, false, nodes, lines, counter, budget, max_depth,
- );
+ walk_cached_bounded(&cached, 0, None, false, nodes, lines, counter, walk_state);
}
}
-#[allow(dead_code)]
-unsafe fn walk_cached(
- element: &IUIAutomationElement,
- depth: usize,
- nodes: &mut Vec<UiaNode>,
- lines: &mut Vec<(usize, String)>,
- counter: &mut usize,
-) {
- walk_cached_bounded(
- element,
- depth,
- None,
- false,
- nodes,
- lines,
- counter,
- &mut WalkBudget::nodes_only(MAX_TOTAL_ELEMENTS),
- MAX_DEPTH,
- );
-}
-
#[allow(clippy::too_many_arguments)]
unsafe fn walk_cached_bounded(
element: &IUIAutomationElement,
@@ -627,12 +608,9 @@ unsafe fn walk_cached_bounded(
nodes: &mut Vec<UiaNode>,
lines: &mut Vec<(usize, String)>,
counter: &mut usize,
- budget: &mut WalkBudget,
- max_depth: usize,
+ walk_state: &mut WalkState<'_>,
) {
- // Node and time budget: a refused node is counted as discovered but not
- // visited, and the walk unwinds with the partial tree it has.
- if depth > max_depth || !budget.admit() {
+ if !walk_state.enter(depth) {
return;
}
@@ -646,24 +624,49 @@ unsafe fn walk_cached_bounded(
counter,
);
- // Recurse using cached children (no additional RPC).
- if let Ok(children) = element.GetCachedChildren() {
- let len = children.Length().unwrap_or(0);
- for i in 0..len {
- if let Ok(child) = children.GetElement(i) {
- walk_cached_bounded(
- &child,
- depth + 1,
- emitted_parent,
- in_web_content || control_type.eq_ignore_ascii_case("Document"),
- nodes,
- lines,
- counter,
- budget,
- max_depth,
- );
- }
+ let children = match cached_children(element) {
+ Ok(Some(children)) => children,
+ Ok(None) => return,
+ Err(_) => {
+ walk_state.mark_incomplete();
+ return;
+ }
+ };
+ let len = match children.Length() {
+ Ok(len) => len,
+ Err(_) => {
+ walk_state.mark_incomplete();
+ return;
}
+ };
+ for i in 0..len {
+ match children.GetElement(i) {
+ Ok(child) => walk_cached_bounded(
+ &child,
+ depth + 1,
+ emitted_parent,
+ in_web_content || control_type.eq_ignore_ascii_case("Document"),
+ nodes,
+ lines,
+ counter,
+ walk_state,
+ ),
+ Err(_) => walk_state.mark_incomplete(),
+ }
+ }
+}
+
+unsafe fn cached_children(
+ element: &IUIAutomationElement,
+) -> windows::core::Result<Option<IUIAutomationElementArray>> {
+ let mut raw = std::ptr::null_mut();
+ let result =
+ (Interface::vtable(element).GetCachedChildren)(Interface::as_raw(element), &mut raw);
+ result.ok()?;
+ if raw.is_null() {
+ Ok(None)
+ } else {
+ Ok(Some(IUIAutomationElementArray::from_raw(raw)))
}
}
@@ -671,8 +674,9 @@ unsafe fn walk_cached_bounded(
/// reads. Callers choose the tree scope and filter.
unsafe fn node_cache_request(
automation: &IUIAutomation,
-) -> windows::core::Result<IUIAutomationCacheRequest> {
+) -> windows::core::Result<(IUIAutomationCacheRequest, bool)> {
let cache_req = automation.CreateCacheRequest()?;
+ let mut complete = true;
for prop in &[
UIA_ControlTypePropertyId,
UIA_NamePropertyId,
@@ -685,7 +689,7 @@ unsafe fn node_cache_request(
UIA_ToggleToggleStatePropertyId,
UIA_SelectionItemIsSelectedPropertyId,
] {
- let _ = cache_req.AddProperty(*prop);
+ complete &= cache_req.AddProperty(*prop).is_ok();
}
for pat in &[
UIA_InvokePatternId,
@@ -697,9 +701,9 @@ unsafe fn node_cache_request(
UIA_TextPatternId,
UIA_ScrollPatternId,
] {
- let _ = cache_req.AddPattern(*pat);
+ complete &= cache_req.AddPattern(*pat).is_ok();
}
- Ok(cache_req)
+ Ok((cache_req, complete))
}
/// Walk only a window's native (non-renderer) accessibility tree.
@@ -722,7 +726,7 @@ pub fn walk_native_tree(hwnd: u64) -> Result<Vec<UiaNode>, String> {
let automation: IUIAutomation =
CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER)
.map_err(|e| format!("UIA init failed: {e}"))?;
- let cache_req = node_cache_request(&automation)
+ let (cache_req, _) = node_cache_request(&automation)
.map_err(|e| format!("CreateCacheRequest failed: {e}"))?;
let _ = cache_req.SetTreeScope(TreeScope(TreeScope_Element.0 | TreeScope_Children.0));
if let Ok(ctrl_cond) = automation.ControlViewCondition() {
@@ -1180,3 +1184,135 @@ fn filter_tree(markdown: &str, query: &str) -> String {
r.push('\n');
r
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn star(child_count: usize) -> Vec<Vec<usize>> {
+ let mut children = vec![Vec::new(); child_count + 1];
+ children[0] = (1..=child_count).collect();
+ children
+ }
+
+ fn chain(len: usize) -> Vec<Vec<usize>> {
+ let mut children = vec![Vec::new(); len];
+ for i in 0..len.saturating_sub(1) {
+ children[i] = vec![i + 1];
+ }
+ children
+ }
+
+ fn walk_synthetic_tree(
+ children: &[Vec<usize>],
+ node: usize,
+ depth: usize,
+ state: &mut WalkState<'_>,
+ ) {
+ if !state.enter(depth) {
+ return;
+ }
+ for child in &children[node] {
+ walk_synthetic_tree(children, *child, depth + 1, state);
+ }
+ }
+
+ #[test]
+ fn walk_state_is_complete_only_when_no_bound_skips_a_node() {
+ struct Case {
+ name: &'static str,
+ children: Vec<Vec<usize>>,
+ max_elements: usize,
+ max_depth: usize,
+ expected_visited: usize,
+ expected_complete: bool,
+ }
+
+ let cases = [
+ Case {
+ name: "default bounds admit a 55-node tree",
+ children: star(54),
+ max_elements: DEFAULT_MAX_TOTAL_ELEMENTS,
+ max_depth: DEFAULT_MAX_DEPTH,
+ expected_visited: 55,
+ expected_complete: true,
+ },
+ Case {
+ name: "visiting exactly max_elements is still complete",
+ children: star(54),
+ max_elements: 55,
+ max_depth: DEFAULT_MAX_DEPTH,
+ expected_visited: 55,
+ expected_complete: true,
+ },
+ Case {
+ name: "skipping a node past max_elements is incomplete",
+ children: star(54),
+ max_elements: 54,
+ max_depth: DEFAULT_MAX_DEPTH,
+ expected_visited: 54,
+ expected_complete: false,
+ },
+ Case {
+ name: "a leaf at exactly max_depth is still complete",
+ children: chain(3),
+ max_elements: 10,
+ max_depth: 2,
+ expected_visited: 3,
+ expected_complete: true,
+ },
+ Case {
+ name: "skipping a descendant past max_depth is incomplete",
+ children: chain(3),
+ max_elements: 10,
+ max_depth: 1,
+ expected_visited: 2,
+ expected_complete: false,
+ },
+ ];
+
+ for case in cases {
+ let mut budget = WalkBudget::nodes_only(case.max_elements);
+ let mut state = WalkState::new(&mut budget, case.max_depth);
+ walk_synthetic_tree(&case.children, 0, 0, &mut state);
+ assert_eq!(
+ state.budget.outcome().nodes_visited,
+ case.expected_visited,
+ "{}: visited",
+ case.name
+ );
+ assert_eq!(
+ state.complete, case.expected_complete,
+ "{}: complete",
+ case.name
+ );
+ }
+ }
+
+ #[test]
+ fn walk_state_keeps_the_callers_budget_across_walks() {
+ let mut budget = WalkBudget::nodes_only(2);
+ assert!(budget.admit());
+ let mut state = WalkState::new(&mut budget, DEFAULT_MAX_DEPTH);
+
+ walk_synthetic_tree(&star(1), 0, 0, &mut state);
+
+ assert!(!state.complete);
+ let outcome = state.budget.outcome();
+ assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (2, 1));
+ assert_eq!(outcome.reason(), Some("node_budget"));
+ }
+
+ #[test]
+ fn timed_out_walk_cannot_report_complete() {
+ let mut budget = WalkBudget::new(100, 10);
+ budget.stop_for_timeout();
+ let mut state = WalkState::new(&mut budget, DEFAULT_MAX_DEPTH);
+
+ walk_synthetic_tree(&star(0), 0, 0, &mut state);
+
+ assert!(!state.complete);
+ assert_eq!(state.budget.outcome().reason(), Some("timeout"));
+ assert_eq!(state.budget.outcome().nodes_visited, 0);
+ }
+}
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index 354a65854d..60205ec695 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -47,7 +47,7 @@
"read_only": true
},
{
- "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the complete snapshot; `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
+ "description": "Walk a running app's UIA tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), or `\"both\"` for both (about twice the size).\n\nSize: a read is bounded by default (`max_elements` 250). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the platform's walk limits (≤5 000 elements, depth ≤25).\n\nDIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Windows yet.\n\nINVARIANT: call `get_window_state` once per turn per (pid, window_id) before any element action against that window. The next snapshot of the same (pid, window_id) replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`. A screenshot-only read (`include_accessibility_tree:false`) keeps the current snapshot's rows and tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\nThe UIA tree walked is the window's tree (HWND-scoped); the screenshot and window bounds reported come from the same `window_id`. This is the source of truth for which window the caller intends to reason about — the driver never picks a window implicitly.\n\n`window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. The driver does not auto-fall-back to a different window.\n\nSet `query` to a case-insensitive substring to project the tree to matching rows plus their ancestor chain. Original element indices are preserved. `total_element_count` reports the unprojected walked snapshot; `returned_element_count` reports the projection. `elements_complete` describes the underlying walk and is unaffected by query projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; pass 0 for native resolution. Legacy `max_dimension` remains a cap on the configured limit.\n\nUses `IUIAutomationCacheRequest` to batch-fetch all element properties in a single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of timing out at 4s with per-property RPCs).\n\nOptional `max_elements` / `max_depth` bound the UIA walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. `elements_complete` is false when a bound skips a node or subtree; ending exactly at a bound remains complete. Default: `max_elements` 250, depth ≤25.\n\nCHROMIUM COVERAGE: a browser-owned permission bubble can be composited outside the requested native window. Chromium-family snapshots therefore describe this limit in structuredContent.capture_coverage. After a verified ineffective window action, call escalate_session, take a fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, then verify with another fresh desktop snapshot. This is separate from page JavaScript dialogs, which remain on browser_dialog.\n\nWindows requires no special permissions.",
"destructive": false,
"idempotent": false,
"input_schema": {
@@ -102,7 +102,7 @@
"type": "integer"
},
"query": {
- "description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.",
+ "description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk.",
"type": "string"
},
"screenshot_out_file": {