Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(list_windows): UIA-first top-level window enumeration on Windows (#1542)

Francesco Bonacci70d899c39cd4
diff --git a/libs/cua-driver-rs/PARITY.md b/libs/cua-driver-rs/PARITY.md
index bd5d3043c8..b08f5d14d2 100644
--- a/libs/cua-driver-rs/PARITY.md
+++ b/libs/cua-driver-rs/PARITY.md
@@ -362,11 +362,36 @@ at least one entry has `active: true` (the foreground app).
- windows=`crates/platform-windows/src/tools/impl_.rs` (ListWindowsTool)
- linux=`crates/platform-linux/src/tools/impl_.rs` (ListWindowsTool, blocked behind Linux compile fix)
- Status:
- - windows: VERIFIED
+ - windows: VERIFIED (EnumWindows-first enumeration; UIA contributes missing HWNDs only)
- macOS: OPEN (audit pending — macOS port already exists)
- linux: OPEN
- Test: `crates/platform-windows/examples/list_windows_parity.rs`
+### Enumeration source (Windows)
+
+`crate::win32::list_windows` walks `EnumWindows` first — that's the Win32
+window manager's canonical top-to-bottom z-order, so it's the authoritative
+source for both window membership and ordering. It then asks UI Automation
+for any top-level windows EnumWindows missed
+(`AutomationElement::RootElement.FindAll(TreeScope::Children, TrueCondition)`,
+filtered to `IsOffscreen == false` + non-empty `GetWindowTextW`) and appends
+those at the end of the merged list, deduped by HWND. UIA elements contribute
+their `NativeWindowHandle` as the canonical HWND so downstream code keyed on
+`(pid, window_id)` keeps working unchanged.
+
+Why UIA at all: modern apps (WebView2-hosted Notepad, packaged-UWP frames,
+some Electron containers) sometimes hide their visible window inside a host
+HWND that `EnumWindows` either skips or surfaces with the wrong title/bounds.
+UIA's desktop-children walk surfaces the real interactable window — but only
+when EnumWindows didn't.
+
+Why EnumWindows-first (not UIA-first): UIA's `FindAll(TreeScope::Children)`
+gives no z-order guarantee; using it as the primary source would produce
+non-deterministic ordering. EnumWindows order IS the Win32 z-order.
+
+`filter_pid` is applied to the merged list, so a UWP app's pid that
+previously returned empty now returns its real window.
+
### Fixed divergences (Windows)
Swift returns per-record: `{window_id, pid, app_name, title, bounds {x,y,width,height}, layer, z_index, is_on_screen, on_current_space?, space_ids?}`
@@ -406,10 +431,10 @@ flat `x/y/width/height`) for any pre-existing callers; remove once they migrate.
Windows Rust does **not** yet enumerate **off-screen / minimized windows**.
Swift's default (`on_screen_only: false`) returns them; Windows currently
-filters them out at the `EnumWindows` callback level (`IsWindowVisible &&
-!IsIconic`). The `on_screen_only` schema field is accepted but has no
-effect. Follow-up to refactor `list_windows` to return everything and
-filter at the tool layer.
+filters them out at both enumeration sources (UIA's `IsOffscreen` flag and
+`EnumWindows`'s `IsWindowVisible && !IsIconic` callback). The
+`on_screen_only` schema field is accepted but has no effect. Follow-up to
+refactor `list_windows` to return everything and filter at the tool layer.
---
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
index 131d3efdbd..144dc87997 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
@@ -265,9 +265,14 @@ impl Tool for ListWindowsTool {
}
}
- // z_index: EnumWindows on Win32 returns top-to-bottom; higher index =
- // farther from front. Swift convention: higher z_index = closer to
- // front. Invert via `(len - 1 - i)`.
+ // z_index: list_windows merges EnumWindows first (which the Win32
+ // window manager returns in canonical top-to-bottom z-order), then
+ // appends any UIA-only HWNDs UIA found that EnumWindows missed
+ // (UIA-only entries land at the bottom of the z-stack — they have
+ // no canonical Win32 ordering). Higher list index = farther from
+ // front. Swift convention: higher z_index = closer to front.
+ // Invert via `(len - 1 - i)` so the front-most window gets the
+ // largest z.
let n = windows.len();
let records: Vec<serde_json::Value> = windows.iter().enumerate().map(|(i, w)| {
let app_name = pid_to_name.get(&w.pid).cloned().unwrap_or_default();
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs
index 6b3c931c57..e98ede89ad 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs
@@ -23,7 +23,9 @@ use windows::Win32::UI::Accessibility::{
};
pub mod cache;
+pub mod windows_enum;
pub use cache::ElementCache;
+pub use windows_enum::enumerate_top_level_windows;
const MAX_DEPTH: usize = 25;
const MAX_TOTAL_ELEMENTS: usize = 5000;
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs b/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs
new file mode 100644
index 0000000000..afc5f97b11
--- /dev/null
+++ b/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs
@@ -0,0 +1,210 @@
+//! UIA-based enumeration of top-level windows.
+//!
+//! Walks the UI Automation tree from the desktop root and returns one entry
+//! per top-level interactable window. UIA surfaces modern containers (WebView2
+//! hosts, packaged-UWP frames, browser windows whose chrome lives inside a
+//! container HWND) with their real title and bounds — which `EnumWindows`
+//! either misses or returns with a misleading parent HWND.
+//!
+//! The result is shape-compatible with `crate::win32::windows::WindowInfo`
+//! (returned as `WindowInfo` directly) so the existing pipeline that consumes
+//! `list_windows` output keeps working unchanged. Each record's `hwnd` is the
+//! UIA element's `NativeWindowHandle` — i.e. an honest Win32 HWND that downstream
+//! code can pass to `GetWindowRect`, `PostMessage`, etc.
+
+use std::cell::RefCell;
+
+use windows::Win32::Foundation::{HWND, RECT};
+use windows::Win32::Graphics::Dwm::{DwmGetWindowAttribute, DWMWA_EXTENDED_FRAME_BOUNDS};
+use windows::Win32::System::Com::{
+ CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED,
+};
+use windows::Win32::UI::Accessibility::{
+ CUIAutomation, IUIAutomation, IUIAutomationElement, TreeScope_Children,
+};
+use windows::Win32::UI::WindowsAndMessaging::{
+ GetWindowRect, GetWindowTextLengthW, GetWindowTextW, GetWindowThreadProcessId,
+};
+
+use crate::win32::windows::WindowInfo;
+
+/// HRESULT for "COM already initialized in another mode on this thread."
+/// Returned by `CoInitializeEx` when something else (a previous call in the
+/// same task, or a library on the same OS thread) picked a different
+/// apartment. Safe to ignore — COM is up either way.
+const RPC_E_CHANGED_MODE: i32 = -2147417850; // 0x80010106
+
+thread_local! {
+ /// Per-thread IUIAutomation instance. UIA / COM objects are
+ /// apartment-bound, so we deliberately do NOT share one across threads —
+ /// each OS thread that calls `enumerate_top_level_windows` initializes
+ /// COM as STA exactly once (the first time the cell is `None`) and caches
+ /// its own IUIAutomation. On failure the cell stays `None`, so the next
+ /// call retries from scratch instead of being stuck with a permanent
+ /// "init failed" sentinel.
+ static UIA_THREAD_LOCAL: RefCell<Option<IUIAutomation>> = const { RefCell::new(None) };
+}
+
+/// Build or fetch the per-thread IUIAutomation instance.
+///
+/// On the first successful call per OS thread this also initializes COM as
+/// STA (UIA's in-process server requires an STA; `RPC_E_CHANGED_MODE` from a
+/// pre-existing apartment is treated as "already up"). Any failure leaves
+/// the thread-local empty so subsequent calls retry.
+fn get_uia() -> Option<IUIAutomation> {
+ UIA_THREAD_LOCAL.with(|cell| {
+ if let Some(uia) = cell.borrow().as_ref() {
+ return Some(uia.clone());
+ }
+ // First (or post-failure) call on this thread: init COM + build UIA.
+ unsafe {
+ let hr = CoInitializeEx(None, COINIT_APARTMENTTHREADED);
+ if hr.is_err() && hr.0 != RPC_E_CHANGED_MODE {
+ tracing::debug!(target: "uia_windows_enum", "CoInitializeEx returned {hr:?}");
+ }
+ }
+ let inst: IUIAutomation = match unsafe {
+ CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER)
+ } {
+ Ok(a) => a,
+ Err(e) => {
+ tracing::warn!(target: "uia_windows_enum", "CoCreateInstance(CUIAutomation) failed: {e}");
+ return None;
+ }
+ };
+ let dup = inst.clone();
+ *cell.borrow_mut() = Some(inst);
+ Some(dup)
+ })
+}
+
+/// Enumerate top-level windows visible to UI Automation.
+///
+/// Returns one `WindowInfo` per non-offscreen child of the UIA desktop root
+/// whose `NativeWindowHandle` is non-null and resolves to a window with a
+/// non-empty title. Windows whose HWND is zero (pure UIA virtual elements,
+/// rare) are skipped because the rest of the driver pipeline keys off HWND.
+///
+/// Returns an empty vec on any UIA failure — callers should treat UIA as a
+/// best-effort source and union with `EnumWindows`.
+pub fn enumerate_top_level_windows() -> Vec<WindowInfo> {
+ let uia = match get_uia() {
+ Some(u) => u,
+ None => return Vec::new(),
+ };
+
+ unsafe {
+ let root = match uia.GetRootElement() {
+ Ok(r) => r,
+ Err(e) => {
+ tracing::debug!(target: "uia_windows_enum", "GetRootElement failed: {e}");
+ return Vec::new();
+ }
+ };
+ let condition = match uia.CreateTrueCondition() {
+ Ok(c) => c,
+ Err(e) => {
+ tracing::debug!(target: "uia_windows_enum", "CreateTrueCondition failed: {e}");
+ return Vec::new();
+ }
+ };
+ let children = match root.FindAll(TreeScope_Children, &condition) {
+ Ok(c) => c,
+ Err(e) => {
+ tracing::debug!(target: "uia_windows_enum", "FindAll(Children) failed: {e}");
+ return Vec::new();
+ }
+ };
+
+ let count = children.Length().unwrap_or(0);
+ let mut out: Vec<WindowInfo> = Vec::with_capacity(count as usize);
+ for i in 0..count {
+ let elem = match children.GetElement(i) {
+ Ok(e) => e,
+ Err(_) => continue,
+ };
+ if let Some(info) = window_info_from_uia_element(&elem) {
+ out.push(info);
+ }
+ }
+ out
+ }
+}
+
+/// Build a `WindowInfo` from a single UIA child element of the desktop root.
+/// Returns `None` if the element doesn't correspond to a real, on-screen,
+/// non-empty-titled HWND.
+unsafe fn window_info_from_uia_element(elem: &IUIAutomationElement) -> Option<WindowInfo> {
+ // NativeWindowHandle is an i32-sized handle in UIA; cast to HWND.
+ let raw = elem.CurrentNativeWindowHandle().ok()?;
+ if raw.0.is_null() {
+ return None;
+ }
+ let hwnd = HWND(raw.0);
+
+ // Drop minimized / off-screen windows. UIA's IsOffscreen flag covers
+ // both "iconic" and "behind another window such that no part is visible"
+ // — for top-level windows it matches the EnumWindows path's intent of
+ // showing only currently-visible candidates.
+ if let Ok(flag) = elem.CurrentIsOffscreen() {
+ if flag.as_bool() {
+ return None;
+ }
+ }
+
+ // Resolve pid via the standard Win32 path. We deliberately do not trust
+ // the UIA ProcessId property here because the rest of the driver indexes
+ // windows by (hwnd, pid) tuples obtained from `GetWindowThreadProcessId`,
+ // and we want bit-identical agreement.
+ let mut pid: u32 = 0;
+ let _ = GetWindowThreadProcessId(hwnd, Some(&mut pid));
+ if pid == 0 {
+ return None;
+ }
+
+ // Title — prefer Win32 GetWindowTextW for parity with the EnumWindows path.
+ // UIA's `CurrentName` sometimes returns the AX-friendly label (e.g. the
+ // tab title) instead of the OS-level window caption, which would diverge
+ // from any caller already keyed on the GetWindowText value.
+ let title_len = GetWindowTextLengthW(hwnd);
+ if title_len == 0 {
+ return None;
+ }
+ let mut buf = vec![0u16; (title_len + 1) as usize];
+ GetWindowTextW(hwnd, &mut buf);
+ let len = buf.iter().position(|&c| c == 0).unwrap_or(buf.len());
+ let title = String::from_utf16_lossy(&buf[..len]);
+ if title.trim().is_empty() {
+ return None;
+ }
+
+ let (x, y, w, h) = window_bounds(hwnd);
+
+ Some(WindowInfo {
+ hwnd: hwnd.0 as u64,
+ pid,
+ title,
+ x,
+ y,
+ width: w,
+ height: h,
+ })
+}
+
+/// Bounds via DWM extended frame (excludes drop-shadow on W11) with
+/// `GetWindowRect` fallback — same logic as the EnumWindows path.
+fn window_bounds(hwnd: HWND) -> (i32, i32, i32, i32) {
+ unsafe {
+ let mut rect = RECT::default();
+ let ok = DwmGetWindowAttribute(
+ hwnd,
+ DWMWA_EXTENDED_FRAME_BOUNDS,
+ &mut rect as *mut RECT as *mut _,
+ std::mem::size_of::<RECT>() as u32,
+ );
+ if ok.is_err() {
+ let _ = GetWindowRect(hwnd, &mut rect);
+ }
+ (rect.left, rect.top, rect.right - rect.left, rect.bottom - rect.top)
+ }
+}
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs b/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs
index 4c9c161e8c..157b924cd6 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs
@@ -1,8 +1,24 @@
//! Enumerate top-level windows on Windows.
//!
-//! Uses EnumWindows + GetWindowTextW + GetWindowThreadProcessId.
-//! Returns only visible, non-minimized windows with a non-empty title.
+//! Two enumeration sources, then union + dedupe by HWND:
+//!
+//! 1. **`EnumWindows`** (canonical z-order) — the Win32 enumerator walks the
+//! window manager's z-order list top-to-bottom, so iteration order IS the
+//! actual z-order. We list these first so the merged array's index reflects
+//! the true Win32 stacking for any HWND the Win32 path can see.
+//!
+//! 2. **UI Automation** (extra coverage) — appended after `EnumWindows`,
+//! contributing only HWNDs Win32 didn't surface. UIA exposes modern
+//! containers (WebView2 hosts, packaged-UWP frames, Electron container
+//! HWNDs) that `EnumWindows` may miss or return as wrapper HWNDs. UIA's
+//! `FindAll(TreeScope::Children, ...)` makes no z-order guarantee, so we
+//! deliberately do NOT let it reorder anything Win32 already reported.
+//!
+//! Both sources apply the same filters (visible, non-iconic, non-empty
+//! title). The `filter_pid` argument is applied to the merged list so the
+//! union/dedupe pipeline runs unconditionally.
+use std::collections::HashSet;
use std::sync::Mutex;
use windows::Win32::Foundation::{BOOL, HWND, LPARAM, RECT, TRUE};
use windows::Win32::Graphics::Dwm::{
@@ -27,13 +43,50 @@ pub struct WindowInfo {
}
struct EnumState {
- filter_pid: Option<u32>,
windows: Vec<WindowInfo>,
}
/// List top-level visible windows. If `filter_pid` is Some, only that process.
+///
+/// `EnumWindows` first — its iteration order is the Win32 window manager's
+/// z-order (top-to-bottom), so the merged array's index doubles as a z_index
+/// for any HWND the Win32 path saw. Then UIA-only entries are appended (UIA
+/// makes no z-order guarantee, so it must not be allowed to reorder anything
+/// EnumWindows already reported). The pid filter is applied to the merged
+/// list.
pub fn list_windows(filter_pid: Option<u32>) -> Vec<WindowInfo> {
- let state = Mutex::new(EnumState { filter_pid, windows: Vec::new() });
+ let win32_windows = enumerate_via_enum_windows();
+ let uia_windows = crate::uia::enumerate_top_level_windows();
+
+ let mut seen: HashSet<u64> = HashSet::with_capacity(uia_windows.len() + win32_windows.len());
+ let mut merged: Vec<WindowInfo> = Vec::with_capacity(uia_windows.len() + win32_windows.len());
+
+ // EnumWindows first — canonical Win32 z-order.
+ for w in win32_windows {
+ if seen.insert(w.hwnd) {
+ merged.push(w);
+ }
+ }
+ // Then any UIA-only HWND that EnumWindows didn't surface (modern
+ // containers, WebView2 hosts, etc.). These get appended (no claim on
+ // z-order priority relative to the Win32 list).
+ for w in uia_windows {
+ if seen.insert(w.hwnd) {
+ merged.push(w);
+ }
+ }
+
+ if let Some(fp) = filter_pid {
+ merged.retain(|w| w.pid == fp);
+ }
+ merged
+}
+
+/// Walk `EnumWindows` and collect every visible, non-iconic, non-empty-titled
+/// top-level window. No pid filter is applied here — the caller does that on
+/// the merged list.
+fn enumerate_via_enum_windows() -> Vec<WindowInfo> {
+ let state = Mutex::new(EnumState { windows: Vec::new() });
let state_ptr = &state as *const Mutex<EnumState> as isize;
unsafe {
let _ = EnumWindows(Some(enum_windows_cb), LPARAM(state_ptr));
@@ -53,14 +106,6 @@ unsafe extern "system" fn enum_windows_cb(hwnd: HWND, lparam: LPARAM) -> BOOL {
let mut pid: u32 = 0;
GetWindowThreadProcessId(hwnd, Some(&mut pid));
- // Filter by pid if requested.
- {
- let s = state.lock().unwrap();
- if let Some(fp) = s.filter_pid {
- if pid != fp { return TRUE; }
- }
- }
-
// Get title (skip empty).
let title_len = GetWindowTextLengthW(hwnd);
if title_len == 0 { return TRUE; }