Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(list_windows): UIA-first top-level window enumeration on Windows (#1542)
Francesco Bonacci70d899c39cd4
diff --git a/libs/cua-driver-rs/PARITY.md b/libs/cua-driver-rs/PARITY.mdindex bd5d3043c8..b08f5d14d2 100644--- a/libs/cua-driver-rs/PARITY.md+++ b/libs/cua-driver-rs/PARITY.md@@ -362,11 +362,36 @@ at least one entry has `active: true` (the foreground app).- windows=`crates/platform-windows/src/tools/impl_.rs` (ListWindowsTool)- linux=`crates/platform-linux/src/tools/impl_.rs` (ListWindowsTool, blocked behind Linux compile fix)- Status:- - windows: VERIFIED+ - windows: VERIFIED (EnumWindows-first enumeration; UIA contributes missing HWNDs only)- macOS: OPEN (audit pending — macOS port already exists)- linux: OPEN- Test: `crates/platform-windows/examples/list_windows_parity.rs`+### Enumeration source (Windows)++`crate::win32::list_windows` walks `EnumWindows` first — that's the Win32+window manager's canonical top-to-bottom z-order, so it's the authoritative+source for both window membership and ordering. It then asks UI Automation+for any top-level windows EnumWindows missed+(`AutomationElement::RootElement.FindAll(TreeScope::Children, TrueCondition)`,+filtered to `IsOffscreen == false` + non-empty `GetWindowTextW`) and appends+those at the end of the merged list, deduped by HWND. UIA elements contribute+their `NativeWindowHandle` as the canonical HWND so downstream code keyed on+`(pid, window_id)` keeps working unchanged.++Why UIA at all: modern apps (WebView2-hosted Notepad, packaged-UWP frames,+some Electron containers) sometimes hide their visible window inside a host+HWND that `EnumWindows` either skips or surfaces with the wrong title/bounds.+UIA's desktop-children walk surfaces the real interactable window — but only+when EnumWindows didn't.++Why EnumWindows-first (not UIA-first): UIA's `FindAll(TreeScope::Children)`+gives no z-order guarantee; using it as the primary source would produce+non-deterministic ordering. EnumWindows order IS the Win32 z-order.++`filter_pid` is applied to the merged list, so a UWP app's pid that+previously returned empty now returns its real window.+### Fixed divergences (Windows)Swift returns per-record: `{window_id, pid, app_name, title, bounds {x,y,width,height}, layer, z_index, is_on_screen, on_current_space?, space_ids?}`@@ -406,10 +431,10 @@ flat `x/y/width/height`) for any pre-existing callers; remove once they migrate.Windows Rust does **not** yet enumerate **off-screen / minimized windows**.Swift's default (`on_screen_only: false`) returns them; Windows currently-filters them out at the `EnumWindows` callback level (`IsWindowVisible &&-!IsIconic`). The `on_screen_only` schema field is accepted but has no-effect. Follow-up to refactor `list_windows` to return everything and-filter at the tool layer.+filters them out at both enumeration sources (UIA's `IsOffscreen` flag and+`EnumWindows`'s `IsWindowVisible && !IsIconic` callback). The+`on_screen_only` schema field is accepted but has no effect. Follow-up to+refactor `list_windows` to return everything and filter at the tool layer.---diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rsindex 131d3efdbd..144dc87997 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs@@ -265,9 +265,14 @@ impl Tool for ListWindowsTool {}}- // z_index: EnumWindows on Win32 returns top-to-bottom; higher index =- // farther from front. Swift convention: higher z_index = closer to- // front. Invert via `(len - 1 - i)`.+ // z_index: list_windows merges EnumWindows first (which the Win32+ // window manager returns in canonical top-to-bottom z-order), then+ // appends any UIA-only HWNDs UIA found that EnumWindows missed+ // (UIA-only entries land at the bottom of the z-stack — they have+ // no canonical Win32 ordering). Higher list index = farther from+ // front. Swift convention: higher z_index = closer to front.+ // Invert via `(len - 1 - i)` so the front-most window gets the+ // largest z.let n = windows.len();let records: Vec<serde_json::Value> = windows.iter().enumerate().map(|(i, w)| {let app_name = pid_to_name.get(&w.pid).cloned().unwrap_or_default();diff --git a/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rsindex 6b3c931c57..e98ede89ad 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs@@ -23,7 +23,9 @@ use windows::Win32::UI::Accessibility::{};pub mod cache;+pub mod windows_enum;pub use cache::ElementCache;+pub use windows_enum::enumerate_top_level_windows;const MAX_DEPTH: usize = 25;const MAX_TOTAL_ELEMENTS: usize = 5000;diff --git a/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs b/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rsnew file mode 100644index 0000000000..afc5f97b11--- /dev/null+++ b/libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs@@ -0,0 +1,210 @@+//! UIA-based enumeration of top-level windows.+//!+//! Walks the UI Automation tree from the desktop root and returns one entry+//! per top-level interactable window. UIA surfaces modern containers (WebView2+//! hosts, packaged-UWP frames, browser windows whose chrome lives inside a+//! container HWND) with their real title and bounds — which `EnumWindows`+//! either misses or returns with a misleading parent HWND.+//!+//! The result is shape-compatible with `crate::win32::windows::WindowInfo`+//! (returned as `WindowInfo` directly) so the existing pipeline that consumes+//! `list_windows` output keeps working unchanged. Each record's `hwnd` is the+//! UIA element's `NativeWindowHandle` — i.e. an honest Win32 HWND that downstream+//! code can pass to `GetWindowRect`, `PostMessage`, etc.++use std::cell::RefCell;++use windows::Win32::Foundation::{HWND, RECT};+use windows::Win32::Graphics::Dwm::{DwmGetWindowAttribute, DWMWA_EXTENDED_FRAME_BOUNDS};+use windows::Win32::System::Com::{+ CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED,+};+use windows::Win32::UI::Accessibility::{+ CUIAutomation, IUIAutomation, IUIAutomationElement, TreeScope_Children,+};+use windows::Win32::UI::WindowsAndMessaging::{+ GetWindowRect, GetWindowTextLengthW, GetWindowTextW, GetWindowThreadProcessId,+};++use crate::win32::windows::WindowInfo;++/// HRESULT for "COM already initialized in another mode on this thread."+/// Returned by `CoInitializeEx` when something else (a previous call in the+/// same task, or a library on the same OS thread) picked a different+/// apartment. Safe to ignore — COM is up either way.+const RPC_E_CHANGED_MODE: i32 = -2147417850; // 0x80010106++thread_local! {+ /// Per-thread IUIAutomation instance. UIA / COM objects are+ /// apartment-bound, so we deliberately do NOT share one across threads —+ /// each OS thread that calls `enumerate_top_level_windows` initializes+ /// COM as STA exactly once (the first time the cell is `None`) and caches+ /// its own IUIAutomation. On failure the cell stays `None`, so the next+ /// call retries from scratch instead of being stuck with a permanent+ /// "init failed" sentinel.+ static UIA_THREAD_LOCAL: RefCell<Option<IUIAutomation>> = const { RefCell::new(None) };+}++/// Build or fetch the per-thread IUIAutomation instance.+///+/// On the first successful call per OS thread this also initializes COM as+/// STA (UIA's in-process server requires an STA; `RPC_E_CHANGED_MODE` from a+/// pre-existing apartment is treated as "already up"). Any failure leaves+/// the thread-local empty so subsequent calls retry.+fn get_uia() -> Option<IUIAutomation> {+ UIA_THREAD_LOCAL.with(|cell| {+ if let Some(uia) = cell.borrow().as_ref() {+ return Some(uia.clone());+ }+ // First (or post-failure) call on this thread: init COM + build UIA.+ unsafe {+ let hr = CoInitializeEx(None, COINIT_APARTMENTTHREADED);+ if hr.is_err() && hr.0 != RPC_E_CHANGED_MODE {+ tracing::debug!(target: "uia_windows_enum", "CoInitializeEx returned {hr:?}");+ }+ }+ let inst: IUIAutomation = match unsafe {+ CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER)+ } {+ Ok(a) => a,+ Err(e) => {+ tracing::warn!(target: "uia_windows_enum", "CoCreateInstance(CUIAutomation) failed: {e}");+ return None;+ }+ };+ let dup = inst.clone();+ *cell.borrow_mut() = Some(inst);+ Some(dup)+ })+}++/// Enumerate top-level windows visible to UI Automation.+///+/// Returns one `WindowInfo` per non-offscreen child of the UIA desktop root+/// whose `NativeWindowHandle` is non-null and resolves to a window with a+/// non-empty title. Windows whose HWND is zero (pure UIA virtual elements,+/// rare) are skipped because the rest of the driver pipeline keys off HWND.+///+/// Returns an empty vec on any UIA failure — callers should treat UIA as a+/// best-effort source and union with `EnumWindows`.+pub fn enumerate_top_level_windows() -> Vec<WindowInfo> {+ let uia = match get_uia() {+ Some(u) => u,+ None => return Vec::new(),+ };++ unsafe {+ let root = match uia.GetRootElement() {+ Ok(r) => r,+ Err(e) => {+ tracing::debug!(target: "uia_windows_enum", "GetRootElement failed: {e}");+ return Vec::new();+ }+ };+ let condition = match uia.CreateTrueCondition() {+ Ok(c) => c,+ Err(e) => {+ tracing::debug!(target: "uia_windows_enum", "CreateTrueCondition failed: {e}");+ return Vec::new();+ }+ };+ let children = match root.FindAll(TreeScope_Children, &condition) {+ Ok(c) => c,+ Err(e) => {+ tracing::debug!(target: "uia_windows_enum", "FindAll(Children) failed: {e}");+ return Vec::new();+ }+ };++ let count = children.Length().unwrap_or(0);+ let mut out: Vec<WindowInfo> = Vec::with_capacity(count as usize);+ for i in 0..count {+ let elem = match children.GetElement(i) {+ Ok(e) => e,+ Err(_) => continue,+ };+ if let Some(info) = window_info_from_uia_element(&elem) {+ out.push(info);+ }+ }+ out+ }+}++/// Build a `WindowInfo` from a single UIA child element of the desktop root.+/// Returns `None` if the element doesn't correspond to a real, on-screen,+/// non-empty-titled HWND.+unsafe fn window_info_from_uia_element(elem: &IUIAutomationElement) -> Option<WindowInfo> {+ // NativeWindowHandle is an i32-sized handle in UIA; cast to HWND.+ let raw = elem.CurrentNativeWindowHandle().ok()?;+ if raw.0.is_null() {+ return None;+ }+ let hwnd = HWND(raw.0);++ // Drop minimized / off-screen windows. UIA's IsOffscreen flag covers+ // both "iconic" and "behind another window such that no part is visible"+ // — for top-level windows it matches the EnumWindows path's intent of+ // showing only currently-visible candidates.+ if let Ok(flag) = elem.CurrentIsOffscreen() {+ if flag.as_bool() {+ return None;+ }+ }++ // Resolve pid via the standard Win32 path. We deliberately do not trust+ // the UIA ProcessId property here because the rest of the driver indexes+ // windows by (hwnd, pid) tuples obtained from `GetWindowThreadProcessId`,+ // and we want bit-identical agreement.+ let mut pid: u32 = 0;+ let _ = GetWindowThreadProcessId(hwnd, Some(&mut pid));+ if pid == 0 {+ return None;+ }++ // Title — prefer Win32 GetWindowTextW for parity with the EnumWindows path.+ // UIA's `CurrentName` sometimes returns the AX-friendly label (e.g. the+ // tab title) instead of the OS-level window caption, which would diverge+ // from any caller already keyed on the GetWindowText value.+ let title_len = GetWindowTextLengthW(hwnd);+ if title_len == 0 {+ return None;+ }+ let mut buf = vec![0u16; (title_len + 1) as usize];+ GetWindowTextW(hwnd, &mut buf);+ let len = buf.iter().position(|&c| c == 0).unwrap_or(buf.len());+ let title = String::from_utf16_lossy(&buf[..len]);+ if title.trim().is_empty() {+ return None;+ }++ let (x, y, w, h) = window_bounds(hwnd);++ Some(WindowInfo {+ hwnd: hwnd.0 as u64,+ pid,+ title,+ x,+ y,+ width: w,+ height: h,+ })+}++/// Bounds via DWM extended frame (excludes drop-shadow on W11) with+/// `GetWindowRect` fallback — same logic as the EnumWindows path.+fn window_bounds(hwnd: HWND) -> (i32, i32, i32, i32) {+ unsafe {+ let mut rect = RECT::default();+ let ok = DwmGetWindowAttribute(+ hwnd,+ DWMWA_EXTENDED_FRAME_BOUNDS,+ &mut rect as *mut RECT as *mut _,+ std::mem::size_of::<RECT>() as u32,+ );+ if ok.is_err() {+ let _ = GetWindowRect(hwnd, &mut rect);+ }+ (rect.left, rect.top, rect.right - rect.left, rect.bottom - rect.top)+ }+}diff --git a/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs b/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rsindex 4c9c161e8c..157b924cd6 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/win32/windows.rs@@ -1,8 +1,24 @@//! Enumerate top-level windows on Windows.//!-//! Uses EnumWindows + GetWindowTextW + GetWindowThreadProcessId.-//! Returns only visible, non-minimized windows with a non-empty title.+//! Two enumeration sources, then union + dedupe by HWND:+//!+//! 1. **`EnumWindows`** (canonical z-order) — the Win32 enumerator walks the+//! window manager's z-order list top-to-bottom, so iteration order IS the+//! actual z-order. We list these first so the merged array's index reflects+//! the true Win32 stacking for any HWND the Win32 path can see.+//!+//! 2. **UI Automation** (extra coverage) — appended after `EnumWindows`,+//! contributing only HWNDs Win32 didn't surface. UIA exposes modern+//! containers (WebView2 hosts, packaged-UWP frames, Electron container+//! HWNDs) that `EnumWindows` may miss or return as wrapper HWNDs. UIA's+//! `FindAll(TreeScope::Children, ...)` makes no z-order guarantee, so we+//! deliberately do NOT let it reorder anything Win32 already reported.+//!+//! Both sources apply the same filters (visible, non-iconic, non-empty+//! title). The `filter_pid` argument is applied to the merged list so the+//! union/dedupe pipeline runs unconditionally.+use std::collections::HashSet;use std::sync::Mutex;use windows::Win32::Foundation::{BOOL, HWND, LPARAM, RECT, TRUE};use windows::Win32::Graphics::Dwm::{@@ -27,13 +43,50 @@ pub struct WindowInfo {}struct EnumState {- filter_pid: Option<u32>,windows: Vec<WindowInfo>,}/// List top-level visible windows. If `filter_pid` is Some, only that process.+///+/// `EnumWindows` first — its iteration order is the Win32 window manager's+/// z-order (top-to-bottom), so the merged array's index doubles as a z_index+/// for any HWND the Win32 path saw. Then UIA-only entries are appended (UIA+/// makes no z-order guarantee, so it must not be allowed to reorder anything+/// EnumWindows already reported). The pid filter is applied to the merged+/// list.pub fn list_windows(filter_pid: Option<u32>) -> Vec<WindowInfo> {- let state = Mutex::new(EnumState { filter_pid, windows: Vec::new() });+ let win32_windows = enumerate_via_enum_windows();+ let uia_windows = crate::uia::enumerate_top_level_windows();++ let mut seen: HashSet<u64> = HashSet::with_capacity(uia_windows.len() + win32_windows.len());+ let mut merged: Vec<WindowInfo> = Vec::with_capacity(uia_windows.len() + win32_windows.len());++ // EnumWindows first — canonical Win32 z-order.+ for w in win32_windows {+ if seen.insert(w.hwnd) {+ merged.push(w);+ }+ }+ // Then any UIA-only HWND that EnumWindows didn't surface (modern+ // containers, WebView2 hosts, etc.). These get appended (no claim on+ // z-order priority relative to the Win32 list).+ for w in uia_windows {+ if seen.insert(w.hwnd) {+ merged.push(w);+ }+ }++ if let Some(fp) = filter_pid {+ merged.retain(|w| w.pid == fp);+ }+ merged+}++/// Walk `EnumWindows` and collect every visible, non-iconic, non-empty-titled+/// top-level window. No pid filter is applied here — the caller does that on+/// the merged list.+fn enumerate_via_enum_windows() -> Vec<WindowInfo> {+ let state = Mutex::new(EnumState { windows: Vec::new() });let state_ptr = &state as *const Mutex<EnumState> as isize;unsafe {let _ = EnumWindows(Some(enum_windows_cb), LPARAM(state_ptr));@@ -53,14 +106,6 @@ unsafe extern "system" fn enum_windows_cb(hwnd: HWND, lparam: LPARAM) -> BOOL {let mut pid: u32 = 0;GetWindowThreadProcessId(hwnd, Some(&mut pid));- // Filter by pid if requested.- {- let s = state.lock().unwrap();- if let Some(fp) = s.filter_pid {- if pid != fp { return TRUE; }- }- }-// Get title (skip empty).let title_len = GetWindowTextLengthW(hwnd);if title_len == 0 { return TRUE; }