Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): scope macOS off-Space AX window recovery (#4085)
Francesco Bonacci2b824d4e0975
diff --git a/docs/content/docs/reference/cua-driver/limits.mdx b/docs/content/docs/reference/cua-driver/limits.mdxindex cca43cc4a6..2542061964 100644--- a/docs/content/docs/reference/cua-driver/limits.mdx+++ b/docs/content/docs/reference/cua-driver/limits.mdx@@ -120,7 +120,9 @@ does not imply arbitrary raw background PX delivery on Wayland.## Off-Space windows keep background input and hide the agent cursor-**Behavior:** macOS omits windows on other Spaces from an application's `AXWindows` list. Cua Driver recovers the requested window by its exact CGWindowID, so `get_window_state`, element actions, and background pixel actions keep working while the window stays on its Space. The user's current Space, frontmost application, and keyboard focus do not change.+**Behavior:** macOS omits windows on other Spaces from an application's `AXWindows` list. For AppKit windows, Cua Driver recovers the requested window by its exact CGWindowID, so `get_window_state`, element actions, and background pixel actions keep working while the window stays on its Space. The user's current Space, frontmost application, and keyboard focus do not change.++**Scope:** recovery runs only for windows that WindowServer reports on another Space, and it probes the first 2,000 accessibility element ids of the application. A window whose element id is beyond that limit, such as a window created late in a long-running browser or Electron app, is not recovered; its request reports `ax_window_unresolved`. SwiftUI windows are recovered but expose little or no accessibility tree while off-Space; see [Off-Space SwiftUI windows strip their AX tree](#off-space-swiftui-windows-strip-their-ax-tree).**Agent cursor:** the overlay does not paint while an action targets a window on another Space, so the cursor never animates over the user's Space at that window's coordinates. It reappears with the next action on a current-Space window.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rsindex 6b5cbcf445..392575e296 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs@@ -803,6 +803,10 @@ pub unsafe fn copy_ax_window_by_remote_token(pid: i32, window_id: u32) -> Optionif copy_string_attr(element, "AXRole").as_deref() == Some("AXWindow")&& ax_get_window_id(element) == Some(window_id){+ // The snapshot walk reads the whole subtree through this element;+ // restore the system default so slow apps are not cut off at the+ // probe's per-candidate timeout (issue #4082).+ AXUIElementSetMessagingTimeout(element, 0.0);return Some(element);}CFRelease(element as CFTypeRef);@@ -810,9 +814,21 @@ pub unsafe fn copy_ax_window_by_remote_token(pid: i32, window_id: u32) -> OptionNone}+/// Whether to run the remote-token probe for a window `AXWindows` omitted.+/// Only windows WindowServer reports on another Space qualify; a current-Space+/// or unknown window that AX cannot map fails fast instead of paying the+/// probe's deadline on every call (issue #4083). `on_current_space` is only+/// queried for unlisted windows, so listed windows skip the WindowServer read.+fn should_probe_off_space_window(+ listed_in_ax_windows: bool,+ on_current_space: impl FnOnce() -> Option<bool>,+) -> bool {+ !listed_in_ax_windows && on_current_space() == Some(false)+}+/// `AXWindows` of `pid`'s application element, plus the requested window when-/// `AXWindows` does not list it (it is on another Space). Returns retained-/// elements the caller must release.+/// `AXWindows` does not list it because it is on another Space. Returns+/// retained elements the caller must release.////// # Safety///@@ -824,10 +840,12 @@ pub unsafe fn copy_ax_windows_including(window_id: u32,) -> Vec<AXUIElementRef> {let mut windows = copy_ax_windows(app);- if !windows+ let listed = windows.iter()- .any(|&window| ax_get_window_id(window) == Some(window_id))- {+ .any(|&window| ax_get_window_id(window) == Some(window_id));+ if should_probe_off_space_window(listed, || {+ crate::windows::window_on_current_space_by_id(window_id)+ }) {windows.extend(copy_ax_window_by_remote_token(pid, window_id));}windows@@ -847,6 +865,16 @@ mod tests {assert_eq!(&token[12..20], &42u64.to_ne_bytes());}+ #[test]+ fn off_space_probe_runs_only_for_unlisted_off_space_windows() {+ assert!(should_probe_off_space_window(false, || Some(false)));+ assert!(!should_probe_off_space_window(false, || Some(true)));+ assert!(!should_probe_off_space_window(false, || None));+ assert!(!should_probe_off_space_window(true, || {+ panic!("listed windows must not query Space membership")+ }));+ }+#[test]fn binary_value_accepts_booleans_and_exact_zero_or_one() {let true_value = CFBoolean::true_value();