Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): scope macOS off-Space AX window recovery (#4085)

Francesco Bonacci2b824d4e0975
diff --git a/docs/content/docs/reference/cua-driver/limits.mdx b/docs/content/docs/reference/cua-driver/limits.mdx
index cca43cc4a6..2542061964 100644
--- a/docs/content/docs/reference/cua-driver/limits.mdx
+++ b/docs/content/docs/reference/cua-driver/limits.mdx
@@ -120,7 +120,9 @@ does not imply arbitrary raw background PX delivery on Wayland.
## Off-Space windows keep background input and hide the agent cursor
-**Behavior:** macOS omits windows on other Spaces from an application's `AXWindows` list. Cua Driver recovers the requested window by its exact CGWindowID, so `get_window_state`, element actions, and background pixel actions keep working while the window stays on its Space. The user's current Space, frontmost application, and keyboard focus do not change.
+**Behavior:** macOS omits windows on other Spaces from an application's `AXWindows` list. For AppKit windows, Cua Driver recovers the requested window by its exact CGWindowID, so `get_window_state`, element actions, and background pixel actions keep working while the window stays on its Space. The user's current Space, frontmost application, and keyboard focus do not change.
+
+**Scope:** recovery runs only for windows that WindowServer reports on another Space, and it probes the first 2,000 accessibility element ids of the application. A window whose element id is beyond that limit, such as a window created late in a long-running browser or Electron app, is not recovered; its request reports `ax_window_unresolved`. SwiftUI windows are recovered but expose little or no accessibility tree while off-Space; see [Off-Space SwiftUI windows strip their AX tree](#off-space-swiftui-windows-strip-their-ax-tree).
**Agent cursor:** the overlay does not paint while an action targets a window on another Space, so the cursor never animates over the user's Space at that window's coordinates. It reappears with the next action on a current-Space window.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
index 6b5cbcf445..392575e296 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs
@@ -803,6 +803,10 @@ pub unsafe fn copy_ax_window_by_remote_token(pid: i32, window_id: u32) -> Option
if copy_string_attr(element, "AXRole").as_deref() == Some("AXWindow")
&& ax_get_window_id(element) == Some(window_id)
{
+ // The snapshot walk reads the whole subtree through this element;
+ // restore the system default so slow apps are not cut off at the
+ // probe's per-candidate timeout (issue #4082).
+ AXUIElementSetMessagingTimeout(element, 0.0);
return Some(element);
}
CFRelease(element as CFTypeRef);
@@ -810,9 +814,21 @@ pub unsafe fn copy_ax_window_by_remote_token(pid: i32, window_id: u32) -> Option
None
}
+/// Whether to run the remote-token probe for a window `AXWindows` omitted.
+/// Only windows WindowServer reports on another Space qualify; a current-Space
+/// or unknown window that AX cannot map fails fast instead of paying the
+/// probe's deadline on every call (issue #4083). `on_current_space` is only
+/// queried for unlisted windows, so listed windows skip the WindowServer read.
+fn should_probe_off_space_window(
+ listed_in_ax_windows: bool,
+ on_current_space: impl FnOnce() -> Option<bool>,
+) -> bool {
+ !listed_in_ax_windows && on_current_space() == Some(false)
+}
+
/// `AXWindows` of `pid`'s application element, plus the requested window when
-/// `AXWindows` does not list it (it is on another Space). Returns retained
-/// elements the caller must release.
+/// `AXWindows` does not list it because it is on another Space. Returns
+/// retained elements the caller must release.
///
/// # Safety
///
@@ -824,10 +840,12 @@ pub unsafe fn copy_ax_windows_including(
window_id: u32,
) -> Vec<AXUIElementRef> {
let mut windows = copy_ax_windows(app);
- if !windows
+ let listed = windows
.iter()
- .any(|&window| ax_get_window_id(window) == Some(window_id))
- {
+ .any(|&window| ax_get_window_id(window) == Some(window_id));
+ if should_probe_off_space_window(listed, || {
+ crate::windows::window_on_current_space_by_id(window_id)
+ }) {
windows.extend(copy_ax_window_by_remote_token(pid, window_id));
}
windows
@@ -847,6 +865,16 @@ mod tests {
assert_eq!(&token[12..20], &42u64.to_ne_bytes());
}
+ #[test]
+ fn off_space_probe_runs_only_for_unlisted_off_space_windows() {
+ assert!(should_probe_off_space_window(false, || Some(false)));
+ assert!(!should_probe_off_space_window(false, || Some(true)));
+ assert!(!should_probe_off_space_window(false, || None));
+ assert!(!should_probe_off_space_window(true, || {
+ panic!("listed windows must not query Space membership")
+ }));
+ }
+
#[test]
fn binary_value_accepts_booleans_and_exact_zero_or_one() {
let true_value = CFBoolean::true_value();