Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): wait for a launching macOS app before its first snapshot (#4309)

Francesco Bonacci1cb7b6fb2415
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index d1a9270286..e4c8207a23 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -71,7 +71,7 @@ Always returns BOTH the element tree AND a screenshot — ground on both and cro
The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.
-The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with `delivery_mode:"foreground"`. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.
+The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with `delivery_mode:"foreground"`. When that pid is an app still launching (its window exists before it answers accessibility), the walk first waits up to `timeout_ms` for it; if it never answers, the tree comes back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and `truncation_reason: app_lookup_timeout`. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.
Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
index d1e36619ee..c78c09135b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs
@@ -16,6 +16,10 @@ pub enum WalkStop {
Timeout,
/// The `max_elements` node budget ran out.
NodeBudget,
+ /// The application could not be reached through the platform
+ /// accessibility API before `timeout_ms` ran out (still launching, or not
+ /// yet registered), so nothing was walked.
+ AppLookupTimeout,
}
impl WalkStop {
@@ -23,6 +27,7 @@ impl WalkStop {
match self {
WalkStop::Timeout => "timeout",
WalkStop::NodeBudget => "node_budget",
+ WalkStop::AppLookupTimeout => "app_lookup_timeout",
}
}
}
@@ -41,6 +46,8 @@ pub struct WalkBudget {
started: Option<Instant>,
timeout: Duration,
timeout_ms: u64,
+ /// False for [`WalkBudget::nodes_only`], whose clock never fires.
+ time_bounded: bool,
max_elements: usize,
visited: usize,
pending: usize,
@@ -55,6 +62,7 @@ impl WalkBudget {
started: None,
timeout: Duration::from_millis(timeout_ms),
timeout_ms,
+ time_bounded: true,
max_elements,
visited: 0,
pending: 0,
@@ -66,7 +74,17 @@ impl WalkBudget {
/// caller-supplied time budget).
pub fn nodes_only(max_elements: usize) -> Self {
// A year: never fires, without overflowing Instant.
- Self::new(365 * 24 * 60 * 60 * 1000, max_elements)
+ Self {
+ time_bounded: false,
+ ..Self::new(365 * 24 * 60 * 60 * 1000, max_elements)
+ }
+ }
+
+ /// The caller's wall-clock budget, or `None` for a node-only budget.
+ /// Backends use it to bound setup that happens before the first node,
+ /// such as waiting for a launching application to become reachable.
+ pub fn time_limit(&self) -> Option<Duration> {
+ self.time_bounded.then_some(self.timeout)
}
/// Admit one node for visiting. `false` means the walk must not visit it:
@@ -101,6 +119,12 @@ impl WalkBudget {
self.stop.get_or_insert(WalkStop::Timeout);
}
+ /// Record that the application never became reachable within the budget,
+ /// so the walk visited nothing.
+ pub fn stop_for_app_lookup_timeout(&mut self) {
+ self.stop.get_or_insert(WalkStop::AppLookupTimeout);
+ }
+
pub fn outcome(&self) -> WalkOutcome {
WalkOutcome {
timeout_ms: self.timeout_ms,
@@ -183,9 +207,10 @@ pub fn truncation_note(
Some("app_unresponsive") => {
"the application stopped answering its accessibility API".to_owned()
}
- Some("app_lookup_timeout") => {
- format!("the application did not register with AT-SPI within {timeout_ms} ms")
- }
+ Some("app_lookup_timeout") => format!(
+ "the application was not reachable through its accessibility API (AT-SPI, AX) \
+ within {timeout_ms} ms; it may still be launching"
+ ),
Some("huge_container") => "a container with more children than can be enumerated \
(e.g. a spreadsheet's cell grid) was not expanded"
.to_owned(),
@@ -215,6 +240,32 @@ mod tests {
assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (2, 2));
}
+ #[test]
+ fn only_a_caller_budget_bounds_setup_waits() {
+ assert_eq!(
+ WalkBudget::new(1500, 10).time_limit(),
+ Some(Duration::from_millis(1500))
+ );
+ assert_eq!(WalkBudget::nodes_only(10).time_limit(), None);
+ }
+
+ #[test]
+ fn an_unreachable_application_reports_app_lookup_timeout() {
+ let mut budget = WalkBudget::new(1000, 10);
+ budget.stop_for_app_lookup_timeout();
+ let outcome = budget.outcome();
+ assert!(outcome.truncated());
+ assert_eq!(outcome.reason(), Some("app_lookup_timeout"));
+ assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (0, 0));
+ let mut structured = json!({});
+ outcome.apply(&mut structured);
+ assert_eq!(structured["truncated"], true);
+ assert_eq!(structured["truncation_reason"], "app_lookup_timeout");
+ let note = outcome.note().unwrap();
+ assert!(note.contains("not reachable through its accessibility API"));
+ assert!(note.contains("1000 ms"));
+ }
+
#[test]
fn expired_deadline_stops_with_timeout() {
let mut budget = WalkBudget::new(0, 100);
@@ -277,6 +328,7 @@ mod tests {
assert!(note.contains("stopped answering"));
let note = truncation_note(Some("app_lookup_timeout"), 250, 0, 0);
assert!(note.contains("250 ms"));
+ assert!(note.contains("still be launching"));
let note = truncation_note(Some("huge_container"), 1000, 1918, 0);
assert!(note.contains("not expanded"));
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
index 09b78b7ca9..12e704fe4a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
@@ -80,6 +80,25 @@ impl Harness {
command_oracle: Option<&Path>,
pointer_oracle: Option<&Path>,
keep_ordered_front: bool,
+ ) -> Self {
+ let harness = Self::spawn(command_oracle, pointer_oracle, keep_ordered_front, None);
+ // Settle for window creation + activation.
+ std::thread::sleep(Duration::from_millis(800));
+ harness.await_presented();
+ harness
+ }
+
+ /// Launch a harness that holds off entering its run loop for `delay`
+ /// after registering its window, without waiting for its launch posture.
+ fn launch_slowly(delay: Duration) -> Self {
+ Self::spawn(None, None, false, Some(delay))
+ }
+
+ fn spawn(
+ command_oracle: Option<&Path>,
+ pointer_oracle: Option<&Path>,
+ keep_ordered_front: bool,
+ launch_delay: Option<Duration>,
) -> Self {
let exe = harness_exe();
assert!(
@@ -100,16 +119,39 @@ impl Harness {
if keep_ordered_front {
command.env("CUA_APPKIT_KEEP_ORDERED_FRONT", "1");
}
+ if let Some(delay) = launch_delay {
+ command.env("CUA_APPKIT_LAUNCH_DELAY_MS", delay.as_millis().to_string());
+ }
let app = command
.spawn()
.unwrap_or_else(|error| panic!("launch AppKit harness {exe:?}: {error}"));
let pid = app.id();
- // Settle for window creation + activation.
- std::thread::sleep(Duration::from_millis(800));
Self { _app: app, pid }
}
+
+ /// Wait until the harness has finished launching and shows its window.
+ ///
+ /// Its window is registered with WindowServer before the app enters its
+ /// run loop, so `find_window` can succeed while every accessibility
+ /// request still fails. On a cold hosted runner the first launch spends
+ /// seconds in that state; a case must not start there.
+ fn await_presented(&self) {
+ let deadline = Instant::now() + HARNESS_LAUNCH_TIMEOUT;
+ while !cua_driver_testkit::observer::macos::application_presented(self.pid) {
+ assert!(
+ Instant::now() < deadline,
+ "AppKit harness pid {} did not finish launching and show its window within {:?}",
+ self.pid,
+ HARNESS_LAUNCH_TIMEOUT
+ );
+ std::thread::sleep(Duration::from_millis(50));
+ }
+ }
}
+/// Upper bound for a harness launch, cold first launches included.
+const HARNESS_LAUNCH_TIMEOUT: Duration = Duration::from_secs(20);
+
impl Drop for Harness {
fn drop(&mut self) {
let _ = self._app.kill();
@@ -616,6 +658,95 @@ fn harness_appkit_smoke() {
);
}
+/// AppKit registers a window with WindowServer when the app constructs it, but
+/// the app answers accessibility only once it enters its run loop; a cold
+/// launch can sit between the two for seconds. The first snapshot waits for
+/// the launch within `timeout_ms`, and a launch that outlasts the budget is
+/// reported as such instead of as an unexplained empty tree.
+#[test]
+#[ignore]
+fn harness_appkit_first_snapshot_waits_for_a_launching_app() {
+ let case = native_readonly_case(
+ "appkit",
+ "launching_app_snapshot",
+ Targeting::Ax,
+ DriverRoute::AxRead,
+ vec![OracleKind::AxState],
+ );
+ execute_case(case, |evidence| {
+ let mut driver = McpDriver::spawn_macos_daemon_proxy_named("appkit-launching-app-snapshot")
+ .expect("start installed macOS daemon proxy");
+ *evidence = recording_evidence(driver.recording_dir());
+ driver.start_behavior_recording();
+ let snapshot = |driver: &mut McpDriver, pid: u32, wid: u64, timeout_ms: u64| {
+ driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "timeout_ms": timeout_ms,
+ "include_screenshot": false
+ }),
+ )
+ };
+ let launching_window = |driver: &mut McpDriver, harness: &Harness| {
+ let (wid, _) = driver
+ .find_window(harness.pid as i64, "CuaTestHarness AppKit")
+ .expect("a launching AppKit window is listed before it can answer AX");
+ assert!(
+ !cua_driver_testkit::observer::macos::application_presented(harness.pid),
+ "the harness must still be launching when the snapshot starts"
+ );
+ wid
+ };
+
+ let stalled = Harness::launch_slowly(Duration::from_secs(4));
+ let wid = launching_window(&mut driver, &stalled);
+ let early = snapshot(&mut driver, stalled.pid, wid, 300);
+ let state = early.structured();
+ assert!(
+ !early.is_error(),
+ "a launching app degrades; it does not error: {}",
+ early.text()
+ );
+ assert_eq!(state["truncated"], true, "{state}");
+ assert_eq!(state["truncation_reason"], "app_lookup_timeout", "{state}");
+ assert_eq!(state["degraded"], true, "{state}");
+ assert!(
+ state["degraded_reason"]
+ .as_str()
+ .is_some_and(|reason| reason.starts_with("ax_app_launching")),
+ "{state}"
+ );
+ assert_eq!(state["elements"], serde_json::json!([]), "{state}");
+ assert!(
+ state.get("snapshot_id").is_none(),
+ "an unresolved launch must not publish element tokens: {state}"
+ );
+ drop(stalled);
+
+ let slow = Harness::launch_slowly(Duration::from_millis(2500));
+ let wid = launching_window(&mut driver, &slow);
+ let started = Instant::now();
+ let first = snapshot(&mut driver, slow.pid, wid, 10_000);
+ let waited = started.elapsed();
+ let state = first.structured();
+ assert!(!first.is_error(), "first snapshot: {}", first.text());
+ assert!(state.get("degraded").is_none(), "{state}");
+ assert_eq!(state["truncated"], false, "{state}");
+ for aid in ["wnd-main", "btn-increment", "btn-clicktarget", "txt-input"] {
+ assert!(
+ has_id(first.tree_text(), aid),
+ "first snapshot of a launching app is missing {aid}: {}",
+ first.tree_text()
+ );
+ }
+ assert!(first.tree_text().contains("slider_value=0"));
+ println!("first snapshot of a launching app returned the whole window after {waited:?}");
+ Observation::delivered(vec![OracleKind::AxState], Evidence::default())
+ });
+}
+
#[test]
#[ignore]
fn harness_appkit_query_projects_structured_elements() {
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
index 058179624e..0619163b11 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
@@ -685,6 +685,29 @@ pub mod macos {
}
}
+ /// Whether AppKit reports `pid` as finished launching and WindowServer
+ /// shows one of its windows on screen, observed without the driver.
+ ///
+ /// AppKit registers a window with WindowServer when the app constructs it,
+ /// before the app enters its run loop, so a fixture that only waits for its
+ /// window to be listed can start a case while every accessibility request
+ /// to the app still fails. Fixtures wait for this posture instead.
+ pub fn application_presented(pid: u32) -> bool {
+ let Ok(pid) = i32::try_from(pid) else {
+ return false;
+ };
+ // SAFETY: plain Objective-C messages on a retained
+ // NSRunningApplication; neither requires the main thread.
+ let finished = unsafe {
+ objc2_app_kit::NSRunningApplication::runningApplicationWithProcessIdentifier(pid)
+ .is_some_and(|app| app.isFinishedLaunching())
+ };
+ finished
+ && window_rows()
+ .iter()
+ .any(|row| row.pid == pid as u32 && row.on_screen)
+ }
+
fn frontmost_pid() -> Option<u64> {
unsafe {
Some(
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/launch.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/launch.rs
new file mode 100644
index 0000000000..823ae86afa
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/launch.rs
@@ -0,0 +1,188 @@
+//! Waiting for a launching application before its first AX walk.
+//!
+//! AppKit creates an `NSWindow` in WindowServer as soon as the app constructs
+//! it, so `list_windows` (and `launch_app`) can report the window while the
+//! application has not yet entered its run loop. Until it does, every AX
+//! request to the application element fails at once with
+//! `kAXErrorCannotComplete`, and a window-scoped walk finds zero `AXWindow`
+//! elements. Without a wait, the first `get_window_state` of a slow-launching
+//! app returned an empty tree that blamed the window scope.
+//!
+//! The walker retries window resolution while AppKit reports the process as
+//! still launching, bounded by the caller's `timeout_ms`. A process that has
+//! finished launching, or that LaunchServices does not know, is never waited
+//! on, so an ordinary unresolved window still fails fast.
+
+use std::time::{Duration, Instant};
+
+/// Interval between window-resolution attempts while an app launches.
+const LAUNCH_POLL: Duration = Duration::from_millis(50);
+
+/// Whether AppKit reports `pid` as an application that has not finished
+/// launching. `false` for finished apps and for processes LaunchServices
+/// does not know.
+pub fn is_still_launching(pid: i32) -> bool {
+ use objc2_app_kit::NSRunningApplication;
+ // SAFETY: both calls are plain Objective-C messages on a retained
+ // NSRunningApplication; neither requires the main thread.
+ unsafe {
+ NSRunningApplication::runningApplicationWithProcessIdentifier(pid)
+ .is_some_and(|app| !app.isFinishedLaunching())
+ }
+}
+
+/// What the walker does after a window-scoped resolution found nothing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum LaunchStep {
+ /// Report the resolution as it is.
+ Proceed,
+ /// The app is still launching and the budget allows another attempt.
+ Retry(Duration),
+ /// The app is still launching and the budget ran out while waiting.
+ TimedOut,
+}
+
+/// The bounded wait for one walk.
+///
+/// The deadline is anchored at the start of the first unresolved attempt, so
+/// an app that resolves immediately pays nothing and the time an attempt
+/// itself takes counts against the budget. An app whose AX server is
+/// registered but busy answers each request only after the messaging timeout;
+/// a retry is refused once another attempt of the same length would overrun
+/// the deadline, so the wait never extends a slow walk by a whole attempt.
+#[derive(Debug)]
+pub struct LaunchWait {
+ limit: Option<Duration>,
+ deadline: Option<Instant>,
+}
+
+impl LaunchWait {
+ /// `limit` is the caller's wall-clock budget; `None` (an internal,
+ /// node-only walk) never waits.
+ pub fn new(limit: Option<Duration>) -> Self {
+ Self {
+ limit,
+ deadline: None,
+ }
+ }
+
+ /// Decide the next step after an unresolved attempt that started at
+ /// `attempt_started` and ended at `now`. `still_launching` is only
+ /// consulted when the budget allows a wait.
+ pub fn step(
+ &mut self,
+ attempt_started: Instant,
+ now: Instant,
+ still_launching: impl FnOnce() -> bool,
+ ) -> LaunchStep {
+ let Some(limit) = self.limit else {
+ return LaunchStep::Proceed;
+ };
+ if !still_launching() {
+ return LaunchStep::Proceed;
+ }
+ let deadline = *self.deadline.get_or_insert(attempt_started + limit);
+ let attempt = now.saturating_duration_since(attempt_started);
+ match deadline
+ .checked_duration_since(now)
+ .and_then(|remaining| remaining.checked_sub(attempt))
+ {
+ Some(slack) if !slack.is_zero() => LaunchStep::Retry(slack.min(LAUNCH_POLL)),
+ _ => LaunchStep::TimedOut,
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const FAST: Duration = Duration::from_millis(5);
+
+ #[test]
+ fn a_node_only_walk_never_waits_or_asks_about_launch_state() {
+ let mut wait = LaunchWait::new(None);
+ let now = Instant::now();
+ let step = wait.step(now, now, || {
+ panic!("a node-only walk must not query launch state")
+ });
+ assert_eq!(step, LaunchStep::Proceed);
+ }
+
+ #[test]
+ fn a_finished_app_fails_fast() {
+ let mut wait = LaunchWait::new(Some(Duration::from_secs(1)));
+ let start = Instant::now();
+ assert_eq!(
+ wait.step(start, start + FAST, || false),
+ LaunchStep::Proceed
+ );
+ // No deadline was armed: a later launching attempt gets the whole budget.
+ let later = start + Duration::from_secs(5);
+ assert_eq!(
+ wait.step(later, later + FAST, || true),
+ LaunchStep::Retry(LAUNCH_POLL)
+ );
+ }
+
+ #[test]
+ fn a_launching_app_is_retried_until_the_budget_runs_out() {
+ let start = Instant::now();
+ let mut wait = LaunchWait::new(Some(Duration::from_millis(120)));
+ assert_eq!(
+ wait.step(start, start + FAST, || true),
+ LaunchStep::Retry(LAUNCH_POLL)
+ );
+ let late = start + Duration::from_millis(95);
+ assert_eq!(
+ wait.step(late, late + FAST, || true),
+ LaunchStep::Retry(Duration::from_millis(15)),
+ "the last pause leaves room for one more attempt of the same length"
+ );
+ let last = start + Duration::from_millis(115);
+ assert_eq!(wait.step(last, last + FAST, || true), LaunchStep::TimedOut);
+ let past = start + Duration::from_millis(500);
+ assert_eq!(wait.step(past, past + FAST, || true), LaunchStep::TimedOut);
+ }
+
+ #[test]
+ fn a_busy_launch_is_not_retried_past_its_budget() {
+ // AX is registered but the app is blocked, so each attempt waits out
+ // the messaging timeout. The first attempt already spent the budget.
+ let start = Instant::now();
+ let mut wait = LaunchWait::new(Some(Duration::from_secs(1)));
+ assert_eq!(
+ wait.step(start, start + Duration::from_secs(4), || true),
+ LaunchStep::TimedOut
+ );
+ // A slow attempt that fits once but not twice is not repeated.
+ let mut wait = LaunchWait::new(Some(Duration::from_secs(5)));
+ assert_eq!(
+ wait.step(start, start + Duration::from_secs(3), || true),
+ LaunchStep::TimedOut
+ );
+ }
+
+ #[test]
+ fn an_app_that_finishes_launching_mid_wait_proceeds() {
+ let start = Instant::now();
+ let mut wait = LaunchWait::new(Some(Duration::from_secs(1)));
+ assert_eq!(
+ wait.step(start, start + FAST, || true),
+ LaunchStep::Retry(LAUNCH_POLL)
+ );
+ let later = start + Duration::from_millis(300);
+ assert_eq!(
+ wait.step(later, later + FAST, || false),
+ LaunchStep::Proceed
+ );
+ }
+
+ #[test]
+ fn the_current_process_is_not_a_launching_app() {
+ // A test binary is not a LaunchServices application, so it must never
+ // be waited on.
+ assert!(!is_still_launching(std::process::id() as i32));
+ assert!(!is_still_launching(0));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
index ee7b6c405a..7640808f6f 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
@@ -20,6 +20,7 @@ pub mod bindings;
pub mod cache;
pub mod enablement;
pub mod exact_target;
+pub mod launch;
pub mod tree;
pub mod window_scope;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
index 84d6aa0d9a..540b105ae9 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
@@ -12,9 +12,11 @@
//! - Tree is walked depth-first; element_index is assigned in DFS order.
use super::bindings::*;
-use super::window_scope::{decide_window_scope, TopLevelCandidate, WindowScope};
+use super::launch::{is_still_launching, LaunchStep, LaunchWait};
+use super::window_scope::{decide_window_scope, ScopeDecision, TopLevelCandidate, WindowScope};
use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};
use cua_driver_core::walk_budget::{WalkBudget, WalkOutcome};
+use std::time::Instant;
/// Default maximum depth for AX tree walks. Deep menus and complex web views
/// can nest deeply; 25 covers realistic app chrome without exploding on
@@ -230,85 +232,57 @@ pub fn walk_tree_budgeted(
}
set_messaging_timeout(app_elem);
- // Chromium/Electron apps (Arc, VS Code, Electron shells) ship their
- // web-content AX tree OFF and only build it once an assistive client
- // asks for it. Without this, the first walk of such an app returns an
- // empty/title-bar-only tree (#1616). Flip the enablement attribute,
- // then — only when the flip actually took and only the first time we
- // see this process lifetime — let the asynchronously-built tree settle
- // before we read it. Native Cocoa apps reject the attribute, so they
- // pay no settle cost. This relies on the MAX_ELEMENTS node cap to keep
- // the now-materialized (potentially large) tree bounded.
- super::enablement::ensure_chromium_ax_enabled(pid, app_elem);
-
- // Union AXChildren + AXWindows — the only way to see background windows.
- // AXChildren omits windows when the app isn't frontmost (AppKit limitation).
- // AXWindows returns the window list regardless of activation state.
- let from_children = copy_children(app_elem);
- // A requested window on another Space is absent from AXWindows; the
- // `_including` variant recovers it by exact CGWindowID.
- let from_windows = match window_id {
- Some(wid) => copy_ax_windows_including(app_elem, pid, wid),
- None => copy_ax_windows(app_elem),
- };
-
- let mut top_level = from_children;
- for w in from_windows {
- // AXChildren and AXWindows can return different proxy pointers for
- // the same native window. CFEqual compares their AX identity;
- // pointer equality alone duplicates the whole subtree and can turn
- // one exact dialog action into a false ambiguity.
- if !top_level
- .iter()
- .any(|&e| CFEqual(e as CFTypeRef, w as CFTypeRef) != 0)
- {
- top_level.push(w);
- } else {
- // Already present — release the extra retain from copy_ax_windows.
- CFRelease(w as CFTypeRef);
- }
- }
-
- // Scope: keep non-window children (menu bar) + the target window —
- // but ONLY once the target window has actually been identified. When
- // nothing claims the requested id, `decide_window_scope` reports why
- // and walks nothing; it must never fall back to "everything that isn't
- // a window", which is how issue #2237 returned menu bars as panels.
- let walk_these: Vec<AXUIElementRef> = if let Some(wid) = window_id {
- let candidates: Vec<TopLevelCandidate> = top_level
- .iter()
- .map(|&child| {
- set_messaging_timeout(child);
- let role = copy_string_attr(child, "AXRole").unwrap_or_default();
- let subrole = copy_string_attr(child, "AXSubrole");
- let identifier = copy_string_attr(child, "AXIdentifier");
- // Match AX window element → CGWindowID via private SPI.
- // Only windows carry one, so skip the round-trip elsewhere.
- let ax_window_id = if role == "AXWindow" {
- ax_get_window_id(child)
- } else {
- None
- };
- TopLevelCandidate {
- role,
- subrole,
- identifier,
- ax_window_id,
+ // A window-scoped walk of an app that is still launching waits, within
+ // the caller's budget, for the app to answer AX (see `super::launch`).
+ let mut launch_wait = LaunchWait::new(budget.time_limit());
+ let (top_level, walk_these) = loop {
+ // Chromium/Electron apps (Arc, VS Code, Electron shells) ship their
+ // web-content AX tree OFF and only build it once an assistive client
+ // asks for it. Without this, the first walk of such an app returns an
+ // empty/title-bar-only tree (#1616). Flip the enablement attribute,
+ // then — only when the flip actually took and only the first time we
+ // see this process lifetime — let the asynchronously-built tree settle
+ // before we read it. Native Cocoa apps reject the attribute, so they
+ // pay no settle cost. This relies on the MAX_ELEMENTS node cap to keep
+ // the now-materialized (potentially large) tree bounded. An app that
+ // has not started answering AX rejects the flip without caching the
+ // refusal, so each launch-wait attempt asks again.
+ let attempt_started = Instant::now();
+ super::enablement::ensure_chromium_ax_enabled(pid, app_elem);
+
+ let top_level = copy_top_level(app_elem, pid, window_id);
+
+ // Scope: keep non-window children (menu bar) + the target window —
+ // but ONLY once the target window has actually been identified. When
+ // nothing claims the requested id, `decide_window_scope` reports why
+ // and walks nothing; it must never fall back to "everything that isn't
+ // a window", which is how issue #2237 returned menu bars as panels.
+ let Some(wid) = window_id else {
+ let walk = top_level.clone();
+ break (top_level, walk);
+ };
+ let decision = scope_top_level(&top_level, pid, wid);
+ if matches!(decision.scope, WindowScope::AxUnresolved { .. }) {
+ match launch_wait.step(attempt_started, Instant::now(), || is_still_launching(pid))
+ {
+ LaunchStep::Retry(pause) => {
+ release_all(top_level);
+ std::thread::sleep(pause);
+ continue;
}
- })
- .collect();
- let decision = decide_window_scope(&candidates, wid, || {
- crate::windows::resolve_window_owner(pid, wid)
- });
+ // Still unresolved because the app never finished
+ // launching: say so instead of blaming the window scope.
+ LaunchStep::TimedOut => budget.stop_for_app_lookup_timeout(),
+ LaunchStep::Proceed => {}
+ }
+ }
let walk = decision
.walk
.iter()
.map(|&index| top_level[index])
.collect();
window_scope = Some(decision.scope);
- walk
- } else {
- top_level.to_vec()
+ break (top_level, walk);
};
// Walk each top-level child at depth 0.
@@ -327,9 +301,7 @@ pub fn walk_tree_budgeted(
}
// Release all top-level elements (copy_children / copy_ax_windows both retain).
- for child in top_level {
- CFRelease(child as CFTypeRef);
- }
+ release_all(top_level);
CFRelease(app_elem as CFTypeRef);
}
@@ -356,6 +328,78 @@ pub fn walk_tree_budgeted(
}
}
+/// The application's top-level AX children: `AXChildren` ∪ `AXWindows`.
+///
+/// `AXChildren` omits windows when the app isn't frontmost (AppKit
+/// limitation); `AXWindows` returns the window list regardless of activation
+/// state, so the union is the only way to see background windows. Every
+/// returned element is retained; release them with [`release_all`].
+unsafe fn copy_top_level(
+ app_elem: AXUIElementRef,
+ pid: i32,
+ window_id: Option<u32>,
+) -> Vec<AXUIElementRef> {
+ let mut top_level = copy_children(app_elem);
+ // A requested window on another Space is absent from AXWindows; the
+ // `_including` variant recovers it by exact CGWindowID.
+ let from_windows = match window_id {
+ Some(wid) => copy_ax_windows_including(app_elem, pid, wid),
+ None => copy_ax_windows(app_elem),
+ };
+ for w in from_windows {
+ // AXChildren and AXWindows can return different proxy pointers for
+ // the same native window. CFEqual compares their AX identity;
+ // pointer equality alone duplicates the whole subtree and can turn
+ // one exact dialog action into a false ambiguity.
+ if !top_level
+ .iter()
+ .any(|&e| CFEqual(e as CFTypeRef, w as CFTypeRef) != 0)
+ {
+ top_level.push(w);
+ } else {
+ // Already present — release the extra retain from copy_ax_windows.
+ CFRelease(w as CFTypeRef);
+ }
+ }
+ top_level
+}
+
+/// Decide which of `top_level` a walk scoped to `wid` covers.
+unsafe fn scope_top_level(top_level: &[AXUIElementRef], pid: i32, wid: u32) -> ScopeDecision {
+ let candidates: Vec<TopLevelCandidate> = top_level
+ .iter()
+ .map(|&child| {
+ set_messaging_timeout(child);
+ let role = copy_string_attr(child, "AXRole").unwrap_or_default();
+ let subrole = copy_string_attr(child, "AXSubrole");
+ let identifier = copy_string_attr(child, "AXIdentifier");
+ // Match AX window element → CGWindowID via private SPI.
+ // Only windows carry one, so skip the round-trip elsewhere.
+ let ax_window_id = if role == "AXWindow" {
+ ax_get_window_id(child)
+ } else {
+ None
+ };
+ TopLevelCandidate {
+ role,
+ subrole,
+ identifier,
+ ax_window_id,
+ }
+ })
+ .collect();
+ decide_window_scope(&candidates, wid, || {
+ crate::windows::resolve_window_owner(pid, wid)
+ })
+}
+
+/// Release elements retained by [`copy_top_level`].
+unsafe fn release_all(elements: Vec<AXUIElementRef>) {
+ for element in elements {
+ CFRelease(element as CFTypeRef);
+ }
+}
+
#[allow(clippy::too_many_arguments)]
unsafe fn walk_element(
element: AXUIElementRef,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index 84d48c7294..1635189fff 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -71,7 +71,11 @@ fn def() -> &'static ToolDef {
this pid but its accessibility surface can't be resolved, the tree comes back \
EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the \
requested window; background input is refused until it resolves, so \
- re-snapshot or act with `delivery_mode:\"foreground\"`. A window on another \
+ re-snapshot or act with `delivery_mode:\"foreground\"`. When that pid is an \
+ app still launching (its window exists before it answers accessibility), the \
+ walk first waits up to `timeout_ms` for it; if it never answers, the tree comes \
+ back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and \
+ `truncation_reason: app_lookup_timeout`. A window on another \
Space still resolves by its exact CGWindowID. This tool never returns another \
surface's elements under your window_id. Before exposing a screenshot, \
its raw dimensions are validated as a coherent 1x/2x representation of \
@@ -311,7 +315,10 @@ impl Tool for GetWindowStateTool {
let payload = crate::ax::cache::CachedSnapshot::from_nodes(&tree.nodes);
(tree, payload)
});
- let backstop = std::time::Duration::from_millis(timeout_ms) + AX_WALK_BACKSTOP_GRACE;
+ // A launching app is waited on for up to `timeout_ms` before the
+ // walk's own `timeout_ms` starts (see `ax::launch`).
+ let backstop =
+ std::time::Duration::from_millis(timeout_ms) * 2 + AX_WALK_BACKSTOP_GRACE;
match tokio::time::timeout(backstop, walk_future).await {
Ok(Ok((tree, payload))) => (Some(tree), Some(payload)),
Ok(Err(e)) => return ToolResult::error(format!("AX tree walk failed: {e}")),
@@ -630,7 +637,15 @@ impl Tool for GetWindowStateTool {
// the screenshot already in this response and click by pixel (x,y).
// macOS can pixel-target in the background, so the recommendation is
// `px`, not `foreground`.
- match degradation_for(tree_result.is_some(), element_count, window_scope.as_ref()) {
+ let app_lookup_timed_out = tree_result
+ .as_ref()
+ .is_some_and(|r| r.walk.reason() == Some("app_lookup_timeout"));
+ match degradation_for(
+ tree_result.is_some(),
+ element_count,
+ window_scope.as_ref(),
+ app_lookup_timed_out,
+ ) {
Degradation::None => {}
Degradation::AxTreeEmpty => {
structured["degraded"] = serde_json::json!(true);
@@ -648,6 +663,24 @@ impl Tool for GetWindowStateTool {
in this response (an element px action)."
});
}
+ Degradation::AxAppLaunching => {
+ structured["degraded"] = serde_json::json!(true);
+ structured["degraded_reason"] = serde_json::json!(format!(
+ "ax_app_launching: window_id {window_id} exists and is owned by pid \
+ {pid}, but that app has not finished launching and did not answer \
+ accessibility within the {timeout_ms} ms timeout_ms budget. The tree is \
+ returned EMPTY because the window's accessibility surface is not \
+ available yet."
+ ));
+ structured["escalation"] = serde_json::json!({
+ "recommended": "foreground",
+ "reason": "observation-only until the app finishes launching: re-snapshot \
+ in a moment or with a larger timeout_ms. Background input \
+ (including px) is refused while the window's AX surface is \
+ unresolved; act with delivery_mode:\"foreground\" only if \
+ you cannot wait."
+ });
+ }
Degradation::AxWindowUnresolved { ax_window_count } => {
structured["degraded"] = serde_json::json!(true);
structured["degraded_reason"] = serde_json::json!(format!(
@@ -819,6 +852,9 @@ enum Degradation {
/// The requested window is live and owned by this pid, but no AXWindow
/// claims its CGWindowID, so the walk deliberately covered nothing.
AxWindowUnresolved { ax_window_count: usize },
+ /// The window scope is unresolved because the app is still launching and
+ /// did not answer accessibility within the caller's budget.
+ AxAppLaunching,
}
/// Decide the degradation rung. Pure: `walk_attempted` is false in the
@@ -829,11 +865,15 @@ fn degradation_for(
walk_attempted: bool,
element_count: usize,
scope: Option<&crate::ax::WindowScope>,
+ app_lookup_timed_out: bool,
) -> Degradation {
if !walk_attempted {
return Degradation::None;
}
if let Some(crate::ax::WindowScope::AxUnresolved { ax_window_count }) = scope {
+ if app_lookup_timed_out {
+ return Degradation::AxAppLaunching;
+ }
return Degradation::AxWindowUnresolved {
ax_window_count: *ax_window_count,
};
@@ -1039,17 +1079,38 @@ mod window_scope_contract_tests {
degradation_for(
true,
0,
- Some(&WindowScope::AxUnresolved { ax_window_count: 3 })
+ Some(&WindowScope::AxUnresolved { ax_window_count: 3 }),
+ false
),
Degradation::AxWindowUnresolved { ax_window_count: 3 }
);
}
+ /// A window that exists before its app answers accessibility is not an
+ /// unscoped window: the degradation names the launch instead.
+ #[test]
+ fn a_launch_that_outlasts_the_budget_degrades_as_app_launching() {
+ assert_eq!(
+ degradation_for(
+ true,
+ 0,
+ Some(&WindowScope::AxUnresolved { ax_window_count: 0 }),
+ true
+ ),
+ Degradation::AxAppLaunching
+ );
+ // A walk cut short for another reason after resolving keeps its rung.
+ assert_eq!(
+ degradation_for(true, 0, Some(&WindowScope::Matched), true),
+ Degradation::AxTreeEmpty
+ );
+ }
+
#[test]
fn empty_tree_still_degrades_as_ax_tree_empty() {
// Back-compat with the pre-existing rung.
assert_eq!(
- degradation_for(true, 0, Some(&WindowScope::Matched)),
+ degradation_for(true, 0, Some(&WindowScope::Matched), false),
Degradation::AxTreeEmpty
);
}
@@ -1057,14 +1118,14 @@ mod window_scope_contract_tests {
#[test]
fn resolved_window_with_elements_is_not_degraded() {
assert_eq!(
- degradation_for(true, 42, Some(&WindowScope::Matched)),
+ degradation_for(true, 42, Some(&WindowScope::Matched), false),
Degradation::None
);
}
#[test]
fn screenshot_only_path_does_not_degrade() {
- assert_eq!(degradation_for(false, 0, None), Degradation::None);
+ assert_eq!(degradation_for(false, 0, None, false), Degradation::None);
}
#[test]
@@ -1074,6 +1135,8 @@ mod window_scope_contract_tests {
"window_id_not_found",
"window_owner_pid_mismatch",
"ax_window_unresolved",
+ "ax_app_launching",
+ "app_lookup_timeout",
] {
assert!(
description.contains(code),
diff --git a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
index f2601dd107..2572a82dcd 100644
--- a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
+++ b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
@@ -775,6 +775,14 @@ struct CuaAppKitHarness {
}
app.activate(ignoringOtherApps: true)
writeBringToFrontWindowReport(main: controller.window, matrix: matrixWindows)
+ if let delay = ProcessInfo.processInfo.environment["CUA_APPKIT_LAUNCH_DELAY_MS"]
+ .flatMap(Double.init), delay > 0 {
+ // A slow launch on demand: the titled window is already registered
+ // with WindowServer, but the app has not entered its run loop, so
+ // it cannot answer accessibility yet. Cold hosted runners reach
+ // this state on their own for seconds.
+ Thread.sleep(forTimeInterval: delay / 1000)
+ }
app.run()
_ = matrixWindows
}
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index 2e895411f8..2abab99699 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -344,6 +344,7 @@ if [[ "${SUITE}" == native || "${SUITE}" == all ]]; then
--ignored --nocapture --test-threads=1
for appkit_test in \
harness_appkit_smoke \
+ harness_appkit_first_snapshot_waits_for_a_launching_app \
harness_appkit_query_projects_structured_elements \
harness_appkit_stale_element_token_fails_closed \
snapshot_publication::harness_appkit_pending_snapshot_cannot_retarget_token \