Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): report a locked screen and keep the display awake while an agent works (#5483)

Francesco Bonaccib1633ef91cb0
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
index 2a20758e05..e1f6251d75 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx
@@ -36,7 +36,7 @@ DIFF READS (use them for every re-read after acting): `since:"latest"`, or `sinc
A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
-Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.
+Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.
**macOS parameters**
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
index 0270e9dffe..eae7b3b7ca 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
@@ -74,6 +74,12 @@ pub struct BackgroundTargetFacts {
pub competing_keyboard_destinations: usize,
/// Ancestry proof for an explicitly addressed element, when one exists.
pub element: ElementAncestry,
+ /// The login session reports its screen as locked (macOS: the screen
+ /// saver, display sleep or a lock put the session behind the lock
+ /// screen). The lock screen owns input while this holds and every app's
+ /// accessibility windows read as empty, so no route can reach the target.
+ /// `false` when the platform cannot tell; the other facts then decide.
+ pub screen_locked: bool,
}
/// The background action classes v1 distinguishes. Route order and trust are
@@ -120,8 +126,21 @@ pub mod refusal_codes {
pub const MINIMIZED_OR_HIDDEN: &str = "minimized_or_hidden_window";
pub const SAME_PID_KEYBOARD_AMBIGUITY: &str = "same_pid_keyboard_ambiguity";
pub const ELEMENT_OUTSIDE_TARGET_WINDOW: &str = "element_outside_target_window";
+ pub const SCREEN_LOCKED: &str = "screen_locked";
}
+/// Shown with every `screen_locked` refusal: what happened and how to keep it
+/// from happening while an agent works. Never advises foreground input, which
+/// would land on the lock screen.
+pub const SCREEN_LOCKED_HINT: &str = "the screen is locked or showing the screen saver, \
+ which also happens when the screen saver starts or the display sleeps with no password \
+ required. While it lasts, macOS hides every app's windows from accessibility and sends \
+ input to the lock screen, so Cua Driver refuses instead of acting; screenshots still \
+ work. Wake and unlock the Mac, then retry. Cua Driver keeps the display awake for 10 \
+ minutes after each tool call (CUA_DRIVER_KEEP_DISPLAY_AWAKE sets the seconds, 0 turns \
+ it off), so this usually means the Mac sat idle between agent turns; running \
+ `caffeinate -d` keeps it awake for longer";
+
/// A typed refusal: no actuator ran and none may run for this decision.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BackgroundRefusal {
@@ -221,6 +240,20 @@ pub fn decide_background_input(
WindowServerOwnership::SamePid => {}
}
+ // A locked session hides every app's AX windows, so the AX checks below
+ // would misreport it as an off-Space window and advise foreground input,
+ // which would land on the lock screen. Report the real cause first.
+ if facts.screen_locked {
+ return refuse(
+ refusal_codes::SCREEN_LOCKED,
+ format!(
+ "window {} cannot receive input: {SCREEN_LOCKED_HINT}",
+ target.window_id
+ ),
+ None,
+ );
+ }
+
match facts.element {
ElementAncestry::NotAddressed | ElementAncestry::ProvenDescendant => {}
ElementAncestry::OutsideTargetWindow | ElementAncestry::Unproven => {
@@ -328,6 +361,7 @@ fn exact_window_status(facts: &BackgroundTargetFacts) -> &'static str {
match &facts.window_server {
WindowServerOwnership::NotFound => "not_found",
WindowServerOwnership::ForeignPid { .. } => "owner_mismatch",
+ WindowServerOwnership::SamePid if facts.screen_locked => "screen_locked",
WindowServerOwnership::SamePid if facts.ax_window_present => "matched",
WindowServerOwnership::SamePid => "ax_unresolved",
}
@@ -394,6 +428,7 @@ mod tests {
app_hidden: Some(false),
competing_keyboard_destinations: 0,
element: ElementAncestry::NotAddressed,
+ screen_locked: false,
}
}
@@ -658,6 +693,7 @@ mod tests {
app_hidden: Some(true),
competing_keyboard_destinations: 3,
element: ElementAncestry::OutsideTargetWindow,
+ screen_locked: false,
};
assert_eq!(
code_of(decide_background_input(
@@ -746,4 +782,52 @@ mod tests {
);
assert_eq!(report["observation"]["one_shot_capture"], "unknown");
}
+
+ /// A locked session (screen saver or display sleep) empties every app's
+ /// AXWindows. Every route refuses with `screen_locked`, not the misleading
+ /// off-Space code, and no refusal advises foreground input, which would
+ /// reach the lock screen.
+ #[test]
+ fn locked_screen_refuses_every_route_without_foreground_advice() {
+ let facts = BackgroundTargetFacts {
+ screen_locked: true,
+ ax_window_present: false,
+ element: ElementAncestry::Unproven,
+ ..matched_facts()
+ };
+ for action in ALL_ACTIONS {
+ match decide_background_input(TARGET, &facts, action) {
+ BackgroundInputDecision::Refuse(refusal) => {
+ assert_eq!(refusal.code, refusal_codes::SCREEN_LOCKED, "{action:?}");
+ assert_eq!(refusal.advice, None, "{action:?}");
+ assert!(refusal.reason.contains("display sleeps"));
+ }
+ BackgroundInputDecision::Execute { .. } => panic!("{action:?} must refuse"),
+ }
+ }
+ let report = background_input_capability_report(TARGET, &facts, Some(true));
+ assert_eq!(report["exact_window"]["status"], "screen_locked");
+ for route in report["routes"].as_array().unwrap() {
+ assert_eq!(route["reason"], refusal_codes::SCREEN_LOCKED);
+ }
+ }
+
+ /// A stale or foreign window id is still reported as such while locked:
+ /// re-targeting is needed either way.
+ #[test]
+ fn ownership_failures_precede_the_lock() {
+ let facts = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::NotFound,
+ screen_locked: true,
+ ..matched_facts()
+ };
+ assert_eq!(
+ code_of(decide_background_input(
+ TARGET,
+ &facts,
+ BackgroundAction::AxSemantic
+ )),
+ refusal_codes::WINDOW_NOT_FOUND
+ );
+ }
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index 611b058030..5b739537ff 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -111,6 +111,7 @@ pub mod socket_io;
pub mod text_sanitize;
pub mod timestamp;
pub mod tool;
+pub mod tool_activity;
pub mod tool_args;
pub mod tool_schema;
pub mod video;
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index 76dcbf27a1..103ebcc5d7 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -2302,6 +2302,7 @@ impl ToolRegistry {
// Desktop pixels read off a capped get_desktop_state image are mapped
// back to the uncapped capture before any platform interprets them.
crate::desktop_capture_scale::map_desktop_args(&mut args);
+ crate::tool_activity::note_tool_activity();
let mut result = crate::recording::scope_dispatch_click_capture(
pending_turn.as_ref(),
tool.invoke(args.clone()),
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rs
new file mode 100644
index 0000000000..303820ca90
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rs
@@ -0,0 +1,25 @@
+//! Per-call activity hook: the platform shell learns that an agent is
+//! working each time the registry dispatches a tool.
+//!
+//! macOS uses it to keep the display awake while an agent works, because the
+//! screen saver and display sleep put the session behind the lock screen,
+//! where no app's windows are reachable. Registered once by the platform
+//! crate; a no-op when nothing is registered.
+
+use std::sync::OnceLock;
+
+type ToolActivityFn = Box<dyn Fn() + Send + Sync>;
+static TOOL_ACTIVITY_FN: OnceLock<ToolActivityFn> = OnceLock::new();
+
+/// Register the activity callback. Returns false when one is already set;
+/// the first registration wins.
+pub fn set_tool_activity_fn(f: impl Fn() + Send + Sync + 'static) -> bool {
+ TOOL_ACTIVITY_FN.set(Box::new(f)).is_ok()
+}
+
+/// Report one tool dispatch. Must stay cheap: it runs on every call.
+pub fn note_tool_activity() {
+ if let Some(f) = TOOL_ACTIVITY_FN.get() {
+ f();
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs
index 74485384b3..7ccdae7944 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs
@@ -355,6 +355,7 @@ pub fn gather_background_facts(
app_hidden,
competing_keyboard_destinations,
element: element.unwrap_or(ElementAncestry::NotAddressed),
+ screen_locked: crate::session::screen_locked(),
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rs b/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rs
new file mode 100644
index 0000000000..9268bd094a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rs
@@ -0,0 +1,221 @@
+//! Keep the display awake while an agent is working.
+//!
+//! On macOS 14 and later the screen saver and display sleep put the login
+//! session behind the lock screen, even when no password is required. While
+//! that holds, every app's accessibility windows read as empty and input goes
+//! to the lock screen, so background actions cannot run. An agent working on
+//! an idle Mac (no one touching the keyboard or mouse) hits this as soon as
+//! the idle timer fires.
+//!
+//! Each tool call holds an IOKit `PreventUserIdleDisplaySleep` assertion (the
+//! one `caffeinate -d` and video players take) until no tool has run for the
+//! idle window. It only stops the *idle* timer: it never wakes a sleeping
+//! display, dismisses the screen saver, or gets past a lock or password, and
+//! the user can still sleep or lock the Mac by hand.
+//!
+//! The idle display-sleep path dims the display first, and loginwindow can
+//! lock on that dim (`kLWLockFromDisplayDim`) without the screen saver ever
+//! running. A `PreventUserIdleDisplaySleep` assertion holds off the idle dim as
+//! well as the sleep that follows (the display stays at full brightness during
+//! video playback, which takes the same assertion), so that lock does not fire
+//! while the guard is held. It is not a system-sleep assertion: keeping the
+//! machine awake is not enough, because the lock comes from the display path.
+//!
+//! `CUA_DRIVER_KEEP_DISPLAY_AWAKE` sets the idle window in seconds (default
+//! 600); `0`, `off` or `false` turns the guard off.
+
+use std::sync::Mutex;
+use std::time::{Duration, Instant};
+
+use core_foundation::base::TCFType;
+use core_foundation::string::{CFString, CFStringRef};
+
+/// Environment variable that tunes or disables the guard.
+pub const KEEP_DISPLAY_AWAKE_ENV: &str = "CUA_DRIVER_KEEP_DISPLAY_AWAKE";
+
+/// How long the display stays awake after the last tool call by default.
+pub const DEFAULT_IDLE_WINDOW: Duration = Duration::from_secs(600);
+
+type IOPMAssertionID = u32;
+type IOReturn = i32;
+const K_IOPM_ASSERTION_LEVEL_ON: u32 = 255;
+const K_IO_RETURN_SUCCESS: IOReturn = 0;
+
+#[link(name = "IOKit", kind = "framework")]
+extern "C" {
+ fn IOPMAssertionCreateWithName(
+ assertion_type: CFStringRef,
+ assertion_level: u32,
+ assertion_name: CFStringRef,
+ assertion_id: *mut IOPMAssertionID,
+ ) -> IOReturn;
+ fn IOPMAssertionRelease(assertion_id: IOPMAssertionID) -> IOReturn;
+}
+
+/// Parse the env value into the idle window, or `None` when disabled.
+pub fn idle_window_from_env(value: Option<&str>) -> Option<Duration> {
+ let Some(raw) = value.map(str::trim).filter(|raw| !raw.is_empty()) else {
+ return Some(DEFAULT_IDLE_WINDOW);
+ };
+ match raw.to_ascii_lowercase().as_str() {
+ "0" | "off" | "false" | "no" => None,
+ "on" | "true" | "yes" => Some(DEFAULT_IDLE_WINDOW),
+ other => match other.parse::<u64>() {
+ Ok(secs) => Some(Duration::from_secs(secs.min(86_400))),
+ Err(_) => {
+ tracing::warn!(
+ "{KEEP_DISPLAY_AWAKE_ENV}={raw:?} is not a number of seconds or off; \
+ using the default"
+ );
+ Some(DEFAULT_IDLE_WINDOW)
+ }
+ },
+ }
+}
+
+struct Guard {
+ assertion: IOPMAssertionID,
+ last_activity: Instant,
+}
+
+static GUARD: Mutex<Option<Guard>> = Mutex::new(None);
+
+/// Register the guard as the registry's tool-activity hook, unless the env
+/// turns it off. Safe to call more than once.
+pub fn install() {
+ let Some(idle) = idle_window_from_env(std::env::var(KEEP_DISPLAY_AWAKE_ENV).ok().as_deref())
+ else {
+ tracing::info!("display-awake guard off ({KEEP_DISPLAY_AWAKE_ENV})");
+ return;
+ };
+ cua_driver_core::tool_activity::set_tool_activity_fn(move || note_activity(idle));
+}
+
+/// Hold the assertion now and keep it until `idle` passes with no activity.
+pub fn note_activity(idle: Duration) {
+ let mut guard = GUARD
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ if let Some(held) = guard.as_mut() {
+ held.last_activity = Instant::now();
+ return;
+ }
+ let Some(assertion) = create_assertion() else {
+ return;
+ };
+ *guard = Some(Guard {
+ assertion,
+ last_activity: Instant::now(),
+ });
+ drop(guard);
+ let spawned = std::thread::Builder::new()
+ .name("cua-display-awake".into())
+ .spawn(move || release_when_idle(idle));
+ if spawned.is_err() {
+ release_now();
+ }
+}
+
+fn release_when_idle(idle: Duration) {
+ loop {
+ let wait = {
+ let guard = GUARD
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ let Some(held) = guard.as_ref() else {
+ return;
+ };
+ remaining(held.last_activity.elapsed(), idle)
+ };
+ match wait {
+ Some(wait) => std::thread::sleep(wait),
+ None => {
+ release_now();
+ return;
+ }
+ }
+ }
+}
+
+/// Time left before release, or `None` once the idle window has passed.
+fn remaining(elapsed: Duration, idle: Duration) -> Option<Duration> {
+ idle.checked_sub(elapsed).filter(|left| !left.is_zero())
+}
+
+fn release_now() {
+ let held = GUARD
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .take();
+ if let Some(held) = held {
+ unsafe { IOPMAssertionRelease(held.assertion) };
+ tracing::debug!("display-awake guard released");
+ }
+}
+
+fn create_assertion() -> Option<IOPMAssertionID> {
+ let kind = CFString::from_static_string("PreventUserIdleDisplaySleep");
+ let name = CFString::from_static_string("Cua Driver: an agent is working on this Mac");
+ let mut id: IOPMAssertionID = 0;
+ let status = unsafe {
+ IOPMAssertionCreateWithName(
+ kind.as_concrete_TypeRef(),
+ K_IOPM_ASSERTION_LEVEL_ON,
+ name.as_concrete_TypeRef(),
+ &mut id,
+ )
+ };
+ if status == K_IO_RETURN_SUCCESS {
+ tracing::debug!("display-awake guard held");
+ Some(id)
+ } else {
+ tracing::warn!("could not keep the display awake: IOPMAssertionCreateWithName {status}");
+ None
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn env_defaults_to_ten_minutes_and_can_be_turned_off() {
+ assert_eq!(idle_window_from_env(None), Some(DEFAULT_IDLE_WINDOW));
+ assert_eq!(idle_window_from_env(Some("")), Some(DEFAULT_IDLE_WINDOW));
+ assert_eq!(idle_window_from_env(Some("on")), Some(DEFAULT_IDLE_WINDOW));
+ for off in ["0", "off", "FALSE", "no"] {
+ assert_eq!(idle_window_from_env(Some(off)), None, "{off}");
+ }
+ assert_eq!(
+ idle_window_from_env(Some("120")),
+ Some(Duration::from_secs(120))
+ );
+ assert_eq!(
+ idle_window_from_env(Some("999999")),
+ Some(Duration::from_secs(86_400))
+ );
+ assert_eq!(
+ idle_window_from_env(Some("soon")),
+ Some(DEFAULT_IDLE_WINDOW)
+ );
+ }
+
+ #[test]
+ fn release_waits_for_the_full_idle_window() {
+ let idle = Duration::from_secs(600);
+ assert_eq!(
+ remaining(Duration::from_secs(100), idle),
+ Some(Duration::from_secs(500))
+ );
+ assert_eq!(remaining(idle, idle), None);
+ assert_eq!(remaining(Duration::from_secs(700), idle), None);
+ }
+
+ /// Holding and releasing the real assertion works and leaves nothing
+ /// held. The idle timer is not reset, so this does not wake the display.
+ #[test]
+ fn assertion_is_created_and_released() {
+ let id = create_assertion().expect("IOPMAssertionCreateWithName");
+ assert_eq!(unsafe { IOPMAssertionRelease(id) }, K_IO_RETURN_SUCCESS);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
index 329c629853..399a636aca 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
@@ -28,6 +28,8 @@ pub mod capture_consent;
#[cfg(target_os = "macos")]
pub mod cursor;
#[cfg(target_os = "macos")]
+pub mod display_awake;
+#[cfg(target_os = "macos")]
pub mod document_edit;
#[cfg(target_os = "macos")]
pub mod focus_guard;
@@ -161,6 +163,9 @@ pub fn register_tools_with_cursor_and_provider(
// that will never arrive. Unconditional: emission depends on the
// registry write path, not on the overlay or on graphic access.
cua_driver_core::cursor_hook::declare_cursor_hook_emitter();
+ // Keep the display awake while tools run, so the screen saver and
+ // display sleep do not lock the session mid-task.
+ display_awake::install();
let mut r = ToolRegistry::new_with_protected_consent_provider(provider);
tools::register_all(
&mut r,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/session.rs b/libs/cua-driver/rust/crates/platform-macos/src/session.rs
index 80a444c913..b03e7138f5 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/session.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/session.rs
@@ -43,10 +43,63 @@ pub fn has_graphic_access() -> bool {
status == 0 && (attributes & SESSION_HAS_GRAPHIC_ACCESS) != 0
}
+#[link(name = "CoreGraphics", kind = "framework")]
+extern "C" {
+ fn CGSessionCopyCurrentDictionary() -> core_foundation::dictionary::CFDictionaryRef;
+}
+
+/// Whether the login session currently reports its screen as locked
+/// (`CGSSessionScreenIsLocked` in `CGSessionCopyCurrentDictionary`).
+///
+/// Read fresh on every call; lock state changes at any time. On macOS 14 and
+/// later the screen saver and display sleep put the session behind the lock
+/// screen (`loginwindow` frontmost) even before a password is required, and
+/// while that holds every app's `AXWindows` reads as empty. The key is the
+/// session's lock flag, not a per-cause one: it covers the screen-saver lock,
+/// a manual lock, and loginwindow's lock on display dim (`kLWLockFromDisplayDim`,
+/// seen in a user's display-sleep logs with `screenSaverIsRunning = 0`). Only
+/// the screen-saver path was reproduced here; the dim path is the same lock
+/// as far as this key is concerned, so no separate check exists. `false` when the
+/// session dictionary is unavailable (no window-server session) or the key is
+/// absent, which is how an unlocked session reports itself.
+pub fn screen_locked() -> bool {
+ use core_foundation::base::TCFType;
+ use core_foundation::boolean::CFBoolean;
+ use core_foundation::dictionary::CFDictionary;
+ use core_foundation::number::CFNumber;
+ use core_foundation::string::CFString;
+
+ let raw = unsafe { CGSessionCopyCurrentDictionary() };
+ if raw.is_null() {
+ return false;
+ }
+ // SAFETY: the "Copy" function returns a +1 reference we now own.
+ let dict: CFDictionary = unsafe { CFDictionary::wrap_under_create_rule(raw) };
+ let key = CFString::from_static_string("CGSSessionScreenIsLocked");
+ let Some(value) = dict.find(key.as_CFTypeRef() as *const _) else {
+ return false;
+ };
+ let value = unsafe { core_foundation::base::CFType::wrap_under_get_rule(*value as _) };
+ if let Some(flag) = value.downcast::<CFBoolean>() {
+ return flag.into();
+ }
+ value
+ .downcast::<CFNumber>()
+ .and_then(|number| number.to_i64())
+ .is_some_and(|number| number != 0)
+}
+
#[cfg(test)]
mod tests {
use super::*;
+ /// Resolves the CoreGraphics symbol and reads the session dictionary
+ /// without crashing, with or without a window-server session.
+ #[test]
+ fn screen_lock_probe_does_not_abort() {
+ let _ = screen_locked();
+ }
+
/// The probe must resolve the `Security` symbol and return a `bool`
/// without aborting. We can't assert the value: it's `true` on a dev GUI
/// session and `false` on a headless CI runner — which is exactly the
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index 8c4d118990..a1a03ed93b 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -35,7 +35,7 @@ fn def() -> &'static ToolDef {
\n\
A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\
\n\
- Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
+ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),
input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({
"type": "object",
"required": ["pid", "window_id"],
@@ -664,13 +664,32 @@ impl Tool for GetWindowStateTool {
let app_lookup_timed_out = tree_result
.as_ref()
.is_some_and(|r| r.walk.reason() == Some("app_lookup_timeout"));
- match degradation_for(
+ let degradation = degradation_for(
tree_result.is_some(),
element_count,
window_scope.as_ref(),
app_lookup_timed_out,
- ) {
+ );
+ let degradation = explain_locked_screen(
+ degradation,
+ !matches!(degradation, Degradation::None) && crate::session::screen_locked(),
+ );
+ match degradation {
Degradation::None => {}
+ Degradation::ScreenLocked => {
+ structured["degraded"] = serde_json::json!(true);
+ structured["degraded_reason"] = serde_json::json!(format!(
+ "screen_locked: window_id {window_id} has no accessibility tree because {}.",
+ cua_driver_core::background_input::SCREEN_LOCKED_HINT
+ ));
+ structured["escalation"] = serde_json::json!({
+ "recommended": "wait_for_unlock",
+ "reason": "observation-only while the screen is locked: no delivery \
+ mode can reach the app, and foreground input would go to \
+ the lock screen. Re-snapshot after the Mac is woken and \
+ unlocked."
+ });
+ }
Degradation::AxTreeEmpty => {
structured["degraded"] = serde_json::json!(true);
structured["degraded_reason"] = serde_json::json!(
@@ -957,6 +976,20 @@ enum Degradation {
/// The window scope is unresolved because the app is still launching and
/// did not answer accessibility within the caller's budget.
AxAppLaunching,
+ /// The session's screen is locked (screen saver, display sleep or a
+ /// lock). macOS then reports no AXWindows for any app, so an empty or
+ /// unresolved tree has this cause rather than the app's own state.
+ ScreenLocked,
+}
+
+/// Name the locked screen as the cause of an empty or unresolved tree.
+/// `locked` is only probed when the walk already degraded.
+fn explain_locked_screen(degradation: Degradation, locked: bool) -> Degradation {
+ if locked && degradation != Degradation::None {
+ Degradation::ScreenLocked
+ } else {
+ degradation
+ }
}
/// Decide the degradation rung. Pure: `walk_attempted` is false in the
@@ -1229,6 +1262,28 @@ mod window_scope_contract_tests {
);
}
+ /// Screen saver or display sleep: every AXWindows list is empty, so the
+ /// window reads as unresolved. The reason names the lock instead, and a
+ /// healthy tree is never relabelled.
+ #[test]
+ fn a_locked_screen_explains_an_empty_or_unresolved_tree() {
+ let unresolved = degradation_for(
+ true,
+ 0,
+ Some(&WindowScope::AxUnresolved { ax_window_count: 0 }),
+ false,
+ );
+ assert_eq!(
+ explain_locked_screen(unresolved, true),
+ Degradation::ScreenLocked
+ );
+ assert_eq!(explain_locked_screen(unresolved, false), unresolved);
+ assert_eq!(
+ explain_locked_screen(Degradation::None, true),
+ Degradation::None
+ );
+ }
+
#[test]
fn empty_tree_still_degrades_as_ax_tree_empty() {
// Back-compat with the pre-existing rung.
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 5d071c22ba..23cadfc474 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -47,7 +47,7 @@
"read_only": true
},
{
- "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",
"destructive": false,
"idempotent": false,
"input_schema": {