Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): report a locked screen and keep the display awake while an agent works (#5483)
Francesco Bonaccib1633ef91cb0
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdxindex 2a20758e05..e1f6251d75 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/window-state.mdx@@ -36,7 +36,7 @@ DIFF READS (use them for every re-read after acting): `since:"latest"`, or `sincA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.-Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.+Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:"foreground"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.**macOS parameters**diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rsindex 0270e9dffe..eae7b3b7ca 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs@@ -74,6 +74,12 @@ pub struct BackgroundTargetFacts {pub competing_keyboard_destinations: usize,/// Ancestry proof for an explicitly addressed element, when one exists.pub element: ElementAncestry,+ /// The login session reports its screen as locked (macOS: the screen+ /// saver, display sleep or a lock put the session behind the lock+ /// screen). The lock screen owns input while this holds and every app's+ /// accessibility windows read as empty, so no route can reach the target.+ /// `false` when the platform cannot tell; the other facts then decide.+ pub screen_locked: bool,}/// The background action classes v1 distinguishes. Route order and trust are@@ -120,8 +126,21 @@ pub mod refusal_codes {pub const MINIMIZED_OR_HIDDEN: &str = "minimized_or_hidden_window";pub const SAME_PID_KEYBOARD_AMBIGUITY: &str = "same_pid_keyboard_ambiguity";pub const ELEMENT_OUTSIDE_TARGET_WINDOW: &str = "element_outside_target_window";+ pub const SCREEN_LOCKED: &str = "screen_locked";}+/// Shown with every `screen_locked` refusal: what happened and how to keep it+/// from happening while an agent works. Never advises foreground input, which+/// would land on the lock screen.+pub const SCREEN_LOCKED_HINT: &str = "the screen is locked or showing the screen saver, \+ which also happens when the screen saver starts or the display sleeps with no password \+ required. While it lasts, macOS hides every app's windows from accessibility and sends \+ input to the lock screen, so Cua Driver refuses instead of acting; screenshots still \+ work. Wake and unlock the Mac, then retry. Cua Driver keeps the display awake for 10 \+ minutes after each tool call (CUA_DRIVER_KEEP_DISPLAY_AWAKE sets the seconds, 0 turns \+ it off), so this usually means the Mac sat idle between agent turns; running \+ `caffeinate -d` keeps it awake for longer";+/// A typed refusal: no actuator ran and none may run for this decision.#[derive(Clone, Debug, PartialEq, Eq)]pub struct BackgroundRefusal {@@ -221,6 +240,20 @@ pub fn decide_background_input(WindowServerOwnership::SamePid => {}}+ // A locked session hides every app's AX windows, so the AX checks below+ // would misreport it as an off-Space window and advise foreground input,+ // which would land on the lock screen. Report the real cause first.+ if facts.screen_locked {+ return refuse(+ refusal_codes::SCREEN_LOCKED,+ format!(+ "window {} cannot receive input: {SCREEN_LOCKED_HINT}",+ target.window_id+ ),+ None,+ );+ }+match facts.element {ElementAncestry::NotAddressed | ElementAncestry::ProvenDescendant => {}ElementAncestry::OutsideTargetWindow | ElementAncestry::Unproven => {@@ -328,6 +361,7 @@ fn exact_window_status(facts: &BackgroundTargetFacts) -> &'static str {match &facts.window_server {WindowServerOwnership::NotFound => "not_found",WindowServerOwnership::ForeignPid { .. } => "owner_mismatch",+ WindowServerOwnership::SamePid if facts.screen_locked => "screen_locked",WindowServerOwnership::SamePid if facts.ax_window_present => "matched",WindowServerOwnership::SamePid => "ax_unresolved",}@@ -394,6 +428,7 @@ mod tests {app_hidden: Some(false),competing_keyboard_destinations: 0,element: ElementAncestry::NotAddressed,+ screen_locked: false,}}@@ -658,6 +693,7 @@ mod tests {app_hidden: Some(true),competing_keyboard_destinations: 3,element: ElementAncestry::OutsideTargetWindow,+ screen_locked: false,};assert_eq!(code_of(decide_background_input(@@ -746,4 +782,52 @@ mod tests {);assert_eq!(report["observation"]["one_shot_capture"], "unknown");}++ /// A locked session (screen saver or display sleep) empties every app's+ /// AXWindows. Every route refuses with `screen_locked`, not the misleading+ /// off-Space code, and no refusal advises foreground input, which would+ /// reach the lock screen.+ #[test]+ fn locked_screen_refuses_every_route_without_foreground_advice() {+ let facts = BackgroundTargetFacts {+ screen_locked: true,+ ax_window_present: false,+ element: ElementAncestry::Unproven,+ ..matched_facts()+ };+ for action in ALL_ACTIONS {+ match decide_background_input(TARGET, &facts, action) {+ BackgroundInputDecision::Refuse(refusal) => {+ assert_eq!(refusal.code, refusal_codes::SCREEN_LOCKED, "{action:?}");+ assert_eq!(refusal.advice, None, "{action:?}");+ assert!(refusal.reason.contains("display sleeps"));+ }+ BackgroundInputDecision::Execute { .. } => panic!("{action:?} must refuse"),+ }+ }+ let report = background_input_capability_report(TARGET, &facts, Some(true));+ assert_eq!(report["exact_window"]["status"], "screen_locked");+ for route in report["routes"].as_array().unwrap() {+ assert_eq!(route["reason"], refusal_codes::SCREEN_LOCKED);+ }+ }++ /// A stale or foreign window id is still reported as such while locked:+ /// re-targeting is needed either way.+ #[test]+ fn ownership_failures_precede_the_lock() {+ let facts = BackgroundTargetFacts {+ window_server: WindowServerOwnership::NotFound,+ screen_locked: true,+ ..matched_facts()+ };+ assert_eq!(+ code_of(decide_background_input(+ TARGET,+ &facts,+ BackgroundAction::AxSemantic+ )),+ refusal_codes::WINDOW_NOT_FOUND+ );+ }}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex 611b058030..5b739537ff 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -111,6 +111,7 @@ pub mod socket_io;pub mod text_sanitize;pub mod timestamp;pub mod tool;+pub mod tool_activity;pub mod tool_args;pub mod tool_schema;pub mod video;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rsindex 76dcbf27a1..103ebcc5d7 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs@@ -2302,6 +2302,7 @@ impl ToolRegistry {// Desktop pixels read off a capped get_desktop_state image are mapped// back to the uncapped capture before any platform interprets them.crate::desktop_capture_scale::map_desktop_args(&mut args);+ crate::tool_activity::note_tool_activity();let mut result = crate::recording::scope_dispatch_click_capture(pending_turn.as_ref(),tool.invoke(args.clone()),diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rsnew file mode 100644index 0000000000..303820ca90--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_activity.rs@@ -0,0 +1,25 @@+//! Per-call activity hook: the platform shell learns that an agent is+//! working each time the registry dispatches a tool.+//!+//! macOS uses it to keep the display awake while an agent works, because the+//! screen saver and display sleep put the session behind the lock screen,+//! where no app's windows are reachable. Registered once by the platform+//! crate; a no-op when nothing is registered.++use std::sync::OnceLock;++type ToolActivityFn = Box<dyn Fn() + Send + Sync>;+static TOOL_ACTIVITY_FN: OnceLock<ToolActivityFn> = OnceLock::new();++/// Register the activity callback. Returns false when one is already set;+/// the first registration wins.+pub fn set_tool_activity_fn(f: impl Fn() + Send + Sync + 'static) -> bool {+ TOOL_ACTIVITY_FN.set(Box::new(f)).is_ok()+}++/// Report one tool dispatch. Must stay cheap: it runs on every call.+pub fn note_tool_activity() {+ if let Some(f) = TOOL_ACTIVITY_FN.get() {+ f();+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rsindex 74485384b3..7ccdae7944 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs@@ -355,6 +355,7 @@ pub fn gather_background_facts(app_hidden,competing_keyboard_destinations,element: element.unwrap_or(ElementAncestry::NotAddressed),+ screen_locked: crate::session::screen_locked(),}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rs b/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rsnew file mode 100644index 0000000000..9268bd094a--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/display_awake.rs@@ -0,0 +1,221 @@+//! Keep the display awake while an agent is working.+//!+//! On macOS 14 and later the screen saver and display sleep put the login+//! session behind the lock screen, even when no password is required. While+//! that holds, every app's accessibility windows read as empty and input goes+//! to the lock screen, so background actions cannot run. An agent working on+//! an idle Mac (no one touching the keyboard or mouse) hits this as soon as+//! the idle timer fires.+//!+//! Each tool call holds an IOKit `PreventUserIdleDisplaySleep` assertion (the+//! one `caffeinate -d` and video players take) until no tool has run for the+//! idle window. It only stops the *idle* timer: it never wakes a sleeping+//! display, dismisses the screen saver, or gets past a lock or password, and+//! the user can still sleep or lock the Mac by hand.+//!+//! The idle display-sleep path dims the display first, and loginwindow can+//! lock on that dim (`kLWLockFromDisplayDim`) without the screen saver ever+//! running. A `PreventUserIdleDisplaySleep` assertion holds off the idle dim as+//! well as the sleep that follows (the display stays at full brightness during+//! video playback, which takes the same assertion), so that lock does not fire+//! while the guard is held. It is not a system-sleep assertion: keeping the+//! machine awake is not enough, because the lock comes from the display path.+//!+//! `CUA_DRIVER_KEEP_DISPLAY_AWAKE` sets the idle window in seconds (default+//! 600); `0`, `off` or `false` turns the guard off.++use std::sync::Mutex;+use std::time::{Duration, Instant};++use core_foundation::base::TCFType;+use core_foundation::string::{CFString, CFStringRef};++/// Environment variable that tunes or disables the guard.+pub const KEEP_DISPLAY_AWAKE_ENV: &str = "CUA_DRIVER_KEEP_DISPLAY_AWAKE";++/// How long the display stays awake after the last tool call by default.+pub const DEFAULT_IDLE_WINDOW: Duration = Duration::from_secs(600);++type IOPMAssertionID = u32;+type IOReturn = i32;+const K_IOPM_ASSERTION_LEVEL_ON: u32 = 255;+const K_IO_RETURN_SUCCESS: IOReturn = 0;++#[link(name = "IOKit", kind = "framework")]+extern "C" {+ fn IOPMAssertionCreateWithName(+ assertion_type: CFStringRef,+ assertion_level: u32,+ assertion_name: CFStringRef,+ assertion_id: *mut IOPMAssertionID,+ ) -> IOReturn;+ fn IOPMAssertionRelease(assertion_id: IOPMAssertionID) -> IOReturn;+}++/// Parse the env value into the idle window, or `None` when disabled.+pub fn idle_window_from_env(value: Option<&str>) -> Option<Duration> {+ let Some(raw) = value.map(str::trim).filter(|raw| !raw.is_empty()) else {+ return Some(DEFAULT_IDLE_WINDOW);+ };+ match raw.to_ascii_lowercase().as_str() {+ "0" | "off" | "false" | "no" => None,+ "on" | "true" | "yes" => Some(DEFAULT_IDLE_WINDOW),+ other => match other.parse::<u64>() {+ Ok(secs) => Some(Duration::from_secs(secs.min(86_400))),+ Err(_) => {+ tracing::warn!(+ "{KEEP_DISPLAY_AWAKE_ENV}={raw:?} is not a number of seconds or off; \+ using the default"+ );+ Some(DEFAULT_IDLE_WINDOW)+ }+ },+ }+}++struct Guard {+ assertion: IOPMAssertionID,+ last_activity: Instant,+}++static GUARD: Mutex<Option<Guard>> = Mutex::new(None);++/// Register the guard as the registry's tool-activity hook, unless the env+/// turns it off. Safe to call more than once.+pub fn install() {+ let Some(idle) = idle_window_from_env(std::env::var(KEEP_DISPLAY_AWAKE_ENV).ok().as_deref())+ else {+ tracing::info!("display-awake guard off ({KEEP_DISPLAY_AWAKE_ENV})");+ return;+ };+ cua_driver_core::tool_activity::set_tool_activity_fn(move || note_activity(idle));+}++/// Hold the assertion now and keep it until `idle` passes with no activity.+pub fn note_activity(idle: Duration) {+ let mut guard = GUARD+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if let Some(held) = guard.as_mut() {+ held.last_activity = Instant::now();+ return;+ }+ let Some(assertion) = create_assertion() else {+ return;+ };+ *guard = Some(Guard {+ assertion,+ last_activity: Instant::now(),+ });+ drop(guard);+ let spawned = std::thread::Builder::new()+ .name("cua-display-awake".into())+ .spawn(move || release_when_idle(idle));+ if spawned.is_err() {+ release_now();+ }+}++fn release_when_idle(idle: Duration) {+ loop {+ let wait = {+ let guard = GUARD+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ let Some(held) = guard.as_ref() else {+ return;+ };+ remaining(held.last_activity.elapsed(), idle)+ };+ match wait {+ Some(wait) => std::thread::sleep(wait),+ None => {+ release_now();+ return;+ }+ }+ }+}++/// Time left before release, or `None` once the idle window has passed.+fn remaining(elapsed: Duration, idle: Duration) -> Option<Duration> {+ idle.checked_sub(elapsed).filter(|left| !left.is_zero())+}++fn release_now() {+ let held = GUARD+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner())+ .take();+ if let Some(held) = held {+ unsafe { IOPMAssertionRelease(held.assertion) };+ tracing::debug!("display-awake guard released");+ }+}++fn create_assertion() -> Option<IOPMAssertionID> {+ let kind = CFString::from_static_string("PreventUserIdleDisplaySleep");+ let name = CFString::from_static_string("Cua Driver: an agent is working on this Mac");+ let mut id: IOPMAssertionID = 0;+ let status = unsafe {+ IOPMAssertionCreateWithName(+ kind.as_concrete_TypeRef(),+ K_IOPM_ASSERTION_LEVEL_ON,+ name.as_concrete_TypeRef(),+ &mut id,+ )+ };+ if status == K_IO_RETURN_SUCCESS {+ tracing::debug!("display-awake guard held");+ Some(id)+ } else {+ tracing::warn!("could not keep the display awake: IOPMAssertionCreateWithName {status}");+ None+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn env_defaults_to_ten_minutes_and_can_be_turned_off() {+ assert_eq!(idle_window_from_env(None), Some(DEFAULT_IDLE_WINDOW));+ assert_eq!(idle_window_from_env(Some("")), Some(DEFAULT_IDLE_WINDOW));+ assert_eq!(idle_window_from_env(Some("on")), Some(DEFAULT_IDLE_WINDOW));+ for off in ["0", "off", "FALSE", "no"] {+ assert_eq!(idle_window_from_env(Some(off)), None, "{off}");+ }+ assert_eq!(+ idle_window_from_env(Some("120")),+ Some(Duration::from_secs(120))+ );+ assert_eq!(+ idle_window_from_env(Some("999999")),+ Some(Duration::from_secs(86_400))+ );+ assert_eq!(+ idle_window_from_env(Some("soon")),+ Some(DEFAULT_IDLE_WINDOW)+ );+ }++ #[test]+ fn release_waits_for_the_full_idle_window() {+ let idle = Duration::from_secs(600);+ assert_eq!(+ remaining(Duration::from_secs(100), idle),+ Some(Duration::from_secs(500))+ );+ assert_eq!(remaining(idle, idle), None);+ assert_eq!(remaining(Duration::from_secs(700), idle), None);+ }++ /// Holding and releasing the real assertion works and leaves nothing+ /// held. The idle timer is not reset, so this does not wake the display.+ #[test]+ fn assertion_is_created_and_released() {+ let id = create_assertion().expect("IOPMAssertionCreateWithName");+ assert_eq!(unsafe { IOPMAssertionRelease(id) }, K_IO_RETURN_SUCCESS);+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rsindex 329c629853..399a636aca 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs@@ -28,6 +28,8 @@ pub mod capture_consent;#[cfg(target_os = "macos")]pub mod cursor;#[cfg(target_os = "macos")]+pub mod display_awake;+#[cfg(target_os = "macos")]pub mod document_edit;#[cfg(target_os = "macos")]pub mod focus_guard;@@ -161,6 +163,9 @@ pub fn register_tools_with_cursor_and_provider(// that will never arrive. Unconditional: emission depends on the// registry write path, not on the overlay or on graphic access.cua_driver_core::cursor_hook::declare_cursor_hook_emitter();+ // Keep the display awake while tools run, so the screen saver and+ // display sleep do not lock the session mid-task.+ display_awake::install();let mut r = ToolRegistry::new_with_protected_consent_provider(provider);tools::register_all(&mut r,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/session.rs b/libs/cua-driver/rust/crates/platform-macos/src/session.rsindex 80a444c913..b03e7138f5 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/session.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/session.rs@@ -43,10 +43,63 @@ pub fn has_graphic_access() -> bool {status == 0 && (attributes & SESSION_HAS_GRAPHIC_ACCESS) != 0}+#[link(name = "CoreGraphics", kind = "framework")]+extern "C" {+ fn CGSessionCopyCurrentDictionary() -> core_foundation::dictionary::CFDictionaryRef;+}++/// Whether the login session currently reports its screen as locked+/// (`CGSSessionScreenIsLocked` in `CGSessionCopyCurrentDictionary`).+///+/// Read fresh on every call; lock state changes at any time. On macOS 14 and+/// later the screen saver and display sleep put the session behind the lock+/// screen (`loginwindow` frontmost) even before a password is required, and+/// while that holds every app's `AXWindows` reads as empty. The key is the+/// session's lock flag, not a per-cause one: it covers the screen-saver lock,+/// a manual lock, and loginwindow's lock on display dim (`kLWLockFromDisplayDim`,+/// seen in a user's display-sleep logs with `screenSaverIsRunning = 0`). Only+/// the screen-saver path was reproduced here; the dim path is the same lock+/// as far as this key is concerned, so no separate check exists. `false` when the+/// session dictionary is unavailable (no window-server session) or the key is+/// absent, which is how an unlocked session reports itself.+pub fn screen_locked() -> bool {+ use core_foundation::base::TCFType;+ use core_foundation::boolean::CFBoolean;+ use core_foundation::dictionary::CFDictionary;+ use core_foundation::number::CFNumber;+ use core_foundation::string::CFString;++ let raw = unsafe { CGSessionCopyCurrentDictionary() };+ if raw.is_null() {+ return false;+ }+ // SAFETY: the "Copy" function returns a +1 reference we now own.+ let dict: CFDictionary = unsafe { CFDictionary::wrap_under_create_rule(raw) };+ let key = CFString::from_static_string("CGSSessionScreenIsLocked");+ let Some(value) = dict.find(key.as_CFTypeRef() as *const _) else {+ return false;+ };+ let value = unsafe { core_foundation::base::CFType::wrap_under_get_rule(*value as _) };+ if let Some(flag) = value.downcast::<CFBoolean>() {+ return flag.into();+ }+ value+ .downcast::<CFNumber>()+ .and_then(|number| number.to_i64())+ .is_some_and(|number| number != 0)+}+#[cfg(test)]mod tests {use super::*;+ /// Resolves the CoreGraphics symbol and reads the session dictionary+ /// without crashing, with or without a window-server session.+ #[test]+ fn screen_lock_probe_does_not_abort() {+ let _ = screen_locked();+ }+/// The probe must resolve the `Security` symbol and return a `bool`/// without aborting. We can't assert the value: it's `true` on a dev GUI/// session and `false` on a headless CI runner — which is exactly thediff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex 8c4d118990..a1a03ed93b 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -35,7 +35,7 @@ fn def() -> &'static ToolDef {\n\A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\\n\- Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),+ Refusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.".into(),input_schema: cua_driver_core::window_state_view::extend_input_schema(serde_json::json!({"type": "object","required": ["pid", "window_id"],@@ -664,13 +664,32 @@ impl Tool for GetWindowStateTool {let app_lookup_timed_out = tree_result.as_ref().is_some_and(|r| r.walk.reason() == Some("app_lookup_timeout"));- match degradation_for(+ let degradation = degradation_for(tree_result.is_some(),element_count,window_scope.as_ref(),app_lookup_timed_out,- ) {+ );+ let degradation = explain_locked_screen(+ degradation,+ !matches!(degradation, Degradation::None) && crate::session::screen_locked(),+ );+ match degradation {Degradation::None => {}+ Degradation::ScreenLocked => {+ structured["degraded"] = serde_json::json!(true);+ structured["degraded_reason"] = serde_json::json!(format!(+ "screen_locked: window_id {window_id} has no accessibility tree because {}.",+ cua_driver_core::background_input::SCREEN_LOCKED_HINT+ ));+ structured["escalation"] = serde_json::json!({+ "recommended": "wait_for_unlock",+ "reason": "observation-only while the screen is locked: no delivery \+ mode can reach the app, and foreground input would go to \+ the lock screen. Re-snapshot after the Mac is woken and \+ unlocked."+ });+ }Degradation::AxTreeEmpty => {structured["degraded"] = serde_json::json!(true);structured["degraded_reason"] = serde_json::json!(@@ -957,6 +976,20 @@ enum Degradation {/// The window scope is unresolved because the app is still launching and/// did not answer accessibility within the caller's budget.AxAppLaunching,+ /// The session's screen is locked (screen saver, display sleep or a+ /// lock). macOS then reports no AXWindows for any app, so an empty or+ /// unresolved tree has this cause rather than the app's own state.+ ScreenLocked,+}++/// Name the locked screen as the cause of an empty or unresolved tree.+/// `locked` is only probed when the walk already degraded.+fn explain_locked_screen(degradation: Degradation, locked: bool) -> Degradation {+ if locked && degradation != Degradation::None {+ Degradation::ScreenLocked+ } else {+ degradation+ }}/// Decide the degradation rung. Pure: `walk_attempted` is false in the@@ -1229,6 +1262,28 @@ mod window_scope_contract_tests {);}+ /// Screen saver or display sleep: every AXWindows list is empty, so the+ /// window reads as unresolved. The reason names the lock instead, and a+ /// healthy tree is never relabelled.+ #[test]+ fn a_locked_screen_explains_an_empty_or_unresolved_tree() {+ let unresolved = degradation_for(+ true,+ 0,+ Some(&WindowScope::AxUnresolved { ax_window_count: 0 }),+ false,+ );+ assert_eq!(+ explain_locked_screen(unresolved, true),+ Degradation::ScreenLocked+ );+ assert_eq!(explain_locked_screen(unresolved, false), unresolved);+ assert_eq!(+ explain_locked_screen(Degradation::None, true),+ Degradation::None+ );+ }+#[test]fn empty_tree_still_degrades_as_ax_tree_empty() {// Back-compat with the pre-existing rung.diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 5d071c22ba..23cadfc474 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -47,7 +47,7 @@"read_only": true},{- "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.",+ "description": "Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (`tree_format:\"markdown\"`): a row `[N]` is addressed with `element_token` `<snapshot_id>:N`, where `snapshot_id` is in the response header and in structuredContent. `tree_format:\"elements\"` returns the structured `elements` array instead (`element_index`, `element_token`, `role`, `label`, `value`, `actions`, `frame`, `parent_index`, `depth`); `\"both\"` returns both (about twice the size). Pass the token to click, type_text, press_key, etc.\n\nSTART NARROW: a read walks at most 250 nodes by default and says `Tree truncated at max_elements=…` (`truncated:true`) when it stops; raise `max_elements`, or use `query` (case-insensitive substring; matching rows plus ancestors) and `max_depth`. Indices and tokens stay valid for the whole snapshot. `verbose:true` adds `_note` and the full `background_input` report; `full_output:true` restores the previous full response (both representations, all metadata, ≤2 000 nodes).\n\nDIFF READS (use them for every re-read after acting): `since:\"latest\"`, or `since:<snapshot_id>` from an earlier read of the same window, returns only what changed: `+` added, `~` changed, `-` removed rows (removed ids are the old snapshot's), a `reindexed:` line if indices shifted, or `no change since …; focused element is …`. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read; `since_status` says why.\n\nA new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (`invalidated_snapshot_ids`). `include_screenshot:false` returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; `include_accessibility_tree:false` returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not `max_elements:1`, when you just need to look); adding `display_only:true` (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with `h:1` frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.\n\nRefusals: `window_id_not_found`; `window_owner_pid_mismatch` names the real `owner_pid` (sandboxed Open/Save panels belong to a panel service). `degraded_reason` `ax_window_unresolved` or `ax_app_launching` (the latter with `truncation_reason: app_lookup_timeout`) means an empty tree: re-snapshot, or act with `delivery_mode:\"foreground\"`. `degraded_reason` `screen_locked` means the screen saver, display sleep or a lock hid every window from accessibility: wake and unlock the Mac, then re-snapshot; foreground input would reach the lock screen. `px_frame_mismatch` / `px_capture_unavailable` omit the unprovable screenshot; the AX payload stays valid.","destructive": false,"idempotent": false,"input_schema": {