Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): honest background refusals on macOS: hover, LibreOffice value writes, drag (#4977)
Francesco Bonacci29823dd6b556
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdxindex 193e8b2904..d56cb0f0fa 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx@@ -229,14 +229,14 @@ Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist><Tab value="macOS">-Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.+Move a cursor to (x, y). With target {kind: "window", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.**macOS parameters**| Parameter | Type | Default | Description || --- | --- | --- | --- || <span id="move_cursor--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "window" (default) moves only the agent cursor overlay; "desktop" moves the real OS pointer. |-| <span id="move_cursor--macos--x"></span>`x` | `number` | required | Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels. |+| <span id="move_cursor--macos--x"></span>`x` | `number` | required | Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels. |</Tab><Tab value="Linux">diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex 21427423ab..9310ebb121 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -114,6 +114,7 @@ is safe even for apps that normally foreground on media-load| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` || Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls || Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |+| Hover (reveal a hover-only control) | Look for the same command elsewhere first (menu item, context menu, shortcut, help text in `get_window_state`). A background app sees no hover; only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovers: it moves only the agent cursor || Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` || Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` || Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |@@ -321,6 +322,17 @@ key focus back within a few tens of milliseconds. Prefer routes that donot ask the app to come forward: `set_value` on the address field or`browser_navigate` instead of `cmd+L`, and `AXOpen` through`double_click` on an element instead of a pixel double-click.+### Hover in the background++macOS apps track hover and show tooltips (help tags) from the real+pointer. Pointer-move events posted to a background app do not reach its+tracking areas, so a control that only appears on hover, or a tooltip,+cannot be revealed in the background. A window-targeted `move_cursor`+moves only the agent cursor and says so. Look for the same command+elsewhere first: a menu item, a context menu, a keyboard shortcut, or the+control's help text in `get_window_state`. Otherwise the step needs the+user's pointer (`move_cursor` with `scope:"desktop"`) or the app in front+(`bring_to_front`); use either only if the user allows it, and say which.### `delivery_mode` on the pointer family (macOS)@@ -424,6 +436,18 @@ There is no backgrounded path that reaches these apps today.AX-addressable elements (links, buttons, toolbar items), use`right_click({pid, element_token})` instead.+### A value write the app does not apply (LibreOffice)++Some apps accept an accessibility value write and ignore it. LibreOffice+Calc cells do: `set_value` returns success at the AX level and the cell+keeps its old content. `set_value` re-reads the value, and when it has+not moved it fails with `code:"write_not_applied"` and+`effect:"not_applied"`, so a `run_script` stops there. LibreOffice also+drops background pointer input, so its cells cannot be edited in the+background at all: the edit needs the window in front (`bring_to_front`,+then `type_text`), only if the user allows it. Otherwise stop and report+the step as blocked.+### Known text-input limits (Catalyst + Electron)On **Catalyst** apps (WhatsApp, Reminders, Notes-via-Catalyst,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex e1e37b0f22..85e3579685 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -40,11 +40,15 @@ const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS supports ongesture, then restore the prior frontmost app. The default \"background\" is refused with \background_unavailable because macOS has no background drag route. Desktop scope ignores this field.";-const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: a drag is \- delivered as real pointer events to the frontmost window, so it cannot run without fronting \- the target. Nothing was sent. Retry with delivery_mode:\"foreground\" and window_id; the \- driver briefly fronts that window, moves the pointer along the path, and restores the prior \- frontmost app.";+const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: \+ WindowServer tracks a drag (including drag-and-drop) with the real pointer over the \+ frontmost window, so it cannot run without fronting the target. Nothing was sent. First \+ look for a route that is not a drag: a Move Up/Down button or menu item, a context-menu \+ command, a keyboard shortcut on the selected item, or set_value on a slider. If the drag is \+ the only route and the user allows the window to come forward, retry with \+ delivery_mode:\"foreground\" and window_id: the driver briefly fronts that window, moves \+ the pointer along the path and restores the prior frontmost app. Otherwise stop and report \+ that the step needs the window in front.";fn background_drag_refusal() -> ToolResult {cua_driver_core::delivery::background_unavailable_result(diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rsindex e52ddbccdc..f03c5c0dde 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs@@ -25,15 +25,18 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "move_cursor".into(),- description: "Move a cursor to (x, y). In window scope (default), moves only the \- agent cursor overlay. With scope=desktop, moves the real OS pointer in native \- get_desktop_state screenshot coordinates.".into(),+ description: "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, \+ x/y are window screenshot pixels (as for click) and only the agent cursor moves \+ there. Without a target, moves only the agent cursor overlay, in screen points. \+ Neither sends input: macOS apps track hover and show tooltips from the real pointer, \+ so a background app sees no hover. With scope=desktop, moves the real OS pointer in \+ native get_desktop_state screenshot coordinates.".into(),input_schema: serde_json::json!({"type": "object","required": ["x", "y"],"properties": {"session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },- "x": { "type": "number", "description": "Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels." },+ "x": { "type": "number", "description": "Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels." },"y": { "type": "number", "description": "Destination Y, in the same space as x." },"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "\"window\" (default) moves only the agent cursor overlay; \"desktop\" moves the real OS pointer." },"cursor_id": { "type": "string", "description": "Cursor instance to move. Default: 'default'." }@@ -87,6 +90,9 @@ impl Tool for MoveCursorTool {Err(error) => ToolResult::error(format!("desktop pointer task failed: {error}")),};}+ if let (Some(pid), Some(wid)) = (args.opt_i64("pid"), args.opt_u64("window_id")) {+ return self.move_over_window(&args, pid as i32, wid as u32).await;+ }if !self.state.cursor_overlay_available {return super::cursor_overlay_unavailable();}@@ -115,6 +121,79 @@ impl Tool for MoveCursorTool {}}+impl MoveCursorTool {+ /// `target: {kind: "window", pid, window_id}`: x/y are window-local+ /// screenshot pixels, as for `click`. Moves the agent cursor there. It+ /// sends no input: AppKit derives hover (tracking areas, tooltips) from+ /// the real pointer, and pointer-move events posted to a background app+ /// do not reach its tracking areas (VM check, 11 Oct 2026: BenchLab saw no+ /// hover through the SkyLight, public or both routes).+ async fn move_over_window(&self, args: &Value, pid: i32, wid: u32) -> ToolResult {+ use cua_driver_core::tool_args::ArgsExt;+ if !self.state.cursor_overlay_available {+ return super::cursor_overlay_unavailable();+ }+ let (x, y) = match (args.require_f64("x"), args.require_f64("y")) {+ (Ok(x), Ok(y)) => (x, y),+ (Err(e), _) | (_, Err(e)) => return e,+ };+ let ratio = match super::screenshot_scale(&self.state, args, pid, Some(wid)) {+ Ok(ratio) => ratio,+ Err(refusal) => return refusal,+ };+ let frame = match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => frame,+ Err(refusal) => return refusal,+ };+ let (screen_x, screen_y, local_x, local_y) = frame.to_screen(x * ratio, y * ratio);+ if local_x < 0.0+ || local_y < 0.0+ || local_x > frame.bounds.width+ || local_y > frame.bounds.height+ {+ return ToolResult::error(format!(+ "move_cursor: window-local point ({local_x:.1}, {local_y:.1}) pt lies outside \+ window {wid}'s {:.0}×{:.0} pt frame. Re-read coordinates from a fresh \+ get_window_state screenshot.",+ frame.bounds.width, frame.bounds.height+ ));+ }+ let cursor_id = super::cursor_tools::resolve_cursor_key(args);+ crate::cursor::overlay::send_command(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),+ );+ crate::cursor::overlay::animate_cursor_to(cursor_id.clone(), screen_x, screen_y).await;+ self.state+ .cursor_registry+ .update_position(&cursor_id, screen_x, screen_y);+ let note = if crate::apps::frontmost_pid() == Some(pid) {+ ""+ } else {+ BACKGROUND_HOVER_NOTE+ };+ ToolResult::text(format!(+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \+ pointer did not move.{note}"+ ))+ .with_structured(serde_json::json!({+ "scope": "window",+ "window_id": wid,+ "screen_point": [screen_x, screen_y],+ "hover_delivered": false,+ "effect": "unverifiable"+ }))+ }+}++/// What a window-target move does not do for a background app.+const BACKGROUND_HOVER_NOTE: &str = " The app is in the background, so it sees no hover: \+ macOS apps track hover and show tooltips from the real pointer, and no background route \+ reaches them. A control that only appears on hover, or a tooltip, cannot be revealed \+ without the user's pointer (scope:\"desktop\") or the app in front (bring_to_front); use \+ either only if the user allows it. Look first for the same command elsewhere: a menu \+ item, a context menu, a keyboard shortcut, or help text in get_window_state.";+/// The app that owns the frontmost ordinary window under a desktop point,/// when that app is not the active one. The driver's own windows (cursor/// overlay) are skipped.@@ -143,6 +222,14 @@ fn inactive_app_note(app: &str) -> String {#[cfg(test)]mod tests {+ #[test]+ fn the_background_hover_note_names_both_routes_and_the_alternatives() {+ let note = super::BACKGROUND_HOVER_NOTE;+ assert!(note.contains("scope:\"desktop\""));+ assert!(note.contains("bring_to_front"));+ assert!(note.contains("only if the user allows it"));+ }+#[test]fn the_inactive_app_note_names_the_route_to_a_tooltip() {let note = super::inactive_app_note("BenchLab");diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rsindex 2d9278cac2..e3a9e002c6 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs@@ -207,6 +207,11 @@ impl Tool for SetValueTool {let changes = snapshot.detect_async().await;match result {+ Ok(Ok(outcome)) if outcome.not_applied && !ax_echo_surface => {+ let vcl = crate::input::pointer_toolkit::detect(pid)+ == Some(crate::input::pointer_toolkit::PointerReadingToolkit::Vcl);+ not_applied_result(element_index, vcl)+ }Ok(Ok(mut outcome)) => {apply_surface_trust(&mut outcome, ax_echo_surface);apply_verification_label(&mut outcome);@@ -327,6 +332,9 @@ struct SetValueOutcome {/// `Some(false)` when the element already held the requested value, so the/// write was a no-op. Lets callers distinguish "idempotent" from "applied".changed: Option<bool>,+ /// The read-back still showed the old value after a short settle: the app+ /// took the accessibility write but did not apply it.+ not_applied: bool,}fn apply_surface_trust(outcome: &mut SetValueOutcome, ax_echo_surface: bool) {@@ -366,6 +374,7 @@ fn set_value_blocking(detail,verified: None,changed: None,+ not_applied: false,}})} else {@@ -393,13 +402,30 @@ fn set_value_blocking(None => unsafe { set_string_attr(element, "AXValue", value) },};if err == kAXErrorSuccess {- let after = unsafe { copy_string_attr(element, "AXValue") };- let (verified, changed) = classify_write(+ let mut after = unsafe { copy_string_attr(element, "AXValue") };+ let (mut verified, mut changed) = classify_write(before.as_deref(),after.as_deref(),value,numeric_target.is_some(),);+ // Some apps apply the write a moment later; give it a short+ // settle before calling it not applied.+ let secure = unsafe { copy_string_attr(element, "AXSubrole") }.as_deref()+ == Some("AXSecureTextField");+ for _ in 0..NOT_APPLIED_REREADS {+ if secure || !write_not_applied(verified, changed) {+ break;+ }+ std::thread::sleep(std::time::Duration::from_millis(100));+ after = unsafe { copy_string_attr(element, "AXValue") };+ (verified, changed) = classify_write(+ before.as_deref(),+ after.as_deref(),+ value,+ numeric_target.is_some(),+ );+ }let suffix = match (verified, changed) {(Some(true), Some(false)) => " Value already matched; write was idempotent.",(Some(true), _) => "",@@ -410,6 +436,9 @@ fn set_value_blocking(detail: format!("✅ Set AXValue on [{element_index}] {role}.{suffix}"),verified,changed,+ // A secure text field reads back empty or masked, so its+ // read-back proves nothing either way.+ not_applied: !secure && write_not_applied(verified, changed),})} else if let Some(target) = numeric_target {// Both direct writes failed for a numeric target — fall back to@@ -424,6 +453,7 @@ fn set_value_blocking(),verified,changed,+ not_applied: false,})} else {anyhow::bail!("AXUIElementSetAttributeValue(AXValue) failed with error {err}")@@ -434,6 +464,41 @@ fn set_value_blocking(}}+/// Re-reads (100 ms apart) before an unchanged value counts as not applied.+const NOT_APPLIED_REREADS: usize = 3;++/// The read-back did not match and the value did not move: the app accepted+/// the accessibility write (LibreOffice Calc cells do) but did not apply it.+fn write_not_applied(verified: Option<bool>, changed: Option<bool>) -> bool {+ verified == Some(false) && changed == Some(false)+}++/// The refusal for a write the app did not apply. Nothing changed, so a+/// script must stop here instead of carrying on as if it had.+fn not_applied_result(element_index: usize, vcl: bool) -> ToolResult {+ let route = if vcl {+ "LibreOffice ignores accessibility value writes and drops background pointer \+ input, so its cells cannot be edited in the background. Editing them needs the \+ window in front: bring_to_front if the user allows it, then type_text into the \+ cell. Otherwise stop and report that the edit is blocked."+ } else {+ "Try the control's own input route instead: click it, then type_text, or use a \+ keyboard shortcut. If that needs the window in front, bring_to_front only if the \+ user allows it."+ };+ ToolResult::error(format!(+ "set_value did not take: element [{element_index}] still shows its old value after \+ the write. The app accepted the accessibility write but did not apply it, so nothing \+ changed. {route}"+ ))+ .with_structured(serde_json::json!({+ "code": "write_not_applied",+ "path": "ax",+ "verified": false,+ "effect": "not_applied",+ }))+}+/// Decide what a post-write AXValue read proves.////// Returns `(verified, changed)`:@@ -954,8 +1019,8 @@ fn hex_digit(n: u8) -> char {mod tests {use super::{apply_surface_trust, apply_verification_label, classify_write, file_name_needs_rename,- is_file_name_cell, is_get_info_name_field, SetValueOutcome, GET_INFO_RENAME_ROUTE,- LIST_RENAME_ROUTE,+ is_file_name_cell, is_get_info_name_field, not_applied_result, write_not_applied,+ SetValueOutcome, GET_INFO_RENAME_ROUTE, LIST_RENAME_ROUTE,};#[test]@@ -1044,6 +1109,29 @@ mod tests {}}+ #[test]+ fn an_unchanged_read_back_is_not_applied_but_an_idempotent_one_is_fine() {+ // Read-back still shows the old value: not applied.+ let (verified, changed) =+ classify_write(Some("pending"), Some("pending"), "matched", false);+ assert!(write_not_applied(verified, changed));+ // Already held the requested value: idempotent, applied.+ let (verified, changed) =+ classify_write(Some("matched"), Some("matched"), "matched", false);+ assert!(!write_not_applied(verified, changed));+ // The app reformatted the value: it moved, so it is not "not applied".+ let (verified, changed) = classify_write(Some(""), Some("1,248.60"), "1248.6", false);+ assert!(!write_not_applied(verified, changed));+ // Unreadable before or after: cannot tell.+ let (verified, changed) = classify_write(None, Some("x"), "y", false);+ assert!(!write_not_applied(verified, changed));+ let refusal = not_applied_result(4, true);+ assert_eq!(refusal.is_error, Some(true));+ let data = refusal.structured_content.unwrap();+ assert_eq!(data["code"], "write_not_applied");+ assert_eq!(data["effect"], "not_applied");+ }+#[test]fn file_name_refusal_names_the_rename_route() {let result = file_name_needs_rename(7, 42, LIST_RENAME_ROUTE);@@ -1136,6 +1224,7 @@ mod tests {detail: "Set value.".to_owned(),verified: Some(true),changed: Some(true),+ not_applied: false,};apply_surface_trust(&mut outcome, true);assert_eq!(outcome.verified, Some(false));@@ -1149,6 +1238,7 @@ mod tests {detail: "Set value.".to_owned(),verified: Some(true),changed: Some(true),+ not_applied: false,};apply_surface_trust(&mut outcome, false);assert_eq!(outcome.verified, Some(true));@@ -1162,6 +1252,7 @@ mod tests {detail: "✅ Set AXValue on [4] AXTextField.".to_owned(),verified: Some(false),changed: Some(false),+ not_applied: false,};apply_verification_label(&mut outcome);assert_eq!(diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex da991e478f..120a69dd0f 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -1406,7 +1406,7 @@"read_only": true},{- "description": "Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",+ "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.","destructive": false,"idempotent": true,"input_schema": {@@ -1462,7 +1462,7 @@"type": "object"},"x": {- "description": "Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels.",+ "description": "Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels.","type": "number"},"y": {