Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): honest background refusals on macOS: hover, LibreOffice value writes, drag (#4977)

Francesco Bonacci29823dd6b556
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index 193e8b2904..d56cb0f0fa 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -229,14 +229,14 @@ Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.
+Move a cursor to (x, y). With target &#123;kind: "window", pid, window_id&#125;, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="move_cursor--macos--scope"></span>`scope` | `"window" \| "desktop"` | `"window"` | "window" (default) moves only the agent cursor overlay; "desktop" moves the real OS pointer. |
-| <span id="move_cursor--macos--x"></span>`x` | `number` | required | Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels. |
+| <span id="move_cursor--macos--x"></span>`x` | `number` | required | Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels. |
</Tab>
<Tab value="Linux">
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 21427423ab..9310ebb121 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -114,6 +114,7 @@ is safe even for apps that normally foreground on media-load
| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |
| Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls |
| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |
+| Hover (reveal a hover-only control) | Look for the same command elsewhere first (menu item, context menu, shortcut, help text in `get_window_state`). A background app sees no hover; only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovers: it moves only the agent cursor |
| Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` |
| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` |
| Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |
@@ -321,6 +322,17 @@ key focus back within a few tens of milliseconds. Prefer routes that do
not ask the app to come forward: `set_value` on the address field or
`browser_navigate` instead of `cmd+L`, and `AXOpen` through
`double_click` on an element instead of a pixel double-click.
+### Hover in the background
+
+macOS apps track hover and show tooltips (help tags) from the real
+pointer. Pointer-move events posted to a background app do not reach its
+tracking areas, so a control that only appears on hover, or a tooltip,
+cannot be revealed in the background. A window-targeted `move_cursor`
+moves only the agent cursor and says so. Look for the same command
+elsewhere first: a menu item, a context menu, a keyboard shortcut, or the
+control's help text in `get_window_state`. Otherwise the step needs the
+user's pointer (`move_cursor` with `scope:"desktop"`) or the app in front
+(`bring_to_front`); use either only if the user allows it, and say which.
### `delivery_mode` on the pointer family (macOS)
@@ -424,6 +436,18 @@ There is no backgrounded path that reaches these apps today.
AX-addressable elements (links, buttons, toolbar items), use
`right_click({pid, element_token})` instead.
+### A value write the app does not apply (LibreOffice)
+
+Some apps accept an accessibility value write and ignore it. LibreOffice
+Calc cells do: `set_value` returns success at the AX level and the cell
+keeps its old content. `set_value` re-reads the value, and when it has
+not moved it fails with `code:"write_not_applied"` and
+`effect:"not_applied"`, so a `run_script` stops there. LibreOffice also
+drops background pointer input, so its cells cannot be edited in the
+background at all: the edit needs the window in front (`bring_to_front`,
+then `type_text`), only if the user allows it. Otherwise stop and report
+the step as blocked.
+
### Known text-input limits (Catalyst + Electron)
On **Catalyst** apps (WhatsApp, Reminders, Notes-via-Catalyst,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index e1e37b0f22..85e3579685 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -40,11 +40,15 @@ const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS supports on
gesture, then restore the prior frontmost app. The default \"background\" is refused with \
background_unavailable because macOS has no background drag route. Desktop scope ignores this field.";
-const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: a drag is \
- delivered as real pointer events to the frontmost window, so it cannot run without fronting \
- the target. Nothing was sent. Retry with delivery_mode:\"foreground\" and window_id; the \
- driver briefly fronts that window, moves the pointer along the path, and restores the prior \
- frontmost app.";
+const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: \
+ WindowServer tracks a drag (including drag-and-drop) with the real pointer over the \
+ frontmost window, so it cannot run without fronting the target. Nothing was sent. First \
+ look for a route that is not a drag: a Move Up/Down button or menu item, a context-menu \
+ command, a keyboard shortcut on the selected item, or set_value on a slider. If the drag is \
+ the only route and the user allows the window to come forward, retry with \
+ delivery_mode:\"foreground\" and window_id: the driver briefly fronts that window, moves \
+ the pointer along the path and restores the prior frontmost app. Otherwise stop and report \
+ that the step needs the window in front.";
fn background_drag_refusal() -> ToolResult {
cua_driver_core::delivery::background_unavailable_result(
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
index e52ddbccdc..f03c5c0dde 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
@@ -25,15 +25,18 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "move_cursor".into(),
- description: "Move a cursor to (x, y). In window scope (default), moves only the \
- agent cursor overlay. With scope=desktop, moves the real OS pointer in native \
- get_desktop_state screenshot coordinates.".into(),
+ description: "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, \
+ x/y are window screenshot pixels (as for click) and only the agent cursor moves \
+ there. Without a target, moves only the agent cursor overlay, in screen points. \
+ Neither sends input: macOS apps track hover and show tooltips from the real pointer, \
+ so a background app sees no hover. With scope=desktop, moves the real OS pointer in \
+ native get_desktop_state screenshot coordinates.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["x", "y"],
"properties": {
"session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
- "x": { "type": "number", "description": "Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels." },
+ "x": { "type": "number", "description": "Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels." },
"y": { "type": "number", "description": "Destination Y, in the same space as x." },
"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "\"window\" (default) moves only the agent cursor overlay; \"desktop\" moves the real OS pointer." },
"cursor_id": { "type": "string", "description": "Cursor instance to move. Default: 'default'." }
@@ -87,6 +90,9 @@ impl Tool for MoveCursorTool {
Err(error) => ToolResult::error(format!("desktop pointer task failed: {error}")),
};
}
+ if let (Some(pid), Some(wid)) = (args.opt_i64("pid"), args.opt_u64("window_id")) {
+ return self.move_over_window(&args, pid as i32, wid as u32).await;
+ }
if !self.state.cursor_overlay_available {
return super::cursor_overlay_unavailable();
}
@@ -115,6 +121,79 @@ impl Tool for MoveCursorTool {
}
}
+impl MoveCursorTool {
+ /// `target: {kind: "window", pid, window_id}`: x/y are window-local
+ /// screenshot pixels, as for `click`. Moves the agent cursor there. It
+ /// sends no input: AppKit derives hover (tracking areas, tooltips) from
+ /// the real pointer, and pointer-move events posted to a background app
+ /// do not reach its tracking areas (VM check, 11 Oct 2026: BenchLab saw no
+ /// hover through the SkyLight, public or both routes).
+ async fn move_over_window(&self, args: &Value, pid: i32, wid: u32) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if !self.state.cursor_overlay_available {
+ return super::cursor_overlay_unavailable();
+ }
+ let (x, y) = match (args.require_f64("x"), args.require_f64("y")) {
+ (Ok(x), Ok(y)) => (x, y),
+ (Err(e), _) | (_, Err(e)) => return e,
+ };
+ let ratio = match super::screenshot_scale(&self.state, args, pid, Some(wid)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ let frame = match super::px_frame::resolve_or_refuse(wid).await {
+ Ok(frame) => frame,
+ Err(refusal) => return refusal,
+ };
+ let (screen_x, screen_y, local_x, local_y) = frame.to_screen(x * ratio, y * ratio);
+ if local_x < 0.0
+ || local_y < 0.0
+ || local_x > frame.bounds.width
+ || local_y > frame.bounds.height
+ {
+ return ToolResult::error(format!(
+ "move_cursor: window-local point ({local_x:.1}, {local_y:.1}) pt lies outside \
+ window {wid}'s {:.0}×{:.0} pt frame. Re-read coordinates from a fresh \
+ get_window_state screenshot.",
+ frame.bounds.width, frame.bounds.height
+ ));
+ }
+ let cursor_id = super::cursor_tools::resolve_cursor_key(args);
+ crate::cursor::overlay::send_command(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
+ crate::cursor::overlay::animate_cursor_to(cursor_id.clone(), screen_x, screen_y).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, screen_x, screen_y);
+ let note = if crate::apps::frontmost_pid() == Some(pid) {
+ ""
+ } else {
+ BACKGROUND_HOVER_NOTE
+ };
+ ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \
+ pointer did not move.{note}"
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "window",
+ "window_id": wid,
+ "screen_point": [screen_x, screen_y],
+ "hover_delivered": false,
+ "effect": "unverifiable"
+ }))
+ }
+}
+
+/// What a window-target move does not do for a background app.
+const BACKGROUND_HOVER_NOTE: &str = " The app is in the background, so it sees no hover: \
+ macOS apps track hover and show tooltips from the real pointer, and no background route \
+ reaches them. A control that only appears on hover, or a tooltip, cannot be revealed \
+ without the user's pointer (scope:\"desktop\") or the app in front (bring_to_front); use \
+ either only if the user allows it. Look first for the same command elsewhere: a menu \
+ item, a context menu, a keyboard shortcut, or help text in get_window_state.";
+
/// The app that owns the frontmost ordinary window under a desktop point,
/// when that app is not the active one. The driver's own windows (cursor
/// overlay) are skipped.
@@ -143,6 +222,14 @@ fn inactive_app_note(app: &str) -> String {
#[cfg(test)]
mod tests {
+ #[test]
+ fn the_background_hover_note_names_both_routes_and_the_alternatives() {
+ let note = super::BACKGROUND_HOVER_NOTE;
+ assert!(note.contains("scope:\"desktop\""));
+ assert!(note.contains("bring_to_front"));
+ assert!(note.contains("only if the user allows it"));
+ }
+
#[test]
fn the_inactive_app_note_names_the_route_to_a_tooltip() {
let note = super::inactive_app_note("BenchLab");
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
index 2d9278cac2..e3a9e002c6 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
@@ -207,6 +207,11 @@ impl Tool for SetValueTool {
let changes = snapshot.detect_async().await;
match result {
+ Ok(Ok(outcome)) if outcome.not_applied && !ax_echo_surface => {
+ let vcl = crate::input::pointer_toolkit::detect(pid)
+ == Some(crate::input::pointer_toolkit::PointerReadingToolkit::Vcl);
+ not_applied_result(element_index, vcl)
+ }
Ok(Ok(mut outcome)) => {
apply_surface_trust(&mut outcome, ax_echo_surface);
apply_verification_label(&mut outcome);
@@ -327,6 +332,9 @@ struct SetValueOutcome {
/// `Some(false)` when the element already held the requested value, so the
/// write was a no-op. Lets callers distinguish "idempotent" from "applied".
changed: Option<bool>,
+ /// The read-back still showed the old value after a short settle: the app
+ /// took the accessibility write but did not apply it.
+ not_applied: bool,
}
fn apply_surface_trust(outcome: &mut SetValueOutcome, ax_echo_surface: bool) {
@@ -366,6 +374,7 @@ fn set_value_blocking(
detail,
verified: None,
changed: None,
+ not_applied: false,
}
})
} else {
@@ -393,13 +402,30 @@ fn set_value_blocking(
None => unsafe { set_string_attr(element, "AXValue", value) },
};
if err == kAXErrorSuccess {
- let after = unsafe { copy_string_attr(element, "AXValue") };
- let (verified, changed) = classify_write(
+ let mut after = unsafe { copy_string_attr(element, "AXValue") };
+ let (mut verified, mut changed) = classify_write(
before.as_deref(),
after.as_deref(),
value,
numeric_target.is_some(),
);
+ // Some apps apply the write a moment later; give it a short
+ // settle before calling it not applied.
+ let secure = unsafe { copy_string_attr(element, "AXSubrole") }.as_deref()
+ == Some("AXSecureTextField");
+ for _ in 0..NOT_APPLIED_REREADS {
+ if secure || !write_not_applied(verified, changed) {
+ break;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ after = unsafe { copy_string_attr(element, "AXValue") };
+ (verified, changed) = classify_write(
+ before.as_deref(),
+ after.as_deref(),
+ value,
+ numeric_target.is_some(),
+ );
+ }
let suffix = match (verified, changed) {
(Some(true), Some(false)) => " Value already matched; write was idempotent.",
(Some(true), _) => "",
@@ -410,6 +436,9 @@ fn set_value_blocking(
detail: format!("✅ Set AXValue on [{element_index}] {role}.{suffix}"),
verified,
changed,
+ // A secure text field reads back empty or masked, so its
+ // read-back proves nothing either way.
+ not_applied: !secure && write_not_applied(verified, changed),
})
} else if let Some(target) = numeric_target {
// Both direct writes failed for a numeric target — fall back to
@@ -424,6 +453,7 @@ fn set_value_blocking(
),
verified,
changed,
+ not_applied: false,
})
} else {
anyhow::bail!("AXUIElementSetAttributeValue(AXValue) failed with error {err}")
@@ -434,6 +464,41 @@ fn set_value_blocking(
}
}
+/// Re-reads (100 ms apart) before an unchanged value counts as not applied.
+const NOT_APPLIED_REREADS: usize = 3;
+
+/// The read-back did not match and the value did not move: the app accepted
+/// the accessibility write (LibreOffice Calc cells do) but did not apply it.
+fn write_not_applied(verified: Option<bool>, changed: Option<bool>) -> bool {
+ verified == Some(false) && changed == Some(false)
+}
+
+/// The refusal for a write the app did not apply. Nothing changed, so a
+/// script must stop here instead of carrying on as if it had.
+fn not_applied_result(element_index: usize, vcl: bool) -> ToolResult {
+ let route = if vcl {
+ "LibreOffice ignores accessibility value writes and drops background pointer \
+ input, so its cells cannot be edited in the background. Editing them needs the \
+ window in front: bring_to_front if the user allows it, then type_text into the \
+ cell. Otherwise stop and report that the edit is blocked."
+ } else {
+ "Try the control's own input route instead: click it, then type_text, or use a \
+ keyboard shortcut. If that needs the window in front, bring_to_front only if the \
+ user allows it."
+ };
+ ToolResult::error(format!(
+ "set_value did not take: element [{element_index}] still shows its old value after \
+ the write. The app accepted the accessibility write but did not apply it, so nothing \
+ changed. {route}"
+ ))
+ .with_structured(serde_json::json!({
+ "code": "write_not_applied",
+ "path": "ax",
+ "verified": false,
+ "effect": "not_applied",
+ }))
+}
+
/// Decide what a post-write AXValue read proves.
///
/// Returns `(verified, changed)`:
@@ -954,8 +1019,8 @@ fn hex_digit(n: u8) -> char {
mod tests {
use super::{
apply_surface_trust, apply_verification_label, classify_write, file_name_needs_rename,
- is_file_name_cell, is_get_info_name_field, SetValueOutcome, GET_INFO_RENAME_ROUTE,
- LIST_RENAME_ROUTE,
+ is_file_name_cell, is_get_info_name_field, not_applied_result, write_not_applied,
+ SetValueOutcome, GET_INFO_RENAME_ROUTE, LIST_RENAME_ROUTE,
};
#[test]
@@ -1044,6 +1109,29 @@ mod tests {
}
}
+ #[test]
+ fn an_unchanged_read_back_is_not_applied_but_an_idempotent_one_is_fine() {
+ // Read-back still shows the old value: not applied.
+ let (verified, changed) =
+ classify_write(Some("pending"), Some("pending"), "matched", false);
+ assert!(write_not_applied(verified, changed));
+ // Already held the requested value: idempotent, applied.
+ let (verified, changed) =
+ classify_write(Some("matched"), Some("matched"), "matched", false);
+ assert!(!write_not_applied(verified, changed));
+ // The app reformatted the value: it moved, so it is not "not applied".
+ let (verified, changed) = classify_write(Some(""), Some("1,248.60"), "1248.6", false);
+ assert!(!write_not_applied(verified, changed));
+ // Unreadable before or after: cannot tell.
+ let (verified, changed) = classify_write(None, Some("x"), "y", false);
+ assert!(!write_not_applied(verified, changed));
+ let refusal = not_applied_result(4, true);
+ assert_eq!(refusal.is_error, Some(true));
+ let data = refusal.structured_content.unwrap();
+ assert_eq!(data["code"], "write_not_applied");
+ assert_eq!(data["effect"], "not_applied");
+ }
+
#[test]
fn file_name_refusal_names_the_rename_route() {
let result = file_name_needs_rename(7, 42, LIST_RENAME_ROUTE);
@@ -1136,6 +1224,7 @@ mod tests {
detail: "Set value.".to_owned(),
verified: Some(true),
changed: Some(true),
+ not_applied: false,
};
apply_surface_trust(&mut outcome, true);
assert_eq!(outcome.verified, Some(false));
@@ -1149,6 +1238,7 @@ mod tests {
detail: "Set value.".to_owned(),
verified: Some(true),
changed: Some(true),
+ not_applied: false,
};
apply_surface_trust(&mut outcome, false);
assert_eq!(outcome.verified, Some(true));
@@ -1162,6 +1252,7 @@ mod tests {
detail: "✅ Set AXValue on [4] AXTextField.".to_owned(),
verified: Some(false),
changed: Some(false),
+ not_applied: false,
};
apply_verification_label(&mut outcome);
assert_eq!(
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index da991e478f..120a69dd0f 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -1406,7 +1406,7 @@
"read_only": true
},
{
- "description": "Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",
+ "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",
"destructive": false,
"idempotent": true,
"input_schema": {
@@ -1462,7 +1462,7 @@
"type": "object"
},
"x": {
- "description": "Destination X. Window scope: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels.",
+ "description": "Destination X. Window target: window screenshot pixels. No target: screen points for the agent cursor overlay. Desktop scope: native get_desktop_state screenshot pixels.",
"type": "number"
},
"y": {