Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): background hover on macOS through AX alternate UI and Electron input (#5005)
Francesco Bonaccie1ed4f41cdc0
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdxindex a942267747..d3af855d31 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx@@ -26,13 +26,13 @@ Click a target pid. Prefer `element_token` (from get_window_state): it identifie- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.-`button` defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.+`button` defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.**macOS parameters**| Parameter | Type | Default | Description || --- | --- | --- | --- |-| <span id="click--macos--action"></span>`action` | `string` | | AX action. |+| <span id="click--macos--action"></span>`action` | `string` | | AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer. || <span id="click--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button. Default left. || <span id="click--macos--capture_id"></span>`capture_id` | `string` | | Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused. || <span id="click--macos--count"></span>`count` | `integer` | | Click count (pixel path only). Default 1. |diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdxindex d56cb0f0fa..8c60d87eba 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx@@ -229,7 +229,7 @@ Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist><Tab value="macOS">-Move a cursor to (x, y). With target {kind: "window", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.+Move a cursor to (x, y). With target {kind: "window", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there; in a background app it also hovers through a route that leaves the real pointer alone where one exists (AXShowAlternateUI on the element under the point, or an Electron page) and says so. Without a target, moves only the agent cursor overlay, in screen points. Other macOS hover and tooltips follow the real pointer, so most background apps see no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.**macOS parameters**diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex 9310ebb121..f01f7e3f96 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -114,7 +114,7 @@ is safe even for apps that normally foreground on media-load| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` || Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls || Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |-| Hover (reveal a hover-only control) | Look for the same command elsewhere first (menu item, context menu, shortcut, help text in `get_window_state`). A background app sees no hover; only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovers: it moves only the agent cursor |+| Hover (reveal a hover-only control) | Window-targeted `move_cursor`; its result says whether it hovered (AX alternate UI or an Electron page). If not, look for the same command elsewhere (menu item, context menu, shortcut, help text in `get_window_state`); only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovered when its result says the app sees no hover || Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` || Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` || Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |@@ -325,12 +325,28 @@ not ask the app to come forward: `set_value` on the address field or### Hover in the backgroundmacOS apps track hover and show tooltips (help tags) from the real-pointer. Pointer-move events posted to a background app do not reach its-tracking areas, so a control that only appears on hover, or a tooltip,-cannot be revealed in the background. A window-targeted `move_cursor`-moves only the agent cursor and says so. Look for the same command-elsewhere first: a menu item, a context menu, a keyboard shortcut, or the-control's help text in `get_window_state`. Otherwise the step needs the+pointer. Pointer-move and enter events posted to a background app reach+the app but never its tracking areas, so most hover-only controls and all+native tooltips cannot be revealed in the background. A window-targeted+`move_cursor` on a background app tries the two routes that leave the+real pointer, focus and z-order alone. Its result says whether it hovered+and how:++- `ax_alternate_ui`: the element under the point (or an ancestor) lists+ `showalternateui`, as sidebar section headers in Finder, Mail, Music and+ System Settings do. The hover-only controls appear in the next+ `get_window_state`; hovering elsewhere hides them. The same element can+ be driven directly with `click` and `action:"show_alternate_ui"` /+ `"show_default_ui"`.+- `electron_input`: an Electron page gets `mouseEnter`/`mouseMove` through+ the app's main process, so page hover effects update. Native tooltips do+ not show.++For a web page in Chrome use `browser_pointer` hover. Otherwise+`move_cursor` moves only the agent cursor and says so. Look for the same+command elsewhere first: a menu item, a context menu, a keyboard+shortcut, or the control's help text in `get_window_state` (toolbar+tooltips are usually there as `help=`). Otherwise the step needs theuser's pointer (`move_cursor` with `scope:"desktop"`) or the app in front(`bring_to_front`); use either only if the user allows it, and say which.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rsindex 2920c64d95..17e0521b55 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs@@ -13,6 +13,44 @@ use tokio::sync::Mutex as AsyncMutex;pub struct CdpClient;impl CdpClient {+ /// Evaluate JavaScript in the Node.js main process behind an inspector+ /// port (Electron's SIGUSR1 inspector), whose `/json` lists a `node`+ /// target and no page.+ pub async fn evaluate_node(javascript: &str, port: u16) -> anyhow::Result<String> {+ let json = http_get_json(port).await?;+ let targets: serde_json::Value = serde_json::from_str(&json)+ .map_err(|e| anyhow::anyhow!("inspector /json parse error: {e}"))?;+ let ws_url = node_target_ws_url(&targets)+ .ok_or_else(|| anyhow::anyhow!("no node target on inspector port {port}"))?;+ let connection = CdpConnection::connect(&ws_url)+ .await+ .map_err(|e| anyhow::anyhow!("WebSocket connect failed: {e}"))?;+ let obj = tokio::time::timeout(+ Duration::from_secs(10),+ connection.call(+ None,+ "Runtime.evaluate",+ serde_json::json!({+ "expression": javascript,+ "returnByValue": true,+ "awaitPromise": true+ }),+ ),+ )+ .await+ .map_err(|_| anyhow::anyhow!("inspector evaluate timed out after 10s"))??;+ parse_cdp_result(&obj)+ }++ /// Whether `port` serves a Node inspector (a `node` target).+ pub async fn is_node_inspector(port: u16) -> bool {+ http_get_json(port)+ .await+ .ok()+ .and_then(|json| serde_json::from_str::<serde_json::Value>(&json).ok())+ .is_some_and(|targets| node_target_ws_url(&targets).is_some())+ }+/// Returns true if a CDP endpoint is listening on `port`.pub async fn is_available(port: u16) -> bool {http_get_json(port).await.is_ok()@@ -496,6 +534,18 @@ async fn ws_url_for_page_target(Ok((ws_url, target_url))}+/// The debugger WebSocket URL of the first `node` target in an inspector's+/// `/json` listing.+fn node_target_ws_url(targets: &serde_json::Value) -> Option<String> {+ targets+ .as_array()?+ .iter()+ .find(|t| t.get("type").and_then(|v| v.as_str()) == Some("node"))?+ .get("webSocketDebuggerUrl")?+ .as_str()+ .map(str::to_owned)+}+fn parse_cdp_result(obj: &serde_json::Value) -> anyhow::Result<String> {if let Some(err) = obj.get("error") {let msg = err@@ -619,6 +669,20 @@ async fn http_get_json(port: u16) -> anyhow::Result<String> {#[cfg(test)]mod tests {+ #[test]+ fn node_target_is_picked_from_an_inspector_listing() {+ let listing = serde_json::json!([+ {"type": "page", "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/page/1"},+ {"type": "node", "webSocketDebuggerUrl": "ws://127.0.0.1:9229/abc"}+ ]);+ assert_eq!(+ super::node_target_ws_url(&listing).as_deref(),+ Some("ws://127.0.0.1:9229/abc")+ );+ let pages_only = serde_json::json!([{"type": "page", "webSocketDebuggerUrl": "ws://x"}]);+ assert!(super::node_target_ws_url(&pages_only).is_none());+ }+use super::CdpSessionCache;use futures_util::{SinkExt, StreamExt};use std::sync::{diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rsindex fd6bde6f85..c22f3e9989 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs@@ -44,6 +44,30 @@ impl ElectronJs {false}+ /// Evaluate JavaScript in the Electron app's main (Node.js) process at+ /// `pid` through its inspector, opening it with SIGUSR1 when no inspector+ /// is listening yet. Apps built with the inspector fuse off refuse.+ pub async fn execute_main(javascript: &str, pid: i32) -> anyhow::Result<String> {+ let existing_ports = listening_ports(pid).await;+ for &port in &existing_ports {+ if CdpClient::is_node_inspector(port).await {+ return CdpClient::evaluate_node(javascript, port).await;+ }+ }+ unsafe {+ libc::kill(pid, libc::SIGUSR1);+ }+ for _ in 0..10 {+ tokio::time::sleep(Duration::from_millis(200)).await;+ for port in listening_ports(pid).await {+ if !existing_ports.contains(&port) && CdpClient::is_node_inspector(port).await {+ return CdpClient::evaluate_node(javascript, port).await;+ }+ }+ }+ anyhow::bail!("no Node inspector for Electron pid {pid} (the app may disable it)")+ }+/// Evaluate JavaScript in the Electron app at `pid` via CDP.pub async fn execute(javascript: &str, pid: i32) -> anyhow::Result<String> {// Only inspect listeners owned by the requested process. Probing commondiff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rsindex 35b9088741..15eb1d2d13 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs@@ -210,7 +210,7 @@ pub fn ensure_ax_action_enabled(element_ptr: usize, action: &str) -> anyhow::Res/// Perform an AX action on a cached element.pub fn perform_ax_action(element_ptr: usize, action: &str) -> anyhow::Result<()> {- let ax_action = map_action(action);+ let ax_action = map_action(action)?;ensure_ax_action_enabled(element_ptr, ax_action)?;let err = unsafe { perform_action(element_ptr as AXUIElementRef, ax_action) };@@ -221,16 +221,31 @@ pub fn perform_ax_action(element_ptr: usize, action: &str) -> anyhow::Result<()>}}-fn map_action(action: &str) -> &'static str {- match action.to_lowercase().as_str() {+/// The AX action named by a tool's `action` argument. Accepts the short+/// names (`press`, `show_menu`, ...) and the tree's spelling (`showalternateui`,+/// `AXShowAlternateUI`). An unknown name is an error: falling back to AXPress+/// would press a control the caller meant only to reveal.+pub fn map_action(action: &str) -> anyhow::Result<&'static str> {+ let key: String = action+ .chars()+ .filter(|c| *c != '_' && *c != '-')+ .collect::<String>()+ .to_lowercase();+ let key = key.strip_prefix("ax").unwrap_or(&key);+ Ok(match key {"press" | "click" => "AXPress",- "show_menu" | "right_click" | "rightclick" => "AXShowMenu",+ "showmenu" | "rightclick" => "AXShowMenu","pick" => "AXPick","confirm" => "AXConfirm","cancel" => "AXCancel","open" => "AXOpen",- _ => "AXPress",- }+ "showalternateui" => "AXShowAlternateUI",+ "showdefaultui" => "AXShowDefaultUI",+ _ => anyhow::bail!(+ "unknown action \"{action}\": use press, show_menu, pick, confirm, cancel, open, \+ show_alternate_ui or show_default_ui"+ ),+ })}/// Set AXFocused=true on an element (for pre-focusing before key press).@@ -280,6 +295,21 @@ pub fn set_ax_value(element_ptr: usize, value: &str) -> anyhow::Result<()> {#[cfg(test)]mod tests {+ #[test]+ fn action_names_accept_the_tree_spelling_and_refuse_unknown_ones() {+ use super::map_action;+ assert_eq!(map_action("press").unwrap(), "AXPress");+ assert_eq!(+ map_action("show_alternate_ui").unwrap(),+ "AXShowAlternateUI"+ );+ assert_eq!(map_action("showalternateui").unwrap(), "AXShowAlternateUI");+ assert_eq!(map_action("AXShowDefaultUI").unwrap(), "AXShowDefaultUI");+ assert_eq!(map_action("right_click").unwrap(), "AXShowMenu");+ // An unknown name must not fall back to pressing the control.+ assert!(map_action("hover").is_err());+ }+use super::*;#[test]diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rsnew file mode 100644index 0000000000..bdec0bf0d3--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rs@@ -0,0 +1,292 @@+//! Background hover routes that leave the user's real pointer alone.+//!+//! AppKit derives hover (tracking areas, tooltips) from the real pointer.+//! Pointer-move and enter events posted to a background app do reach+//! `-[NSApplication sendEvent:]`, but AppKit never turns them into tracking+//! callbacks (VM probe, 11 Oct 2026). Two routes do work without moving the+//! pointer, focusing or raising anything:+//!+//! * **AX alternate UI.** Many AppKit views model their hover-only controls as+//! transient UI (`NSAccessibilityContainsTransientUI`) and advertise+//! `AXShowAlternateUI` / `AXShowDefaultUI`. Sidebar section headers in Finder,+//! Mail, Music and System Settings reveal their hover-only Show/Hide button+//! this way.+//! * **Electron input.** `webContents.sendInputEvent` in the app's main process+//! delivers `mouseEnter`/`mouseMove` straight to the renderer.++use std::sync::Mutex;++use core_foundation::base::{CFEqual, CFRelease, CFTypeRef};++use crate::ax::bindings::{+ ax_get_window_id, copy_action_names, copy_bool_attr, copy_children, copy_element_attr,+ copy_string_attr, kAXErrorSuccess, perform_action, AXUIElementCopyElementAtPosition,+ AXUIElementCreateApplication, AXUIElementRef, AXUIElementSetMessagingTimeout,+};++/// How far up from the hit element to look for an element with alternate UI.+const ANCESTOR_DEPTH: usize = 6;+const AX_TIMEOUT_S: f32 = 0.25;+const SHOW_ALTERNATE: &str = "AXShowAlternateUI";+const SHOW_DEFAULT: &str = "AXShowDefaultUI";+/// How deep to count the target's subtree when checking for new controls.+const SUBTREE_DEPTH: usize = 4;++/// The element whose alternate UI the last hover revealed: (pid, retained+/// AXUIElementRef as usize). Moving the hover elsewhere restores its default+/// UI, the way the pointer leaving it would.+static REVEALED: Mutex<Option<(i32, usize)>> = Mutex::new(None);++/// What the AX route did.+#[derive(Debug, Clone, PartialEq)]+pub struct AxReveal {+ pub role: String,+ pub label: String,+ /// `AXAlternateUIVisible` read back after the action, when the app exposes it.+ pub visible: Option<bool>,+ /// Elements in the target's subtree before and after the action. More+ /// after means hover-only controls appeared.+ pub elements_before: usize,+ pub elements_after: usize,+}++/// Whether an element's advertised actions let the AX route reveal its+/// hover-only UI.+pub fn advertises_alternate_ui(actions: &[String]) -> bool {+ actions.iter().any(|a| a == SHOW_ALTERNATE)+}++/// Reveal the hover-only UI of the element under a screen point in `pid`'s+/// window `window_id`, through `AXShowAlternateUI` on the hit element or one of+/// its ancestors. Restores the previously revealed element first. Returns+/// `None` when nothing under the point offers alternate UI.+pub fn reveal_alternate_ui(pid: i32, window_id: u32, x: f64, y: f64) -> Option<AxReveal> {+ unsafe {+ let Some(target) = alternate_ui_target(pid, window_id, x, y) else {+ // Hovering where nothing reveals: the last revealed element loses hover.+ restore_default_ui(None);+ return None;+ };+ restore_default_ui(Some(target));+ let elements_before = subtree_size(target, SUBTREE_DEPTH);+ if perform_action(target, SHOW_ALTERNATE) != kAXErrorSuccess {+ CFRelease(target as CFTypeRef);+ return None;+ }+ let reveal = AxReveal {+ role: copy_string_attr(target, "AXRole").unwrap_or_default(),+ label: first_label(target),+ visible: copy_bool_attr(target, "AXAlternateUIVisible"),+ elements_before,+ elements_after: {+ std::thread::sleep(std::time::Duration::from_millis(150));+ subtree_size(target, SUBTREE_DEPTH)+ },+ };+ let mut slot = REVEALED.lock().unwrap_or_else(|e| e.into_inner());+ match *slot {+ Some((_, prev)) if CFEqual(prev as CFTypeRef, target as CFTypeRef) != 0 => {+ CFRelease(target as CFTypeRef);+ }+ _ => *slot = Some((pid, target as usize)),+ }+ Some(reveal)+ }+}++/// The element under the point in window `window_id`, or its nearest+/// ancestor, that advertises `AXShowAlternateUI`. Owned (+1) by the caller.+unsafe fn alternate_ui_target(pid: i32, window_id: u32, x: f64, y: f64) -> Option<AXUIElementRef> {+ let app = AXUIElementCreateApplication(pid);+ if app.is_null() {+ return None;+ }+ AXUIElementSetMessagingTimeout(app, AX_TIMEOUT_S);+ let mut hit: AXUIElementRef = std::ptr::null_mut();+ let err = AXUIElementCopyElementAtPosition(app, x as f32, y as f32, &mut hit);+ CFRelease(app as CFTypeRef);+ if err != kAXErrorSuccess || hit.is_null() {+ return None;+ }+ // The point must land in the target window, not another window of the app.+ let in_window = copy_element_attr(hit, "AXWindow").map(|w| {+ let id = ax_get_window_id(w);+ CFRelease(w as CFTypeRef);+ id == Some(window_id)+ });+ if in_window != Some(true) {+ CFRelease(hit as CFTypeRef);+ return None;+ }+ let mut current = Some(hit);+ for _ in 0..=ANCESTOR_DEPTH {+ let el = current.take()?;+ if advertises_alternate_ui(©_action_names(el)) {+ return Some(el);+ }+ current = copy_element_attr(el, "AXParent");+ CFRelease(el as CFTypeRef);+ }+ if let Some(el) = current {+ CFRelease(el as CFTypeRef);+ }+ None+}++/// Put the previously revealed element back to its default UI, unless it is+/// `keep` (the element about to be revealed again).+unsafe fn restore_default_ui(keep: Option<AXUIElementRef>) {+ let mut slot = REVEALED.lock().unwrap_or_else(|e| e.into_inner());+ let Some((_, prev)) = *slot else { return };+ if let Some(k) = keep {+ if CFEqual(prev as CFTypeRef, k as CFTypeRef) != 0 {+ return;+ }+ }+ perform_action(prev as AXUIElementRef, SHOW_DEFAULT);+ CFRelease(prev as CFTypeRef);+ *slot = None;+}++unsafe fn first_label(element: AXUIElementRef) -> String {+ for attr in ["AXTitle", "AXDescription", "AXValue"] {+ if let Some(s) = copy_string_attr(element, attr).filter(|s| !s.trim().is_empty()) {+ return s;+ }+ }+ // Outline rows carry their label on a child (or grandchild) text element.+ let mut label = String::new();+ for child in copy_children(element) {+ if label.is_empty() {+ label = text_of(child);+ if label.is_empty() {+ for grandchild in copy_children(child) {+ if label.is_empty() {+ label = text_of(grandchild);+ }+ CFRelease(grandchild as CFTypeRef);+ }+ }+ }+ CFRelease(child as CFTypeRef);+ }+ label+}++unsafe fn text_of(element: AXUIElementRef) -> String {+ copy_string_attr(element, "AXValue")+ .or_else(|| copy_string_attr(element, "AXTitle"))+ .filter(|s| !s.trim().is_empty())+ .unwrap_or_default()+}++/// Number of elements under `element`, `depth` levels deep (bounded).+unsafe fn subtree_size(element: AXUIElementRef, depth: usize) -> usize {+ if depth == 0 {+ return 0;+ }+ let mut n = 0;+ for child in copy_children(element) {+ n += 1 + subtree_size(child, depth - 1);+ CFRelease(child as CFTypeRef);+ }+ n+}++impl AxReveal {+ /// Whether the action made new controls appear (or the app says its+ /// alternate UI is visible).+ pub fn revealed_new_controls(&self) -> bool {+ self.elements_after > self.elements_before || self.visible == Some(true)+ }+}++/// Main-process JavaScript that hovers the renderer of the BrowserWindow whose+/// native window number is `window_id`, at window-local point (`x`, `y`) in+/// points. Returns a JSON string: `{"ok":true,"x":..,"y":..}` or+/// `{"ok":false,"reason":..}`.+pub fn electron_hover_js(window_id: u32, x: f64, y: f64) -> String {+ format!(+ r#"(() => {{+ const req = (typeof require === 'function') ? require : (process.mainModule && process.mainModule.require.bind(process.mainModule));+ if (!req) return JSON.stringify({{ ok: false, reason: 'no_require' }});+ const {{ BrowserWindow }} = req('electron');+ const w = BrowserWindow.getAllWindows().find((w) => {{+ try {{ return String(w.getMediaSourceId()).split(':')[1] === '{window_id}'; }} catch (e) {{ return false; }}+ }});+ if (!w) return JSON.stringify({{ ok: false, reason: 'no_window' }});+ const b = w.getBounds(), c = w.getContentBounds();+ const x = Math.round({x} - (c.x - b.x)), y = Math.round({y} - (c.y - b.y));+ if (x < 0 || y < 0 || x >= c.width || y >= c.height) return JSON.stringify({{ ok: false, reason: 'outside_content' }});+ w.webContents.sendInputEvent({{ type: 'mouseEnter', x, y }});+ w.webContents.sendInputEvent({{ type: 'mouseMove', x, y }});+ return JSON.stringify({{ ok: true, x, y }});+}})()"#,+ window_id = window_id,+ x = x,+ y = y+ )+}++/// Whether the main-process reply says the hover was delivered.+pub fn electron_hover_delivered(reply: &str) -> bool {+ let trimmed = reply.trim().trim_matches('"').replace("\\\"", "\"");+ serde_json::from_str::<serde_json::Value>(&trimmed)+ .ok()+ .and_then(|v| v.get("ok").and_then(|ok| ok.as_bool()))+ .unwrap_or(false)+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn a_reveal_counts_only_when_controls_appear_or_the_app_says_so() {+ let r = |before, after, visible| AxReveal {+ role: "AXRow".into(),+ label: String::new(),+ visible,+ elements_before: before,+ elements_after: after,+ };+ assert!(r(3, 4, None).revealed_new_controls());+ assert!(r(3, 3, Some(true)).revealed_new_controls());+ assert!(!r(3, 3, None).revealed_new_controls());+ assert!(!r(3, 3, Some(false)).revealed_new_controls());+ }++ #[test]+ fn only_show_alternate_ui_qualifies_an_element() {+ assert!(advertises_alternate_ui(&[+ "AXShowDefaultUI".into(),+ "AXShowAlternateUI".into()+ ]));+ assert!(!advertises_alternate_ui(&[+ "AXPress".into(),+ "AXShowMenu".into()+ ]));+ }++ #[test]+ fn electron_js_targets_the_window_by_media_source_id_and_content_offset() {+ let js = electron_hover_js(227, 150.0, 178.5);+ assert!(js.contains("split(':')[1] === '227'"));+ assert!(js.contains("Math.round(150 - (c.x - b.x))"));+ assert!(js.contains("Math.round(178.5 - (c.y - b.y))"));+ assert!(js.contains("type: 'mouseEnter'"));+ assert!(js.contains("type: 'mouseMove'"));+ }++ #[test]+ fn electron_reply_parsing_accepts_raw_and_quoted_json() {+ assert!(electron_hover_delivered(r#"{"ok":true,"x":1,"y":2}"#));+ assert!(electron_hover_delivered(+ r#""{\"ok\":true,\"x\":1,\"y\":2}""#+ ));+ assert!(!electron_hover_delivered(+ r#"{"ok":false,"reason":"no_window"}"#+ ));+ assert!(!electron_hover_delivered("undefined"));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rsindex 123289a225..e3faeaeee7 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs@@ -9,6 +9,7 @@//! the activity-monitor tickle required for live-input detection.pub mod ax_actions;+pub mod hover;pub mod interactive;pub mod keyboard;pub mod mouse;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex a0a79e18a9..996e625db6 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -244,7 +244,7 @@ fn def() -> &'static ToolDef {- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n\- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\\n\- `button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.".into(),+ `button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.".into(),input_schema: serde_json::json!({"type": "object",// `pid` is conditionally required — needed for window/element clicks@@ -261,7 +261,7 @@ fn def() -> &'static ToolDef {"capture_id": { "type": "string", "description": "Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused." },"x": { "type": "number", "description": "X in screenshot pixels (get_window_state PNG, or get_desktop_state PNG for a desktop target)." },"y": { "type": "number", "description": "Y in screenshot pixels from the image selected by target." },- "action": { "type": "string", "description": "AX action." },+ "action": { "type": "string", "description": "AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer." },"button": {"type": "string","enum": ["left", "right", "middle"],@@ -478,6 +478,9 @@ impl Tool for ClickTool {.opt_f64("y").or_else(|| args.opt_i64("y").map(|i| i as f64));let action = args.str_or("action", "press");+ if let Err(error) = crate::input::ax_actions::map_action(&action) {+ return ToolResult::error(format!("click: {error}"));+ }// Surface 5: optional `button` arg, default "left" preserves legacy behaviour.// Pixel path: routes to left/right/middle CGEvent primitives.// AX path: "right" delegates to AXShowMenu (same surface as right_click);@@ -1428,7 +1431,7 @@ fn perform_ax_click() -> anyhow::Result<(String, bool, bool, bool, bool)> {let (element_ptr, idx) = element;let (pid, window_id) = window;- let ax_action = map_action(action_str);+ let ax_action = crate::input::ax_actions::map_action(action_str)?;let element = element_ptr as AXUIElementRef;// Check the live value immediately before dispatch. Foreground assist can@@ -1727,18 +1730,6 @@ mod selection_fallback_tests {}}-fn map_action(action: &str) -> &'static str {- match action.to_lowercase().as_str() {- "press" | "click" => "AXPress",- "show_menu" | "right_click" => "AXShowMenu",- "pick" => "AXPick",- "confirm" => "AXConfirm",- "cancel" => "AXCancel",- "open" => "AXOpen",- _ => "AXPress",- }-}-// ── Tests ────────────────────────────────────────────────────────────────────#[cfg(test)]diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rsindex f03c5c0dde..88b3cf0e8e 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs@@ -27,10 +27,12 @@ fn def() -> &'static ToolDef {name: "move_cursor".into(),description: "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, \x/y are window screenshot pixels (as for click) and only the agent cursor moves \- there. Without a target, moves only the agent cursor overlay, in screen points. \- Neither sends input: macOS apps track hover and show tooltips from the real pointer, \- so a background app sees no hover. With scope=desktop, moves the real OS pointer in \- native get_desktop_state screenshot coordinates.".into(),+ there; in a background app it also hovers through a route that leaves the real \+ pointer alone where one exists (AXShowAlternateUI on the element under the point, \+ or an Electron page) and says so. Without a target, moves only the agent cursor \+ overlay, in screen points. Other macOS hover and tooltips follow the real pointer, \+ so most background apps see no hover. With scope=desktop, moves the real OS pointer \+ in native get_desktop_state screenshot coordinates.".into(),input_schema: serde_json::json!({"type": "object","required": ["x", "y"],@@ -167,14 +169,90 @@ impl MoveCursorTool {self.state.cursor_registry.update_position(&cursor_id, screen_x, screen_y);- let note = if crate::apps::frontmost_pid() == Some(pid) {- ""- } else {- BACKGROUND_HOVER_NOTE- };+ if crate::apps::frontmost_pid() == Some(pid) {+ return ToolResult::text(format!(+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \+ pointer did not move."+ ))+ .with_structured(serde_json::json!({+ "scope": "window",+ "window_id": wid,+ "screen_point": [screen_x, screen_y],+ "hover_delivered": false,+ "effect": "unverifiable"+ }));+ }+ // Background app: try the hover routes that leave the real pointer alone.+ let ax = tokio::task::spawn_blocking(move || {+ crate::input::hover::reveal_alternate_ui(pid, wid, screen_x, screen_y)+ })+ .await+ .ok()+ .flatten();+ if let Some(reveal) = ax {+ let label = if reveal.label.is_empty() {+ String::new()+ } else {+ format!(" \"{}\"", reveal.label)+ };+ let revealed = reveal.revealed_new_controls();+ let outcome = if revealed {+ format!(+ "Revealed the hover-only controls of {}{label} through AXShowAlternateUI \+ ({} new element(s)), without the real pointer, focus or raise. Call \+ get_window_state to see them; hovering elsewhere hides them again.",+ reveal.role,+ reveal.elements_after.saturating_sub(reveal.elements_before)+ )+ } else {+ format!(+ "Sent AXShowAlternateUI to {}{label}, the background hover route, but no new \+ control appeared in its accessibility tree, so it may have no hover-only \+ controls. The real pointer did not move.",+ reveal.role+ )+ };+ return ToolResult::text(format!(+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}). {outcome}"+ ))+ .with_structured(serde_json::json!({+ "scope": "window",+ "window_id": wid,+ "screen_point": [screen_x, screen_y],+ "hover_delivered": true,+ "hover_route": "ax_alternate_ui",+ "alternate_ui_visible": reveal.visible,+ "new_elements": reveal.elements_after.saturating_sub(reveal.elements_before),+ "effect": if revealed { "confirmed" } else { "unverifiable" }+ }));+ }+ if tokio::task::spawn_blocking(move || crate::browser::ElectronJs::is_electron(pid))+ .await+ .unwrap_or(false)+ {+ let js = crate::input::hover::electron_hover_js(wid, local_x, local_y);+ if let Ok(reply) = crate::browser::ElectronJs::execute_main(&js, pid).await {+ if crate::input::hover::electron_hover_delivered(&reply) {+ return ToolResult::text(format!(+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}). \+ Hovered the Electron page there with webContents.sendInputEvent, without \+ the real pointer, focus or raise. Hover effects in the page update; \+ native tooltips do not show."+ ))+ .with_structured(serde_json::json!({+ "scope": "window",+ "window_id": wid,+ "screen_point": [screen_x, screen_y],+ "hover_delivered": true,+ "hover_route": "electron_input",+ "effect": "unverifiable"+ }));+ }+ }+ }ToolResult::text(format!("Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \- pointer did not move.{note}"+ pointer did not move.{BACKGROUND_HOVER_NOTE}")).with_structured(serde_json::json!({"scope": "window",@@ -188,11 +266,13 @@ impl MoveCursorTool {/// What a window-target move does not do for a background app.const BACKGROUND_HOVER_NOTE: &str = " The app is in the background, so it sees no hover: \- macOS apps track hover and show tooltips from the real pointer, and no background route \- reaches them. A control that only appears on hover, or a tooltip, cannot be revealed \- without the user's pointer (scope:\"desktop\") or the app in front (bring_to_front); use \- either only if the user allows it. Look first for the same command elsewhere: a menu \- item, a context menu, a keyboard shortcut, or help text in get_window_state.";+ macOS apps track hover and show tooltips from the real pointer, and nothing under this \+ point offers a background route (no AXShowAlternateUI, not an Electron page). For a web \+ page in Chrome use browser_pointer hover. A control that only appears on hover, or a \+ tooltip, cannot be revealed without the user's pointer (scope:\"desktop\") or the app in \+ front (bring_to_front); use either only if the user allows it. Look first for the same \+ command elsewhere: a menu item, a context menu, a keyboard shortcut, or help text in \+ get_window_state.";/// The app that owns the frontmost ordinary window under a desktop point,/// when that app is not the active one. The driver's own windows (cursordiff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 120a69dd0f..d40d8bbd9f 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -486,14 +486,14 @@"read_only": false},{- "description": "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\n- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\n`button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.",+ "description": "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\n- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\n`button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.","destructive": true,"idempotent": false,"input_schema": {"additionalProperties": false,"properties": {"action": {- "description": "AX action.",+ "description": "AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer.","type": "string"},"button": {@@ -1406,7 +1406,7 @@"read_only": true},{- "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",+ "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there; in a background app it also hovers through a route that leaves the real pointer alone where one exists (AXShowAlternateUI on the element under the point, or an Electron page) and says so. Without a target, moves only the agent cursor overlay, in screen points. Other macOS hover and tooltips follow the real pointer, so most background apps see no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.","destructive": false,"idempotent": true,"input_schema": {