Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): background hover on macOS through AX alternate UI and Electron input (#5005)

Francesco Bonaccie1ed4f41cdc0
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
index a942267747..d3af855d31 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/click.mdx
@@ -26,13 +26,13 @@ Click a target pid. Prefer `element_token` (from get_window_state): it identifie
- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.
- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.
-`button` defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.
+`button` defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| <span id="click--macos--action"></span>`action` | `string` | | AX action. |
+| <span id="click--macos--action"></span>`action` | `string` | | AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer. |
| <span id="click--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button. Default left. |
| <span id="click--macos--capture_id"></span>`capture_id` | `string` | | Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused. |
| <span id="click--macos--count"></span>`count` | `integer` | | Click count (pixel path only). Default 1. |
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index d56cb0f0fa..8c60d87eba 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -229,7 +229,7 @@ Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-Move a cursor to (x, y). With target &#123;kind: "window", pid, window_id&#125;, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.
+Move a cursor to (x, y). With target &#123;kind: "window", pid, window_id&#125;, x/y are window screenshot pixels (as for click) and only the agent cursor moves there; in a background app it also hovers through a route that leaves the real pointer alone where one exists (AXShowAlternateUI on the element under the point, or an Electron page) and says so. Without a target, moves only the agent cursor overlay, in screen points. Other macOS hover and tooltips follow the real pointer, so most background apps see no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.
**macOS parameters**
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 9310ebb121..f01f7e3f96 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -114,7 +114,7 @@ is safe even for apps that normally foreground on media-load
| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |
| Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls |
| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |
-| Hover (reveal a hover-only control) | Look for the same command elsewhere first (menu item, context menu, shortcut, help text in `get_window_state`). A background app sees no hover; only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovers: it moves only the agent cursor |
+| Hover (reveal a hover-only control) | Window-targeted `move_cursor`; its result says whether it hovered (AX alternate UI or an Electron page). If not, look for the same command elsewhere (menu item, context menu, shortcut, help text in `get_window_state`); only `move_cursor({scope:"desktop"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovered when its result says the app sees no hover |
| Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` |
| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` |
| Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |
@@ -325,12 +325,28 @@ not ask the app to come forward: `set_value` on the address field or
### Hover in the background
macOS apps track hover and show tooltips (help tags) from the real
-pointer. Pointer-move events posted to a background app do not reach its
-tracking areas, so a control that only appears on hover, or a tooltip,
-cannot be revealed in the background. A window-targeted `move_cursor`
-moves only the agent cursor and says so. Look for the same command
-elsewhere first: a menu item, a context menu, a keyboard shortcut, or the
-control's help text in `get_window_state`. Otherwise the step needs the
+pointer. Pointer-move and enter events posted to a background app reach
+the app but never its tracking areas, so most hover-only controls and all
+native tooltips cannot be revealed in the background. A window-targeted
+`move_cursor` on a background app tries the two routes that leave the
+real pointer, focus and z-order alone. Its result says whether it hovered
+and how:
+
+- `ax_alternate_ui`: the element under the point (or an ancestor) lists
+ `showalternateui`, as sidebar section headers in Finder, Mail, Music and
+ System Settings do. The hover-only controls appear in the next
+ `get_window_state`; hovering elsewhere hides them. The same element can
+ be driven directly with `click` and `action:"show_alternate_ui"` /
+ `"show_default_ui"`.
+- `electron_input`: an Electron page gets `mouseEnter`/`mouseMove` through
+ the app's main process, so page hover effects update. Native tooltips do
+ not show.
+
+For a web page in Chrome use `browser_pointer` hover. Otherwise
+`move_cursor` moves only the agent cursor and says so. Look for the same
+command elsewhere first: a menu item, a context menu, a keyboard
+shortcut, or the control's help text in `get_window_state` (toolbar
+tooltips are usually there as `help=`). Otherwise the step needs the
user's pointer (`move_cursor` with `scope:"desktop"`) or the app in front
(`bring_to_front`); use either only if the user allows it, and say which.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs
index 2920c64d95..17e0521b55 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/cdp_client.rs
@@ -13,6 +13,44 @@ use tokio::sync::Mutex as AsyncMutex;
pub struct CdpClient;
impl CdpClient {
+ /// Evaluate JavaScript in the Node.js main process behind an inspector
+ /// port (Electron's SIGUSR1 inspector), whose `/json` lists a `node`
+ /// target and no page.
+ pub async fn evaluate_node(javascript: &str, port: u16) -> anyhow::Result<String> {
+ let json = http_get_json(port).await?;
+ let targets: serde_json::Value = serde_json::from_str(&json)
+ .map_err(|e| anyhow::anyhow!("inspector /json parse error: {e}"))?;
+ let ws_url = node_target_ws_url(&targets)
+ .ok_or_else(|| anyhow::anyhow!("no node target on inspector port {port}"))?;
+ let connection = CdpConnection::connect(&ws_url)
+ .await
+ .map_err(|e| anyhow::anyhow!("WebSocket connect failed: {e}"))?;
+ let obj = tokio::time::timeout(
+ Duration::from_secs(10),
+ connection.call(
+ None,
+ "Runtime.evaluate",
+ serde_json::json!({
+ "expression": javascript,
+ "returnByValue": true,
+ "awaitPromise": true
+ }),
+ ),
+ )
+ .await
+ .map_err(|_| anyhow::anyhow!("inspector evaluate timed out after 10s"))??;
+ parse_cdp_result(&obj)
+ }
+
+ /// Whether `port` serves a Node inspector (a `node` target).
+ pub async fn is_node_inspector(port: u16) -> bool {
+ http_get_json(port)
+ .await
+ .ok()
+ .and_then(|json| serde_json::from_str::<serde_json::Value>(&json).ok())
+ .is_some_and(|targets| node_target_ws_url(&targets).is_some())
+ }
+
/// Returns true if a CDP endpoint is listening on `port`.
pub async fn is_available(port: u16) -> bool {
http_get_json(port).await.is_ok()
@@ -496,6 +534,18 @@ async fn ws_url_for_page_target(
Ok((ws_url, target_url))
}
+/// The debugger WebSocket URL of the first `node` target in an inspector's
+/// `/json` listing.
+fn node_target_ws_url(targets: &serde_json::Value) -> Option<String> {
+ targets
+ .as_array()?
+ .iter()
+ .find(|t| t.get("type").and_then(|v| v.as_str()) == Some("node"))?
+ .get("webSocketDebuggerUrl")?
+ .as_str()
+ .map(str::to_owned)
+}
+
fn parse_cdp_result(obj: &serde_json::Value) -> anyhow::Result<String> {
if let Some(err) = obj.get("error") {
let msg = err
@@ -619,6 +669,20 @@ async fn http_get_json(port: u16) -> anyhow::Result<String> {
#[cfg(test)]
mod tests {
+ #[test]
+ fn node_target_is_picked_from_an_inspector_listing() {
+ let listing = serde_json::json!([
+ {"type": "page", "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/page/1"},
+ {"type": "node", "webSocketDebuggerUrl": "ws://127.0.0.1:9229/abc"}
+ ]);
+ assert_eq!(
+ super::node_target_ws_url(&listing).as_deref(),
+ Some("ws://127.0.0.1:9229/abc")
+ );
+ let pages_only = serde_json::json!([{"type": "page", "webSocketDebuggerUrl": "ws://x"}]);
+ assert!(super::node_target_ws_url(&pages_only).is_none());
+ }
+
use super::CdpSessionCache;
use futures_util::{SinkExt, StreamExt};
use std::sync::{
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs
index fd6bde6f85..c22f3e9989 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/electron_js.rs
@@ -44,6 +44,30 @@ impl ElectronJs {
false
}
+ /// Evaluate JavaScript in the Electron app's main (Node.js) process at
+ /// `pid` through its inspector, opening it with SIGUSR1 when no inspector
+ /// is listening yet. Apps built with the inspector fuse off refuse.
+ pub async fn execute_main(javascript: &str, pid: i32) -> anyhow::Result<String> {
+ let existing_ports = listening_ports(pid).await;
+ for &port in &existing_ports {
+ if CdpClient::is_node_inspector(port).await {
+ return CdpClient::evaluate_node(javascript, port).await;
+ }
+ }
+ unsafe {
+ libc::kill(pid, libc::SIGUSR1);
+ }
+ for _ in 0..10 {
+ tokio::time::sleep(Duration::from_millis(200)).await;
+ for port in listening_ports(pid).await {
+ if !existing_ports.contains(&port) && CdpClient::is_node_inspector(port).await {
+ return CdpClient::evaluate_node(javascript, port).await;
+ }
+ }
+ }
+ anyhow::bail!("no Node inspector for Electron pid {pid} (the app may disable it)")
+ }
+
/// Evaluate JavaScript in the Electron app at `pid` via CDP.
pub async fn execute(javascript: &str, pid: i32) -> anyhow::Result<String> {
// Only inspect listeners owned by the requested process. Probing common
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
index 35b9088741..15eb1d2d13 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
@@ -210,7 +210,7 @@ pub fn ensure_ax_action_enabled(element_ptr: usize, action: &str) -> anyhow::Res
/// Perform an AX action on a cached element.
pub fn perform_ax_action(element_ptr: usize, action: &str) -> anyhow::Result<()> {
- let ax_action = map_action(action);
+ let ax_action = map_action(action)?;
ensure_ax_action_enabled(element_ptr, ax_action)?;
let err = unsafe { perform_action(element_ptr as AXUIElementRef, ax_action) };
@@ -221,16 +221,31 @@ pub fn perform_ax_action(element_ptr: usize, action: &str) -> anyhow::Result<()>
}
}
-fn map_action(action: &str) -> &'static str {
- match action.to_lowercase().as_str() {
+/// The AX action named by a tool's `action` argument. Accepts the short
+/// names (`press`, `show_menu`, ...) and the tree's spelling (`showalternateui`,
+/// `AXShowAlternateUI`). An unknown name is an error: falling back to AXPress
+/// would press a control the caller meant only to reveal.
+pub fn map_action(action: &str) -> anyhow::Result<&'static str> {
+ let key: String = action
+ .chars()
+ .filter(|c| *c != '_' && *c != '-')
+ .collect::<String>()
+ .to_lowercase();
+ let key = key.strip_prefix("ax").unwrap_or(&key);
+ Ok(match key {
"press" | "click" => "AXPress",
- "show_menu" | "right_click" | "rightclick" => "AXShowMenu",
+ "showmenu" | "rightclick" => "AXShowMenu",
"pick" => "AXPick",
"confirm" => "AXConfirm",
"cancel" => "AXCancel",
"open" => "AXOpen",
- _ => "AXPress",
- }
+ "showalternateui" => "AXShowAlternateUI",
+ "showdefaultui" => "AXShowDefaultUI",
+ _ => anyhow::bail!(
+ "unknown action \"{action}\": use press, show_menu, pick, confirm, cancel, open, \
+ show_alternate_ui or show_default_ui"
+ ),
+ })
}
/// Set AXFocused=true on an element (for pre-focusing before key press).
@@ -280,6 +295,21 @@ pub fn set_ax_value(element_ptr: usize, value: &str) -> anyhow::Result<()> {
#[cfg(test)]
mod tests {
+ #[test]
+ fn action_names_accept_the_tree_spelling_and_refuse_unknown_ones() {
+ use super::map_action;
+ assert_eq!(map_action("press").unwrap(), "AXPress");
+ assert_eq!(
+ map_action("show_alternate_ui").unwrap(),
+ "AXShowAlternateUI"
+ );
+ assert_eq!(map_action("showalternateui").unwrap(), "AXShowAlternateUI");
+ assert_eq!(map_action("AXShowDefaultUI").unwrap(), "AXShowDefaultUI");
+ assert_eq!(map_action("right_click").unwrap(), "AXShowMenu");
+ // An unknown name must not fall back to pressing the control.
+ assert!(map_action("hover").is_err());
+ }
+
use super::*;
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rs
new file mode 100644
index 0000000000..bdec0bf0d3
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/hover.rs
@@ -0,0 +1,292 @@
+//! Background hover routes that leave the user's real pointer alone.
+//!
+//! AppKit derives hover (tracking areas, tooltips) from the real pointer.
+//! Pointer-move and enter events posted to a background app do reach
+//! `-[NSApplication sendEvent:]`, but AppKit never turns them into tracking
+//! callbacks (VM probe, 11 Oct 2026). Two routes do work without moving the
+//! pointer, focusing or raising anything:
+//!
+//! * **AX alternate UI.** Many AppKit views model their hover-only controls as
+//! transient UI (`NSAccessibilityContainsTransientUI`) and advertise
+//! `AXShowAlternateUI` / `AXShowDefaultUI`. Sidebar section headers in Finder,
+//! Mail, Music and System Settings reveal their hover-only Show/Hide button
+//! this way.
+//! * **Electron input.** `webContents.sendInputEvent` in the app's main process
+//! delivers `mouseEnter`/`mouseMove` straight to the renderer.
+
+use std::sync::Mutex;
+
+use core_foundation::base::{CFEqual, CFRelease, CFTypeRef};
+
+use crate::ax::bindings::{
+ ax_get_window_id, copy_action_names, copy_bool_attr, copy_children, copy_element_attr,
+ copy_string_attr, kAXErrorSuccess, perform_action, AXUIElementCopyElementAtPosition,
+ AXUIElementCreateApplication, AXUIElementRef, AXUIElementSetMessagingTimeout,
+};
+
+/// How far up from the hit element to look for an element with alternate UI.
+const ANCESTOR_DEPTH: usize = 6;
+const AX_TIMEOUT_S: f32 = 0.25;
+const SHOW_ALTERNATE: &str = "AXShowAlternateUI";
+const SHOW_DEFAULT: &str = "AXShowDefaultUI";
+/// How deep to count the target's subtree when checking for new controls.
+const SUBTREE_DEPTH: usize = 4;
+
+/// The element whose alternate UI the last hover revealed: (pid, retained
+/// AXUIElementRef as usize). Moving the hover elsewhere restores its default
+/// UI, the way the pointer leaving it would.
+static REVEALED: Mutex<Option<(i32, usize)>> = Mutex::new(None);
+
+/// What the AX route did.
+#[derive(Debug, Clone, PartialEq)]
+pub struct AxReveal {
+ pub role: String,
+ pub label: String,
+ /// `AXAlternateUIVisible` read back after the action, when the app exposes it.
+ pub visible: Option<bool>,
+ /// Elements in the target's subtree before and after the action. More
+ /// after means hover-only controls appeared.
+ pub elements_before: usize,
+ pub elements_after: usize,
+}
+
+/// Whether an element's advertised actions let the AX route reveal its
+/// hover-only UI.
+pub fn advertises_alternate_ui(actions: &[String]) -> bool {
+ actions.iter().any(|a| a == SHOW_ALTERNATE)
+}
+
+/// Reveal the hover-only UI of the element under a screen point in `pid`'s
+/// window `window_id`, through `AXShowAlternateUI` on the hit element or one of
+/// its ancestors. Restores the previously revealed element first. Returns
+/// `None` when nothing under the point offers alternate UI.
+pub fn reveal_alternate_ui(pid: i32, window_id: u32, x: f64, y: f64) -> Option<AxReveal> {
+ unsafe {
+ let Some(target) = alternate_ui_target(pid, window_id, x, y) else {
+ // Hovering where nothing reveals: the last revealed element loses hover.
+ restore_default_ui(None);
+ return None;
+ };
+ restore_default_ui(Some(target));
+ let elements_before = subtree_size(target, SUBTREE_DEPTH);
+ if perform_action(target, SHOW_ALTERNATE) != kAXErrorSuccess {
+ CFRelease(target as CFTypeRef);
+ return None;
+ }
+ let reveal = AxReveal {
+ role: copy_string_attr(target, "AXRole").unwrap_or_default(),
+ label: first_label(target),
+ visible: copy_bool_attr(target, "AXAlternateUIVisible"),
+ elements_before,
+ elements_after: {
+ std::thread::sleep(std::time::Duration::from_millis(150));
+ subtree_size(target, SUBTREE_DEPTH)
+ },
+ };
+ let mut slot = REVEALED.lock().unwrap_or_else(|e| e.into_inner());
+ match *slot {
+ Some((_, prev)) if CFEqual(prev as CFTypeRef, target as CFTypeRef) != 0 => {
+ CFRelease(target as CFTypeRef);
+ }
+ _ => *slot = Some((pid, target as usize)),
+ }
+ Some(reveal)
+ }
+}
+
+/// The element under the point in window `window_id`, or its nearest
+/// ancestor, that advertises `AXShowAlternateUI`. Owned (+1) by the caller.
+unsafe fn alternate_ui_target(pid: i32, window_id: u32, x: f64, y: f64) -> Option<AXUIElementRef> {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ return None;
+ }
+ AXUIElementSetMessagingTimeout(app, AX_TIMEOUT_S);
+ let mut hit: AXUIElementRef = std::ptr::null_mut();
+ let err = AXUIElementCopyElementAtPosition(app, x as f32, y as f32, &mut hit);
+ CFRelease(app as CFTypeRef);
+ if err != kAXErrorSuccess || hit.is_null() {
+ return None;
+ }
+ // The point must land in the target window, not another window of the app.
+ let in_window = copy_element_attr(hit, "AXWindow").map(|w| {
+ let id = ax_get_window_id(w);
+ CFRelease(w as CFTypeRef);
+ id == Some(window_id)
+ });
+ if in_window != Some(true) {
+ CFRelease(hit as CFTypeRef);
+ return None;
+ }
+ let mut current = Some(hit);
+ for _ in 0..=ANCESTOR_DEPTH {
+ let el = current.take()?;
+ if advertises_alternate_ui(&copy_action_names(el)) {
+ return Some(el);
+ }
+ current = copy_element_attr(el, "AXParent");
+ CFRelease(el as CFTypeRef);
+ }
+ if let Some(el) = current {
+ CFRelease(el as CFTypeRef);
+ }
+ None
+}
+
+/// Put the previously revealed element back to its default UI, unless it is
+/// `keep` (the element about to be revealed again).
+unsafe fn restore_default_ui(keep: Option<AXUIElementRef>) {
+ let mut slot = REVEALED.lock().unwrap_or_else(|e| e.into_inner());
+ let Some((_, prev)) = *slot else { return };
+ if let Some(k) = keep {
+ if CFEqual(prev as CFTypeRef, k as CFTypeRef) != 0 {
+ return;
+ }
+ }
+ perform_action(prev as AXUIElementRef, SHOW_DEFAULT);
+ CFRelease(prev as CFTypeRef);
+ *slot = None;
+}
+
+unsafe fn first_label(element: AXUIElementRef) -> String {
+ for attr in ["AXTitle", "AXDescription", "AXValue"] {
+ if let Some(s) = copy_string_attr(element, attr).filter(|s| !s.trim().is_empty()) {
+ return s;
+ }
+ }
+ // Outline rows carry their label on a child (or grandchild) text element.
+ let mut label = String::new();
+ for child in copy_children(element) {
+ if label.is_empty() {
+ label = text_of(child);
+ if label.is_empty() {
+ for grandchild in copy_children(child) {
+ if label.is_empty() {
+ label = text_of(grandchild);
+ }
+ CFRelease(grandchild as CFTypeRef);
+ }
+ }
+ }
+ CFRelease(child as CFTypeRef);
+ }
+ label
+}
+
+unsafe fn text_of(element: AXUIElementRef) -> String {
+ copy_string_attr(element, "AXValue")
+ .or_else(|| copy_string_attr(element, "AXTitle"))
+ .filter(|s| !s.trim().is_empty())
+ .unwrap_or_default()
+}
+
+/// Number of elements under `element`, `depth` levels deep (bounded).
+unsafe fn subtree_size(element: AXUIElementRef, depth: usize) -> usize {
+ if depth == 0 {
+ return 0;
+ }
+ let mut n = 0;
+ for child in copy_children(element) {
+ n += 1 + subtree_size(child, depth - 1);
+ CFRelease(child as CFTypeRef);
+ }
+ n
+}
+
+impl AxReveal {
+ /// Whether the action made new controls appear (or the app says its
+ /// alternate UI is visible).
+ pub fn revealed_new_controls(&self) -> bool {
+ self.elements_after > self.elements_before || self.visible == Some(true)
+ }
+}
+
+/// Main-process JavaScript that hovers the renderer of the BrowserWindow whose
+/// native window number is `window_id`, at window-local point (`x`, `y`) in
+/// points. Returns a JSON string: `{"ok":true,"x":..,"y":..}` or
+/// `{"ok":false,"reason":..}`.
+pub fn electron_hover_js(window_id: u32, x: f64, y: f64) -> String {
+ format!(
+ r#"(() => {{
+ const req = (typeof require === 'function') ? require : (process.mainModule && process.mainModule.require.bind(process.mainModule));
+ if (!req) return JSON.stringify({{ ok: false, reason: 'no_require' }});
+ const {{ BrowserWindow }} = req('electron');
+ const w = BrowserWindow.getAllWindows().find((w) => {{
+ try {{ return String(w.getMediaSourceId()).split(':')[1] === '{window_id}'; }} catch (e) {{ return false; }}
+ }});
+ if (!w) return JSON.stringify({{ ok: false, reason: 'no_window' }});
+ const b = w.getBounds(), c = w.getContentBounds();
+ const x = Math.round({x} - (c.x - b.x)), y = Math.round({y} - (c.y - b.y));
+ if (x < 0 || y < 0 || x >= c.width || y >= c.height) return JSON.stringify({{ ok: false, reason: 'outside_content' }});
+ w.webContents.sendInputEvent({{ type: 'mouseEnter', x, y }});
+ w.webContents.sendInputEvent({{ type: 'mouseMove', x, y }});
+ return JSON.stringify({{ ok: true, x, y }});
+}})()"#,
+ window_id = window_id,
+ x = x,
+ y = y
+ )
+}
+
+/// Whether the main-process reply says the hover was delivered.
+pub fn electron_hover_delivered(reply: &str) -> bool {
+ let trimmed = reply.trim().trim_matches('"').replace("\\\"", "\"");
+ serde_json::from_str::<serde_json::Value>(&trimmed)
+ .ok()
+ .and_then(|v| v.get("ok").and_then(|ok| ok.as_bool()))
+ .unwrap_or(false)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn a_reveal_counts_only_when_controls_appear_or_the_app_says_so() {
+ let r = |before, after, visible| AxReveal {
+ role: "AXRow".into(),
+ label: String::new(),
+ visible,
+ elements_before: before,
+ elements_after: after,
+ };
+ assert!(r(3, 4, None).revealed_new_controls());
+ assert!(r(3, 3, Some(true)).revealed_new_controls());
+ assert!(!r(3, 3, None).revealed_new_controls());
+ assert!(!r(3, 3, Some(false)).revealed_new_controls());
+ }
+
+ #[test]
+ fn only_show_alternate_ui_qualifies_an_element() {
+ assert!(advertises_alternate_ui(&[
+ "AXShowDefaultUI".into(),
+ "AXShowAlternateUI".into()
+ ]));
+ assert!(!advertises_alternate_ui(&[
+ "AXPress".into(),
+ "AXShowMenu".into()
+ ]));
+ }
+
+ #[test]
+ fn electron_js_targets_the_window_by_media_source_id_and_content_offset() {
+ let js = electron_hover_js(227, 150.0, 178.5);
+ assert!(js.contains("split(':')[1] === '227'"));
+ assert!(js.contains("Math.round(150 - (c.x - b.x))"));
+ assert!(js.contains("Math.round(178.5 - (c.y - b.y))"));
+ assert!(js.contains("type: 'mouseEnter'"));
+ assert!(js.contains("type: 'mouseMove'"));
+ }
+
+ #[test]
+ fn electron_reply_parsing_accepts_raw_and_quoted_json() {
+ assert!(electron_hover_delivered(r#"{"ok":true,"x":1,"y":2}"#));
+ assert!(electron_hover_delivered(
+ r#""{\"ok\":true,\"x\":1,\"y\":2}""#
+ ));
+ assert!(!electron_hover_delivered(
+ r#"{"ok":false,"reason":"no_window"}"#
+ ));
+ assert!(!electron_hover_delivered("undefined"));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
index 123289a225..e3faeaeee7 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
@@ -9,6 +9,7 @@
//! the activity-monitor tickle required for live-input detection.
pub mod ax_actions;
+pub mod hover;
pub mod interactive;
pub mod keyboard;
pub mod mouse;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index a0a79e18a9..996e625db6 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -244,7 +244,7 @@ fn def() -> &'static ToolDef {
- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n\
- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\
\n\
- `button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.".into(),
+ `button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.".into(),
input_schema: serde_json::json!({
"type": "object",
// `pid` is conditionally required — needed for window/element clicks
@@ -261,7 +261,7 @@ fn def() -> &'static ToolDef {
"capture_id": { "type": "string", "description": "Optional capture ID from get_window_state or get_desktop_state. With x,y it is consumed atomically; stale or mismatched captures are refused." },
"x": { "type": "number", "description": "X in screenshot pixels (get_window_state PNG, or get_desktop_state PNG for a desktop target)." },
"y": { "type": "number", "description": "Y in screenshot pixels from the image selected by target." },
- "action": { "type": "string", "description": "AX action." },
+ "action": { "type": "string", "description": "AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer." },
"button": {
"type": "string",
"enum": ["left", "right", "middle"],
@@ -478,6 +478,9 @@ impl Tool for ClickTool {
.opt_f64("y")
.or_else(|| args.opt_i64("y").map(|i| i as f64));
let action = args.str_or("action", "press");
+ if let Err(error) = crate::input::ax_actions::map_action(&action) {
+ return ToolResult::error(format!("click: {error}"));
+ }
// Surface 5: optional `button` arg, default "left" preserves legacy behaviour.
// Pixel path: routes to left/right/middle CGEvent primitives.
// AX path: "right" delegates to AXShowMenu (same surface as right_click);
@@ -1428,7 +1431,7 @@ fn perform_ax_click(
) -> anyhow::Result<(String, bool, bool, bool, bool)> {
let (element_ptr, idx) = element;
let (pid, window_id) = window;
- let ax_action = map_action(action_str);
+ let ax_action = crate::input::ax_actions::map_action(action_str)?;
let element = element_ptr as AXUIElementRef;
// Check the live value immediately before dispatch. Foreground assist can
@@ -1727,18 +1730,6 @@ mod selection_fallback_tests {
}
}
-fn map_action(action: &str) -> &'static str {
- match action.to_lowercase().as_str() {
- "press" | "click" => "AXPress",
- "show_menu" | "right_click" => "AXShowMenu",
- "pick" => "AXPick",
- "confirm" => "AXConfirm",
- "cancel" => "AXCancel",
- "open" => "AXOpen",
- _ => "AXPress",
- }
-}
-
// ── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
index f03c5c0dde..88b3cf0e8e 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
@@ -27,10 +27,12 @@ fn def() -> &'static ToolDef {
name: "move_cursor".into(),
description: "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, \
x/y are window screenshot pixels (as for click) and only the agent cursor moves \
- there. Without a target, moves only the agent cursor overlay, in screen points. \
- Neither sends input: macOS apps track hover and show tooltips from the real pointer, \
- so a background app sees no hover. With scope=desktop, moves the real OS pointer in \
- native get_desktop_state screenshot coordinates.".into(),
+ there; in a background app it also hovers through a route that leaves the real \
+ pointer alone where one exists (AXShowAlternateUI on the element under the point, \
+ or an Electron page) and says so. Without a target, moves only the agent cursor \
+ overlay, in screen points. Other macOS hover and tooltips follow the real pointer, \
+ so most background apps see no hover. With scope=desktop, moves the real OS pointer \
+ in native get_desktop_state screenshot coordinates.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["x", "y"],
@@ -167,14 +169,90 @@ impl MoveCursorTool {
self.state
.cursor_registry
.update_position(&cursor_id, screen_x, screen_y);
- let note = if crate::apps::frontmost_pid() == Some(pid) {
- ""
- } else {
- BACKGROUND_HOVER_NOTE
- };
+ if crate::apps::frontmost_pid() == Some(pid) {
+ return ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \
+ pointer did not move."
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "window",
+ "window_id": wid,
+ "screen_point": [screen_x, screen_y],
+ "hover_delivered": false,
+ "effect": "unverifiable"
+ }));
+ }
+ // Background app: try the hover routes that leave the real pointer alone.
+ let ax = tokio::task::spawn_blocking(move || {
+ crate::input::hover::reveal_alternate_ui(pid, wid, screen_x, screen_y)
+ })
+ .await
+ .ok()
+ .flatten();
+ if let Some(reveal) = ax {
+ let label = if reveal.label.is_empty() {
+ String::new()
+ } else {
+ format!(" \"{}\"", reveal.label)
+ };
+ let revealed = reveal.revealed_new_controls();
+ let outcome = if revealed {
+ format!(
+ "Revealed the hover-only controls of {}{label} through AXShowAlternateUI \
+ ({} new element(s)), without the real pointer, focus or raise. Call \
+ get_window_state to see them; hovering elsewhere hides them again.",
+ reveal.role,
+ reveal.elements_after.saturating_sub(reveal.elements_before)
+ )
+ } else {
+ format!(
+ "Sent AXShowAlternateUI to {}{label}, the background hover route, but no new \
+ control appeared in its accessibility tree, so it may have no hover-only \
+ controls. The real pointer did not move.",
+ reveal.role
+ )
+ };
+ return ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}). {outcome}"
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "window",
+ "window_id": wid,
+ "screen_point": [screen_x, screen_y],
+ "hover_delivered": true,
+ "hover_route": "ax_alternate_ui",
+ "alternate_ui_visible": reveal.visible,
+ "new_elements": reveal.elements_after.saturating_sub(reveal.elements_before),
+ "effect": if revealed { "confirmed" } else { "unverifiable" }
+ }));
+ }
+ if tokio::task::spawn_blocking(move || crate::browser::ElectronJs::is_electron(pid))
+ .await
+ .unwrap_or(false)
+ {
+ let js = crate::input::hover::electron_hover_js(wid, local_x, local_y);
+ if let Ok(reply) = crate::browser::ElectronJs::execute_main(&js, pid).await {
+ if crate::input::hover::electron_hover_delivered(&reply) {
+ return ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}). \
+ Hovered the Electron page there with webContents.sendInputEvent, without \
+ the real pointer, focus or raise. Hover effects in the page update; \
+ native tooltips do not show."
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "window",
+ "window_id": wid,
+ "screen_point": [screen_x, screen_y],
+ "hover_delivered": true,
+ "hover_route": "electron_input",
+ "effect": "unverifiable"
+ }));
+ }
+ }
+ }
ToolResult::text(format!(
"Agent cursor '{cursor_id}' moved over window {wid} at ({x:.0}, {y:.0}); the real \
- pointer did not move.{note}"
+ pointer did not move.{BACKGROUND_HOVER_NOTE}"
))
.with_structured(serde_json::json!({
"scope": "window",
@@ -188,11 +266,13 @@ impl MoveCursorTool {
/// What a window-target move does not do for a background app.
const BACKGROUND_HOVER_NOTE: &str = " The app is in the background, so it sees no hover: \
- macOS apps track hover and show tooltips from the real pointer, and no background route \
- reaches them. A control that only appears on hover, or a tooltip, cannot be revealed \
- without the user's pointer (scope:\"desktop\") or the app in front (bring_to_front); use \
- either only if the user allows it. Look first for the same command elsewhere: a menu \
- item, a context menu, a keyboard shortcut, or help text in get_window_state.";
+ macOS apps track hover and show tooltips from the real pointer, and nothing under this \
+ point offers a background route (no AXShowAlternateUI, not an Electron page). For a web \
+ page in Chrome use browser_pointer hover. A control that only appears on hover, or a \
+ tooltip, cannot be revealed without the user's pointer (scope:\"desktop\") or the app in \
+ front (bring_to_front); use either only if the user allows it. Look first for the same \
+ command elsewhere: a menu item, a context menu, a keyboard shortcut, or help text in \
+ get_window_state.";
/// The app that owns the frontmost ordinary window under a desktop point,
/// when that app is not the active one. The driver's own windows (cursor
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 120a69dd0f..d40d8bbd9f 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -486,14 +486,14 @@
"read_only": false
},
{
- "description": "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\n- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\n`button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.",
+ "description": "Click a target pid. Prefer `element_token` (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use `x, y` only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.\n\n- element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.\n- x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. `modifier` takes cmd/shift/option/ctrl.\n\n`button` defaults to left; on the AX path \"right\" maps to AXShowMenu and \"middle\" falls back to a pixel click at the element center. `action`: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. `from_zoom:true` translates zoom-image pixels after a `zoom` call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use `set_value`.",
"destructive": true,
"idempotent": false,
"input_schema": {
"additionalProperties": false,
"properties": {
"action": {
- "description": "AX action.",
+ "description": "AX action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui. show_alternate_ui reveals hover-only controls on an element that lists it, without the pointer.",
"type": "string"
},
"button": {
@@ -1406,7 +1406,7 @@
"read_only": true
},
{
- "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there. Without a target, moves only the agent cursor overlay, in screen points. Neither sends input: macOS apps track hover and show tooltips from the real pointer, so a background app sees no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",
+ "description": "Move a cursor to (x, y). With target {kind: \"window\", pid, window_id}, x/y are window screenshot pixels (as for click) and only the agent cursor moves there; in a background app it also hovers through a route that leaves the real pointer alone where one exists (AXShowAlternateUI on the element under the point, or an Electron page) and says so. Without a target, moves only the agent cursor overlay, in screen points. Other macOS hover and tooltips follow the real pointer, so most background apps see no hover. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.",
"destructive": false,
"idempotent": true,
"input_schema": {