Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): select macOS collection items reliably

Francesco Bonacci4be90ea11345
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 58848b4059..92ee5434ac 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -220,7 +220,7 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin
- `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle".
- `count` (integer, optional): Click count (pixel path only). Default 1.
- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.
-- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, so its action effect remains unverifiable — confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".
+- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".
- `element_index` (integer, optional): Element index from last get_window_state. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token` of the last get_window_state. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded — re-snapshot in that case.
- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
index 1c8e428bac..73b1dd6bc7 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
@@ -1,6 +1,101 @@
//! AX action dispatch — the preferred click/interaction path for indexed elements.
use crate::ax::bindings::*;
+use core_foundation::base::{CFRelease, CFTypeRef};
+
+const MAX_SELECTION_ANCESTORS: usize = 8;
+
+fn is_selectable_container_role(role: &str) -> bool {
+ matches!(role, "AXRow" | "AXCell" | "AXListItem" | "AXImage")
+}
+
+/// Select the nearest list-like element at or above `element_ptr` and confirm
+/// the write through `AXSelected` read-back.
+///
+/// Finder and other AppKit collection views commonly expose an item's label as
+/// an actionable child (`AXTextField`) while the selectable object is its
+/// parent `AXRow`. Neither object necessarily advertises `AXPress`, and Finder
+/// can return `kAXErrorCannotComplete` for a press on the row. A pointer click
+/// selects that row, so the AX equivalent is to set the row's `AXSelected`
+/// attribute rather than treating the failed press as terminal.
+///
+/// Finder icon views expose each selectable file directly as an `AXImage` with
+/// an `AXSelected` attribute, so that role is included alongside the standard
+/// row-like containers. The fallback remains bounded and requires a successful
+/// `AXSelected=true` read-back, so an arbitrary failed image/button press cannot
+/// become a claimed success.
+pub fn select_nearest_container(element_ptr: usize) -> Option<String> {
+ let mut current = element_ptr as AXUIElementRef;
+ let mut owns_current = false;
+
+ for _ in 0..MAX_SELECTION_ANCESTORS {
+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();
+ if is_selectable_container_role(&role)
+ && unsafe { copy_bool_attr(current, "AXSelected") }.is_some()
+ {
+ let err = unsafe { set_bool_attr_true(current, "AXSelected") };
+ if err == kAXErrorSuccess
+ && unsafe { copy_bool_attr(current, "AXSelected") } == Some(true)
+ {
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ return Some(role);
+ }
+ }
+
+ let parent = unsafe { copy_element_attr(current, "AXParent") };
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ let Some(parent) = parent else {
+ return None;
+ };
+ current = parent;
+ owns_current = true;
+ }
+
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ None
+}
+
+/// Read the selection state of the nearest collection-like element without
+/// mutating it. This is used to verify a coordinate fallback when AppKit
+/// exposes `AXSelected` but refuses to set it directly (notably Finder icon
+/// views).
+pub fn nearest_container_selection_state(element_ptr: usize) -> Option<(String, bool)> {
+ let mut current = element_ptr as AXUIElementRef;
+ let mut owns_current = false;
+
+ for _ in 0..MAX_SELECTION_ANCESTORS {
+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();
+ if is_selectable_container_role(&role) {
+ if let Some(selected) = unsafe { copy_bool_attr(current, "AXSelected") } {
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ return Some((role, selected));
+ }
+ }
+
+ let parent = unsafe { copy_element_attr(current, "AXParent") };
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ let Some(parent) = parent else {
+ return None;
+ };
+ current = parent;
+ owns_current = true;
+ }
+
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ None
+}
fn ensure_ax_enabled(enabled: Option<bool>, action: &str) -> anyhow::Result<()> {
if enabled == Some(false) {
@@ -65,6 +160,16 @@ mod tests {
assert!(ensure_ax_enabled(Some(true), "AXPress").is_ok());
assert!(ensure_ax_enabled(None, "AXPress").is_ok());
}
+
+ #[test]
+ fn selection_fallback_is_limited_to_collection_item_roles() {
+ for role in ["AXRow", "AXCell", "AXListItem", "AXImage"] {
+ assert!(is_selectable_container_role(role), "{role}");
+ }
+ for role in ["AXButton", "AXTextField", "AXWindow", "AXOutline"] {
+ assert!(!is_selectable_container_role(role), "{role}");
+ }
+ }
}
/// Set AXFocused=true on an element (for pre-focusing before key press).
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index 662faf3f91..aec9ce0047 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -58,6 +58,25 @@ enum PixelActivationPolicy {
ForegroundAssist,
}
+#[derive(Clone, Copy, Debug)]
+struct SelectionPixelTarget {
+ screen_x: f64,
+ screen_y: f64,
+ window_x: f64,
+ window_y: f64,
+}
+
+fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -> bool {
+ if has_modifiers {
+ after != before
+ } else {
+ after
+ }
+}
+
+const SELECTION_READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(500);
+const SELECTION_READBACK_POLL: std::time::Duration = std::time::Duration::from_millis(25);
+
fn pixel_activation_policy(
button: &str,
effective_foreground: bool,
@@ -163,7 +182,7 @@ fn def() -> &'static ToolDef {
"delivery_mode": {
"type": "string",
"enum": ["background", "foreground"],
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, so its action effect remains unverifiable — confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."
},
"scope": {
"type": "string",
@@ -457,6 +476,39 @@ impl Tool for ClickTool {
.update_position(&cursor_key, cx, cy);
}
+ // Finder icon/list items can expose a readable AXSelected state
+ // while refusing both AXSelected writes and AXPress. Resolve a
+ // verified coordinate frame only for those collection-like
+ // elements so perform_ax_click can cross that one failed semantic
+ // rung internally and confirm the result by AX read-back.
+ let selection_candidate = if effective_action == "press" {
+ tokio::task::spawn_blocking(move || {
+ crate::input::ax_actions::nearest_container_selection_state(element_ptr)
+ .is_some()
+ })
+ .await
+ .unwrap_or(false)
+ } else {
+ false
+ };
+ let selection_pixel = if selection_candidate {
+ if let Some((cx, cy)) = center {
+ super::px_frame::resolve_or_refuse(wid)
+ .await
+ .ok()
+ .map(|frame| SelectionPixelTarget {
+ screen_x: cx,
+ screen_y: cy,
+ window_x: cx - frame.bounds.x,
+ window_y: cy - frame.bounds.y,
+ })
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Capture prior frontmost, arm the wildcard suppressor in the
// snapshot, then arm a targeted suppressor across the AX action
@@ -479,6 +531,7 @@ impl Tool for ClickTool {
// not the shared "default" one (which would light the wrong cursor
// and stomp default for a non-default session).
let ck = cursor_key.clone();
+ let selection_modifiers = modifiers.clone();
let result = focus_guard::with_focus_suppressed(
if foreground { None } else { Some(pid) },
prior_front,
@@ -498,6 +551,8 @@ impl Tool for ClickTool {
wid,
&action_clone,
&ck,
+ selection_pixel,
+ &selection_modifiers,
)?);
std::thread::sleep(std::time::Duration::from_millis(150));
Ok(())
@@ -508,8 +563,17 @@ impl Tool for ClickTool {
})?;
Ok((outcome, fronted))
} else {
- perform_ax_click(element_ptr, idx, pid, wid, &action_clone, &ck)
- .map(|outcome| (outcome, false))
+ perform_ax_click(
+ element_ptr,
+ idx,
+ pid,
+ wid,
+ &action_clone,
+ &ck,
+ selection_pixel,
+ &selection_modifiers,
+ )
+ .map(|outcome| (outcome, false))
}
})
.await
@@ -521,7 +585,16 @@ impl Tool for ClickTool {
let changes = super::finish_window_observation(snapshot, &args).await;
match result {
- Ok(Ok(((mut msg, needs_webkit_delay, suspected_noop), fronted))) => {
+ Ok(Ok((
+ (
+ mut msg,
+ needs_webkit_delay,
+ suspected_noop,
+ selection_verified,
+ selection_via_pixel,
+ ),
+ fronted,
+ ))) => {
// For text inputs, wait 800ms for WebKit DOM focus to settle
// before returning — matches the Swift reference behaviour.
if needs_webkit_delay {
@@ -538,10 +611,27 @@ impl Tool for ClickTool {
// * unverifiable — dispatched fine, driver just can't confirm;
// the caller verifies via screenshot.
let mut structured = serde_json::json!({
- "path": if fronted { "ax_fg" } else { "ax" },
- "verified": false,
- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },
+ "path": if selection_via_pixel {
+ if fronted { "cgevent_fg" } else { "cgevent" }
+ } else if fronted {
+ "ax_fg"
+ } else {
+ "ax"
+ },
+ "verified": selection_verified,
+ "effect": if selection_verified {
+ "confirmed"
+ } else if suspected_noop {
+ "suspected_noop"
+ } else {
+ "unverifiable"
+ },
});
+ if selection_verified {
+ structured["evidence"] = serde_json::json!([
+ { "kind": "accessibility_readback" }
+ ]);
+ }
if suspected_noop {
structured["escalation"] = serde_json::json!({
"recommended": "px",
@@ -916,7 +1006,8 @@ impl Tool for ClickTool {
// ── AX click implementation (blocking) ───────────────────────────────────────
-/// Returns `(summary_text, needs_webkit_delay, suspected_noop)`.
+/// Returns `(summary_text, needs_webkit_delay, suspected_noop,
+/// selection_verified, selection_via_pixel)`.
///
/// `suspected_noop` is true when the element did not advertise the action we
/// dispatched — AXUIElementPerformAction returns success regardless, so this is
@@ -930,7 +1021,9 @@ fn perform_ax_click(
window_id: u32,
action_str: &str,
cursor_key: &str,
-) -> anyhow::Result<(String, bool, bool)> {
+ selection_pixel: Option<SelectionPixelTarget>,
+ modifiers: &[String],
+) -> anyhow::Result<(String, bool, bool, bool, bool)> {
let ax_action = map_action(action_str);
let element = element_ptr as AXUIElementRef;
@@ -944,14 +1037,100 @@ fn perform_ax_click(
// (AX returns success even when the element doesn't advertise the action).
let advertised = unsafe { copy_action_names(element) };
+ let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();
+ let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();
+
+ // A click on an AppKit collection item is frequently represented by a
+ // label child or row that does not advertise AXPress. Prefer a bounded,
+ // read-back-verified AXSelected write over dispatching a known hollow press
+ // or forcing the caller onto a less stable pixel coordinate.
+ if ax_action == "AXPress" && !advertised.iter().any(|action| action == ax_action) {
+ if modifiers.is_empty() {
+ if let Some(selected_role) =
+ crate::input::ax_actions::select_nearest_container(element_ptr)
+ {
+ return Ok((
+ format!(
+ "✅ Selected nearest {selected_role} for [{idx}] {role} \"{title}\"; \
+ confirmed AXSelected=true."
+ ),
+ false,
+ false,
+ true,
+ false,
+ ));
+ }
+ }
+
+ if let (Some(target), Some((selected_role, before))) = (
+ selection_pixel,
+ crate::input::ax_actions::nearest_container_selection_state(element_ptr),
+ ) {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::mouse::click_at_xy_with_window_local(
+ pid,
+ target.screen_x,
+ target.screen_y,
+ target.window_x,
+ target.window_y,
+ window_id,
+ 1,
+ &modifier_refs,
+ )?;
+ let deadline = std::time::Instant::now() + SELECTION_READBACK_TIMEOUT;
+ loop {
+ if let Some((_, after)) =
+ crate::input::ax_actions::nearest_container_selection_state(element_ptr)
+ {
+ let verified =
+ selection_readback_confirms(before, after, !modifiers.is_empty());
+ if verified {
+ return Ok((
+ format!(
+ "✅ Selected nearest {selected_role} for [{idx}] {role} \
+ \"{title}\"; AX selection write was unavailable, so a \
+ coordinate click was delivered and confirmed by AXSelected \
+ read-back."
+ ),
+ false,
+ false,
+ true,
+ true,
+ ));
+ }
+ }
+ if std::time::Instant::now() >= deadline {
+ break;
+ }
+ std::thread::sleep(SELECTION_READBACK_POLL);
+ }
+ }
+ }
+
let err = unsafe { crate::ax::bindings::perform_action(element, ax_action) };
if err != crate::ax::bindings::kAXErrorSuccess {
+ // Some collection rows claim a click-like action but Finder returns
+ // kAXErrorCannotComplete. Use the same verified selection fallback
+ // before surfacing the dispatch error.
+ if ax_action == "AXPress" {
+ if let Some(selected_role) =
+ crate::input::ax_actions::select_nearest_container(element_ptr)
+ {
+ return Ok((
+ format!(
+ "✅ Selected nearest {selected_role} for [{idx}] {role} \"{title}\" \
+ after AXPress returned {err}; confirmed AXSelected=true."
+ ),
+ false,
+ false,
+ true,
+ false,
+ ));
+ }
+ }
anyhow::bail!("AXUIElementPerformAction({ax_action}) returned {err}");
}
- let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();
- let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();
-
let mut summary = format!("✅ Performed {ax_action} on [{idx}] {role} \"{title}\".");
// AXPopUpButton: list available options, redirect to set_value.
@@ -1033,7 +1212,26 @@ fn perform_ax_click(
let _ = pid;
let _ = window_id; // used by caller context
- Ok((summary, needs_webkit_delay, suspected_noop))
+ Ok((summary, needs_webkit_delay, suspected_noop, false, false))
+}
+
+#[cfg(test)]
+mod selection_fallback_tests {
+ use super::selection_readback_confirms;
+
+ #[test]
+ fn plain_click_requires_selected_readback() {
+ assert!(selection_readback_confirms(false, true, false));
+ assert!(selection_readback_confirms(true, true, false));
+ assert!(!selection_readback_confirms(false, false, false));
+ }
+
+ #[test]
+ fn modified_click_requires_a_selection_transition() {
+ assert!(selection_readback_confirms(false, true, true));
+ assert!(selection_readback_confirms(true, false, true));
+ assert!(!selection_readback_confirms(true, true, true));
+ }
}
fn map_action(action: &str) -> &'static str {