Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): select macOS collection items reliably
Francesco Bonacci4be90ea11345
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 58848b4059..92ee5434ac 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -220,7 +220,7 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin- `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle".- `count` (integer, optional): Click count (pixel path only). Default 1.- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.-- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, so its action effect remains unverifiable — confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".+- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".- `element_index` (integer, optional): Element index from last get_window_state. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token` of the last get_window_state. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded — re-snapshot in that case.- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rsindex 1c8e428bac..73b1dd6bc7 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs@@ -1,6 +1,101 @@//! AX action dispatch — the preferred click/interaction path for indexed elements.use crate::ax::bindings::*;+use core_foundation::base::{CFRelease, CFTypeRef};++const MAX_SELECTION_ANCESTORS: usize = 8;++fn is_selectable_container_role(role: &str) -> bool {+ matches!(role, "AXRow" | "AXCell" | "AXListItem" | "AXImage")+}++/// Select the nearest list-like element at or above `element_ptr` and confirm+/// the write through `AXSelected` read-back.+///+/// Finder and other AppKit collection views commonly expose an item's label as+/// an actionable child (`AXTextField`) while the selectable object is its+/// parent `AXRow`. Neither object necessarily advertises `AXPress`, and Finder+/// can return `kAXErrorCannotComplete` for a press on the row. A pointer click+/// selects that row, so the AX equivalent is to set the row's `AXSelected`+/// attribute rather than treating the failed press as terminal.+///+/// Finder icon views expose each selectable file directly as an `AXImage` with+/// an `AXSelected` attribute, so that role is included alongside the standard+/// row-like containers. The fallback remains bounded and requires a successful+/// `AXSelected=true` read-back, so an arbitrary failed image/button press cannot+/// become a claimed success.+pub fn select_nearest_container(element_ptr: usize) -> Option<String> {+ let mut current = element_ptr as AXUIElementRef;+ let mut owns_current = false;++ for _ in 0..MAX_SELECTION_ANCESTORS {+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();+ if is_selectable_container_role(&role)+ && unsafe { copy_bool_attr(current, "AXSelected") }.is_some()+ {+ let err = unsafe { set_bool_attr_true(current, "AXSelected") };+ if err == kAXErrorSuccess+ && unsafe { copy_bool_attr(current, "AXSelected") } == Some(true)+ {+ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ return Some(role);+ }+ }++ let parent = unsafe { copy_element_attr(current, "AXParent") };+ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ let Some(parent) = parent else {+ return None;+ };+ current = parent;+ owns_current = true;+ }++ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ None+}++/// Read the selection state of the nearest collection-like element without+/// mutating it. This is used to verify a coordinate fallback when AppKit+/// exposes `AXSelected` but refuses to set it directly (notably Finder icon+/// views).+pub fn nearest_container_selection_state(element_ptr: usize) -> Option<(String, bool)> {+ let mut current = element_ptr as AXUIElementRef;+ let mut owns_current = false;++ for _ in 0..MAX_SELECTION_ANCESTORS {+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();+ if is_selectable_container_role(&role) {+ if let Some(selected) = unsafe { copy_bool_attr(current, "AXSelected") } {+ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ return Some((role, selected));+ }+ }++ let parent = unsafe { copy_element_attr(current, "AXParent") };+ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ let Some(parent) = parent else {+ return None;+ };+ current = parent;+ owns_current = true;+ }++ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ None+}fn ensure_ax_enabled(enabled: Option<bool>, action: &str) -> anyhow::Result<()> {if enabled == Some(false) {@@ -65,6 +160,16 @@ mod tests {assert!(ensure_ax_enabled(Some(true), "AXPress").is_ok());assert!(ensure_ax_enabled(None, "AXPress").is_ok());}++ #[test]+ fn selection_fallback_is_limited_to_collection_item_roles() {+ for role in ["AXRow", "AXCell", "AXListItem", "AXImage"] {+ assert!(is_selectable_container_role(role), "{role}");+ }+ for role in ["AXButton", "AXTextField", "AXWindow", "AXOutline"] {+ assert!(!is_selectable_container_role(role), "{role}");+ }+ }}/// Set AXFocused=true on an element (for pre-focusing before key press).diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex 662faf3f91..aec9ce0047 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -58,6 +58,25 @@ enum PixelActivationPolicy {ForegroundAssist,}+#[derive(Clone, Copy, Debug)]+struct SelectionPixelTarget {+ screen_x: f64,+ screen_y: f64,+ window_x: f64,+ window_y: f64,+}++fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -> bool {+ if has_modifiers {+ after != before+ } else {+ after+ }+}++const SELECTION_READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(500);+const SELECTION_READBACK_POLL: std::time::Duration = std::time::Duration::from_millis(25);+fn pixel_activation_policy(button: &str,effective_foreground: bool,@@ -163,7 +182,7 @@ fn def() -> &'static ToolDef {"delivery_mode": {"type": "string","enum": ["background", "foreground"],- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, so its action effect remains unverifiable — confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."},"scope": {"type": "string",@@ -457,6 +476,39 @@ impl Tool for ClickTool {.update_position(&cursor_key, cx, cy);}+ // Finder icon/list items can expose a readable AXSelected state+ // while refusing both AXSelected writes and AXPress. Resolve a+ // verified coordinate frame only for those collection-like+ // elements so perform_ax_click can cross that one failed semantic+ // rung internally and confirm the result by AX read-back.+ let selection_candidate = if effective_action == "press" {+ tokio::task::spawn_blocking(move || {+ crate::input::ax_actions::nearest_container_selection_state(element_ptr)+ .is_some()+ })+ .await+ .unwrap_or(false)+ } else {+ false+ };+ let selection_pixel = if selection_candidate {+ if let Some((cx, cy)) = center {+ super::px_frame::resolve_or_refuse(wid)+ .await+ .ok()+ .map(|frame| SelectionPixelTarget {+ screen_x: cx,+ screen_y: cy,+ window_x: cx - frame.bounds.x,+ window_y: cy - frame.bounds.y,+ })+ } else {+ None+ }+ } else {+ None+ };+// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// Capture prior frontmost, arm the wildcard suppressor in the// snapshot, then arm a targeted suppressor across the AX action@@ -479,6 +531,7 @@ impl Tool for ClickTool {// not the shared "default" one (which would light the wrong cursor// and stomp default for a non-default session).let ck = cursor_key.clone();+ let selection_modifiers = modifiers.clone();let result = focus_guard::with_focus_suppressed(if foreground { None } else { Some(pid) },prior_front,@@ -498,6 +551,8 @@ impl Tool for ClickTool {wid,&action_clone,&ck,+ selection_pixel,+ &selection_modifiers,)?);std::thread::sleep(std::time::Duration::from_millis(150));Ok(())@@ -508,8 +563,17 @@ impl Tool for ClickTool {})?;Ok((outcome, fronted))} else {- perform_ax_click(element_ptr, idx, pid, wid, &action_clone, &ck)- .map(|outcome| (outcome, false))+ perform_ax_click(+ element_ptr,+ idx,+ pid,+ wid,+ &action_clone,+ &ck,+ selection_pixel,+ &selection_modifiers,+ )+ .map(|outcome| (outcome, false))}}).await@@ -521,7 +585,16 @@ impl Tool for ClickTool {let changes = super::finish_window_observation(snapshot, &args).await;match result {- Ok(Ok(((mut msg, needs_webkit_delay, suspected_noop), fronted))) => {+ Ok(Ok((+ (+ mut msg,+ needs_webkit_delay,+ suspected_noop,+ selection_verified,+ selection_via_pixel,+ ),+ fronted,+ ))) => {// For text inputs, wait 800ms for WebKit DOM focus to settle// before returning — matches the Swift reference behaviour.if needs_webkit_delay {@@ -538,10 +611,27 @@ impl Tool for ClickTool {// * unverifiable — dispatched fine, driver just can't confirm;// the caller verifies via screenshot.let mut structured = serde_json::json!({- "path": if fronted { "ax_fg" } else { "ax" },- "verified": false,- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },+ "path": if selection_via_pixel {+ if fronted { "cgevent_fg" } else { "cgevent" }+ } else if fronted {+ "ax_fg"+ } else {+ "ax"+ },+ "verified": selection_verified,+ "effect": if selection_verified {+ "confirmed"+ } else if suspected_noop {+ "suspected_noop"+ } else {+ "unverifiable"+ },});+ if selection_verified {+ structured["evidence"] = serde_json::json!([+ { "kind": "accessibility_readback" }+ ]);+ }if suspected_noop {structured["escalation"] = serde_json::json!({"recommended": "px",@@ -916,7 +1006,8 @@ impl Tool for ClickTool {// ── AX click implementation (blocking) ───────────────────────────────────────-/// Returns `(summary_text, needs_webkit_delay, suspected_noop)`.+/// Returns `(summary_text, needs_webkit_delay, suspected_noop,+/// selection_verified, selection_via_pixel)`.////// `suspected_noop` is true when the element did not advertise the action we/// dispatched — AXUIElementPerformAction returns success regardless, so this is@@ -930,7 +1021,9 @@ fn perform_ax_click(window_id: u32,action_str: &str,cursor_key: &str,-) -> anyhow::Result<(String, bool, bool)> {+ selection_pixel: Option<SelectionPixelTarget>,+ modifiers: &[String],+) -> anyhow::Result<(String, bool, bool, bool, bool)> {let ax_action = map_action(action_str);let element = element_ptr as AXUIElementRef;@@ -944,14 +1037,100 @@ fn perform_ax_click(// (AX returns success even when the element doesn't advertise the action).let advertised = unsafe { copy_action_names(element) };+ let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();+ let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();++ // A click on an AppKit collection item is frequently represented by a+ // label child or row that does not advertise AXPress. Prefer a bounded,+ // read-back-verified AXSelected write over dispatching a known hollow press+ // or forcing the caller onto a less stable pixel coordinate.+ if ax_action == "AXPress" && !advertised.iter().any(|action| action == ax_action) {+ if modifiers.is_empty() {+ if let Some(selected_role) =+ crate::input::ax_actions::select_nearest_container(element_ptr)+ {+ return Ok((+ format!(+ "✅ Selected nearest {selected_role} for [{idx}] {role} \"{title}\"; \+ confirmed AXSelected=true."+ ),+ false,+ false,+ true,+ false,+ ));+ }+ }++ if let (Some(target), Some((selected_role, before))) = (+ selection_pixel,+ crate::input::ax_actions::nearest_container_selection_state(element_ptr),+ ) {+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::mouse::click_at_xy_with_window_local(+ pid,+ target.screen_x,+ target.screen_y,+ target.window_x,+ target.window_y,+ window_id,+ 1,+ &modifier_refs,+ )?;+ let deadline = std::time::Instant::now() + SELECTION_READBACK_TIMEOUT;+ loop {+ if let Some((_, after)) =+ crate::input::ax_actions::nearest_container_selection_state(element_ptr)+ {+ let verified =+ selection_readback_confirms(before, after, !modifiers.is_empty());+ if verified {+ return Ok((+ format!(+ "✅ Selected nearest {selected_role} for [{idx}] {role} \+ \"{title}\"; AX selection write was unavailable, so a \+ coordinate click was delivered and confirmed by AXSelected \+ read-back."+ ),+ false,+ false,+ true,+ true,+ ));+ }+ }+ if std::time::Instant::now() >= deadline {+ break;+ }+ std::thread::sleep(SELECTION_READBACK_POLL);+ }+ }+ }+let err = unsafe { crate::ax::bindings::perform_action(element, ax_action) };if err != crate::ax::bindings::kAXErrorSuccess {+ // Some collection rows claim a click-like action but Finder returns+ // kAXErrorCannotComplete. Use the same verified selection fallback+ // before surfacing the dispatch error.+ if ax_action == "AXPress" {+ if let Some(selected_role) =+ crate::input::ax_actions::select_nearest_container(element_ptr)+ {+ return Ok((+ format!(+ "✅ Selected nearest {selected_role} for [{idx}] {role} \"{title}\" \+ after AXPress returned {err}; confirmed AXSelected=true."+ ),+ false,+ false,+ true,+ false,+ ));+ }+ }anyhow::bail!("AXUIElementPerformAction({ax_action}) returned {err}");}- let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();- let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();-let mut summary = format!("✅ Performed {ax_action} on [{idx}] {role} \"{title}\".");// AXPopUpButton: list available options, redirect to set_value.@@ -1033,7 +1212,26 @@ fn perform_ax_click(let _ = pid;let _ = window_id; // used by caller context- Ok((summary, needs_webkit_delay, suspected_noop))+ Ok((summary, needs_webkit_delay, suspected_noop, false, false))+}++#[cfg(test)]+mod selection_fallback_tests {+ use super::selection_readback_confirms;++ #[test]+ fn plain_click_requires_selected_readback() {+ assert!(selection_readback_confirms(false, true, false));+ assert!(selection_readback_confirms(true, true, false));+ assert!(!selection_readback_confirms(false, false, false));+ }++ #[test]+ fn modified_click_requires_a_selection_transition() {+ assert!(selection_readback_confirms(false, true, true));+ assert!(selection_readback_confirms(true, false, true));+ assert!(!selection_readback_confirms(true, true, true));+ }}fn map_action(action: &str) -> &'static str {