Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): make native selection and editing foreground-safe (#2789)
Francesco Bonacci815013a88c02
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 5bc8134ecc..ce934961d3 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -238,7 +238,7 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin- `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle".- `count` (integer, optional): Click count (pixel path only). Default 1.- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.-- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".+- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require "foreground" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.@@ -366,7 +366,7 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's compPress and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_index it focuses that AX element first. `window_id` only targets; it does not raise.-• `foreground`: guard and briefly front the exact window, send a genuine HID key transition so Chromium content and native menu equivalents both receive it, then restore prior frontmost. Requires window_id (and no element_index).+• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id.A key press is never driver-verifiable → effect:"unverifiable"; confirm via screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex b72279cd73..da2f332218 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -298,8 +298,16 @@ breadth Windows and Linux already exposed (`type_text` / `press_key` /no raise, no focus steal. `"foreground"` briefly fronts the owning app,acts, then restores the prior frontmost — the explicit last resort for asurface that only accepts events while frontmost (the canvas/viewport/game-case below). Element-indexed (AX) actions are inherently background and-hold the no-foreground contract without the flag.+case below). Unmodified element-indexed (AX) actions remain background-capable+and hold the no-foreground contract without the flag.++Modified clicks are the deliberate exception: pass+`delivery_mode:"foreground"` and a concrete `window_id`. macOS applications+can discard PID-routed modifier state after initially publishing a transient+selection, so Cua Driver refuses that background combination. The foreground+rung holds physical HID modifier keys around the click, restores the hardware+cursor and prior foreground app, and confirms list-like selection changes with+a stable AX readback.macOS-specific residuals worth knowing (the rest of the capture/dispatch/addressing params are a shared cross-platform contract — see `SKILL.md` →diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex b3a392bd91..04aa2fb05a 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -94,6 +94,35 @@ that boundary, do not replace Cua's targeted and verified actions with shellscripts that mutate the app UI. A shell is a capability of the calling agent,not of the Cua Driver MCP server; an MCP-only client must not assume one exists.+### Filesystem outcomes and GUI fallbacks++When the requested outcome is a filesystem change and the caller has a+headless filesystem or command capability, keep it on rung 0. Enumerate the+exact source set, decide the destination-conflict policy before changing+anything, perform one batch-safe operation, then independently read back both+source and destination manifests. Do not open a file manager merely to mimic a+move, copy, or rename that the caller can execute and verify directly.++If the caller has no such capability, use the file manager as a GUI fallback+and keep each claim narrow:++1. After entering an inline rename and setting its value, commit it with the+ platform's confirmation key, then take a fresh snapshot. Value readback from+ the inline editor proves only that the editor changed; it does not prove the+ filesystem rename committed.+2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on+ Windows/Linux). On macOS and Windows, issue that modified click with+ `delivery_mode:"foreground"` so the target observes physical modifier state;+ a refused background attempt is an escalation signal, not a failed action to+ trust or repeat. Re-snapshot before the next operation. Continue only when+ every intended item is selected and the prior selection was preserved.+3. After a cross-window drag or paste, verify the destination contains the+ complete expected set and the source reflects copy-versus-move semantics.+ A delivered drag, keypress, or menu action is not file-operation proof.+4. If a destination conflict presents an unrecognized policy or ambiguous+ partial result, stop that GUI path and surface the unresolved state instead+ of retrying blindly.+## The no-foreground principle (window phase)In a strict `window` session, and during the initial window phase of andiff --git a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs b/libs/cua-driver/rust/crates/cua-driver/src/skills.rsindex 0a4aff5f9f..017090405e 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/skills.rs@@ -841,6 +841,27 @@ mod tests {}}+ #[test]+ fn bundled_skill_keeps_filesystem_outcome_ladder_and_gui_proof_boundaries() {+ let crate_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));+ let skill = std::fs::read_to_string(crate_dir.join("../../Skills/cua-driver/SKILL.md"))+ .expect("canonical skill must be readable");++ for required in [+ "headless filesystem or command capability",+ "perform one batch-safe operation",+ "filesystem rename committed",+ "issue that modified click with",+ "every intended item is selected",+ "source reflects copy-versus-move semantics",+ ] {+ assert!(+ skill.contains(required),+ "skill lost required filesystem outcome guidance: {required}"+ );+ }+ }+#[test]fn extract_flat_tarball_v_0_2_20_plus() {// Post-fix shape: one wrapper dir, files directly under it.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rsindex 95f6eefe32..fb46502f1a 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs@@ -452,6 +452,186 @@ fn harness_appkit_text_input() {);}+#[test]+#[ignore]+fn harness_appkit_element_foreground_press_key_commits_edit() {+ run_case(+ native_foreground_case(+ "appkit",+ "press_key_commit",+ Targeting::Ax,+ DriverRoute::MacosCgEventHid,+ ),+ |pid, wid, driver| {+ let first = snapshot_elements(driver, pid, wid);+ let field = element_token_by_id(&first, "txt-input");+ let set = driver.call(+ "set_value",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "element_token": field,+ "value": "inline-cua"+ }),+ );+ assert!(!set.is_error(), "set_value failed: {}", set.text());++ let second = snapshot_elements(driver, pid, wid);+ assert!(+ second.tree_text().contains("inline-cua"),+ "transient edit value was not readable:\n{}",+ second.tree_text()+ );+ assert!(+ second.tree_text().contains("committed=none"),+ "fixture reported a commit before Return:\n{}",+ second.tree_text()+ );+ let field = element_token_by_id(&second, "txt-input");+ let commit = driver.call(+ "press_key",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "element_token": field,+ "key": "return",+ "delivery_mode": "foreground"+ }),+ );+ assert!(+ !commit.is_error(),+ "foreground element press_key failed: {}",+ commit.text()+ );+ assert_eq!(+ commit.action_route(),+ Some("global_input"),+ "foreground press_key used the wrong public route: {}",+ commit.raw+ );+ assert_eq!(+ commit.action_delivery_mode(),+ Some("foreground"),+ "foreground press_key reported the wrong delivery: {}",+ commit.raw+ );+ assert_eq!(+ commit.action_effect(),+ Some("unverifiable"),+ "press_key claimed more truth than the tool itself observed: {}",+ commit.raw+ );++ std::thread::sleep(Duration::from_millis(250));+ let post = snapshot_elements(driver, pid, wid);+ assert!(+ post.tree_text().contains("committed=inline-cua"),+ "Return did not commit the addressed edit:\n{}",+ post.tree_text()+ );+ Observation::delivered_with_fixture_state(Vec::new())+ },+ );+}++#[test]+#[ignore]+fn harness_appkit_modified_click_preserves_selection() {+ run_case(+ native_foreground_case(+ "appkit",+ "modified_click_selection",+ Targeting::Ax,+ DriverRoute::MacosCgEventHid,+ ),+ |pid, wid, driver| {+ let first = snapshot_elements(driver, pid, wid);+ let alpha = element_token_by_id(&first, "selection-alpha");+ let select_alpha = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "element_token": alpha+ }),+ );+ assert!(+ !select_alpha.is_error(),+ "select alpha failed: {}",+ select_alpha.text()+ );+ std::thread::sleep(Duration::from_millis(200));++ let second = snapshot_elements(driver, pid, wid);+ assert!(+ second.tree_text().contains("selection=alpha"),+ "alpha was not selected:\n{}",+ second.tree_text()+ );+ let beta = element_token_by_id(&second, "selection-beta");+ let refused_background = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "element_token": beta,+ "modifier": ["cmd"]+ }),+ );+ assert!(+ refused_background.is_error(),+ "background modified click was not refused: {}",+ refused_background.text()+ );+ assert_eq!(+ refused_background.structured()["code"],+ "background_unavailable",+ "background modified click returned the wrong refusal: {}",+ refused_background.structured()+ );+ std::thread::sleep(Duration::from_millis(300));+ let after_refusal = snapshot_elements(driver, pid, wid);+ assert!(+ after_refusal.tree_text().contains("selection=alpha"),+ "refused modified click changed the prior selection:\n{}",+ after_refusal.tree_text()+ );++ let beta = element_token_by_id(&after_refusal, "selection-beta");+ let add_beta = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": wid,+ "element_token": beta,+ "modifier": ["cmd"],+ "delivery_mode": "foreground"+ }),+ );+ assert!(+ !add_beta.is_error(),+ "foreground modified click failed: {}",+ add_beta.text()+ );+ assert_eq!(+ add_beta.structured()["effect"],+ "confirmed",+ "modified click lacked settled selection proof: {}",+ add_beta.structured()+ );++ std::thread::sleep(Duration::from_millis(250));+ let post = snapshot_elements(driver, pid, wid);+ assert!(+ post.tree_text().contains("selection=alpha,beta"),+ "modified click replaced or lost the prior selection:\n{}",+ post.tree_text()+ );+ Observation::delivered_with_fixture_state(Vec::new())+ },+ );+}+/// type_text: synthesize a keystroke into the NSTextField (CGEvent/// path, distinct from set_value's AX path). Verifies the keyboard/// dispatch chain reaches a backgrounded Cocoa text input.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rsindex 0d838ee4af..f612f27ad0 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs@@ -284,6 +284,117 @@ fn harness_gtk3_query_projects_structured_elements() {);}+#[test]+#[ignore]+fn harness_gtk3_modified_click_preserves_selection() {+ let wayland_refusal = DisplayServer::current() == DisplayServer::Wayland;+ let case = if wayland_refusal {+ native_background_case(+ "gtk3",+ "modified_click_selection",+ Targeting::Ax,+ DriverRoute::LinuxWaylandVirtualPointer,+ )+ .expecting_refusal(vec![RefusalCode::BackgroundUnavailable])+ } else {+ native_foreground_case(+ "gtk3",+ "modified_click_selection",+ Targeting::Ax,+ DriverRoute::LinuxXTest,+ )+ };+ run_case(case, |pid, window_id, driver| {+ let first = snapshot(driver, pid, window_id);+ if wayland_refusal {+ let beta = element_token(&first, "selection-beta");+ let (add_beta, mut passed) = run_with_background_oracles(+ driver,+ TargetWindow {+ pid,+ native_id: window_id,+ },+ |driver| {+ driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": window_id,+ "element_token": beta,+ "delivery_mode": "background",+ "modifier": ["ctrl"]+ }),+ )+ },+ )+ .unwrap_or_else(|error| panic!("Wayland background contract failed: {error}"));+ assert!(+ add_beta.is_error(),+ "native Wayland unexpectedly accepted a modified pointer click: {}",+ add_beta.text()+ );+ assert!(+ add_beta+ .text()+ .contains("modified element clicks are unavailable on native Wayland"),+ "native Wayland returned the wrong refusal: {}",+ add_beta.text()+ );+ std::thread::sleep(Duration::from_millis(250));+ let post = snapshot(driver, pid, window_id);+ assert!(+ post.tree_text().contains("selection=none"),+ "refused modified click mutated the selection:\n{}",+ post.tree_text()+ );+ passed.push(OracleKind::FixtureState);+ return Observation::refused(+ RefusalCode::BackgroundUnavailable,+ passed,+ "native Wayland cannot pair virtual-pointer delivery with modifier state",+ Evidence::default(),+ );+ }++ let alpha = element_token(&first, "selection-alpha");+ let select_alpha = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": window_id,+ "element_token": alpha,+ "delivery_mode": "foreground"+ }),+ );+ assert!(+ !select_alpha.is_error(),+ "select alpha failed: {}",+ select_alpha.text()+ );+ wait_for_state(driver, pid, window_id, "selection=alpha");++ let second = snapshot(driver, pid, window_id);+ let beta = element_token(&second, "selection-beta");+ let add_beta = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64,+ "window_id": window_id,+ "element_token": beta,+ "delivery_mode": "foreground",+ "modifier": ["ctrl"]+ }),+ );+ assert!(+ !add_beta.is_error(),+ "modified click failed: {}",+ add_beta.text()+ );+ wait_for_state(driver, pid, window_id, "selection=alpha,beta");+ Observation::delivered_with_fixture_state(Vec::new())+ });+}+#[test]#[ignore]fn harness_gtk3_stale_element_token_fails_closed() {diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rsindex a7dd659a86..c37857fc00 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs@@ -1593,6 +1593,92 @@ fn harness_wpf_listbox_select() {);}+#[test]+#[ignore]+fn harness_wpf_modified_click_preserves_selection() {+ run_foreground_case(+ "modified_click_selection",+ Targeting::Ax,+ DriverRoute::WindowsSendInput,+ Vec::new(),+ |pid, wid, driver| {+ focus_harness(driver, pid, wid);+ let first = snapshot(driver, pid, wid);+ let apple =+ ax::element_index_by_id(first.text(), "lst-apple").expect("lst-apple missing");+ let select_apple = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64, "window_id": wid, "element_index": apple,+ "snapshot_id": first.snapshot_id(), "delivery_mode": "foreground"+ }),+ );+ assert!(+ !select_apple.is_error(),+ "select apple failed: {}",+ select_apple.text()+ );+ std::thread::sleep(Duration::from_millis(250));++ let second = snapshot(driver, pid, wid);+ let banana =+ ax::element_index_by_id(second.text(), "lst-banana").expect("lst-banana missing");+ let refused_background = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64, "window_id": wid, "element_index": banana,+ "snapshot_id": second.snapshot_id(), "modifier": ["ctrl"]+ }),+ );+ assert!(+ refused_background.is_error(),+ "background modified click was not refused: {}",+ refused_background.text()+ );+ assert_eq!(+ refused_background.structured()["code"],+ "background_unavailable",+ "background modified click returned the wrong refusal: {}",+ refused_background.structured()+ );+ std::thread::sleep(Duration::from_millis(250));+ let after_refusal = snapshot(driver, pid, wid);+ assert!(+ after_refusal.text().contains("selected=apple"),+ "refused modified click changed the prior selection: {}",+ after_refusal.text()+ );+ let banana = ax::element_index_by_id(after_refusal.text(), "lst-banana")+ .expect("lst-banana missing after refusal");+ let add_banana = driver.call(+ "click",+ serde_json::json!({+ "pid": pid as i64, "window_id": wid, "element_index": banana,+ "snapshot_id": after_refusal.snapshot_id(), "delivery_mode": "foreground",+ "modifier": ["ctrl"]+ }),+ );+ assert!(+ !add_banana.is_error(),+ "modified click failed: {}",+ add_banana.text()+ );+ std::thread::sleep(Duration::from_millis(300));+ let post = snapshot(driver, pid, wid);+ assert!(+ post.text().contains("selected=apple,banana"),+ "modified click replaced or lost the prior selection: {}",+ post.text()+ .lines()+ .filter(|line| line.contains("selected="))+ .collect::<Vec<_>>()+ .join(" / ")+ );+ Vec::new()+ },+ );+}+#[test]#[ignore]fn harness_wpf_invoke_menu_live_path() {diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rsindex 9b0b70e3a7..0c4d30d75e 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs@@ -131,6 +131,7 @@ struct Visited<'a> {checked: Option<bool>,enabled: Option<bool>,selected: Option<bool>,+ selectable: bool,actions: Vec<String>,has_editable: bool,has_value: bool,@@ -503,6 +504,10 @@ async fn collect_visited_bounded<'a>(.as_ref().and_then(|state| state.as_ref().ok()).map(|state| state.contains(State::Enabled) && state.contains(State::Sensitive));+ let selectable = state_r+ .as_ref()+ .and_then(|state| state.as_ref().ok())+ .is_some_and(|state| state.contains(State::Selectable));let selected = if role_lower.contains("check") {checked} else if role_lower.contains("radio")@@ -603,6 +608,7 @@ async fn collect_visited_bounded<'a>(checked,enabled,selected,+ selectable,actions,has_editable,has_value,@@ -708,11 +714,11 @@ fn format_value(v: f64) -> String {/// Whether a walked node is exposed as an indexed, usable element.////// Historically this was "the node advertises AT-SPI Actions" (buttons, menu-/// items, links). That silently dropped every **Value**-only widget — GTK-/// `GtkScale` sliders, scroll bars, spin buttons, progress bars expose the-/// `Value` interface but NO `Action`, while some text fields expose-/// `EditableText` without either. Omitting those interfaces makes controls the-/// driver can operate impossible to address by `element_index`.+/// items, links). That silently dropped Value-only widgets, editable text, and+/// selectable list rows. GTK list rows expose Component + Selectable state but+/// no Action even though a coordinate click on their bounds is operable. Keep+/// every such control in the shared index space so physical-input fallbacks can+/// address it without inventing pixels in the caller.////// This predicate is the single source of truth for the element-index space and/// MUST be applied identically in `render` and in every `action_nodes` filter@@ -723,6 +729,7 @@ fn is_indexable(v: &Visited) -> bool {!v.actions.is_empty(),v.has_editable,v.has_value,+ v.selectable,v.enabled,)}@@ -731,9 +738,10 @@ fn is_indexable_capabilities(has_action: bool,has_editable: bool,has_value: bool,+ has_selectable_state: bool,enabled: Option<bool>,) -> bool {- (has_action || has_editable || has_value) && enabled != Some(false)+ (has_action || has_editable || has_value || has_selectable_state) && enabled != Some(false)}// ── Public (sync) entry points ───────────────────────────────────────────────@@ -2486,12 +2494,43 @@ mod coord_tests {};#[test]- fn editable_only_nodes_are_addressable() {- assert!(is_indexable_capabilities(false, true, false, Some(true)));- assert!(is_indexable_capabilities(true, false, false, None));- assert!(is_indexable_capabilities(false, false, true, Some(true)));- assert!(!is_indexable_capabilities(false, false, false, Some(true)));- assert!(!is_indexable_capabilities(true, false, false, Some(false)));+ fn operable_nodes_are_addressable() {+ assert!(is_indexable_capabilities(+ false,+ true,+ false,+ false,+ Some(true)+ ));+ assert!(is_indexable_capabilities(true, false, false, false, None));+ assert!(is_indexable_capabilities(+ false,+ false,+ true,+ false,+ Some(true)+ ));+ assert!(is_indexable_capabilities(+ false,+ false,+ false,+ true,+ Some(true)+ ));+ assert!(!is_indexable_capabilities(+ false,+ false,+ false,+ false,+ Some(true)+ ));+ assert!(!is_indexable_capabilities(+ true,+ false,+ false,+ false,+ Some(false)+ ));}#[test]diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rsindex 3b2497a421..4725ad77cf 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs@@ -1605,8 +1605,23 @@ pub fn send_focus_out(xid: u64) -> Result<()> {/// Send a button click (down + up) to a window at window-local coordinates.pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<()> {+ send_click_with_modifiers(xid, x, y, count, button, &[])+}++/// Send a target-addressed X11 click whose event-state mask carries the named+/// modifiers. Unlike a plain AT-SPI action, this preserves multi-selection+/// semantics without changing the X input focus.+pub fn send_click_with_modifiers(+ xid: u64,+ x: i32,+ y: i32,+ count: usize,+ button: u8,+ modifiers: &[&str],+) -> Result<()> {let (conn, _) = connect_x11_for_input()?;let root = conn.setup().roots[0].root;+ let modifier_state = modifiers_to_state(modifiers);for _ in 0..count {let target = resolve_event_target(&conn, xid, x, y)?;@@ -1622,7 +1637,7 @@ pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<root_y: target.root_y,event_x: target.local_x,event_y: target.local_y,- state: KeyButMask::from(0u16),+ state: modifier_state,same_screen: true,};@@ -1638,7 +1653,9 @@ pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<root_y: target.root_y,event_x: target.local_x,event_y: target.local_y,- state: button_state_mask(button),+ state: KeyButMask::from(+ u16::from(modifier_state) | u16::from(button_state_mask(button)),+ ),same_screen: true,};@@ -2056,22 +2073,67 @@ pub fn send_key_xtest(key: &str, modifiers: &[&str]) -> Result<()> {/// by vision on the whole screen and issues a real screen-absolute pointer/// click. `button` is an X button number (1=left, 2=middle, 3=right).pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> Result<()> {+ send_click_xtest_desktop_with_modifiers(x, y, button, count, &[])+}++/// Real XTest click with physical modifier down/up transitions around the+/// pointer gesture. Used only after the caller selected foreground delivery.+pub fn send_click_xtest_desktop_with_modifiers(+ x: i32,+ y: i32,+ button: u8,+ count: usize,+ modifiers: &[&str],+) -> Result<()> {use x11rb::protocol::xtest::ConnectionExt as _;let (conn, screen_num) = connect_x11_for_input()?;let root = conn.setup().roots[screen_num].root;- // Absolute pointer warp (MotionNotify, detail=0 => absolute) so the button- // events that follow are delivered at (x, y).- conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;- let count = count.max(1);- for click_index in 0..count {- conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;- conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;- if click_index + 1 < count {- // Chromium needs the first pair to reach the server before the- // second pair. A zero-gap batch produces two click events but no- // DOM dblclick event under Xvfb/Openbox.- conn.flush()?;- sleep(Duration::from_millis(DOUBLE_CLICK_DELAY_MS));+ let mapping = conn.get_keyboard_mapping(8, 248)?.reply()?;+ let mut guards = Vec::new();+ let mut modifier_keycodes = Vec::new();+ for modifier in modifiers {+ let keysym = key_name_to_keysym(modifier)?;+ let (keycode, guard) = keycode_for_keysym(&conn, &mapping, keysym, modifier)?;+ if let Some(guard) = guard {+ guards.push(guard);+ }+ modifier_keycodes.push(keycode);+ }+ let mut pressed = Vec::new();+ let gesture_result = (|| -> Result<()> {+ for &keycode in &modifier_keycodes {+ conn.xtest_fake_input(KEY_PRESS_EVENT, keycode, 0, x11rb::NONE, 0, 0, 0)?;+ pressed.push(keycode);+ }+ // Absolute pointer warp (MotionNotify, detail=0 => absolute) so the+ // button events that follow are delivered at (x, y).+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;+ let count = count.max(1);+ for click_index in 0..count {+ conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;+ conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;+ if click_index + 1 < count {+ // Chromium needs the first pair to reach the server before the+ // second pair. A zero-gap batch produces two click events but+ // no DOM dblclick event under Xvfb/Openbox.+ conn.flush()?;+ sleep(Duration::from_millis(DOUBLE_CLICK_DELAY_MS));+ }+ }+ Ok(())+ })();++ // Always attempt to release every modifier that was successfully queued,+ // including when a later pointer request fails. A failed gesture must not+ // leave the desktop with a logically stuck Ctrl/Shift/Alt/Super key.+ let mut release_result: Result<()> = Ok(());+ for &keycode in pressed.iter().rev() {+ if let Err(error) =+ conn.xtest_fake_input(KEY_RELEASE_EVENT, keycode, 0, x11rb::NONE, 0, 0, 0)+ {+ if release_result.is_ok() {+ release_result = Err(error.into());+ }}}conn.flush()?;@@ -2080,6 +2142,9 @@ pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> Res// under Xtigervnc where keyboard events did not (see send_key_xtest), but make// it explicit so the desktop click is reliable across X servers too.let _ = conn.get_input_focus()?.reply();+ drop(guards);+ gesture_result?;+ release_result?;Ok(())}@@ -2595,8 +2660,24 @@ exit 0"#,#[cfg(test)]mod path_tests {use super::{- path_cumulative, point_on_path, real_pointer_capabilities_available, sample_function,+ modifiers_to_state, path_cumulative, point_on_path, real_pointer_capabilities_available,+ sample_function,};+ use x11rb::protocol::xproto::KeyButMask;++ #[test]+ fn click_modifier_state_combines_canonical_names_and_aliases() {+ let state = modifiers_to_state(&["ctrl", "shift", "meta"]);+ assert!(state.contains(KeyButMask::CONTROL));+ assert!(state.contains(KeyButMask::SHIFT));+ assert!(state.contains(KeyButMask::MOD4));+ assert!(!state.contains(KeyButMask::MOD1));++ assert_eq!(+ modifiers_to_state(&["control", "alt"]),+ KeyButMask::from(u16::from(KeyButMask::CONTROL) | u16::from(KeyButMask::MOD1))+ );+ }#[test]fn real_pointer_capabilities_require_uinput_access() {diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex 12ea98659d..636c4b4baa 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -1559,6 +1559,17 @@ fn x11_pixel_click_no_focus_steal(crate::input::send_click(xid, lx, ly, count, button)}+fn x11_pixel_click_no_focus_steal_modifiers(+ xid: u64,+ lx: i32,+ ly: i32,+ button: u8,+ count: usize,+ modifiers: &[&str],+) -> anyhow::Result<()> {+ crate::input::send_click_with_modifiers(xid, lx, ly, count, button, modifiers)+}+fn mouse_button_name(button: u8) -> &'static str {match button {3 => "right",@@ -1854,7 +1865,9 @@ impl Tool for ClickTool {field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release \with the matching button code. Native Wayland: only left-button is supported \via the virtual-pointer protocol — right/middle return an error rather than \- silently degrading to left.".into(),+ silently degrading to left. `modifier` holds ctrl/shift/alt/super for the \+ click on X11. Native Wayland refuses modified pointer clicks until its input \+ protocol can carry keyboard modifier state.".into(),input_schema: json!({// `pid` is conditionally required (validated in code: needed for// window/element clicks, omitted for windowless scope="desktop"),@@ -1875,6 +1888,7 @@ impl Tool for ClickTool {// back-compat prose the click button-schema test asserts on."button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},"count":{"type":"integer"},+ "modifier": cua_driver_core::tool_schema::modifier_schema(),"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()@@ -1886,6 +1900,7 @@ impl Tool for ClickTool {async fn invoke(&self, args: Value) -> ToolResult {let cursor_id = resolve_cursor_key(&args);+ let modifiers: Vec<String> = args.str_array("modifier");// ── Window-less screen-absolute branch (capture_scope="desktop") ──────// x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,@@ -1927,9 +1942,22 @@ impl Tool for ClickTool {overlay_glide_to_for(&cursor_id, sx as f64, sy as f64).await;let r = tokio::task::spawn_blocking(move || {if crate::wayland::wayland_input_enabled() {+ if !modifiers.is_empty() {+ anyhow::bail!(+ "modified desktop clicks are unavailable on native Wayland: \+ the virtual-pointer route cannot carry keyboard modifier state"+ );+ }crate::wayland::click_desktop(sx, sy, n as u32, button)} else {- crate::input::send_click_xtest_desktop(sx, sy, button, n)+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::send_click_xtest_desktop_with_modifiers(+ sx,+ sy,+ button,+ n,+ &modifier_refs,+ )}}).await;@@ -2033,19 +2061,22 @@ impl Tool for ClickTool {// Chromium can execute a genuine AT-SPI action without focus. Try// that route before applying its background synthetic-input gate.- let ax_result =- tokio::task::spawn_blocking(move || crate::atspi::perform_action(pid, idx)).await;- if let Ok(Ok((_action, suspected_noop))) = ax_result {- let mut structured = json!({- "path": "ax",- "verified": false,- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },- });- if suspected_noop {- structured["escalation"] = non_ax_escalation();+ if modifiers.is_empty() {+ let ax_result =+ tokio::task::spawn_blocking(move || crate::atspi::perform_action(pid, idx))+ .await;+ if let Ok(Ok((_action, suspected_noop))) = ax_result {+ let mut structured = json!({+ "path": "ax",+ "verified": false,+ "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },+ });+ if suspected_noop {+ structured["escalation"] = non_ax_escalation();+ }+ return ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))+ .with_structured(structured);}- return ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))- .with_structured(structured);}if let Some(refusal) = unavailable_chromium_background(pid, delivery) {return refusal;@@ -2055,7 +2086,38 @@ impl Tool for ClickTool {// X11 event for toolkits that accept it.let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {let (xid2, lx, ly) = resolve_element_local_coords(pid, idx, xid_hint)?;- crate::input::send_click(xid2, lx as i32, ly as i32, count, button)+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {+ anyhow::bail!(+ "modified element clicks are unavailable on native Wayland: \+ the pointer route cannot carry keyboard modifier state"+ );+ }+ // An explicit X11 foreground request needs the real XTest path+ // even for a plain click. Some native widgets (notably GTK+ // selectable rows) expose bounds but no AT-SPI Action and+ // ignore a targeted XSendEvent; limiting XTest to modified+ // clicks made those rows addressable but not selectable.+ if delivery.is_foreground() && !crate::wayland::wayland_input_enabled() {+ crate::input::with_x11_foreground(xid2, 80, || {+ crate::input::send_click_xtest_desktop_with_modifiers(+ sx.round() as i32,+ sy.round() as i32,+ button,+ count,+ &modifier_refs,+ )+ })+ } else {+ crate::input::send_click_with_modifiers(+ xid2,+ lx as i32,+ ly as i32,+ count,+ button,+ &modifier_refs,+ )+ }}).await;return match result {@@ -2143,11 +2205,18 @@ impl Tool for ClickTool {let (xi, yi) = (x as i32, y as i32);let (output_x, output_y) = wayland_output_point.unwrap_or((xi, yi));let cursor_id_for_task = cursor_id.clone();+ let modifiers_for_task = modifiers.clone();// delivery_mode: background (default) = no-focus-steal injection;// foreground = activate the target window (EWMH) first, then inject,// then restore prior active. Mirrors macOS/Windows.let result = tokio::task::spawn_blocking(move || -> anyhow::Result<&'static str> {if crate::wayland::wayland_input_enabled() {+ if !modifiers_for_task.is_empty() {+ anyhow::bail!(+ "modified coordinate clicks are unavailable on native Wayland: \+ the pointer route cannot carry keyboard modifier state"+ );+ }// Vision/pixel click on native Wayland. Mutter drops synthetic// virtual-pointer events (the `wayland::click` warp doesn't land),// so for a plain left single click resolve the screen pixel to the@@ -2183,7 +2252,7 @@ impl Tool for ClickTool {// Foreground skips the AT-SPI shortcut and does a real activated pixel// click (the agent's escalation when background didn't land).let inject = |fg: bool| -> anyhow::Result<&'static str> {- if !fg && button == 1 && count == 1 {+ if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {if let Ok(Some(_)) = crate::atspi::perform_action_at_point(pid, xi, yi) {return Ok("x11_atspi");}@@ -2197,16 +2266,39 @@ impl Tool for ClickTool {// widget. XTest is accepted as real input and gives the// widget keyboard focus, so a following type lands.if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64) {- crate::input::send_click_xtest_desktop(+ let modifier_refs: Vec<&str> =+ modifiers_for_task.iter().map(String::as_str).collect();+ crate::input::send_click_xtest_desktop_with_modifiers(sx.round() as i32,sy.round() as i32,button,count,+ &modifier_refs,)?;return Ok("x11_xtest_fg");}}- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, button, count)?;+ if modifiers_for_task.is_empty() {+ x11_pixel_click_no_focus_steal(+ &cursor_id_for_task,+ xid,+ xi,+ yi,+ button,+ count,+ )?;+ } else {+ let modifier_refs: Vec<&str> =+ modifiers_for_task.iter().map(String::as_str).collect();+ x11_pixel_click_no_focus_steal_modifiers(+ xid,+ xi,+ yi,+ button,+ count,+ &modifier_refs,+ )?;+ }Ok(if fg { "x11_pixel_fg" } else { "x11_pixel" })};if delivery.is_foreground() {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rsindex 73b1dd6bc7..724217369f 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs@@ -1,7 +1,7 @@//! AX action dispatch — the preferred click/interaction path for indexed elements.use crate::ax::bindings::*;-use core_foundation::base::{CFRelease, CFTypeRef};+use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};const MAX_SELECTION_ANCESTORS: usize = 8;@@ -97,6 +97,106 @@ pub fn nearest_container_selection_state(element_ptr: usize) -> Option<(String,None}+/// A retained selection context for proving the settled result of a modified+/// pointer click.+///+/// Reading only the target's `AXSelected` bit is insufficient for multi-select:+/// AppKit can expose a transient target transition while it is still resolving+/// the synthetic gesture, and a modifier-less outcome can replace the prior+/// selection with the target. Keep the target and every selected sibling alive+/// across delivery so callers can require both the intended target transition+/// and preservation of the pre-existing selection.+pub struct SelectionReadback {+ role: String,+ target: AXUIElementRef,+ peer_model_observed: bool,+ previously_selected_peers: Vec<AXUIElementRef>,+}++impl SelectionReadback {+ pub fn role(&self) -> &str {+ &self.role+ }++ pub fn observe(&self) -> Option<(bool, bool)> {+ let target_selected = unsafe { copy_bool_attr(self.target, "AXSelected") }?;+ let peers_preserved = self.peer_model_observed+ && self+ .previously_selected_peers+ .iter()+ .all(|&peer| unsafe { copy_bool_attr(peer, "AXSelected") } == Some(true));+ Some((target_selected, peers_preserved))+ }+}++impl Drop for SelectionReadback {+ fn drop(&mut self) {+ unsafe {+ CFRelease(self.target as CFTypeRef);+ for peer in self.previously_selected_peers.drain(..) {+ CFRelease(peer as CFTypeRef);+ }+ }+ }+}++/// Capture the nearest selectable container and its currently-selected peers.+/// Returns `None` when the platform does not expose a readable selection model;+/// callers must then leave the action unverifiable instead of inventing proof.+pub fn capture_nearest_container_selection(element_ptr: usize) -> Option<SelectionReadback> {+ let mut current = element_ptr as AXUIElementRef;+ let mut owns_current = false;++ for _ in 0..MAX_SELECTION_ANCESTORS {+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();+ if is_selectable_container_role(&role)+ && unsafe { copy_bool_attr(current, "AXSelected") }.is_some()+ {+ if !owns_current {+ unsafe { CFRetain(current as CFTypeRef) };+ }+ let target = current;+ let parent = unsafe { copy_element_attr(target, "AXParent") };+ let mut peers = Vec::new();+ let mut peer_model_observed = false;+ if let Some(parent) = parent {+ for child in unsafe { copy_children(parent) } {+ let is_target =+ unsafe { CFEqual(child as CFTypeRef, target as CFTypeRef) != 0 };+ peer_model_observed |= is_target;+ if !is_target && unsafe { copy_bool_attr(child, "AXSelected") } == Some(true) {+ peers.push(child);+ } else {+ unsafe { CFRelease(child as CFTypeRef) };+ }+ }+ unsafe { CFRelease(parent as CFTypeRef) };+ }+ return Some(SelectionReadback {+ role,+ target,+ peer_model_observed,+ previously_selected_peers: peers,+ });+ }++ let parent = unsafe { copy_element_attr(current, "AXParent") };+ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ let Some(parent) = parent else {+ return None;+ };+ current = parent;+ owns_current = true;+ }++ if owns_current {+ unsafe { CFRelease(current as CFTypeRef) };+ }+ None+}+fn ensure_ax_enabled(enabled: Option<bool>, action: &str) -> anyhow::Result<()> {if enabled == Some(false) {anyhow::bail!(diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rsindex 857ef2400f..2b9baf8e55 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs@@ -216,6 +216,53 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {result}+/// Hold physical modifier keys on the global HID queue around one pointer+/// gesture, then release them in reverse order even when the gesture fails.+///+/// This is only safe inside an exact-window foreground activation guard. The+/// caller receives the live modifier flags so every mouse event in the gesture+/// carries the same state as the physical key transitions.+pub(super) fn with_global_modifier_keys<T>(+ modifiers: &[&str],+ gesture: impl FnOnce(CGEventFlags) -> anyhow::Result<T>,+) -> anyhow::Result<T> {+ use core_graphics::event::CGEventTapLocation;++ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;+ let mut active_flags = CGEventFlags::CGEventFlagNull;+ let mut pressed = Vec::new();++ for modifier in modifiers {+ let Some((key_code, flag)) = modifier_key_code_and_flag(modifier) else {+ continue;+ };+ if pressed+ .iter()+ .any(|(pressed_code, _)| *pressed_code == key_code)+ {+ continue;+ }+ active_flags |= flag;+ if let Err(error) = post_global_key(+ &source,+ key_code,+ true,+ active_flags,+ CGEventTapLocation::HID,+ ) {+ release_global_modifiers(&source, &pressed, active_flags, CGEventTapLocation::HID);+ return Err(error);+ }+ pressed.push((key_code, flag));+ std::thread::sleep(std::time::Duration::from_millis(8));+ }++ let result = gesture(active_flags);+ release_global_modifiers(&source, &pressed, active_flags, CGEventTapLocation::HID);+ result+}+fn post_global_key(source: &CGEventSource,key_code: u16,@@ -257,6 +304,57 @@ fn modifier_key_code_and_flag(modifier: &str) -> Option<(u16, CGEventFlags)> {}}+/// Hold PID-routed modifier keys around one pointer gesture and release them in+/// reverse order even when the gesture fails.+///+/// Mouse-event flag bits alone are not a sufficient physical-modifier model for+/// every AppKit host (Finder's collection views are a notable example). Pairing+/// the flagged mouse events with real modifier down/up transitions gives the+/// target the same ordered state it receives for a keyboard chord without+/// putting those transitions on the global HID queue.+pub(super) fn with_pid_modifier_keys<T>(+ pid: i32,+ modifiers: &[&str],+ gesture: impl FnOnce() -> anyhow::Result<T>,+) -> anyhow::Result<T> {+ let mut active_flags = CGEventFlags::CGEventFlagNull;+ let mut pressed = Vec::new();+ for modifier in modifiers {+ let Some((key_code, flag)) = modifier_key_code_and_flag(modifier) else {+ continue;+ };+ if pressed+ .iter()+ .any(|(pressed_code, _)| *pressed_code == key_code)+ {+ continue;+ }+ active_flags |= flag;+ if let Err(error) = post_key(pid, key_code, true, active_flags) {+ release_pid_modifiers(pid, &pressed, active_flags);+ return Err(error);+ }+ pressed.push((key_code, flag));+ std::thread::sleep(std::time::Duration::from_millis(8));+ }++ let result = gesture();+ release_pid_modifiers(pid, &pressed, active_flags);+ result+}++fn release_pid_modifiers(+ pid: i32,+ pressed: &[(u16, CGEventFlags)],+ mut active_flags: CGEventFlags,+) {+ for &(key_code, flag) in pressed.iter().rev() {+ active_flags.remove(flag);+ let _ = post_key(pid, key_code, false, active_flags);+ std::thread::sleep(std::time::Duration::from_millis(8));+ }+}+/// Type Unicode text into the frontmost application through the global HID/// queue. This is the desktop-scope counterpart to PID-routed `type_text` and/// mirrors computer-server's frontmost pynput typing behavior.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rsindex 6a210126e0..f3a2e3f2aa 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs@@ -43,10 +43,54 @@ pub fn click_at_xy(/// (it lands on whatever is visually on top at the point) — exactly the/// foreground, vision-driven model that complements the background contract.pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhow::Result<()> {+ click_at_xy_desktop_inner(x, y, count, button, &[], false)+}++/// Global HID click with physical modifier transitions. The caller must guard+/// the exact foreground window because this transport has no pid addressing.+pub fn click_at_xy_desktop_with_modifiers(+ x: f64,+ y: f64,+ count: usize,+ button: &str,+ modifiers: &[&str],+) -> anyhow::Result<()> {+ click_at_xy_desktop_inner(x, y, count, button, modifiers, false)+}++/// Global HID modifier click that restores the user's hardware cursor after+/// the event pair has been queued.+pub fn click_at_xy_desktop_with_modifiers_preserving_cursor(+ x: f64,+ y: f64,+ count: usize,+ button: &str,+ modifiers: &[&str],+) -> anyhow::Result<()> {+ click_at_xy_desktop_inner(x, y, count, button, modifiers, true)+}++fn click_at_xy_desktop_inner(+ x: f64,+ y: f64,+ count: usize,+ button: &str,+ modifiers: &[&str],+ preserve_cursor: bool,+) -> anyhow::Result<()> {use core_graphics::display::CGDisplay;use core_graphics::event::CGEventTapLocation;let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState).map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;+ let prior = if preserve_cursor {+ Some(+ CGEvent::new(source.clone())+ .map_err(|_| anyhow::anyhow!("CGEvent::new failed"))?+ .location(),+ )+ } else {+ None+ };let point = CGPoint::new(x, y);let (down_ty, up_ty, btn) = match button {"right" => (@@ -75,16 +119,40 @@ pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhow// warped point, not the pre-warp one.unsafe { CGAssociateMouseAndMouseCursorPosition(true) };std::thread::sleep(std::time::Duration::from_millis(40));- for _ in 0..count.max(1) {- let down = CGEvent::new_mouse_event(source.clone(), down_ty, point, btn)- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;- down.post(CGEventTapLocation::HID);- std::thread::sleep(std::time::Duration::from_millis(20));- let up = CGEvent::new_mouse_event(source.clone(), up_ty, point, btn)- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;- up.post(CGEventTapLocation::HID);+ let result = super::keyboard::with_global_modifier_keys(modifiers, |flags| {+ for pair_index in 0..count.max(1) {+ let down = CGEvent::new_mouse_event(source.clone(), down_ty, point, btn)+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;+ down.set_flags(flags);+ down.set_integer_value_field(+ core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE,+ (pair_index + 1) as i64,+ );+ down.post(CGEventTapLocation::HID);+ std::thread::sleep(std::time::Duration::from_millis(28));+ let up = CGEvent::new_mouse_event(source.clone(), up_ty, point, btn)+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;+ up.set_flags(flags);+ up.set_integer_value_field(+ core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE,+ (pair_index + 1) as i64,+ );+ up.post(CGEventTapLocation::HID);+ if count > 1 {+ std::thread::sleep(std::time::Duration::from_millis(80));+ }+ }+ Ok(())+ });++ // Let AppKit consume the up event before the pointer is restored. The+ // exact foreground guard remains active around this entire helper.+ std::thread::sleep(std::time::Duration::from_millis(40));+ if let Some(prior) = prior {+ let _ = CGDisplay::warp_mouse_cursor_position(prior);+ unsafe { CGAssociateMouseAndMouseCursorPosition(true) };}- Ok(())+ result}/// Move the real hardware cursor to a logical desktop point.@@ -145,17 +213,7 @@ pub fn scroll_wheel_desktop(/// used only by approved, bounded setup flows after an accessibility element/// has proven the exact target point and window.pub fn click_at_xy_desktop_preserving_cursor(x: f64, y: f64) -> anyhow::Result<()> {- use core_graphics::display::CGDisplay;-- let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)- .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;- let prior = CGEvent::new(source)- .map_err(|_| anyhow::anyhow!("CGEvent::new failed"))?- .location();- let result = click_at_xy_desktop(x, y, 1, "left");- let _ = CGDisplay::warp_mouse_cursor_position(prior);- unsafe { CGAssociateMouseAndMouseCursorPosition(true) };- result+ click_at_xy_desktop_inner(x, y, 1, "left", &[], true)}extern "C" {@@ -219,70 +277,71 @@ fn click_at_xy_inner(.subsec_nanos() as i64});- // Prime the target window's cursor-tracking state with a leading mouseMoved- // so an AppKit NSButton / NSView hit-tests the down at the right point- // (Swift recipe Step 3). Without it the synthetic mouseDown on a- // backgrounded AppKit control is silently ignored.- post_mouse_moved_primer(pid, &source, point, window_local, wid, click_group_id);- std::thread::sleep(std::time::Duration::from_millis(12));-- for pair_index in 0..count {- let click_state = (pair_index + 1) as i64;-- let down = CGEvent::new_mouse_event(- source.clone(),- CGEventType::LeftMouseDown,- point,- CGMouseButton::Left,- )- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;- if flags != CGEventFlags::CGEventFlagNull {- down.set_flags(flags);- }-- post_mouse_event(- pid,- &down,- window_local,- wid,- click_group_id,- click_state,- 0,- 3,- );- // 28 ms down→up gap: an NSButton's mouseDown enters a modal tracking- // loop that polls for the matching mouseUp; too tight a gap can race the- // loop's first poll and the click is dropped. 16 ms was under that- // threshold for some controls.- std::thread::sleep(std::time::Duration::from_millis(28));-- let up = CGEvent::new_mouse_event(- source.clone(),- CGEventType::LeftMouseUp,- point,- CGMouseButton::Left,- )- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;- if flags != CGEventFlags::CGEventFlagNull {- up.set_flags(flags);- }+ super::keyboard::with_pid_modifier_keys(pid, modifiers, || {+ // Prime the target window's cursor-tracking state with a leading+ // mouseMoved so an AppKit NSButton / NSView hit-tests the down at the+ // right point. Without it the synthetic mouseDown on a backgrounded+ // AppKit control is silently ignored.+ post_mouse_moved_primer(pid, &source, point, window_local, wid, click_group_id);+ std::thread::sleep(std::time::Duration::from_millis(12));++ for pair_index in 0..count {+ let click_state = (pair_index + 1) as i64;++ let down = CGEvent::new_mouse_event(+ source.clone(),+ CGEventType::LeftMouseDown,+ point,+ CGMouseButton::Left,+ )+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;+ if flags != CGEventFlags::CGEventFlagNull {+ down.set_flags(flags);+ }++ post_mouse_event(+ pid,+ &down,+ window_local,+ wid,+ click_group_id,+ click_state,+ 0,+ 3,+ );+ // 28 ms down→up gap: an NSButton's mouseDown enters a modal+ // tracking loop that polls for the matching mouseUp; too tight a+ // gap can race the loop's first poll and the click is dropped.+ std::thread::sleep(std::time::Duration::from_millis(28));++ let up = CGEvent::new_mouse_event(+ source.clone(),+ CGEventType::LeftMouseUp,+ point,+ CGMouseButton::Left,+ )+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;+ if flags != CGEventFlags::CGEventFlagNull {+ up.set_flags(flags);+ }++ post_mouse_event(+ pid,+ &up,+ window_local,+ wid,+ click_group_id,+ click_state,+ 0,+ 3,+ );- post_mouse_event(- pid,- &up,- window_local,- wid,- click_group_id,- click_state,- 0,- 3,- );-- if count > 1 {- std::thread::sleep(std::time::Duration::from_millis(80));+ if count > 1 {+ std::thread::sleep(std::time::Duration::from_millis(80));+ }}- }- Ok(())+ Ok(())+ })}/// Prepare a raw background pixel click by making the target AppKit-activediff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex 17dd6fcff3..67f038d4dd 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -66,9 +66,14 @@ struct SelectionPixelTarget {window_y: f64,}-fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -> bool {+fn selection_readback_confirms(+ before: bool,+ after: bool,+ has_modifiers: bool,+ prior_selected_peers_preserved: bool,+) -> bool {if has_modifiers {- after != before+ after != before && prior_selected_peers_preserved} else {after}@@ -76,6 +81,8 @@ fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -const SELECTION_READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(500);const SELECTION_READBACK_POLL: std::time::Duration = std::time::Duration::from_millis(25);+const SELECTION_READBACK_SETTLE: std::time::Duration = std::time::Duration::from_millis(200);+const SELECTION_READBACK_STABILITY: std::time::Duration = std::time::Duration::from_millis(250);fn pixel_activation_policy(button: &str,@@ -183,7 +190,7 @@ fn def() -> &'static ToolDef {"delivery_mode": {"type": "string","enum": ["background", "foreground"],- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require \"foreground\" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."},"scope": {"type": "string",@@ -294,12 +301,21 @@ impl Tool for ClickTool {.update_position(&cursor_key, sx, sy);let btn = button.clone();+ let desktop_modifiers: Vec<String> = args.str_array("modifier");let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {// Desktop scope is explicitly foreground and vision-driven: post// at the global HID tap so WindowServer delivers to the window// actually visible at this point. PID-posting here would silently// turn the foreground contract back into background delivery.- crate::input::mouse::click_at_xy_desktop(sx, sy, count, &btn)+ let modifier_refs: Vec<&str> =+ desktop_modifiers.iter().map(String::as_str).collect();+ crate::input::mouse::click_at_xy_desktop_with_modifiers(+ sx,+ sy,+ count,+ &btn,+ &modifier_refs,+ )}).await;let button_label = match button.as_str() {@@ -386,6 +402,25 @@ impl Tool for ClickTool {let debug_image_out = args.opt_str("debug_image_out");let modifiers: Vec<String> = args.str_array("modifier");+ // PID-routed key transitions can look correct for one AX poll and then+ // collapse to a plain click once AppKit resolves the gesture. Refuse+ // that false-success path. The explicit foreground rung uses physical+ // HID modifier transitions under an exact-window activation guard.+ if !modifiers.is_empty() && (!delivery_mode.is_foreground() || window_id.is_none()) {+ return ToolResult::error(+ "click modifiers require delivery_mode:\"foreground\" and window_id on macOS; \+ background PID-routed events cannot preserve live modifier-key state",+ )+ .with_structured(serde_json::json!({+ "code": "background_unavailable",+ "effect": "refused",+ "escalation": {+ "recommended": "foreground",+ "reason": "macOS modifier clicks require exact-window HID delivery so the target observes live modifier state"+ }+ }));+ }+if let (Some(idx), Some(wid)) = (element_index, window_id) {// ── AX element path ────────────────────────────────────────────// Retain the element out of the cache so it can't be freed by a@@ -447,9 +482,22 @@ impl Tool for ClickTool {.update_position(&cursor_key, cx, cy);let mods_owned = modifiers.clone();+ let foreground = delivery_mode.is_foreground();let result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();- crate::input::mouse::middle_click_at_xy(pid, cx, cy, &m)+ if foreground && !m.is_empty() {+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ || {+ crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(+ cx, cy, 1, "middle", &m,+ )+ },+ )+ } else {+ crate::input::mouse::middle_click_at_xy(pid, cx, cy, &m)+ }}).await;return match result {@@ -542,24 +590,36 @@ impl Tool for ClickTool {tokio::task::spawn_blocking(move || {if foreground {let mut outcome = None;- let fronted = crate::input::skylight::with_foreground_assist(- pid as libc::pid_t,- wid,- || {- outcome = Some(perform_ax_click(- element_ptr,- idx,- pid,- wid,- &action_clone,- &ck,- selection_pixel,- &selection_modifiers,- )?);- std::thread::sleep(std::time::Duration::from_millis(150));- Ok(())- },- )?;+ let has_modifiers = !selection_modifiers.is_empty();+ let action = || {+ outcome = Some(perform_ax_click(+ element_ptr,+ idx,+ pid,+ wid,+ &action_clone,+ &ck,+ selection_pixel,+ &selection_modifiers,+ foreground,+ )?);+ std::thread::sleep(std::time::Duration::from_millis(150));+ Ok(())+ };+ let fronted = if has_modifiers {+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ action,+ )?;+ true+ } else {+ crate::input::skylight::with_foreground_assist(+ pid as libc::pid_t,+ wid,+ action,+ )?+ };let outcome = outcome.ok_or_else(|| {anyhow::anyhow!("foreground AX click did not execute")})?;@@ -574,6 +634,7 @@ impl Tool for ClickTool {&ck,selection_pixel,&selection_modifiers,+ false,).map(|outcome| (outcome, false))}@@ -885,8 +946,18 @@ impl Tool for ClickTool {"click.pixel",|| async move {tokio::task::spawn_blocking(move || {+ let has_modifiers = !mods_owned.is_empty();let do_click = move || -> anyhow::Result<()> {let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();+ if fg && !m.is_empty() {+ return crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(+ screen_x,+ screen_y,+ count,+ &button_kind,+ &m,+ );+ }match button_kind.as_str() {"right" => {if let Some(wid) = window_id {@@ -931,10 +1002,22 @@ impl Tool for ClickTool {// Foreground rung: brief front → click → restore.// Returns whether the window was ACTUALLY fronted, so the// reported `path` honestly reflects the rung that ran.- match (fg, window_id) {- (true, Some(wid)) => crate::input::skylight::with_foreground_assist(- pid as libc::pid_t, wid, do_click,- ),+ match (fg, window_id, has_modifiers) {+ (true, Some(wid), true) => {+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ do_click,+ )+ .map(|_| true)+ }+ (true, Some(wid), false) => {+ crate::input::skylight::with_foreground_assist(+ pid as libc::pid_t,+ wid,+ do_click,+ )+ }_ => do_click().map(|_| false),}})@@ -1025,6 +1108,7 @@ fn perform_ax_click(cursor_key: &str,selection_pixel: Option<SelectionPixelTarget>,modifiers: &[String],+ foreground: bool,) -> anyhow::Result<(String, bool, bool, bool, bool)> {let ax_action = map_action(action_str);let element = element_ptr as AXUIElementRef;@@ -1064,41 +1148,75 @@ fn perform_ax_click(}}- if let (Some(target), Some((selected_role, before))) = (+ if let (Some(target), Some(selection)) = (selection_pixel,- crate::input::ax_actions::nearest_container_selection_state(element_ptr),+ crate::input::ax_actions::capture_nearest_container_selection(element_ptr),) {+ let selected_role = selection.role().to_owned();+ let Some((before, _)) = selection.observe() else {+ anyhow::bail!("selection target stopped exposing AXSelected before delivery");+ };let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();- crate::input::mouse::click_at_xy_with_window_local(- pid,- target.screen_x,- target.screen_y,- target.window_x,- target.window_y,- window_id,- 1,- &modifier_refs,- )?;+ if foreground && !modifier_refs.is_empty() {+ crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(+ target.screen_x,+ target.screen_y,+ 1,+ "left",+ &modifier_refs,+ )?;+ } else {+ crate::input::mouse::click_at_xy_with_window_local(+ pid,+ target.screen_x,+ target.screen_y,+ target.window_x,+ target.window_y,+ window_id,+ 1,+ &modifier_refs,+ )?;+ }+ // AppKit may publish a transient AXSelected transition while the+ // event queue is still resolving the gesture. Let it settle before+ // accepting a candidate, then require the same state to survive a+ // second observation. A modified selection additionally preserves+ // every peer that was selected before delivery.+ std::thread::sleep(SELECTION_READBACK_SETTLE);let deadline = std::time::Instant::now() + SELECTION_READBACK_TIMEOUT;loop {- if let Some((_, after)) =- crate::input::ax_actions::nearest_container_selection_state(element_ptr)- {- let verified =- selection_readback_confirms(before, after, !modifiers.is_empty());+ if let Some((after, peers_preserved)) = selection.observe() {+ let verified = selection_readback_confirms(+ before,+ after,+ !modifiers.is_empty(),+ peers_preserved,+ );if verified {- return Ok((- format!(- "✅ Selected nearest {selected_role} for [{idx}] {role} \- \"{title}\"; AX selection write was unavailable, so a \- coordinate click was delivered and confirmed by AXSelected \- read-back."- ),- false,- false,- true,- true,- ));+ std::thread::sleep(SELECTION_READBACK_STABILITY);+ if let Some((stable_after, stable_peers_preserved)) = selection.observe() {+ if stable_after == after+ && selection_readback_confirms(+ before,+ stable_after,+ !modifiers.is_empty(),+ stable_peers_preserved,+ )+ {+ return Ok((+ format!(+ "✅ Selected nearest {selected_role} for [{idx}] {role} \+ \"{title}\"; AX selection write was unavailable, so a \+ coordinate click was delivered and confirmed by stable \+ AXSelected read-back."+ ),+ false,+ false,+ true,+ true,+ ));+ }+ }}}if std::time::Instant::now() >= deadline {@@ -1106,6 +1224,17 @@ fn perform_ax_click(}std::thread::sleep(SELECTION_READBACK_POLL);}+ if modifiers.is_empty() {+ anyhow::bail!(+ "coordinate click did not produce a stable AXSelected transition; \+ retry after a fresh snapshot"+ );+ }+ anyhow::bail!(+ "foreground modified coordinate click did not produce a stable AXSelected \+ transition while preserving the prior selection; take a fresh snapshot \+ before retrying"+ );}}@@ -1114,7 +1243,7 @@ fn perform_ax_click(// Some collection rows claim a click-like action but Finder returns// kAXErrorCannotComplete. Use the same verified selection fallback// before surfacing the dispatch error.- if ax_action == "AXPress" {+ if ax_action == "AXPress" && modifiers.is_empty() {if let Some(selected_role) =crate::input::ax_actions::select_nearest_container(element_ptr){@@ -1223,16 +1352,17 @@ mod selection_fallback_tests {#[test]fn plain_click_requires_selected_readback() {- assert!(selection_readback_confirms(false, true, false));- assert!(selection_readback_confirms(true, true, false));- assert!(!selection_readback_confirms(false, false, false));+ assert!(selection_readback_confirms(false, true, false, true));+ assert!(selection_readback_confirms(true, true, false, true));+ assert!(!selection_readback_confirms(false, false, false, true));}#[test]- fn modified_click_requires_a_selection_transition() {- assert!(selection_readback_confirms(false, true, true));- assert!(selection_readback_confirms(true, false, true));- assert!(!selection_readback_confirms(true, true, true));+ fn modified_click_requires_a_transition_and_preserves_prior_selection() {+ assert!(selection_readback_confirms(false, true, true, true));+ assert!(selection_readback_confirms(true, false, true, true));+ assert!(!selection_readback_confirms(true, true, true, true));+ assert!(!selection_readback_confirms(false, true, true, false));}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rsindex ed6f812713..6d01dc1b52 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs@@ -238,25 +238,43 @@ fn focus_exact_window(pid: i32, window_id: u32) -> Result<(), String> {CFRelease(target as CFTypeRef);}- let deadline = std::time::Instant::now() + Duration::from_millis(600);+ // AXFocusedWindow can lead AppKit's native `isKeyWindow` state while the+ // WindowServer activation is still settling. Menu validation observes the+ // latter. Require the exact app/window pair to remain stable briefly so a+ // loaded desktop cannot expose a transiently disabled contextual item.+ let deadline = std::time::Instant::now() + Duration::from_millis(800);+ let mut stable_since = None;loop {- if exact_window_is_focused(+ let now = std::time::Instant::now();+ if exact_window_is_ready(+ crate::apps::frontmost_pid(),+ pid,crate::ax::bindings::focused_window_id_of_pid(pid),window_id,) {- return Ok(());+ let since = stable_since.get_or_insert(now);+ if now.duration_since(*since) >= Duration::from_millis(120) {+ return Ok(());+ }+ } else {+ stable_since = None;}- if std::time::Instant::now() >= deadline {+ if now >= deadline {return Err(format!(- "invoke_menu: target window {window_id} did not become the focused window"+ "invoke_menu: target window {window_id} did not become stably key and frontmost"));}std::thread::sleep(Duration::from_millis(20));}}-fn exact_window_is_focused(focused_window_id: Option<u32>, target_window_id: u32) -> bool {- focused_window_id == Some(target_window_id)+fn exact_window_is_ready(+ frontmost_pid: Option<i32>,+ target_pid: i32,+ focused_window_id: Option<u32>,+ target_window_id: u32,+) -> bool {+ frontmost_pid == Some(target_pid) && focused_window_id == Some(target_window_id)}fn refusal(message: String) -> ToolResult {@@ -371,9 +389,10 @@ mod tests {}#[test]- fn menu_focus_requires_the_exact_requested_window() {- assert!(exact_window_is_focused(Some(42), 42));- assert!(!exact_window_is_focused(Some(41), 42));- assert!(!exact_window_is_focused(None, 42));+ fn menu_focus_requires_the_exact_frontmost_app_and_window() {+ assert!(exact_window_is_ready(Some(7), 7, Some(42), 42));+ assert!(!exact_window_is_ready(Some(8), 7, Some(42), 42));+ assert!(!exact_window_is_ready(Some(7), 7, Some(41), 42));+ assert!(!exact_window_is_ready(Some(7), 7, None, 42));}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rsindex 2e9f63a977..18e70f8467 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs@@ -35,9 +35,10 @@ fn def() -> &'static ToolDef {• `background` (default): post to the pid WITHOUT fronting/raising — the \auth-message path (Chromium-safe). With element_index it focuses that AX \element first. `window_id` only targets; it does not raise.\n\- • `foreground`: guard and briefly front the exact window, send a genuine HID \- key transition so Chromium content and native menu equivalents both receive \- it, then restore prior frontmost. Requires window_id (and no element_index).\n\n\+ • `foreground`: guard and briefly front the exact window, focus an addressed AX \+ element when supplied, send a genuine HID key transition so Chromium content, \+ inline editors, and native menu equivalents receive it, then restore prior \+ frontmost. Requires window_id.\n\n\A key press is never driver-verifiable → effect:\"unverifiable\"; confirm via \screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, \home, end, pageup, pagedown, f1-f12, plus any letter or digit. \@@ -241,16 +242,26 @@ impl Tool for PressKeyTool {// is accepted by both. Skipped when px-focus already handled the// target-specific foreground transition.if fg && !px_focus {- if let Some(wid) = window_id {- if element_index.is_none() {- crate::input::skylight::with_foreground_hid_activation(- pid as libc::pid_t,- wid,- || crate::input::keyboard::press_key_bare_global(&key, &m),- )?;- return Ok(());- }- }+ let wid = window_id.ok_or_else(|| {+ anyhow::anyhow!(+ "delivery_mode=foreground requires window_id for press_key"+ )+ })?;+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ || {+ // Activation can change the first responder, so+ // repeat the best-effort AX focus write inside+ // the guarded foreground interval immediately+ // before the physical key transition.+ if let Some(element_ptr) = pre_focus_ptr {+ let _ = crate::input::ax_actions::focus_element(element_ptr);+ }+ crate::input::keyboard::press_key_bare_global(&key, &m)+ },+ )?;+ return Ok(());}// background (default): auth-envelope post, no raise.crate::input::keyboard::press_key(pid, &key, &m)diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rsindex 0bc83e52f6..c2b1d4c9a0 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs@@ -597,14 +597,18 @@ fn send_click_synthesized_mods_impl(// poll GetKeyState (WPF, Chromium) observe the held state, then released// after the click loop below.let (mod_downs, mod_ups) = crate::input::keyboard::modifier_hold_inputs(modifiers);+ let mut sent_ok = true;if !mod_downs.is_empty() {- SendInput(&mod_downs, std::mem::size_of::<INPUT>() as i32);+ let sent = SendInput(&mod_downs, std::mem::size_of::<INPUT>() as i32);+ sent_ok = sent as usize == mod_downs.len();sleep(Duration::from_millis(5));}let count = count.max(1);- let mut sent_ok = true;for i in 0..count {+ if !sent_ok {+ break;+ }// Only the move record carries absolute coordinates. Button-only// records act at the current pointer position; adding ABSOLUTE to// them can prevent retained-mode controls from seeing the press.@@ -621,7 +625,13 @@ fn send_click_synthesized_mods_impl(// Release any held modifiers (reverse order) before restoring z-order.if !mod_ups.is_empty() {- SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);+ let released = SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);+ if released as usize != mod_ups.len() {+ // A second release attempt is safe and reduces the chance of a+ // partially inserted chord leaving system modifier state held.+ let _ = SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);+ sent_ok = false;+ }}// Let the target process mouse-up before any background-route restore.@@ -656,7 +666,10 @@ fn send_click_synthesized_mods_impl(}drop(noactivate);if !sent_ok {- bail!("SendInput inserted fewer mouse events than expected for the foreground click.");+ bail!(+ "SendInput inserted fewer modifier or mouse events than expected for the \+ foreground click."+ );}if activate {let foreground_root = GetAncestor(GetForegroundWindow(), GA_ROOT);diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 2927ff2994..5999e35322 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -2801,6 +2801,20 @@ impl Tool for ClickTool {// a modifier passed to a background click is necessarily ignored there.let modifiers: Vec<String> = args.str_array("modifier");let delivery = DeliveryMode::from_args(&args);+ if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {+ return ToolResult::error(+ "click modifiers require delivery_mode:\"foreground\" on Windows; UIA and \+ PostMessage cannot carry live modifier-key state",+ )+ .with_structured(json!({+ "code": "background_unavailable",+ "effect": "refused",+ "escalation": {+ "recommended": "foreground",+ "reason": "Windows modifier clicks require SendInput so the target observes live modifier state"+ }+ }));+ }// For every non-foreground click, mark the target window// non-activatable (WS_EX_NOACTIVATE) for the duration so a target that// self-activates in its UIA-Invoke / click handler (WPF@@ -2922,8 +2936,19 @@ impl Tool for ClickTool {let prev_fg_addr = unsafe {windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize};+ let mods_owned = modifiers.clone();+ let activate = delivery == DeliveryMode::Foreground;let send_result = tokio::task::spawn_blocking(move || {- crate::input::send_click_synthesized(hwnd, tx, ty, count, &btn_fg)+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();+ if activate && !mod_refs.is_empty() {+ crate::input::send_click_synthesized_active_mods(+ hwnd, tx, ty, count, &btn_fg, &mod_refs,+ )+ } else {+ crate::input::send_click_synthesized_mods(+ hwnd, tx, ty, count, &btn_fg, &mod_refs,+ )+ }}).await;tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));@@ -3028,6 +3053,27 @@ impl Tool for ClickTool {// Foreground delivery is a real SendInput tap at (cx,cy); if the// element is off-screen, scroll it into view and re-resolve so the// tap doesn't land on the taskbar, else keep the clean failure.+ // Modified selection needs one stronger preflight: some WPF+ // ScrollViewers report a clipped item as on-screen because its+ // stale rectangle is still inside the outer HWND. Ask the item+ // to scroll itself into view before trusting that rectangle.+ let (cx, cy) = if !modifiers.is_empty() {+ self.state+ .element_cache+ .get_element_retained(pid, hwnd, idx)+ .and_then(|retained| {+ retained.is_uia().then(|| unsafe {+ crate::uia::scroll::scroll_into_view_and_recenter(+ hwnd,+ retained.as_ptr(),+ )+ })+ })+ .flatten()+ .unwrap_or((cx, cy))+ } else {+ (cx, cy)+ };let (cx, cy) = match resolve_onscreen_point_with_scroll(&self.state.element_cache,pid,@@ -3041,11 +3087,19 @@ impl Tool for ClickTool {Err(msg) => return ToolResult::error(msg),};let btn_fg = btn.clone();+ let mods_owned = modifiers.clone();let prev_fg_addr = unsafe {windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize};let send_result = tokio::task::spawn_blocking(move || {- crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();+ if mod_refs.is_empty() {+ crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)+ } else {+ crate::input::send_click_synthesized_active_mods(+ hwnd, cx, cy, count, &btn_fg, &mod_refs,+ )+ }}).await;tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));@@ -3350,9 +3404,15 @@ impl Tool for ClickTool {let mods_owned = modifiers.clone();let send_result = tokio::task::spawn_blocking(move || {let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();- crate::input::send_click_synthesized_active_mods(- hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,- )+ if mod_refs.is_empty() {+ crate::input::send_click_synthesized(+ hwnd, sx as i32, sy as i32, count, &btn,+ )+ } else {+ crate::input::send_click_synthesized_active_mods(+ hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,+ )+ }}).await;tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));diff --git a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.pyindex bd3ea9e7e8..6ec9bfc269 100755--- a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py+++ b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py@@ -127,6 +127,22 @@ class HarnessWindow(Gtk.Window):self.chk_status = Gtk.Label(label="agreed=False", xalign=0)root.pack_start(self.chk_status, False, False, 0)+ # ── multi-selection ───────────────────────────────────────────────+ section(root, "multi selection")+ self.selection = Gtk.ListBox()+ self.selection.set_selection_mode(Gtk.SelectionMode.MULTIPLE)+ for value in ("alpha", "beta", "gamma"):+ row = aid(Gtk.ListBoxRow(), f"selection-{value}")+ row.set_activatable(True)+ row.set_selectable(True)+ row.set_can_focus(True)+ row.add(Gtk.Label(label=value, xalign=0))+ self.selection.add(row)+ self.selection.connect("selected-rows-changed", self.on_selection_changed)+ root.pack_start(self.selection, False, False, 0)+ self.selection_status = Gtk.Label(label="selection=none", xalign=0)+ root.pack_start(self.selection_status, False, False, 0)+# ── context_menu ──────────────────────────────────────────────────section(root, "context menu")self.ctx = aid(Gtk.Button(label="Right-click for context menu"), "btn-context")@@ -218,6 +234,14 @@ class HarnessWindow(Gtk.Window):def on_chk(self, c):self.chk_status.set_text(f"agreed={c.get_active()}")+ def on_selection_changed(self, list_box):+ values = sorted(+ row.get_child().get_text() for row in list_box.get_selected_rows()+ )+ self.selection_status.set_text(+ "selection=" + (",".join(values) if values else "none")+ )+def on_ctx_press(self, _w, ev):if ev.button == 3:self.menu.popup_at_pointer(ev)diff --git a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swiftindex 71852fefb1..fade434faf 100644--- a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift+++ b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift@@ -32,6 +32,7 @@ let kTextBodyAID = "txt-body"let kTextBodyMarker = "HARNESS_TEXT_MARKER_v1"let kTextInputAID = "txt-input"let kTextInputMirrorAID = "lbl-input-mirror"+let kTextInputCommitAID = "lbl-input-commit"let kClickTargetAID = "btn-clicktarget"let kLastActionAID = "lbl-last-action"let kClickCountAID = "lbl-click-count"@@ -39,6 +40,7 @@ let kSliderAID = "sld-value"let kSliderValueAID = "lbl-slider-value"let kCheckboxAID = "chk-agree"let kCheckStateAID = "lbl-chk-state"+let kSelectionStateAID = "lbl-selection-state"let kContextButtonAID = "btn-context"let kMenuActionAID = "lbl-menu-action"let kScrollerAID = "scroll-tall"@@ -51,17 +53,21 @@ let kMenuItemTitle = "Harness Test Item"// MARK: - Controller-final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemValidation {+final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDataSource, NSTableViewDelegate, NSMenuItemValidation {let window: NSWindowlet counterLabel = NSTextField(labelWithString: "counter=0")var counterValue = 0let textInput = NSTextField(string: "")let textInputMirror = NSTextField(labelWithString: "")+ let textInputCommit = NSTextField(labelWithString: "committed=none")let lastActionLabel = NSTextField(labelWithString: "last_action=none")let clickCountLabel = NSTextField(labelWithString: "clicks=0")var clicks = 0let sliderValueLabel = NSTextField(labelWithString: "slider_value=0")let checkStateLabel = NSTextField(labelWithString: "agreed=false")+ let selectionItems = ["alpha", "beta", "gamma"]+ let selectionTable = NSTableView()+ let selectionStateLabel = NSTextField(labelWithString: "selection=none")let menuActionLabel = NSTextField(labelWithString: "menu_action=none")let scrollOffsetLabel = NSTextField(labelWithString: "scroll_offset=0")let accelCountLabel = NSTextField(labelWithString: "accel_fired=0")@@ -146,11 +152,14 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVatextInput.translatesAutoresizingMaskIntoConstraints = falsetextInputMirror.setAccessibilityIdentifier(kTextInputMirrorAID)textInputMirror.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)+ textInputCommit.setAccessibilityIdentifier(kTextInputCommitAID)+ textInputCommit.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)let inputRow = NSStackView()inputRow.orientation = .horizontalinputRow.spacing = 12inputRow.addArrangedSubview(textInput)inputRow.addArrangedSubview(textInputMirror)+ inputRow.addArrangedSubview(textInputCommit)NSLayoutConstraint.activate([textInput.widthAnchor.constraint(equalToConstant: 240),])@@ -210,6 +219,26 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVacheckRow.spacing = 12checkRow.addArrangedSubview(checkbox)checkRow.addArrangedSubview(checkStateLabel)+ selectionTable.headerView = nil+ selectionTable.allowsMultipleSelection = true+ selectionTable.dataSource = self+ selectionTable.delegate = self+ let selectionColumn = NSTableColumn(identifier: NSUserInterfaceItemIdentifier("selection-column"))+ selectionColumn.width = 140+ selectionTable.addTableColumn(selectionColumn)+ let selectionScroll = NSScrollView()+ selectionScroll.documentView = selectionTable+ selectionScroll.hasVerticalScroller = true+ selectionScroll.borderType = .lineBorder+ selectionScroll.translatesAutoresizingMaskIntoConstraints = false+ selectionStateLabel.setAccessibilityIdentifier(kSelectionStateAID)+ selectionStateLabel.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)+ checkRow.addArrangedSubview(selectionScroll)+ checkRow.addArrangedSubview(selectionStateLabel)+ NSLayoutConstraint.activate([+ selectionScroll.widthAnchor.constraint(equalToConstant: 150),+ selectionScroll.heightAnchor.constraint(equalToConstant: 68),+ ])content.addArrangedSubview(checkRow)// context_menu — NSButton with an attached NSMenu. Right-click opens the@@ -359,6 +388,26 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVacheckStateLabel.stringValue = "agreed=\(sender.state == .on)"}+ func numberOfRows(in tableView: NSTableView) -> Int {+ selectionItems.count+ }++ func tableView(_ tableView: NSTableView,+ viewFor tableColumn: NSTableColumn?, row: Int) -> NSView? {+ let value = selectionItems[row]+ let field = NSTextField(labelWithString: value)+ field.isSelectable = true+ field.setAccessibilityIdentifier("selection-\(value)")+ return field+ }++ func tableViewSelectionDidChange(_ notification: Notification) {+ let values = selectionTable.selectedRowIndexes.map { selectionItems[$0] }+ selectionStateLabel.stringValue = values.isEmpty+ ? "selection=none"+ : "selection=\(values.joined(separator: ","))"+ }+@objc private func onContextItem(_ sender: NSMenuItem) {menuActionLabel.stringValue = "menu_action=\(sender.title)"}@@ -385,6 +434,13 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVa}}+ func controlTextDidEndEditing(_ obj: Notification) {+ guard let field = obj.object as? NSTextField else { return }+ if field === textInput {+ textInputCommit.stringValue = "committed=\(field.stringValue)"+ }+ }+// Click target — single source of truth for all three actions.@objc func onClickTarget() { recordClick("click") }func clickTargetSawDouble() { recordClick("double_click") }diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xamlindex eb1f4dfef1..1f797c410c 100644--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml@@ -207,6 +207,7 @@<StackPanel><ListBox x:Name="LstItems"AutomationProperties.AutomationId="lst-items"+ SelectionMode="Extended"Width="220" Height="80" HorizontalAlignment="Left"SelectionChanged="OnListChanged"><ListBoxItem Content="apple" AutomationProperties.AutomationId="lst-apple"/>diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.csindex 155fbeebe8..e49ce58257 100644--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs@@ -1,6 +1,7 @@using System;using System.Collections.Generic;using System.IO;+using System.Linq;using System.Runtime.InteropServices;using System.Text.Json;using System.Windows;@@ -267,10 +268,11 @@ public partial class MainWindow : Windowprivate void OnListChanged(object sender, SelectionChangedEventArgs e){if (LblListValue is null) return;- if (LstItems?.SelectedItem is ListBoxItem item)- {- LblListValue.Text = $"selected={item.Content}";- }+ var selected = LstItems?.SelectedItems+ .OfType<ListBoxItem>()+ .Select(item => item.Content?.ToString() ?? "")+ .Where(value => value.Length > 0);+ LblListValue.Text = $"selected={string.Join(",", selected ?? Enumerable.Empty<string>())}";}private void OnMenuFileNew(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=file_new";diff --git a/libs/cua-driver/tests/fixtures/shared/scenarios.json b/libs/cua-driver/tests/fixtures/shared/scenarios.jsonindex 129253d05f..358ebc8e9d 100644--- a/libs/cua-driver/tests/fixtures/shared/scenarios.json+++ b/libs/cua-driver/tests/fixtures/shared/scenarios.json@@ -338,7 +338,8 @@"description": "Editable NSTextField with a mirror label. Tests type_text and set_value via AXValue on AppKit.","controls": {"textbox_aid": "txt-input",- "mirror_label_aid": "lbl-input-mirror"+ "mirror_label_aid": "lbl-input-mirror",+ "commit_label_aid": "lbl-input-commit"}},{diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.shindex 5669a6cdb6..7a36eb1fba 100755--- a/scripts/ci/macos/run-rust-e2e.sh+++ b/scripts/ci/macos/run-rust-e2e.sh@@ -322,6 +322,8 @@ if [[ "${SUITE}" == native || "${SUITE}" == all ]]; thenharness_appkit_stale_element_token_fails_closed \harness_appkit_invoke_menu_live_path \harness_appkit_text_input \+ harness_appkit_element_foreground_press_key_commits_edit \+ harness_appkit_modified_click_preserves_selection \harness_appkit_type_text_background \harness_appkit_scroll_foreground \harness_appkit_scroll_background \