Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): make native selection and editing foreground-safe (#2789)

Francesco Bonacci815013a88c02
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 5bc8134ecc..ce934961d3 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -238,7 +238,7 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin
- `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle".
- `count` (integer, optional): Click count (pixel path only). Default 1.
- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.
-- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".
+- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require "foreground" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground".
- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.
@@ -366,7 +366,7 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp
Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:
• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_index it focuses that AX element first. `window_id` only targets; it does not raise.
-• `foreground`: guard and briefly front the exact window, send a genuine HID key transition so Chromium content and native menu equivalents both receive it, then restore prior frontmost. Requires window_id (and no element_index).
+• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id.
A key press is never driver-verifiable → effect:"unverifiable"; confirm via screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index b72279cd73..da2f332218 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -298,8 +298,16 @@ breadth Windows and Linux already exposed (`type_text` / `press_key` /
no raise, no focus steal. `"foreground"` briefly fronts the owning app,
acts, then restores the prior frontmost — the explicit last resort for a
surface that only accepts events while frontmost (the canvas/viewport/game
-case below). Element-indexed (AX) actions are inherently background and
-hold the no-foreground contract without the flag.
+case below). Unmodified element-indexed (AX) actions remain background-capable
+and hold the no-foreground contract without the flag.
+
+Modified clicks are the deliberate exception: pass
+`delivery_mode:"foreground"` and a concrete `window_id`. macOS applications
+can discard PID-routed modifier state after initially publishing a transient
+selection, so Cua Driver refuses that background combination. The foreground
+rung holds physical HID modifier keys around the click, restores the hardware
+cursor and prior foreground app, and confirms list-like selection changes with
+a stable AX readback.
macOS-specific residuals worth knowing (the rest of the capture/dispatch/
addressing params are a shared cross-platform contract — see `SKILL.md` →
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index b3a392bd91..04aa2fb05a 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -94,6 +94,35 @@ that boundary, do not replace Cua's targeted and verified actions with shell
scripts that mutate the app UI. A shell is a capability of the calling agent,
not of the Cua Driver MCP server; an MCP-only client must not assume one exists.
+### Filesystem outcomes and GUI fallbacks
+
+When the requested outcome is a filesystem change and the caller has a
+headless filesystem or command capability, keep it on rung 0. Enumerate the
+exact source set, decide the destination-conflict policy before changing
+anything, perform one batch-safe operation, then independently read back both
+source and destination manifests. Do not open a file manager merely to mimic a
+move, copy, or rename that the caller can execute and verify directly.
+
+If the caller has no such capability, use the file manager as a GUI fallback
+and keep each claim narrow:
+
+1. After entering an inline rename and setting its value, commit it with the
+ platform's confirmation key, then take a fresh snapshot. Value readback from
+ the inline editor proves only that the editor changed; it does not prove the
+ filesystem rename committed.
+2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on
+ Windows/Linux). On macOS and Windows, issue that modified click with
+ `delivery_mode:"foreground"` so the target observes physical modifier state;
+ a refused background attempt is an escalation signal, not a failed action to
+ trust or repeat. Re-snapshot before the next operation. Continue only when
+ every intended item is selected and the prior selection was preserved.
+3. After a cross-window drag or paste, verify the destination contains the
+ complete expected set and the source reflects copy-versus-move semantics.
+ A delivered drag, keypress, or menu action is not file-operation proof.
+4. If a destination conflict presents an unrecognized policy or ambiguous
+ partial result, stop that GUI path and surface the unresolved state instead
+ of retrying blindly.
+
## The no-foreground principle (window phase)
In a strict `window` session, and during the initial window phase of an
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs b/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
index 0a4aff5f9f..017090405e 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
@@ -841,6 +841,27 @@ mod tests {
}
}
+ #[test]
+ fn bundled_skill_keeps_filesystem_outcome_ladder_and_gui_proof_boundaries() {
+ let crate_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+ let skill = std::fs::read_to_string(crate_dir.join("../../Skills/cua-driver/SKILL.md"))
+ .expect("canonical skill must be readable");
+
+ for required in [
+ "headless filesystem or command capability",
+ "perform one batch-safe operation",
+ "filesystem rename committed",
+ "issue that modified click with",
+ "every intended item is selected",
+ "source reflects copy-versus-move semantics",
+ ] {
+ assert!(
+ skill.contains(required),
+ "skill lost required filesystem outcome guidance: {required}"
+ );
+ }
+ }
+
#[test]
fn extract_flat_tarball_v_0_2_20_plus() {
// Post-fix shape: one wrapper dir, files directly under it.
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
index 95f6eefe32..fb46502f1a 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs
@@ -452,6 +452,186 @@ fn harness_appkit_text_input() {
);
}
+#[test]
+#[ignore]
+fn harness_appkit_element_foreground_press_key_commits_edit() {
+ run_case(
+ native_foreground_case(
+ "appkit",
+ "press_key_commit",
+ Targeting::Ax,
+ DriverRoute::MacosCgEventHid,
+ ),
+ |pid, wid, driver| {
+ let first = snapshot_elements(driver, pid, wid);
+ let field = element_token_by_id(&first, "txt-input");
+ let set = driver.call(
+ "set_value",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "element_token": field,
+ "value": "inline-cua"
+ }),
+ );
+ assert!(!set.is_error(), "set_value failed: {}", set.text());
+
+ let second = snapshot_elements(driver, pid, wid);
+ assert!(
+ second.tree_text().contains("inline-cua"),
+ "transient edit value was not readable:\n{}",
+ second.tree_text()
+ );
+ assert!(
+ second.tree_text().contains("committed=none"),
+ "fixture reported a commit before Return:\n{}",
+ second.tree_text()
+ );
+ let field = element_token_by_id(&second, "txt-input");
+ let commit = driver.call(
+ "press_key",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "element_token": field,
+ "key": "return",
+ "delivery_mode": "foreground"
+ }),
+ );
+ assert!(
+ !commit.is_error(),
+ "foreground element press_key failed: {}",
+ commit.text()
+ );
+ assert_eq!(
+ commit.action_route(),
+ Some("global_input"),
+ "foreground press_key used the wrong public route: {}",
+ commit.raw
+ );
+ assert_eq!(
+ commit.action_delivery_mode(),
+ Some("foreground"),
+ "foreground press_key reported the wrong delivery: {}",
+ commit.raw
+ );
+ assert_eq!(
+ commit.action_effect(),
+ Some("unverifiable"),
+ "press_key claimed more truth than the tool itself observed: {}",
+ commit.raw
+ );
+
+ std::thread::sleep(Duration::from_millis(250));
+ let post = snapshot_elements(driver, pid, wid);
+ assert!(
+ post.tree_text().contains("committed=inline-cua"),
+ "Return did not commit the addressed edit:\n{}",
+ post.tree_text()
+ );
+ Observation::delivered_with_fixture_state(Vec::new())
+ },
+ );
+}
+
+#[test]
+#[ignore]
+fn harness_appkit_modified_click_preserves_selection() {
+ run_case(
+ native_foreground_case(
+ "appkit",
+ "modified_click_selection",
+ Targeting::Ax,
+ DriverRoute::MacosCgEventHid,
+ ),
+ |pid, wid, driver| {
+ let first = snapshot_elements(driver, pid, wid);
+ let alpha = element_token_by_id(&first, "selection-alpha");
+ let select_alpha = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "element_token": alpha
+ }),
+ );
+ assert!(
+ !select_alpha.is_error(),
+ "select alpha failed: {}",
+ select_alpha.text()
+ );
+ std::thread::sleep(Duration::from_millis(200));
+
+ let second = snapshot_elements(driver, pid, wid);
+ assert!(
+ second.tree_text().contains("selection=alpha"),
+ "alpha was not selected:\n{}",
+ second.tree_text()
+ );
+ let beta = element_token_by_id(&second, "selection-beta");
+ let refused_background = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "element_token": beta,
+ "modifier": ["cmd"]
+ }),
+ );
+ assert!(
+ refused_background.is_error(),
+ "background modified click was not refused: {}",
+ refused_background.text()
+ );
+ assert_eq!(
+ refused_background.structured()["code"],
+ "background_unavailable",
+ "background modified click returned the wrong refusal: {}",
+ refused_background.structured()
+ );
+ std::thread::sleep(Duration::from_millis(300));
+ let after_refusal = snapshot_elements(driver, pid, wid);
+ assert!(
+ after_refusal.tree_text().contains("selection=alpha"),
+ "refused modified click changed the prior selection:\n{}",
+ after_refusal.tree_text()
+ );
+
+ let beta = element_token_by_id(&after_refusal, "selection-beta");
+ let add_beta = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": wid,
+ "element_token": beta,
+ "modifier": ["cmd"],
+ "delivery_mode": "foreground"
+ }),
+ );
+ assert!(
+ !add_beta.is_error(),
+ "foreground modified click failed: {}",
+ add_beta.text()
+ );
+ assert_eq!(
+ add_beta.structured()["effect"],
+ "confirmed",
+ "modified click lacked settled selection proof: {}",
+ add_beta.structured()
+ );
+
+ std::thread::sleep(Duration::from_millis(250));
+ let post = snapshot_elements(driver, pid, wid);
+ assert!(
+ post.tree_text().contains("selection=alpha,beta"),
+ "modified click replaced or lost the prior selection:\n{}",
+ post.tree_text()
+ );
+ Observation::delivered_with_fixture_state(Vec::new())
+ },
+ );
+}
+
/// type_text: synthesize a keystroke into the NSTextField (CGEvent
/// path, distinct from set_value's AX path). Verifies the keyboard
/// dispatch chain reaches a backgrounded Cocoa text input.
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
index 0d838ee4af..f612f27ad0 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs
@@ -284,6 +284,117 @@ fn harness_gtk3_query_projects_structured_elements() {
);
}
+#[test]
+#[ignore]
+fn harness_gtk3_modified_click_preserves_selection() {
+ let wayland_refusal = DisplayServer::current() == DisplayServer::Wayland;
+ let case = if wayland_refusal {
+ native_background_case(
+ "gtk3",
+ "modified_click_selection",
+ Targeting::Ax,
+ DriverRoute::LinuxWaylandVirtualPointer,
+ )
+ .expecting_refusal(vec![RefusalCode::BackgroundUnavailable])
+ } else {
+ native_foreground_case(
+ "gtk3",
+ "modified_click_selection",
+ Targeting::Ax,
+ DriverRoute::LinuxXTest,
+ )
+ };
+ run_case(case, |pid, window_id, driver| {
+ let first = snapshot(driver, pid, window_id);
+ if wayland_refusal {
+ let beta = element_token(&first, "selection-beta");
+ let (add_beta, mut passed) = run_with_background_oracles(
+ driver,
+ TargetWindow {
+ pid,
+ native_id: window_id,
+ },
+ |driver| {
+ driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": window_id,
+ "element_token": beta,
+ "delivery_mode": "background",
+ "modifier": ["ctrl"]
+ }),
+ )
+ },
+ )
+ .unwrap_or_else(|error| panic!("Wayland background contract failed: {error}"));
+ assert!(
+ add_beta.is_error(),
+ "native Wayland unexpectedly accepted a modified pointer click: {}",
+ add_beta.text()
+ );
+ assert!(
+ add_beta
+ .text()
+ .contains("modified element clicks are unavailable on native Wayland"),
+ "native Wayland returned the wrong refusal: {}",
+ add_beta.text()
+ );
+ std::thread::sleep(Duration::from_millis(250));
+ let post = snapshot(driver, pid, window_id);
+ assert!(
+ post.tree_text().contains("selection=none"),
+ "refused modified click mutated the selection:\n{}",
+ post.tree_text()
+ );
+ passed.push(OracleKind::FixtureState);
+ return Observation::refused(
+ RefusalCode::BackgroundUnavailable,
+ passed,
+ "native Wayland cannot pair virtual-pointer delivery with modifier state",
+ Evidence::default(),
+ );
+ }
+
+ let alpha = element_token(&first, "selection-alpha");
+ let select_alpha = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": window_id,
+ "element_token": alpha,
+ "delivery_mode": "foreground"
+ }),
+ );
+ assert!(
+ !select_alpha.is_error(),
+ "select alpha failed: {}",
+ select_alpha.text()
+ );
+ wait_for_state(driver, pid, window_id, "selection=alpha");
+
+ let second = snapshot(driver, pid, window_id);
+ let beta = element_token(&second, "selection-beta");
+ let add_beta = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64,
+ "window_id": window_id,
+ "element_token": beta,
+ "delivery_mode": "foreground",
+ "modifier": ["ctrl"]
+ }),
+ );
+ assert!(
+ !add_beta.is_error(),
+ "modified click failed: {}",
+ add_beta.text()
+ );
+ wait_for_state(driver, pid, window_id, "selection=alpha,beta");
+ Observation::delivered_with_fixture_state(Vec::new())
+ });
+}
+
#[test]
#[ignore]
fn harness_gtk3_stale_element_token_fails_closed() {
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
index a7dd659a86..c37857fc00 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
@@ -1593,6 +1593,92 @@ fn harness_wpf_listbox_select() {
);
}
+#[test]
+#[ignore]
+fn harness_wpf_modified_click_preserves_selection() {
+ run_foreground_case(
+ "modified_click_selection",
+ Targeting::Ax,
+ DriverRoute::WindowsSendInput,
+ Vec::new(),
+ |pid, wid, driver| {
+ focus_harness(driver, pid, wid);
+ let first = snapshot(driver, pid, wid);
+ let apple =
+ ax::element_index_by_id(first.text(), "lst-apple").expect("lst-apple missing");
+ let select_apple = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64, "window_id": wid, "element_index": apple,
+ "snapshot_id": first.snapshot_id(), "delivery_mode": "foreground"
+ }),
+ );
+ assert!(
+ !select_apple.is_error(),
+ "select apple failed: {}",
+ select_apple.text()
+ );
+ std::thread::sleep(Duration::from_millis(250));
+
+ let second = snapshot(driver, pid, wid);
+ let banana =
+ ax::element_index_by_id(second.text(), "lst-banana").expect("lst-banana missing");
+ let refused_background = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64, "window_id": wid, "element_index": banana,
+ "snapshot_id": second.snapshot_id(), "modifier": ["ctrl"]
+ }),
+ );
+ assert!(
+ refused_background.is_error(),
+ "background modified click was not refused: {}",
+ refused_background.text()
+ );
+ assert_eq!(
+ refused_background.structured()["code"],
+ "background_unavailable",
+ "background modified click returned the wrong refusal: {}",
+ refused_background.structured()
+ );
+ std::thread::sleep(Duration::from_millis(250));
+ let after_refusal = snapshot(driver, pid, wid);
+ assert!(
+ after_refusal.text().contains("selected=apple"),
+ "refused modified click changed the prior selection: {}",
+ after_refusal.text()
+ );
+ let banana = ax::element_index_by_id(after_refusal.text(), "lst-banana")
+ .expect("lst-banana missing after refusal");
+ let add_banana = driver.call(
+ "click",
+ serde_json::json!({
+ "pid": pid as i64, "window_id": wid, "element_index": banana,
+ "snapshot_id": after_refusal.snapshot_id(), "delivery_mode": "foreground",
+ "modifier": ["ctrl"]
+ }),
+ );
+ assert!(
+ !add_banana.is_error(),
+ "modified click failed: {}",
+ add_banana.text()
+ );
+ std::thread::sleep(Duration::from_millis(300));
+ let post = snapshot(driver, pid, wid);
+ assert!(
+ post.text().contains("selected=apple,banana"),
+ "modified click replaced or lost the prior selection: {}",
+ post.text()
+ .lines()
+ .filter(|line| line.contains("selected="))
+ .collect::<Vec<_>>()
+ .join(" / ")
+ );
+ Vec::new()
+ },
+ );
+}
+
#[test]
#[ignore]
fn harness_wpf_invoke_menu_live_path() {
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
index 9b0b70e3a7..0c4d30d75e 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
@@ -131,6 +131,7 @@ struct Visited<'a> {
checked: Option<bool>,
enabled: Option<bool>,
selected: Option<bool>,
+ selectable: bool,
actions: Vec<String>,
has_editable: bool,
has_value: bool,
@@ -503,6 +504,10 @@ async fn collect_visited_bounded<'a>(
.as_ref()
.and_then(|state| state.as_ref().ok())
.map(|state| state.contains(State::Enabled) && state.contains(State::Sensitive));
+ let selectable = state_r
+ .as_ref()
+ .and_then(|state| state.as_ref().ok())
+ .is_some_and(|state| state.contains(State::Selectable));
let selected = if role_lower.contains("check") {
checked
} else if role_lower.contains("radio")
@@ -603,6 +608,7 @@ async fn collect_visited_bounded<'a>(
checked,
enabled,
selected,
+ selectable,
actions,
has_editable,
has_value,
@@ -708,11 +714,11 @@ fn format_value(v: f64) -> String {
/// Whether a walked node is exposed as an indexed, usable element.
///
/// Historically this was "the node advertises AT-SPI Actions" (buttons, menu
-/// items, links). That silently dropped every **Value**-only widget — GTK
-/// `GtkScale` sliders, scroll bars, spin buttons, progress bars expose the
-/// `Value` interface but NO `Action`, while some text fields expose
-/// `EditableText` without either. Omitting those interfaces makes controls the
-/// driver can operate impossible to address by `element_index`.
+/// items, links). That silently dropped Value-only widgets, editable text, and
+/// selectable list rows. GTK list rows expose Component + Selectable state but
+/// no Action even though a coordinate click on their bounds is operable. Keep
+/// every such control in the shared index space so physical-input fallbacks can
+/// address it without inventing pixels in the caller.
///
/// This predicate is the single source of truth for the element-index space and
/// MUST be applied identically in `render` and in every `action_nodes` filter
@@ -723,6 +729,7 @@ fn is_indexable(v: &Visited) -> bool {
!v.actions.is_empty(),
v.has_editable,
v.has_value,
+ v.selectable,
v.enabled,
)
}
@@ -731,9 +738,10 @@ fn is_indexable_capabilities(
has_action: bool,
has_editable: bool,
has_value: bool,
+ has_selectable_state: bool,
enabled: Option<bool>,
) -> bool {
- (has_action || has_editable || has_value) && enabled != Some(false)
+ (has_action || has_editable || has_value || has_selectable_state) && enabled != Some(false)
}
// ── Public (sync) entry points ───────────────────────────────────────────────
@@ -2486,12 +2494,43 @@ mod coord_tests {
};
#[test]
- fn editable_only_nodes_are_addressable() {
- assert!(is_indexable_capabilities(false, true, false, Some(true)));
- assert!(is_indexable_capabilities(true, false, false, None));
- assert!(is_indexable_capabilities(false, false, true, Some(true)));
- assert!(!is_indexable_capabilities(false, false, false, Some(true)));
- assert!(!is_indexable_capabilities(true, false, false, Some(false)));
+ fn operable_nodes_are_addressable() {
+ assert!(is_indexable_capabilities(
+ false,
+ true,
+ false,
+ false,
+ Some(true)
+ ));
+ assert!(is_indexable_capabilities(true, false, false, false, None));
+ assert!(is_indexable_capabilities(
+ false,
+ false,
+ true,
+ false,
+ Some(true)
+ ));
+ assert!(is_indexable_capabilities(
+ false,
+ false,
+ false,
+ true,
+ Some(true)
+ ));
+ assert!(!is_indexable_capabilities(
+ false,
+ false,
+ false,
+ false,
+ Some(true)
+ ));
+ assert!(!is_indexable_capabilities(
+ true,
+ false,
+ false,
+ false,
+ Some(false)
+ ));
}
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
index 3b2497a421..4725ad77cf 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
@@ -1605,8 +1605,23 @@ pub fn send_focus_out(xid: u64) -> Result<()> {
/// Send a button click (down + up) to a window at window-local coordinates.
pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<()> {
+ send_click_with_modifiers(xid, x, y, count, button, &[])
+}
+
+/// Send a target-addressed X11 click whose event-state mask carries the named
+/// modifiers. Unlike a plain AT-SPI action, this preserves multi-selection
+/// semantics without changing the X input focus.
+pub fn send_click_with_modifiers(
+ xid: u64,
+ x: i32,
+ y: i32,
+ count: usize,
+ button: u8,
+ modifiers: &[&str],
+) -> Result<()> {
let (conn, _) = connect_x11_for_input()?;
let root = conn.setup().roots[0].root;
+ let modifier_state = modifiers_to_state(modifiers);
for _ in 0..count {
let target = resolve_event_target(&conn, xid, x, y)?;
@@ -1622,7 +1637,7 @@ pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<
root_y: target.root_y,
event_x: target.local_x,
event_y: target.local_y,
- state: KeyButMask::from(0u16),
+ state: modifier_state,
same_screen: true,
};
@@ -1638,7 +1653,9 @@ pub fn send_click(xid: u64, x: i32, y: i32, count: usize, button: u8) -> Result<
root_y: target.root_y,
event_x: target.local_x,
event_y: target.local_y,
- state: button_state_mask(button),
+ state: KeyButMask::from(
+ u16::from(modifier_state) | u16::from(button_state_mask(button)),
+ ),
same_screen: true,
};
@@ -2056,22 +2073,67 @@ pub fn send_key_xtest(key: &str, modifiers: &[&str]) -> Result<()> {
/// by vision on the whole screen and issues a real screen-absolute pointer
/// click. `button` is an X button number (1=left, 2=middle, 3=right).
pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> Result<()> {
+ send_click_xtest_desktop_with_modifiers(x, y, button, count, &[])
+}
+
+/// Real XTest click with physical modifier down/up transitions around the
+/// pointer gesture. Used only after the caller selected foreground delivery.
+pub fn send_click_xtest_desktop_with_modifiers(
+ x: i32,
+ y: i32,
+ button: u8,
+ count: usize,
+ modifiers: &[&str],
+) -> Result<()> {
use x11rb::protocol::xtest::ConnectionExt as _;
let (conn, screen_num) = connect_x11_for_input()?;
let root = conn.setup().roots[screen_num].root;
- // Absolute pointer warp (MotionNotify, detail=0 => absolute) so the button
- // events that follow are delivered at (x, y).
- conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;
- let count = count.max(1);
- for click_index in 0..count {
- conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;
- conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;
- if click_index + 1 < count {
- // Chromium needs the first pair to reach the server before the
- // second pair. A zero-gap batch produces two click events but no
- // DOM dblclick event under Xvfb/Openbox.
- conn.flush()?;
- sleep(Duration::from_millis(DOUBLE_CLICK_DELAY_MS));
+ let mapping = conn.get_keyboard_mapping(8, 248)?.reply()?;
+ let mut guards = Vec::new();
+ let mut modifier_keycodes = Vec::new();
+ for modifier in modifiers {
+ let keysym = key_name_to_keysym(modifier)?;
+ let (keycode, guard) = keycode_for_keysym(&conn, &mapping, keysym, modifier)?;
+ if let Some(guard) = guard {
+ guards.push(guard);
+ }
+ modifier_keycodes.push(keycode);
+ }
+ let mut pressed = Vec::new();
+ let gesture_result = (|| -> Result<()> {
+ for &keycode in &modifier_keycodes {
+ conn.xtest_fake_input(KEY_PRESS_EVENT, keycode, 0, x11rb::NONE, 0, 0, 0)?;
+ pressed.push(keycode);
+ }
+ // Absolute pointer warp (MotionNotify, detail=0 => absolute) so the
+ // button events that follow are delivered at (x, y).
+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;
+ let count = count.max(1);
+ for click_index in 0..count {
+ conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;
+ conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;
+ if click_index + 1 < count {
+ // Chromium needs the first pair to reach the server before the
+ // second pair. A zero-gap batch produces two click events but
+ // no DOM dblclick event under Xvfb/Openbox.
+ conn.flush()?;
+ sleep(Duration::from_millis(DOUBLE_CLICK_DELAY_MS));
+ }
+ }
+ Ok(())
+ })();
+
+ // Always attempt to release every modifier that was successfully queued,
+ // including when a later pointer request fails. A failed gesture must not
+ // leave the desktop with a logically stuck Ctrl/Shift/Alt/Super key.
+ let mut release_result: Result<()> = Ok(());
+ for &keycode in pressed.iter().rev() {
+ if let Err(error) =
+ conn.xtest_fake_input(KEY_RELEASE_EVENT, keycode, 0, x11rb::NONE, 0, 0, 0)
+ {
+ if release_result.is_ok() {
+ release_result = Err(error.into());
+ }
}
}
conn.flush()?;
@@ -2080,6 +2142,9 @@ pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> Res
// under Xtigervnc where keyboard events did not (see send_key_xtest), but make
// it explicit so the desktop click is reliable across X servers too.
let _ = conn.get_input_focus()?.reply();
+ drop(guards);
+ gesture_result?;
+ release_result?;
Ok(())
}
@@ -2595,8 +2660,24 @@ exit 0"#,
#[cfg(test)]
mod path_tests {
use super::{
- path_cumulative, point_on_path, real_pointer_capabilities_available, sample_function,
+ modifiers_to_state, path_cumulative, point_on_path, real_pointer_capabilities_available,
+ sample_function,
};
+ use x11rb::protocol::xproto::KeyButMask;
+
+ #[test]
+ fn click_modifier_state_combines_canonical_names_and_aliases() {
+ let state = modifiers_to_state(&["ctrl", "shift", "meta"]);
+ assert!(state.contains(KeyButMask::CONTROL));
+ assert!(state.contains(KeyButMask::SHIFT));
+ assert!(state.contains(KeyButMask::MOD4));
+ assert!(!state.contains(KeyButMask::MOD1));
+
+ assert_eq!(
+ modifiers_to_state(&["control", "alt"]),
+ KeyButMask::from(u16::from(KeyButMask::CONTROL) | u16::from(KeyButMask::MOD1))
+ );
+ }
#[test]
fn real_pointer_capabilities_require_uinput_access() {
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 12ea98659d..636c4b4baa 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1559,6 +1559,17 @@ fn x11_pixel_click_no_focus_steal(
crate::input::send_click(xid, lx, ly, count, button)
}
+fn x11_pixel_click_no_focus_steal_modifiers(
+ xid: u64,
+ lx: i32,
+ ly: i32,
+ button: u8,
+ count: usize,
+ modifiers: &[&str],
+) -> anyhow::Result<()> {
+ crate::input::send_click_with_modifiers(xid, lx, ly, count, button, modifiers)
+}
+
fn mouse_button_name(button: u8) -> &'static str {
match button {
3 => "right",
@@ -1854,7 +1865,9 @@ impl Tool for ClickTool {
field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release \
with the matching button code. Native Wayland: only left-button is supported \
via the virtual-pointer protocol — right/middle return an error rather than \
- silently degrading to left.".into(),
+ silently degrading to left. `modifier` holds ctrl/shift/alt/super for the \
+ click on X11. Native Wayland refuses modified pointer clicks until its input \
+ protocol can carry keyboard modifier state.".into(),
input_schema: json!({
// `pid` is conditionally required (validated in code: needed for
// window/element clicks, omitted for windowless scope="desktop"),
@@ -1875,6 +1888,7 @@ impl Tool for ClickTool {
// back-compat prose the click button-schema test asserts on.
"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},
"count":{"type":"integer"},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
"scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
@@ -1886,6 +1900,7 @@ impl Tool for ClickTool {
async fn invoke(&self, args: Value) -> ToolResult {
let cursor_id = resolve_cursor_key(&args);
+ let modifiers: Vec<String> = args.str_array("modifier");
// ── Window-less screen-absolute branch (capture_scope="desktop") ──────
// x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,
@@ -1927,9 +1942,22 @@ impl Tool for ClickTool {
overlay_glide_to_for(&cursor_id, sx as f64, sy as f64).await;
let r = tokio::task::spawn_blocking(move || {
if crate::wayland::wayland_input_enabled() {
+ if !modifiers.is_empty() {
+ anyhow::bail!(
+ "modified desktop clicks are unavailable on native Wayland: \
+ the virtual-pointer route cannot carry keyboard modifier state"
+ );
+ }
crate::wayland::click_desktop(sx, sy, n as u32, button)
} else {
- crate::input::send_click_xtest_desktop(sx, sy, button, n)
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx,
+ sy,
+ button,
+ n,
+ &modifier_refs,
+ )
}
})
.await;
@@ -2033,19 +2061,22 @@ impl Tool for ClickTool {
// Chromium can execute a genuine AT-SPI action without focus. Try
// that route before applying its background synthetic-input gate.
- let ax_result =
- tokio::task::spawn_blocking(move || crate::atspi::perform_action(pid, idx)).await;
- if let Ok(Ok((_action, suspected_noop))) = ax_result {
- let mut structured = json!({
- "path": "ax",
- "verified": false,
- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },
- });
- if suspected_noop {
- structured["escalation"] = non_ax_escalation();
+ if modifiers.is_empty() {
+ let ax_result =
+ tokio::task::spawn_blocking(move || crate::atspi::perform_action(pid, idx))
+ .await;
+ if let Ok(Ok((_action, suspected_noop))) = ax_result {
+ let mut structured = json!({
+ "path": "ax",
+ "verified": false,
+ "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },
+ });
+ if suspected_noop {
+ structured["escalation"] = non_ax_escalation();
+ }
+ return ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))
+ .with_structured(structured);
}
- return ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))
- .with_structured(structured);
}
if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
return refusal;
@@ -2055,7 +2086,38 @@ impl Tool for ClickTool {
// X11 event for toolkits that accept it.
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
let (xid2, lx, ly) = resolve_element_local_coords(pid, idx, xid_hint)?;
- crate::input::send_click(xid2, lx as i32, ly as i32, count, button)
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {
+ anyhow::bail!(
+ "modified element clicks are unavailable on native Wayland: \
+ the pointer route cannot carry keyboard modifier state"
+ );
+ }
+ // An explicit X11 foreground request needs the real XTest path
+ // even for a plain click. Some native widgets (notably GTK
+ // selectable rows) expose bounds but no AT-SPI Action and
+ // ignore a targeted XSendEvent; limiting XTest to modified
+ // clicks made those rows addressable but not selectable.
+ if delivery.is_foreground() && !crate::wayland::wayland_input_enabled() {
+ crate::input::with_x11_foreground(xid2, 80, || {
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx.round() as i32,
+ sy.round() as i32,
+ button,
+ count,
+ &modifier_refs,
+ )
+ })
+ } else {
+ crate::input::send_click_with_modifiers(
+ xid2,
+ lx as i32,
+ ly as i32,
+ count,
+ button,
+ &modifier_refs,
+ )
+ }
})
.await;
return match result {
@@ -2143,11 +2205,18 @@ impl Tool for ClickTool {
let (xi, yi) = (x as i32, y as i32);
let (output_x, output_y) = wayland_output_point.unwrap_or((xi, yi));
let cursor_id_for_task = cursor_id.clone();
+ let modifiers_for_task = modifiers.clone();
// delivery_mode: background (default) = no-focus-steal injection;
// foreground = activate the target window (EWMH) first, then inject,
// then restore prior active. Mirrors macOS/Windows.
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<&'static str> {
if crate::wayland::wayland_input_enabled() {
+ if !modifiers_for_task.is_empty() {
+ anyhow::bail!(
+ "modified coordinate clicks are unavailable on native Wayland: \
+ the pointer route cannot carry keyboard modifier state"
+ );
+ }
// Vision/pixel click on native Wayland. Mutter drops synthetic
// virtual-pointer events (the `wayland::click` warp doesn't land),
// so for a plain left single click resolve the screen pixel to the
@@ -2183,7 +2252,7 @@ impl Tool for ClickTool {
// Foreground skips the AT-SPI shortcut and does a real activated pixel
// click (the agent's escalation when background didn't land).
let inject = |fg: bool| -> anyhow::Result<&'static str> {
- if !fg && button == 1 && count == 1 {
+ if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {
if let Ok(Some(_)) = crate::atspi::perform_action_at_point(pid, xi, yi) {
return Ok("x11_atspi");
}
@@ -2197,16 +2266,39 @@ impl Tool for ClickTool {
// widget. XTest is accepted as real input and gives the
// widget keyboard focus, so a following type lands.
if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64) {
- crate::input::send_click_xtest_desktop(
+ let modifier_refs: Vec<&str> =
+ modifiers_for_task.iter().map(String::as_str).collect();
+ crate::input::send_click_xtest_desktop_with_modifiers(
sx.round() as i32,
sy.round() as i32,
button,
count,
+ &modifier_refs,
)?;
return Ok("x11_xtest_fg");
}
}
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, button, count)?;
+ if modifiers_for_task.is_empty() {
+ x11_pixel_click_no_focus_steal(
+ &cursor_id_for_task,
+ xid,
+ xi,
+ yi,
+ button,
+ count,
+ )?;
+ } else {
+ let modifier_refs: Vec<&str> =
+ modifiers_for_task.iter().map(String::as_str).collect();
+ x11_pixel_click_no_focus_steal_modifiers(
+ xid,
+ xi,
+ yi,
+ button,
+ count,
+ &modifier_refs,
+ )?;
+ }
Ok(if fg { "x11_pixel_fg" } else { "x11_pixel" })
};
if delivery.is_foreground() {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
index 73b1dd6bc7..724217369f 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
@@ -1,7 +1,7 @@
//! AX action dispatch — the preferred click/interaction path for indexed elements.
use crate::ax::bindings::*;
-use core_foundation::base::{CFRelease, CFTypeRef};
+use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};
const MAX_SELECTION_ANCESTORS: usize = 8;
@@ -97,6 +97,106 @@ pub fn nearest_container_selection_state(element_ptr: usize) -> Option<(String,
None
}
+/// A retained selection context for proving the settled result of a modified
+/// pointer click.
+///
+/// Reading only the target's `AXSelected` bit is insufficient for multi-select:
+/// AppKit can expose a transient target transition while it is still resolving
+/// the synthetic gesture, and a modifier-less outcome can replace the prior
+/// selection with the target. Keep the target and every selected sibling alive
+/// across delivery so callers can require both the intended target transition
+/// and preservation of the pre-existing selection.
+pub struct SelectionReadback {
+ role: String,
+ target: AXUIElementRef,
+ peer_model_observed: bool,
+ previously_selected_peers: Vec<AXUIElementRef>,
+}
+
+impl SelectionReadback {
+ pub fn role(&self) -> &str {
+ &self.role
+ }
+
+ pub fn observe(&self) -> Option<(bool, bool)> {
+ let target_selected = unsafe { copy_bool_attr(self.target, "AXSelected") }?;
+ let peers_preserved = self.peer_model_observed
+ && self
+ .previously_selected_peers
+ .iter()
+ .all(|&peer| unsafe { copy_bool_attr(peer, "AXSelected") } == Some(true));
+ Some((target_selected, peers_preserved))
+ }
+}
+
+impl Drop for SelectionReadback {
+ fn drop(&mut self) {
+ unsafe {
+ CFRelease(self.target as CFTypeRef);
+ for peer in self.previously_selected_peers.drain(..) {
+ CFRelease(peer as CFTypeRef);
+ }
+ }
+ }
+}
+
+/// Capture the nearest selectable container and its currently-selected peers.
+/// Returns `None` when the platform does not expose a readable selection model;
+/// callers must then leave the action unverifiable instead of inventing proof.
+pub fn capture_nearest_container_selection(element_ptr: usize) -> Option<SelectionReadback> {
+ let mut current = element_ptr as AXUIElementRef;
+ let mut owns_current = false;
+
+ for _ in 0..MAX_SELECTION_ANCESTORS {
+ let role = unsafe { copy_string_attr(current, "AXRole") }.unwrap_or_default();
+ if is_selectable_container_role(&role)
+ && unsafe { copy_bool_attr(current, "AXSelected") }.is_some()
+ {
+ if !owns_current {
+ unsafe { CFRetain(current as CFTypeRef) };
+ }
+ let target = current;
+ let parent = unsafe { copy_element_attr(target, "AXParent") };
+ let mut peers = Vec::new();
+ let mut peer_model_observed = false;
+ if let Some(parent) = parent {
+ for child in unsafe { copy_children(parent) } {
+ let is_target =
+ unsafe { CFEqual(child as CFTypeRef, target as CFTypeRef) != 0 };
+ peer_model_observed |= is_target;
+ if !is_target && unsafe { copy_bool_attr(child, "AXSelected") } == Some(true) {
+ peers.push(child);
+ } else {
+ unsafe { CFRelease(child as CFTypeRef) };
+ }
+ }
+ unsafe { CFRelease(parent as CFTypeRef) };
+ }
+ return Some(SelectionReadback {
+ role,
+ target,
+ peer_model_observed,
+ previously_selected_peers: peers,
+ });
+ }
+
+ let parent = unsafe { copy_element_attr(current, "AXParent") };
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ let Some(parent) = parent else {
+ return None;
+ };
+ current = parent;
+ owns_current = true;
+ }
+
+ if owns_current {
+ unsafe { CFRelease(current as CFTypeRef) };
+ }
+ None
+}
+
fn ensure_ax_enabled(enabled: Option<bool>, action: &str) -> anyhow::Result<()> {
if enabled == Some(false) {
anyhow::bail!(
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
index 857ef2400f..2b9baf8e55 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
@@ -216,6 +216,53 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {
result
}
+/// Hold physical modifier keys on the global HID queue around one pointer
+/// gesture, then release them in reverse order even when the gesture fails.
+///
+/// This is only safe inside an exact-window foreground activation guard. The
+/// caller receives the live modifier flags so every mouse event in the gesture
+/// carries the same state as the physical key transitions.
+pub(super) fn with_global_modifier_keys<T>(
+ modifiers: &[&str],
+ gesture: impl FnOnce(CGEventFlags) -> anyhow::Result<T>,
+) -> anyhow::Result<T> {
+ use core_graphics::event::CGEventTapLocation;
+
+ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
+ let mut active_flags = CGEventFlags::CGEventFlagNull;
+ let mut pressed = Vec::new();
+
+ for modifier in modifiers {
+ let Some((key_code, flag)) = modifier_key_code_and_flag(modifier) else {
+ continue;
+ };
+ if pressed
+ .iter()
+ .any(|(pressed_code, _)| *pressed_code == key_code)
+ {
+ continue;
+ }
+ active_flags |= flag;
+ if let Err(error) = post_global_key(
+ &source,
+ key_code,
+ true,
+ active_flags,
+ CGEventTapLocation::HID,
+ ) {
+ release_global_modifiers(&source, &pressed, active_flags, CGEventTapLocation::HID);
+ return Err(error);
+ }
+ pressed.push((key_code, flag));
+ std::thread::sleep(std::time::Duration::from_millis(8));
+ }
+
+ let result = gesture(active_flags);
+ release_global_modifiers(&source, &pressed, active_flags, CGEventTapLocation::HID);
+ result
+}
+
fn post_global_key(
source: &CGEventSource,
key_code: u16,
@@ -257,6 +304,57 @@ fn modifier_key_code_and_flag(modifier: &str) -> Option<(u16, CGEventFlags)> {
}
}
+/// Hold PID-routed modifier keys around one pointer gesture and release them in
+/// reverse order even when the gesture fails.
+///
+/// Mouse-event flag bits alone are not a sufficient physical-modifier model for
+/// every AppKit host (Finder's collection views are a notable example). Pairing
+/// the flagged mouse events with real modifier down/up transitions gives the
+/// target the same ordered state it receives for a keyboard chord without
+/// putting those transitions on the global HID queue.
+pub(super) fn with_pid_modifier_keys<T>(
+ pid: i32,
+ modifiers: &[&str],
+ gesture: impl FnOnce() -> anyhow::Result<T>,
+) -> anyhow::Result<T> {
+ let mut active_flags = CGEventFlags::CGEventFlagNull;
+ let mut pressed = Vec::new();
+ for modifier in modifiers {
+ let Some((key_code, flag)) = modifier_key_code_and_flag(modifier) else {
+ continue;
+ };
+ if pressed
+ .iter()
+ .any(|(pressed_code, _)| *pressed_code == key_code)
+ {
+ continue;
+ }
+ active_flags |= flag;
+ if let Err(error) = post_key(pid, key_code, true, active_flags) {
+ release_pid_modifiers(pid, &pressed, active_flags);
+ return Err(error);
+ }
+ pressed.push((key_code, flag));
+ std::thread::sleep(std::time::Duration::from_millis(8));
+ }
+
+ let result = gesture();
+ release_pid_modifiers(pid, &pressed, active_flags);
+ result
+}
+
+fn release_pid_modifiers(
+ pid: i32,
+ pressed: &[(u16, CGEventFlags)],
+ mut active_flags: CGEventFlags,
+) {
+ for &(key_code, flag) in pressed.iter().rev() {
+ active_flags.remove(flag);
+ let _ = post_key(pid, key_code, false, active_flags);
+ std::thread::sleep(std::time::Duration::from_millis(8));
+ }
+}
+
/// Type Unicode text into the frontmost application through the global HID
/// queue. This is the desktop-scope counterpart to PID-routed `type_text` and
/// mirrors computer-server's frontmost pynput typing behavior.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
index 6a210126e0..f3a2e3f2aa 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
@@ -43,10 +43,54 @@ pub fn click_at_xy(
/// (it lands on whatever is visually on top at the point) — exactly the
/// foreground, vision-driven model that complements the background contract.
pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhow::Result<()> {
+ click_at_xy_desktop_inner(x, y, count, button, &[], false)
+}
+
+/// Global HID click with physical modifier transitions. The caller must guard
+/// the exact foreground window because this transport has no pid addressing.
+pub fn click_at_xy_desktop_with_modifiers(
+ x: f64,
+ y: f64,
+ count: usize,
+ button: &str,
+ modifiers: &[&str],
+) -> anyhow::Result<()> {
+ click_at_xy_desktop_inner(x, y, count, button, modifiers, false)
+}
+
+/// Global HID modifier click that restores the user's hardware cursor after
+/// the event pair has been queued.
+pub fn click_at_xy_desktop_with_modifiers_preserving_cursor(
+ x: f64,
+ y: f64,
+ count: usize,
+ button: &str,
+ modifiers: &[&str],
+) -> anyhow::Result<()> {
+ click_at_xy_desktop_inner(x, y, count, button, modifiers, true)
+}
+
+fn click_at_xy_desktop_inner(
+ x: f64,
+ y: f64,
+ count: usize,
+ button: &str,
+ modifiers: &[&str],
+ preserve_cursor: bool,
+) -> anyhow::Result<()> {
use core_graphics::display::CGDisplay;
use core_graphics::event::CGEventTapLocation;
let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
+ let prior = if preserve_cursor {
+ Some(
+ CGEvent::new(source.clone())
+ .map_err(|_| anyhow::anyhow!("CGEvent::new failed"))?
+ .location(),
+ )
+ } else {
+ None
+ };
let point = CGPoint::new(x, y);
let (down_ty, up_ty, btn) = match button {
"right" => (
@@ -75,16 +119,40 @@ pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhow
// warped point, not the pre-warp one.
unsafe { CGAssociateMouseAndMouseCursorPosition(true) };
std::thread::sleep(std::time::Duration::from_millis(40));
- for _ in 0..count.max(1) {
- let down = CGEvent::new_mouse_event(source.clone(), down_ty, point, btn)
- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;
- down.post(CGEventTapLocation::HID);
- std::thread::sleep(std::time::Duration::from_millis(20));
- let up = CGEvent::new_mouse_event(source.clone(), up_ty, point, btn)
- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;
- up.post(CGEventTapLocation::HID);
+ let result = super::keyboard::with_global_modifier_keys(modifiers, |flags| {
+ for pair_index in 0..count.max(1) {
+ let down = CGEvent::new_mouse_event(source.clone(), down_ty, point, btn)
+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;
+ down.set_flags(flags);
+ down.set_integer_value_field(
+ core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE,
+ (pair_index + 1) as i64,
+ );
+ down.post(CGEventTapLocation::HID);
+ std::thread::sleep(std::time::Duration::from_millis(28));
+ let up = CGEvent::new_mouse_event(source.clone(), up_ty, point, btn)
+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;
+ up.set_flags(flags);
+ up.set_integer_value_field(
+ core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE,
+ (pair_index + 1) as i64,
+ );
+ up.post(CGEventTapLocation::HID);
+ if count > 1 {
+ std::thread::sleep(std::time::Duration::from_millis(80));
+ }
+ }
+ Ok(())
+ });
+
+ // Let AppKit consume the up event before the pointer is restored. The
+ // exact foreground guard remains active around this entire helper.
+ std::thread::sleep(std::time::Duration::from_millis(40));
+ if let Some(prior) = prior {
+ let _ = CGDisplay::warp_mouse_cursor_position(prior);
+ unsafe { CGAssociateMouseAndMouseCursorPosition(true) };
}
- Ok(())
+ result
}
/// Move the real hardware cursor to a logical desktop point.
@@ -145,17 +213,7 @@ pub fn scroll_wheel_desktop(
/// used only by approved, bounded setup flows after an accessibility element
/// has proven the exact target point and window.
pub fn click_at_xy_desktop_preserving_cursor(x: f64, y: f64) -> anyhow::Result<()> {
- use core_graphics::display::CGDisplay;
-
- let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
- .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
- let prior = CGEvent::new(source)
- .map_err(|_| anyhow::anyhow!("CGEvent::new failed"))?
- .location();
- let result = click_at_xy_desktop(x, y, 1, "left");
- let _ = CGDisplay::warp_mouse_cursor_position(prior);
- unsafe { CGAssociateMouseAndMouseCursorPosition(true) };
- result
+ click_at_xy_desktop_inner(x, y, 1, "left", &[], true)
}
extern "C" {
@@ -219,70 +277,71 @@ fn click_at_xy_inner(
.subsec_nanos() as i64
});
- // Prime the target window's cursor-tracking state with a leading mouseMoved
- // so an AppKit NSButton / NSView hit-tests the down at the right point
- // (Swift recipe Step 3). Without it the synthetic mouseDown on a
- // backgrounded AppKit control is silently ignored.
- post_mouse_moved_primer(pid, &source, point, window_local, wid, click_group_id);
- std::thread::sleep(std::time::Duration::from_millis(12));
-
- for pair_index in 0..count {
- let click_state = (pair_index + 1) as i64;
-
- let down = CGEvent::new_mouse_event(
- source.clone(),
- CGEventType::LeftMouseDown,
- point,
- CGMouseButton::Left,
- )
- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;
- if flags != CGEventFlags::CGEventFlagNull {
- down.set_flags(flags);
- }
-
- post_mouse_event(
- pid,
- &down,
- window_local,
- wid,
- click_group_id,
- click_state,
- 0,
- 3,
- );
- // 28 ms down→up gap: an NSButton's mouseDown enters a modal tracking
- // loop that polls for the matching mouseUp; too tight a gap can race the
- // loop's first poll and the click is dropped. 16 ms was under that
- // threshold for some controls.
- std::thread::sleep(std::time::Duration::from_millis(28));
-
- let up = CGEvent::new_mouse_event(
- source.clone(),
- CGEventType::LeftMouseUp,
- point,
- CGMouseButton::Left,
- )
- .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;
- if flags != CGEventFlags::CGEventFlagNull {
- up.set_flags(flags);
- }
+ super::keyboard::with_pid_modifier_keys(pid, modifiers, || {
+ // Prime the target window's cursor-tracking state with a leading
+ // mouseMoved so an AppKit NSButton / NSView hit-tests the down at the
+ // right point. Without it the synthetic mouseDown on a backgrounded
+ // AppKit control is silently ignored.
+ post_mouse_moved_primer(pid, &source, point, window_local, wid, click_group_id);
+ std::thread::sleep(std::time::Duration::from_millis(12));
+
+ for pair_index in 0..count {
+ let click_state = (pair_index + 1) as i64;
+
+ let down = CGEvent::new_mouse_event(
+ source.clone(),
+ CGEventType::LeftMouseDown,
+ point,
+ CGMouseButton::Left,
+ )
+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(down) failed"))?;
+ if flags != CGEventFlags::CGEventFlagNull {
+ down.set_flags(flags);
+ }
+
+ post_mouse_event(
+ pid,
+ &down,
+ window_local,
+ wid,
+ click_group_id,
+ click_state,
+ 0,
+ 3,
+ );
+ // 28 ms down→up gap: an NSButton's mouseDown enters a modal
+ // tracking loop that polls for the matching mouseUp; too tight a
+ // gap can race the loop's first poll and the click is dropped.
+ std::thread::sleep(std::time::Duration::from_millis(28));
+
+ let up = CGEvent::new_mouse_event(
+ source.clone(),
+ CGEventType::LeftMouseUp,
+ point,
+ CGMouseButton::Left,
+ )
+ .map_err(|_| anyhow::anyhow!("CGEvent::new_mouse_event(up) failed"))?;
+ if flags != CGEventFlags::CGEventFlagNull {
+ up.set_flags(flags);
+ }
+
+ post_mouse_event(
+ pid,
+ &up,
+ window_local,
+ wid,
+ click_group_id,
+ click_state,
+ 0,
+ 3,
+ );
- post_mouse_event(
- pid,
- &up,
- window_local,
- wid,
- click_group_id,
- click_state,
- 0,
- 3,
- );
-
- if count > 1 {
- std::thread::sleep(std::time::Duration::from_millis(80));
+ if count > 1 {
+ std::thread::sleep(std::time::Duration::from_millis(80));
+ }
}
- }
- Ok(())
+ Ok(())
+ })
}
/// Prepare a raw background pixel click by making the target AppKit-active
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index 17dd6fcff3..67f038d4dd 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -66,9 +66,14 @@ struct SelectionPixelTarget {
window_y: f64,
}
-fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -> bool {
+fn selection_readback_confirms(
+ before: bool,
+ after: bool,
+ has_modifiers: bool,
+ prior_selected_peers_preserved: bool,
+) -> bool {
if has_modifiers {
- after != before
+ after != before && prior_selected_peers_preserved
} else {
after
}
@@ -76,6 +81,8 @@ fn selection_readback_confirms(before: bool, after: bool, has_modifiers: bool) -
const SELECTION_READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(500);
const SELECTION_READBACK_POLL: std::time::Duration = std::time::Duration::from_millis(25);
+const SELECTION_READBACK_SETTLE: std::time::Duration = std::time::Duration::from_millis(200);
+const SELECTION_READBACK_STABILITY: std::time::Duration = std::time::Duration::from_millis(250);
fn pixel_activation_policy(
button: &str,
@@ -183,7 +190,7 @@ fn def() -> &'static ToolDef {
"delivery_mode": {
"type": "string",
"enum": ["background", "foreground"],
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require \"foreground\" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\"."
},
"scope": {
"type": "string",
@@ -294,12 +301,21 @@ impl Tool for ClickTool {
.update_position(&cursor_key, sx, sy);
let btn = button.clone();
+ let desktop_modifiers: Vec<String> = args.str_array("modifier");
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
// Desktop scope is explicitly foreground and vision-driven: post
// at the global HID tap so WindowServer delivers to the window
// actually visible at this point. PID-posting here would silently
// turn the foreground contract back into background delivery.
- crate::input::mouse::click_at_xy_desktop(sx, sy, count, &btn)
+ let modifier_refs: Vec<&str> =
+ desktop_modifiers.iter().map(String::as_str).collect();
+ crate::input::mouse::click_at_xy_desktop_with_modifiers(
+ sx,
+ sy,
+ count,
+ &btn,
+ &modifier_refs,
+ )
})
.await;
let button_label = match button.as_str() {
@@ -386,6 +402,25 @@ impl Tool for ClickTool {
let debug_image_out = args.opt_str("debug_image_out");
let modifiers: Vec<String> = args.str_array("modifier");
+ // PID-routed key transitions can look correct for one AX poll and then
+ // collapse to a plain click once AppKit resolves the gesture. Refuse
+ // that false-success path. The explicit foreground rung uses physical
+ // HID modifier transitions under an exact-window activation guard.
+ if !modifiers.is_empty() && (!delivery_mode.is_foreground() || window_id.is_none()) {
+ return ToolResult::error(
+ "click modifiers require delivery_mode:\"foreground\" and window_id on macOS; \
+ background PID-routed events cannot preserve live modifier-key state",
+ )
+ .with_structured(serde_json::json!({
+ "code": "background_unavailable",
+ "effect": "refused",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "macOS modifier clicks require exact-window HID delivery so the target observes live modifier state"
+ }
+ }));
+ }
+
if let (Some(idx), Some(wid)) = (element_index, window_id) {
// ── AX element path ────────────────────────────────────────────
// Retain the element out of the cache so it can't be freed by a
@@ -447,9 +482,22 @@ impl Tool for ClickTool {
.update_position(&cursor_key, cx, cy);
let mods_owned = modifiers.clone();
+ let foreground = delivery_mode.is_foreground();
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- crate::input::mouse::middle_click_at_xy(pid, cx, cy, &m)
+ if foreground && !m.is_empty() {
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ || {
+ crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(
+ cx, cy, 1, "middle", &m,
+ )
+ },
+ )
+ } else {
+ crate::input::mouse::middle_click_at_xy(pid, cx, cy, &m)
+ }
})
.await;
return match result {
@@ -542,24 +590,36 @@ impl Tool for ClickTool {
tokio::task::spawn_blocking(move || {
if foreground {
let mut outcome = None;
- let fronted = crate::input::skylight::with_foreground_assist(
- pid as libc::pid_t,
- wid,
- || {
- outcome = Some(perform_ax_click(
- element_ptr,
- idx,
- pid,
- wid,
- &action_clone,
- &ck,
- selection_pixel,
- &selection_modifiers,
- )?);
- std::thread::sleep(std::time::Duration::from_millis(150));
- Ok(())
- },
- )?;
+ let has_modifiers = !selection_modifiers.is_empty();
+ let action = || {
+ outcome = Some(perform_ax_click(
+ element_ptr,
+ idx,
+ pid,
+ wid,
+ &action_clone,
+ &ck,
+ selection_pixel,
+ &selection_modifiers,
+ foreground,
+ )?);
+ std::thread::sleep(std::time::Duration::from_millis(150));
+ Ok(())
+ };
+ let fronted = if has_modifiers {
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ action,
+ )?;
+ true
+ } else {
+ crate::input::skylight::with_foreground_assist(
+ pid as libc::pid_t,
+ wid,
+ action,
+ )?
+ };
let outcome = outcome.ok_or_else(|| {
anyhow::anyhow!("foreground AX click did not execute")
})?;
@@ -574,6 +634,7 @@ impl Tool for ClickTool {
&ck,
selection_pixel,
&selection_modifiers,
+ false,
)
.map(|outcome| (outcome, false))
}
@@ -885,8 +946,18 @@ impl Tool for ClickTool {
"click.pixel",
|| async move {
tokio::task::spawn_blocking(move || {
+ let has_modifiers = !mods_owned.is_empty();
let do_click = move || -> anyhow::Result<()> {
let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ if fg && !m.is_empty() {
+ return crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(
+ screen_x,
+ screen_y,
+ count,
+ &button_kind,
+ &m,
+ );
+ }
match button_kind.as_str() {
"right" => {
if let Some(wid) = window_id {
@@ -931,10 +1002,22 @@ impl Tool for ClickTool {
// Foreground rung: brief front → click → restore.
// Returns whether the window was ACTUALLY fronted, so the
// reported `path` honestly reflects the rung that ran.
- match (fg, window_id) {
- (true, Some(wid)) => crate::input::skylight::with_foreground_assist(
- pid as libc::pid_t, wid, do_click,
- ),
+ match (fg, window_id, has_modifiers) {
+ (true, Some(wid), true) => {
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ do_click,
+ )
+ .map(|_| true)
+ }
+ (true, Some(wid), false) => {
+ crate::input::skylight::with_foreground_assist(
+ pid as libc::pid_t,
+ wid,
+ do_click,
+ )
+ }
_ => do_click().map(|_| false),
}
})
@@ -1025,6 +1108,7 @@ fn perform_ax_click(
cursor_key: &str,
selection_pixel: Option<SelectionPixelTarget>,
modifiers: &[String],
+ foreground: bool,
) -> anyhow::Result<(String, bool, bool, bool, bool)> {
let ax_action = map_action(action_str);
let element = element_ptr as AXUIElementRef;
@@ -1064,41 +1148,75 @@ fn perform_ax_click(
}
}
- if let (Some(target), Some((selected_role, before))) = (
+ if let (Some(target), Some(selection)) = (
selection_pixel,
- crate::input::ax_actions::nearest_container_selection_state(element_ptr),
+ crate::input::ax_actions::capture_nearest_container_selection(element_ptr),
) {
+ let selected_role = selection.role().to_owned();
+ let Some((before, _)) = selection.observe() else {
+ anyhow::bail!("selection target stopped exposing AXSelected before delivery");
+ };
let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::mouse::click_at_xy_with_window_local(
- pid,
- target.screen_x,
- target.screen_y,
- target.window_x,
- target.window_y,
- window_id,
- 1,
- &modifier_refs,
- )?;
+ if foreground && !modifier_refs.is_empty() {
+ crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(
+ target.screen_x,
+ target.screen_y,
+ 1,
+ "left",
+ &modifier_refs,
+ )?;
+ } else {
+ crate::input::mouse::click_at_xy_with_window_local(
+ pid,
+ target.screen_x,
+ target.screen_y,
+ target.window_x,
+ target.window_y,
+ window_id,
+ 1,
+ &modifier_refs,
+ )?;
+ }
+ // AppKit may publish a transient AXSelected transition while the
+ // event queue is still resolving the gesture. Let it settle before
+ // accepting a candidate, then require the same state to survive a
+ // second observation. A modified selection additionally preserves
+ // every peer that was selected before delivery.
+ std::thread::sleep(SELECTION_READBACK_SETTLE);
let deadline = std::time::Instant::now() + SELECTION_READBACK_TIMEOUT;
loop {
- if let Some((_, after)) =
- crate::input::ax_actions::nearest_container_selection_state(element_ptr)
- {
- let verified =
- selection_readback_confirms(before, after, !modifiers.is_empty());
+ if let Some((after, peers_preserved)) = selection.observe() {
+ let verified = selection_readback_confirms(
+ before,
+ after,
+ !modifiers.is_empty(),
+ peers_preserved,
+ );
if verified {
- return Ok((
- format!(
- "✅ Selected nearest {selected_role} for [{idx}] {role} \
- \"{title}\"; AX selection write was unavailable, so a \
- coordinate click was delivered and confirmed by AXSelected \
- read-back."
- ),
- false,
- false,
- true,
- true,
- ));
+ std::thread::sleep(SELECTION_READBACK_STABILITY);
+ if let Some((stable_after, stable_peers_preserved)) = selection.observe() {
+ if stable_after == after
+ && selection_readback_confirms(
+ before,
+ stable_after,
+ !modifiers.is_empty(),
+ stable_peers_preserved,
+ )
+ {
+ return Ok((
+ format!(
+ "✅ Selected nearest {selected_role} for [{idx}] {role} \
+ \"{title}\"; AX selection write was unavailable, so a \
+ coordinate click was delivered and confirmed by stable \
+ AXSelected read-back."
+ ),
+ false,
+ false,
+ true,
+ true,
+ ));
+ }
+ }
}
}
if std::time::Instant::now() >= deadline {
@@ -1106,6 +1224,17 @@ fn perform_ax_click(
}
std::thread::sleep(SELECTION_READBACK_POLL);
}
+ if modifiers.is_empty() {
+ anyhow::bail!(
+ "coordinate click did not produce a stable AXSelected transition; \
+ retry after a fresh snapshot"
+ );
+ }
+ anyhow::bail!(
+ "foreground modified coordinate click did not produce a stable AXSelected \
+ transition while preserving the prior selection; take a fresh snapshot \
+ before retrying"
+ );
}
}
@@ -1114,7 +1243,7 @@ fn perform_ax_click(
// Some collection rows claim a click-like action but Finder returns
// kAXErrorCannotComplete. Use the same verified selection fallback
// before surfacing the dispatch error.
- if ax_action == "AXPress" {
+ if ax_action == "AXPress" && modifiers.is_empty() {
if let Some(selected_role) =
crate::input::ax_actions::select_nearest_container(element_ptr)
{
@@ -1223,16 +1352,17 @@ mod selection_fallback_tests {
#[test]
fn plain_click_requires_selected_readback() {
- assert!(selection_readback_confirms(false, true, false));
- assert!(selection_readback_confirms(true, true, false));
- assert!(!selection_readback_confirms(false, false, false));
+ assert!(selection_readback_confirms(false, true, false, true));
+ assert!(selection_readback_confirms(true, true, false, true));
+ assert!(!selection_readback_confirms(false, false, false, true));
}
#[test]
- fn modified_click_requires_a_selection_transition() {
- assert!(selection_readback_confirms(false, true, true));
- assert!(selection_readback_confirms(true, false, true));
- assert!(!selection_readback_confirms(true, true, true));
+ fn modified_click_requires_a_transition_and_preserves_prior_selection() {
+ assert!(selection_readback_confirms(false, true, true, true));
+ assert!(selection_readback_confirms(true, false, true, true));
+ assert!(!selection_readback_confirms(true, true, true, true));
+ assert!(!selection_readback_confirms(false, true, true, false));
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs
index ed6f812713..6d01dc1b52 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/invoke_menu.rs
@@ -238,25 +238,43 @@ fn focus_exact_window(pid: i32, window_id: u32) -> Result<(), String> {
CFRelease(target as CFTypeRef);
}
- let deadline = std::time::Instant::now() + Duration::from_millis(600);
+ // AXFocusedWindow can lead AppKit's native `isKeyWindow` state while the
+ // WindowServer activation is still settling. Menu validation observes the
+ // latter. Require the exact app/window pair to remain stable briefly so a
+ // loaded desktop cannot expose a transiently disabled contextual item.
+ let deadline = std::time::Instant::now() + Duration::from_millis(800);
+ let mut stable_since = None;
loop {
- if exact_window_is_focused(
+ let now = std::time::Instant::now();
+ if exact_window_is_ready(
+ crate::apps::frontmost_pid(),
+ pid,
crate::ax::bindings::focused_window_id_of_pid(pid),
window_id,
) {
- return Ok(());
+ let since = stable_since.get_or_insert(now);
+ if now.duration_since(*since) >= Duration::from_millis(120) {
+ return Ok(());
+ }
+ } else {
+ stable_since = None;
}
- if std::time::Instant::now() >= deadline {
+ if now >= deadline {
return Err(format!(
- "invoke_menu: target window {window_id} did not become the focused window"
+ "invoke_menu: target window {window_id} did not become stably key and frontmost"
));
}
std::thread::sleep(Duration::from_millis(20));
}
}
-fn exact_window_is_focused(focused_window_id: Option<u32>, target_window_id: u32) -> bool {
- focused_window_id == Some(target_window_id)
+fn exact_window_is_ready(
+ frontmost_pid: Option<i32>,
+ target_pid: i32,
+ focused_window_id: Option<u32>,
+ target_window_id: u32,
+) -> bool {
+ frontmost_pid == Some(target_pid) && focused_window_id == Some(target_window_id)
}
fn refusal(message: String) -> ToolResult {
@@ -371,9 +389,10 @@ mod tests {
}
#[test]
- fn menu_focus_requires_the_exact_requested_window() {
- assert!(exact_window_is_focused(Some(42), 42));
- assert!(!exact_window_is_focused(Some(41), 42));
- assert!(!exact_window_is_focused(None, 42));
+ fn menu_focus_requires_the_exact_frontmost_app_and_window() {
+ assert!(exact_window_is_ready(Some(7), 7, Some(42), 42));
+ assert!(!exact_window_is_ready(Some(8), 7, Some(42), 42));
+ assert!(!exact_window_is_ready(Some(7), 7, Some(41), 42));
+ assert!(!exact_window_is_ready(Some(7), 7, None, 42));
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index 2e9f63a977..18e70f8467 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -35,9 +35,10 @@ fn def() -> &'static ToolDef {
• `background` (default): post to the pid WITHOUT fronting/raising — the \
auth-message path (Chromium-safe). With element_index it focuses that AX \
element first. `window_id` only targets; it does not raise.\n\
- • `foreground`: guard and briefly front the exact window, send a genuine HID \
- key transition so Chromium content and native menu equivalents both receive \
- it, then restore prior frontmost. Requires window_id (and no element_index).\n\n\
+ • `foreground`: guard and briefly front the exact window, focus an addressed AX \
+ element when supplied, send a genuine HID key transition so Chromium content, \
+ inline editors, and native menu equivalents receive it, then restore prior \
+ frontmost. Requires window_id.\n\n\
A key press is never driver-verifiable → effect:\"unverifiable\"; confirm via \
screenshot. Key names: return, tab, escape, up/down/left/right, space, delete, \
home, end, pageup, pagedown, f1-f12, plus any letter or digit. \
@@ -241,16 +242,26 @@ impl Tool for PressKeyTool {
// is accepted by both. Skipped when px-focus already handled the
// target-specific foreground transition.
if fg && !px_focus {
- if let Some(wid) = window_id {
- if element_index.is_none() {
- crate::input::skylight::with_foreground_hid_activation(
- pid as libc::pid_t,
- wid,
- || crate::input::keyboard::press_key_bare_global(&key, &m),
- )?;
- return Ok(());
- }
- }
+ let wid = window_id.ok_or_else(|| {
+ anyhow::anyhow!(
+ "delivery_mode=foreground requires window_id for press_key"
+ )
+ })?;
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ || {
+ // Activation can change the first responder, so
+ // repeat the best-effort AX focus write inside
+ // the guarded foreground interval immediately
+ // before the physical key transition.
+ if let Some(element_ptr) = pre_focus_ptr {
+ let _ = crate::input::ax_actions::focus_element(element_ptr);
+ }
+ crate::input::keyboard::press_key_bare_global(&key, &m)
+ },
+ )?;
+ return Ok(());
}
// background (default): auth-envelope post, no raise.
crate::input::keyboard::press_key(pid, &key, &m)
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
index 0bc83e52f6..c2b1d4c9a0 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
@@ -597,14 +597,18 @@ fn send_click_synthesized_mods_impl(
// poll GetKeyState (WPF, Chromium) observe the held state, then released
// after the click loop below.
let (mod_downs, mod_ups) = crate::input::keyboard::modifier_hold_inputs(modifiers);
+ let mut sent_ok = true;
if !mod_downs.is_empty() {
- SendInput(&mod_downs, std::mem::size_of::<INPUT>() as i32);
+ let sent = SendInput(&mod_downs, std::mem::size_of::<INPUT>() as i32);
+ sent_ok = sent as usize == mod_downs.len();
sleep(Duration::from_millis(5));
}
let count = count.max(1);
- let mut sent_ok = true;
for i in 0..count {
+ if !sent_ok {
+ break;
+ }
// Only the move record carries absolute coordinates. Button-only
// records act at the current pointer position; adding ABSOLUTE to
// them can prevent retained-mode controls from seeing the press.
@@ -621,7 +625,13 @@ fn send_click_synthesized_mods_impl(
// Release any held modifiers (reverse order) before restoring z-order.
if !mod_ups.is_empty() {
- SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);
+ let released = SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);
+ if released as usize != mod_ups.len() {
+ // A second release attempt is safe and reduces the chance of a
+ // partially inserted chord leaving system modifier state held.
+ let _ = SendInput(&mod_ups, std::mem::size_of::<INPUT>() as i32);
+ sent_ok = false;
+ }
}
// Let the target process mouse-up before any background-route restore.
@@ -656,7 +666,10 @@ fn send_click_synthesized_mods_impl(
}
drop(noactivate);
if !sent_ok {
- bail!("SendInput inserted fewer mouse events than expected for the foreground click.");
+ bail!(
+ "SendInput inserted fewer modifier or mouse events than expected for the \
+ foreground click."
+ );
}
if activate {
let foreground_root = GetAncestor(GetForegroundWindow(), GA_ROOT);
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 2927ff2994..5999e35322 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -2801,6 +2801,20 @@ impl Tool for ClickTool {
// a modifier passed to a background click is necessarily ignored there.
let modifiers: Vec<String> = args.str_array("modifier");
let delivery = DeliveryMode::from_args(&args);
+ if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {
+ return ToolResult::error(
+ "click modifiers require delivery_mode:\"foreground\" on Windows; UIA and \
+ PostMessage cannot carry live modifier-key state",
+ )
+ .with_structured(json!({
+ "code": "background_unavailable",
+ "effect": "refused",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "Windows modifier clicks require SendInput so the target observes live modifier state"
+ }
+ }));
+ }
// For every non-foreground click, mark the target window
// non-activatable (WS_EX_NOACTIVATE) for the duration so a target that
// self-activates in its UIA-Invoke / click handler (WPF
@@ -2922,8 +2936,19 @@ impl Tool for ClickTool {
let prev_fg_addr = unsafe {
windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
};
+ let mods_owned = modifiers.clone();
+ let activate = delivery == DeliveryMode::Foreground;
let send_result = tokio::task::spawn_blocking(move || {
- crate::input::send_click_synthesized(hwnd, tx, ty, count, &btn_fg)
+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ if activate && !mod_refs.is_empty() {
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, tx, ty, count, &btn_fg, &mod_refs,
+ )
+ } else {
+ crate::input::send_click_synthesized_mods(
+ hwnd, tx, ty, count, &btn_fg, &mod_refs,
+ )
+ }
})
.await;
tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
@@ -3028,6 +3053,27 @@ impl Tool for ClickTool {
// Foreground delivery is a real SendInput tap at (cx,cy); if the
// element is off-screen, scroll it into view and re-resolve so the
// tap doesn't land on the taskbar, else keep the clean failure.
+ // Modified selection needs one stronger preflight: some WPF
+ // ScrollViewers report a clipped item as on-screen because its
+ // stale rectangle is still inside the outer HWND. Ask the item
+ // to scroll itself into view before trusting that rectangle.
+ let (cx, cy) = if !modifiers.is_empty() {
+ self.state
+ .element_cache
+ .get_element_retained(pid, hwnd, idx)
+ .and_then(|retained| {
+ retained.is_uia().then(|| unsafe {
+ crate::uia::scroll::scroll_into_view_and_recenter(
+ hwnd,
+ retained.as_ptr(),
+ )
+ })
+ })
+ .flatten()
+ .unwrap_or((cx, cy))
+ } else {
+ (cx, cy)
+ };
let (cx, cy) = match resolve_onscreen_point_with_scroll(
&self.state.element_cache,
pid,
@@ -3041,11 +3087,19 @@ impl Tool for ClickTool {
Err(msg) => return ToolResult::error(msg),
};
let btn_fg = btn.clone();
+ let mods_owned = modifiers.clone();
let prev_fg_addr = unsafe {
windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
};
let send_result = tokio::task::spawn_blocking(move || {
- crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)
+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ if mod_refs.is_empty() {
+ crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)
+ } else {
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, cx, cy, count, &btn_fg, &mod_refs,
+ )
+ }
})
.await;
tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
@@ -3350,9 +3404,15 @@ impl Tool for ClickTool {
let mods_owned = modifiers.clone();
let send_result = tokio::task::spawn_blocking(move || {
let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- crate::input::send_click_synthesized_active_mods(
- hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,
- )
+ if mod_refs.is_empty() {
+ crate::input::send_click_synthesized(
+ hwnd, sx as i32, sy as i32, count, &btn,
+ )
+ } else {
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,
+ )
+ }
})
.await;
tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
diff --git a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
index bd3ea9e7e8..6ec9bfc269 100755
--- a/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
+++ b/libs/cua-driver/tests/fixtures/apps/linux/gtk3/main.py
@@ -127,6 +127,22 @@ class HarnessWindow(Gtk.Window):
self.chk_status = Gtk.Label(label="agreed=False", xalign=0)
root.pack_start(self.chk_status, False, False, 0)
+ # ── multi-selection ───────────────────────────────────────────────
+ section(root, "multi selection")
+ self.selection = Gtk.ListBox()
+ self.selection.set_selection_mode(Gtk.SelectionMode.MULTIPLE)
+ for value in ("alpha", "beta", "gamma"):
+ row = aid(Gtk.ListBoxRow(), f"selection-{value}")
+ row.set_activatable(True)
+ row.set_selectable(True)
+ row.set_can_focus(True)
+ row.add(Gtk.Label(label=value, xalign=0))
+ self.selection.add(row)
+ self.selection.connect("selected-rows-changed", self.on_selection_changed)
+ root.pack_start(self.selection, False, False, 0)
+ self.selection_status = Gtk.Label(label="selection=none", xalign=0)
+ root.pack_start(self.selection_status, False, False, 0)
+
# ── context_menu ──────────────────────────────────────────────────
section(root, "context menu")
self.ctx = aid(Gtk.Button(label="Right-click for context menu"), "btn-context")
@@ -218,6 +234,14 @@ class HarnessWindow(Gtk.Window):
def on_chk(self, c):
self.chk_status.set_text(f"agreed={c.get_active()}")
+ def on_selection_changed(self, list_box):
+ values = sorted(
+ row.get_child().get_text() for row in list_box.get_selected_rows()
+ )
+ self.selection_status.set_text(
+ "selection=" + (",".join(values) if values else "none")
+ )
+
def on_ctx_press(self, _w, ev):
if ev.button == 3:
self.menu.popup_at_pointer(ev)
diff --git a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
index 71852fefb1..fade434faf 100644
--- a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
+++ b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift
@@ -32,6 +32,7 @@ let kTextBodyAID = "txt-body"
let kTextBodyMarker = "HARNESS_TEXT_MARKER_v1"
let kTextInputAID = "txt-input"
let kTextInputMirrorAID = "lbl-input-mirror"
+let kTextInputCommitAID = "lbl-input-commit"
let kClickTargetAID = "btn-clicktarget"
let kLastActionAID = "lbl-last-action"
let kClickCountAID = "lbl-click-count"
@@ -39,6 +40,7 @@ let kSliderAID = "sld-value"
let kSliderValueAID = "lbl-slider-value"
let kCheckboxAID = "chk-agree"
let kCheckStateAID = "lbl-chk-state"
+let kSelectionStateAID = "lbl-selection-state"
let kContextButtonAID = "btn-context"
let kMenuActionAID = "lbl-menu-action"
let kScrollerAID = "scroll-tall"
@@ -51,17 +53,21 @@ let kMenuItemTitle = "Harness Test Item"
// MARK: - Controller
-final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemValidation {
+final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDataSource, NSTableViewDelegate, NSMenuItemValidation {
let window: NSWindow
let counterLabel = NSTextField(labelWithString: "counter=0")
var counterValue = 0
let textInput = NSTextField(string: "")
let textInputMirror = NSTextField(labelWithString: "")
+ let textInputCommit = NSTextField(labelWithString: "committed=none")
let lastActionLabel = NSTextField(labelWithString: "last_action=none")
let clickCountLabel = NSTextField(labelWithString: "clicks=0")
var clicks = 0
let sliderValueLabel = NSTextField(labelWithString: "slider_value=0")
let checkStateLabel = NSTextField(labelWithString: "agreed=false")
+ let selectionItems = ["alpha", "beta", "gamma"]
+ let selectionTable = NSTableView()
+ let selectionStateLabel = NSTextField(labelWithString: "selection=none")
let menuActionLabel = NSTextField(labelWithString: "menu_action=none")
let scrollOffsetLabel = NSTextField(labelWithString: "scroll_offset=0")
let accelCountLabel = NSTextField(labelWithString: "accel_fired=0")
@@ -146,11 +152,14 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVa
textInput.translatesAutoresizingMaskIntoConstraints = false
textInputMirror.setAccessibilityIdentifier(kTextInputMirrorAID)
textInputMirror.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)
+ textInputCommit.setAccessibilityIdentifier(kTextInputCommitAID)
+ textInputCommit.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)
let inputRow = NSStackView()
inputRow.orientation = .horizontal
inputRow.spacing = 12
inputRow.addArrangedSubview(textInput)
inputRow.addArrangedSubview(textInputMirror)
+ inputRow.addArrangedSubview(textInputCommit)
NSLayoutConstraint.activate([
textInput.widthAnchor.constraint(equalToConstant: 240),
])
@@ -210,6 +219,26 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVa
checkRow.spacing = 12
checkRow.addArrangedSubview(checkbox)
checkRow.addArrangedSubview(checkStateLabel)
+ selectionTable.headerView = nil
+ selectionTable.allowsMultipleSelection = true
+ selectionTable.dataSource = self
+ selectionTable.delegate = self
+ let selectionColumn = NSTableColumn(identifier: NSUserInterfaceItemIdentifier("selection-column"))
+ selectionColumn.width = 140
+ selectionTable.addTableColumn(selectionColumn)
+ let selectionScroll = NSScrollView()
+ selectionScroll.documentView = selectionTable
+ selectionScroll.hasVerticalScroller = true
+ selectionScroll.borderType = .lineBorder
+ selectionScroll.translatesAutoresizingMaskIntoConstraints = false
+ selectionStateLabel.setAccessibilityIdentifier(kSelectionStateAID)
+ selectionStateLabel.font = NSFont.monospacedSystemFont(ofSize: 14, weight: .regular)
+ checkRow.addArrangedSubview(selectionScroll)
+ checkRow.addArrangedSubview(selectionStateLabel)
+ NSLayoutConstraint.activate([
+ selectionScroll.widthAnchor.constraint(equalToConstant: 150),
+ selectionScroll.heightAnchor.constraint(equalToConstant: 68),
+ ])
content.addArrangedSubview(checkRow)
// context_menu — NSButton with an attached NSMenu. Right-click opens the
@@ -359,6 +388,26 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVa
checkStateLabel.stringValue = "agreed=\(sender.state == .on)"
}
+ func numberOfRows(in tableView: NSTableView) -> Int {
+ selectionItems.count
+ }
+
+ func tableView(_ tableView: NSTableView,
+ viewFor tableColumn: NSTableColumn?, row: Int) -> NSView? {
+ let value = selectionItems[row]
+ let field = NSTextField(labelWithString: value)
+ field.isSelectable = true
+ field.setAccessibilityIdentifier("selection-\(value)")
+ return field
+ }
+
+ func tableViewSelectionDidChange(_ notification: Notification) {
+ let values = selectionTable.selectedRowIndexes.map { selectionItems[$0] }
+ selectionStateLabel.stringValue = values.isEmpty
+ ? "selection=none"
+ : "selection=\(values.joined(separator: ","))"
+ }
+
@objc private func onContextItem(_ sender: NSMenuItem) {
menuActionLabel.stringValue = "menu_action=\(sender.title)"
}
@@ -385,6 +434,13 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSMenuItemVa
}
}
+ func controlTextDidEndEditing(_ obj: Notification) {
+ guard let field = obj.object as? NSTextField else { return }
+ if field === textInput {
+ textInputCommit.stringValue = "committed=\(field.stringValue)"
+ }
+ }
+
// Click target — single source of truth for all three actions.
@objc func onClickTarget() { recordClick("click") }
func clickTargetSawDouble() { recordClick("double_click") }
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
index eb1f4dfef1..1f797c410c 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
@@ -207,6 +207,7 @@
<StackPanel>
<ListBox x:Name="LstItems"
AutomationProperties.AutomationId="lst-items"
+ SelectionMode="Extended"
Width="220" Height="80" HorizontalAlignment="Left"
SelectionChanged="OnListChanged">
<ListBoxItem Content="apple" AutomationProperties.AutomationId="lst-apple"/>
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
index 155fbeebe8..e49ce58257 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
@@ -1,6 +1,7 @@
using System;
using System.Collections.Generic;
using System.IO;
+using System.Linq;
using System.Runtime.InteropServices;
using System.Text.Json;
using System.Windows;
@@ -267,10 +268,11 @@ public partial class MainWindow : Window
private void OnListChanged(object sender, SelectionChangedEventArgs e)
{
if (LblListValue is null) return;
- if (LstItems?.SelectedItem is ListBoxItem item)
- {
- LblListValue.Text = $"selected={item.Content}";
- }
+ var selected = LstItems?.SelectedItems
+ .OfType<ListBoxItem>()
+ .Select(item => item.Content?.ToString() ?? "")
+ .Where(value => value.Length > 0);
+ LblListValue.Text = $"selected={string.Join(",", selected ?? Enumerable.Empty<string>())}";
}
private void OnMenuFileNew(object sender, RoutedEventArgs e) => LblMenuAction.Text = "menu_action=file_new";
diff --git a/libs/cua-driver/tests/fixtures/shared/scenarios.json b/libs/cua-driver/tests/fixtures/shared/scenarios.json
index 129253d05f..358ebc8e9d 100644
--- a/libs/cua-driver/tests/fixtures/shared/scenarios.json
+++ b/libs/cua-driver/tests/fixtures/shared/scenarios.json
@@ -338,7 +338,8 @@
"description": "Editable NSTextField with a mirror label. Tests type_text and set_value via AXValue on AppKit.",
"controls": {
"textbox_aid": "txt-input",
- "mirror_label_aid": "lbl-input-mirror"
+ "mirror_label_aid": "lbl-input-mirror",
+ "commit_label_aid": "lbl-input-commit"
}
},
{
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index 5669a6cdb6..7a36eb1fba 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -322,6 +322,8 @@ if [[ "${SUITE}" == native || "${SUITE}" == all ]]; then
harness_appkit_stale_element_token_fails_closed \
harness_appkit_invoke_menu_live_path \
harness_appkit_text_input \
+ harness_appkit_element_foreground_press_key_commits_edit \
+ harness_appkit_modified_click_preserves_selection \
harness_appkit_type_text_background \
harness_appkit_scroll_foreground \
harness_appkit_scroll_background \