Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): verify foreground focus before input (#3068)
Francesco Bonaccif9192dde69dd
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex ef3a8f7d38..37641828c0 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -413,8 +413,8 @@ than silently changing coordinate spaces.### `hotkey`Press a key combination — e.g. `["cmd", "c"]` for Copy, `["cmd", "shift", "4"]` for screenshot selection. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:-• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. No focus steal. `window_id` here only targets the combo; it does not raise.-• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.+• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. `window_id` here only targets the combo; it does not raise.+• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.A combo is never driver-verifiable (no read-back) → effect:"unverifiable"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field — to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then `type_text`. If an app only accepts paste, call `clipboard_write`, then `clipboard_read` and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V.@@ -423,10 +423,13 @@ Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modi**Arguments:**- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"]. items: 2–unbounded- `pid` (integer, optional): Target process ID.- `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application. default: `"window"`- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.- `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action.`display_id="primary"` is the portable desktop target in this release.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rsindex 6746c4fbb3..dc82e03304 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs@@ -795,6 +795,7 @@ fn run_text_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> O}thread::sleep(Duration::from_millis(250));assert_fixture_contains(fixture, &format!("mirror={text}"));+ assert_fixture_value(fixture, "number-input", "42");Observation::delivered(passed, Evidence::default())}@@ -818,6 +819,7 @@ fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &sttype_response.text());assert_fixture_value(fixture, "keyboard-input", &text);+ assert_fixture_value(fixture, "number-input", "42");let post_type = snapshot(fixture);let mut enter_args =@@ -837,6 +839,7 @@ fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &stenter_response.text());assert_fixture_contains(fixture, &format!("key_state=submitted:{text}"));+ assert_fixture_value(fixture, "number-input", "42");let mut passed = vec![OracleKind::FixtureState];if addressing == "px" {@@ -873,6 +876,7 @@ fn run_press_key_action(fixture: &mut Fixture, addressing: &str, delivery: &str)let mut passed = vec![OracleKind::FixtureState];passed.extend(unverified_background_protocol_oracle(&response, delivery));assert_fixture_contains(fixture, "key_state=enter");+ assert_fixture_value(fixture, "number-input", "42");Observation::delivered(passed, Evidence::default())}@@ -896,6 +900,7 @@ fn run_hotkey_action(fixture: &mut Fixture, addressing: &str, delivery: &str) ->response.text());assert_fixture_contains(fixture, "key_state=hotkey");+ assert_fixture_value(fixture, "number-input", "42");#[cfg(target_os = "macos")]if fixture.name == "electron" && addressing == "px" && delivery == "foreground" {run_macos_selection_hotkeys(fixture);diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rsindex 3726992c75..31e9da2cd0 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs@@ -718,53 +718,24 @@ fn harness_wpf_type_text() {let snap = snapshot(driver, pid, wid);let idx = ax::element_index_by_id(snap.text(), "txt-input").expect("txt-input not in snapshot");-- // WPF's TextBox needs *keyboard focus* for WM_CHAR delivery — and- // PostMessage(WM_LBUTTONDOWN) doesn't reliably transfer keyboard- // focus (WPF's input system treats posted events differently from- // real ones). Use delivery_mode:"foreground" → SendInput synthesizes- // an OS-level click that WPF treats identically to a user mouse,- // landing actual keyboard focus on the TextBox.- let _ = driver.call(- "bring_to_front",- serde_json::json!({- "pid": pid as i64, "window_id": wid- }),- );- std::thread::sleep(Duration::from_millis(300));driver.start_behavior_recording();- let _ = driver.call(- "click",- serde_json::json!({- "pid": pid as i64, "window_id": wid, "element_index": idx,- "snapshot_id": snap.snapshot_id(),- "delivery_mode": "foreground"- }),- );- std::thread::sleep(Duration::from_millis(400));-- // SendInput's restore_foreground_polling_best_effort may yank- // foreground back from the harness window between click and- // type_text. Re-assert foreground so PostMessage WM_CHAR finds- // the TextBox with keyboard focus.- let _ = driver.call(- "bring_to_front",- serde_json::json!({- "pid": pid as i64, "window_id": wid- }),- );- std::thread::sleep(Duration::from_millis(300));-+ // The fixture deliberately starts with txt-deferred-input focused.+ // A single indexed type_text call must atomically activate the WPF+ // window, focus txt-input, confirm that focus, and only then inject.let resp = driver.call("type_text",serde_json::json!({"pid": pid as i64,+ "window_id": wid,+ "element_index": idx,+ "snapshot_id": snap.snapshot_id(),"text": "harness-typed","delivery_mode": "foreground"}),);println!("type_text: {}", resp.text());+ assert!(!resp.is_error(), "type_text failed: {}", resp.text());std::thread::sleep(Duration::from_millis(700));let post = snapshot(driver, pid, wid);@@ -778,6 +749,20 @@ fn harness_wpf_type_text() {"TextBox mirror did not reflect typed text. Mirror/input lines: {:?}",mirror_lines);+ let decoy_idx = ax::element_index_by_id(text, "txt-deferred-input")+ .expect("txt-deferred-input not in post-action snapshot");+ let decoy = post.structured()["elements"]+ .as_array()+ .and_then(|elements| {+ elements+ .iter()+ .find(|element| element["element_index"].as_u64() == Some(decoy_idx))+ })+ .expect("txt-deferred-input missing from structured elements");+ assert!(+ decoy["value"].as_str().unwrap_or_default().is_empty(),+ "foreground type_text leaked into the pre-focused decoy: {decoy}"+ );println!("✅ harness_wpf_type_text: TextBox mirror advanced to 'harness-typed'");Vec::new()},diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rsindex 1310c9e200..01dfc99795 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs@@ -2029,8 +2029,7 @@ pub fn scroll_element(pid: u32, idx: usize, direction: &str, amount: usize) -> R/// updates its renderer-owned focused control. Sending key events immediately/// after the acknowledgement can therefore split one string between the old/// and new controls. Wait for the target's Focused state to become observable;-/// if a toolkit does not publish that state, retain the historical successful-/// result after a bounded settling interval.+/// an acknowledgement without read-back is not sufficient for global input.pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {bounded(async {@@ -2078,7 +2077,7 @@ pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {}}}- Ok(true)+ Ok(false)},|| Err(anyhow!("focus_element timed out for pid {pid}")),)diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rsindex f408864f1f..4983a721bb 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs@@ -967,10 +967,11 @@ fn ewmh_activate_window(/// primitives (proper `x_server_time` stamping beats the WM's focus-stealing/// prevention).///-/// Best-effort: if no X display can be opened the body still runs (without-/// activation) so a headless/Wayland path degrades rather than hard-fails.-/// `settle_ms` is the pause after activation before the first injected event —-/// the WM needs a moment to complete the focus swap (mirrors the macOS settle).+/// The transition is confirmed from both EWMH active-window state and the X11+/// core input-focus tree before `body` runs. A fixed delay or a successful+/// `XSetInputFocus` return is not evidence that global XTest input is safe.+/// `settle_ms` is retained as a compatibility hint and folded into the bounded+/// confirmation timeout; it is no longer an unconditional sleep.pub fn with_x11_foreground<T>(xid: u64,settle_ms: u64,@@ -978,16 +979,18 @@ pub fn with_x11_foreground<T>() -> Result<T> {let display = unsafe { x11::xlib::XOpenDisplay(ptr::null()) };if display.is_null() {- return body();+ bail!("foreground_unavailable: cannot open DISPLAY to verify exact X11 input focus");}let prior = ewmh_active_window(display);+ let mut prior_core_focus: x11::xlib::Window = 0;+ let mut prior_revert = 0;+ unsafe {+ x11::xlib::XGetInputFocus(display, &mut prior_core_focus, &mut prior_revert);+ }ewmh_activate_window(display, xid as x11::xlib::Window, prior.unwrap_or(0));unsafe {x11::xlib::XSync(display, 0);}- if settle_ms > 0 {- std::thread::sleep(std::time::Duration::from_millis(settle_ms));- }// EWMH `_NET_ACTIVE_WINDOW` is honored as *raise-only* by WMs with// focus-stealing prevention (e.g. KWin): the window reaches the top of the// stack — enough for a coordinate click, which lands by stacking — but the X@@ -1008,12 +1011,45 @@ pub fn with_x11_foreground<T>(x11::xlib::XSync(display, 0);x11::xlib::XSetErrorHandler(prev_handler);}- let result = body();- // Restore the prior active window (brief swap, like macOS/Windows).+ let timeout = std::time::Duration::from_millis(settle_ms.max(400));+ let deadline = std::time::Instant::now() + timeout;+ let target = xid as x11::xlib::Window;+ let focused = loop {+ let active = ewmh_active_window(display) == Some(target);+ if active && x11_focus_is_within(display, target) {+ break true;+ }+ if std::time::Instant::now() >= deadline {+ break false;+ }+ std::thread::sleep(std::time::Duration::from_millis(10));+ };+ let result = if focused {+ body()+ } else {+ let active = ewmh_active_window(display).unwrap_or(0);+ Err(anyhow::anyhow!(+ "foreground_unavailable: X11 did not confirm active window and input focus within \+ exact target 0x{xid:x} before the {:?} deadline (active=0x{active:x}); no input was sent",+ timeout+ ))+ };++ // Restore both the EWMH active toplevel and the exact prior core focus.if let Some(p) = prior {ewmh_activate_window(display, p, xid as x11::xlib::Window);+ }+ if prior_core_focus != 0 {unsafe {+ let previous_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));+ x11::xlib::XSetInputFocus(+ display,+ prior_core_focus,+ prior_revert,+ x11::xlib::CurrentTime,+ );x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);}}unsafe {@@ -1022,6 +1058,47 @@ pub fn with_x11_foreground<T>(result}+fn x11_focus_is_within(display: *mut x11::xlib::Display, target: x11::xlib::Window) -> bool {+ let mut focused: x11::xlib::Window = 0;+ let mut revert_to = 0;+ unsafe {+ x11::xlib::XGetInputFocus(display, &mut focused, &mut revert_to);+ }+ if focused == target {+ return true;+ }+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ while focused != 0 && focused != root {+ let mut query_root = 0;+ let mut parent = 0;+ let mut children: *mut x11::xlib::Window = ptr::null_mut();+ let mut child_count = 0;+ let status = unsafe {+ x11::xlib::XQueryTree(+ display,+ focused,+ &mut query_root,+ &mut parent,+ &mut children,+ &mut child_count,+ )+ };+ if !children.is_null() {+ unsafe {+ x11::xlib::XFree(children.cast());+ }+ }+ if status == 0 || parent == 0 || parent == focused {+ return false;+ }+ if parent == target {+ return true;+ }+ focused = parent;+ }+ false+}+/// Activate `xid` and LEAVE it active (no restore) — the persistent foreground/// swap behind `bring_to_front`. Returns the window that was active before, so/// the caller can report/inspect it. Best-effort; returns `None` prior on adiff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex 046592ba68..514b0d1812 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -2995,24 +2995,18 @@ impl Tool for TypeTextTool {&& (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid)){if let Some(idx) = resolved_elem_idx {- let focused =- tokio::task::spawn_blocking(move || crate::atspi::focus_element(pid, idx))- .await;- match focused {- Ok(Ok(true)) => {}- Ok(Ok(false)) => {- return ToolResult::error(format!(- "AT-SPI Component.GrabFocus returned false for element {idx}"- ))- }- Ok(Err(error)) => return ToolResult::error(error.to_string()),- Err(error) => return ToolResult::error(format!("Task error: {error}")),- }-let text_w = text.clone();- let result =- tokio::task::spawn_blocking(move || crate::wayland::type_text(xid, &text_w))- .await;+ let result = tokio::task::spawn_blocking(move || {+ crate::wayland::with_target_foreground(pid, xid, || {+ if !crate::atspi::focus_element(pid, idx)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {idx}"+ );+ }+ crate::wayland::type_text_focused(&text_w)+ })+ })+ .await;return match result {Ok(Ok(())) => ToolResult::text(format!("Typed {text_len} character(s) (via Wayland virtual-keyboard)."@@ -3062,8 +3056,20 @@ impl Tool for TypeTextTool {);}let text_w = text.clone();- let result =- tokio::task::spawn_blocking(move || crate::wayland::type_text(xid, &text_w)).await;+ let idx = resolved_elem_idx;+ let result = tokio::task::spawn_blocking(move || {+ crate::wayland::with_target_foreground(pid, xid, || {+ if let Some(idx) = idx {+ if !crate::atspi::focus_element(pid, idx)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {idx}"+ );+ }+ }+ crate::wayland::type_text_focused(&text_w)+ })+ })+ .await;return match result {Ok(Ok(())) => ToolResult::text(format!("Typed {text_len} character(s) (via Wayland virtual-keyboard)."@@ -3131,24 +3137,17 @@ impl Tool for TypeTextTool {// producing the renderer input event, so web embedders use real XTest// key events. Native toolkits keep their verifiable AT-SPI path below.if delivery.is_foreground() && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid)) {- if let Some(idx) = resolved_elem_idx {- let focused =- tokio::task::spawn_blocking(move || crate::atspi::focus_element(pid, idx))- .await;- match focused {- Ok(Ok(true)) => {}- Ok(Ok(false)) => {- return ToolResult::error(format!(- "AT-SPI Component.GrabFocus returned false for element {idx}"- ))- }- Ok(Err(e)) => return ToolResult::error(e.to_string()),- Err(e) => return ToolResult::error(format!("Task error: {e}")),- }- }let text_f = text.clone();+ let idx = resolved_elem_idx;let result = tokio::task::spawn_blocking(move || {crate::input::with_x11_foreground(xid, 80, || {+ if let Some(idx) = idx {+ if !crate::atspi::focus_element(pid, idx)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {idx}"+ );+ }+ }crate::input::send_type_text_xtest(&text_f)})})@@ -3445,10 +3444,10 @@ impl Tool for PressKeyTool {};let mods: Vec<String> = args.str_array("modifiers");- // Surface 6: resolve element_token / element_index for the window-id- // hint. press_key targets a window via XSendEvent, so we only need- // the resolved window_id — element_index itself is not used to- // address an AX node here (no focus-grab path).+ // Surface 6: resolve the element token/index into both its owning+ // window and exact child. Foreground delivery establishes child focus+ // inside the verified top-level activation transaction; background+ // XSendEvent retains the historical direct-window path.let element_token_arg = args.opt_str("element_token");let window_id_arg = args.opt_u64("window_id");let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);@@ -3469,6 +3468,12 @@ impl Tool for PressKeyTool {}cua_driver_core::element_token::ResolvedElement::None => window_id_arg,};+ let resolved_element_idx = match &resolved {+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {+ Some(*element_index)+ }+ cua_driver_core::element_token::ResolvedElement::None => None,+ };let xid = match xid_opt {Some(x) => x,None => {@@ -3511,7 +3516,7 @@ impl Tool for PressKeyTool {if px.is_some() != py.is_some() {return ToolResult::error("Pass both x and y to press_key, or neither.");}- if px.is_some() && element_index_arg.is_some() {+ if px.is_some() && resolved_element_idx.is_some() {return ToolResult::error("Pass either element_index (ax) or x,y (px) to press_key, not both.",);@@ -3521,7 +3526,7 @@ impl Tool for PressKeyTool {// Preserve legacy modifiers by promoting the request to a chord.if crate::wayland::is_inject_mode() {if let Err(error) =- focus_nested_inject_target(pid, xid, element_index_arg, px.zip(py)).await+ focus_nested_inject_target(pid, xid, resolved_element_idx, px.zip(py)).await{return error;}@@ -3549,28 +3554,6 @@ impl Tool for PressKeyTool {};}- // An element-addressed keypress needs to establish the target's- // focus before the window-level X11 key event is sent. AT-SPI's- // primary action is the focus-free route for editable web controls;- // resolving the element only for its window ID silently loses the key- // on Chromium inputs whose window is backgrounded.- if let Some(element_index) = element_index_arg {- let focused = tokio::task::spawn_blocking(move || {- crate::atspi::focus_element(pid, element_index)- })- .await;- match focused {- Ok(Ok(true)) => {}- Ok(Ok(false)) => {- return ToolResult::error(format!(- "AT-SPI Component.GrabFocus returned false for element {element_index}"- ))- }- Ok(Err(e)) => return ToolResult::error(e.to_string()),- Err(e) => return ToolResult::error(format!("Task error: {e}")),- }- }-let px_target = {if let (Some(cx), Some(cy)) = (px, py) {let from_zoom = args.bool_or("from_zoom", false);@@ -3599,16 +3582,25 @@ impl Tool for PressKeyTool {// to become a chord here — otherwise the modifiers are dropped and the// bare key is typed as a character (Ctrl+S inserts a literal "s").if crate::wayland::wayland_input_enabled() {- let result = match press_key_chord(&mods, &key) {- None => {- let key_w = key.clone();- tokio::task::spawn_blocking(move || crate::wayland::press_key(xid, &key_w))- .await- }- Some(chord) => {- tokio::task::spawn_blocking(move || crate::wayland::hotkey(xid, &chord)).await- }- };+ let key_w = key.clone();+ let chord = press_key_chord(&mods, &key);+ let idx = resolved_element_idx;+ let result = tokio::task::spawn_blocking(move || {+ crate::wayland::with_target_foreground(pid, xid, || {+ if let Some(idx) = idx {+ if !crate::atspi::focus_element(pid, idx)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {idx}"+ );+ }+ }+ match chord {+ Some(keys) => crate::wayland::hotkey_focused(&keys),+ None => crate::wayland::press_key_focused(&key_w),+ }+ })+ })+ .await;return match result {Ok(Ok(())) => ToolResult::text(format!("Pressed key '{key}' (via Wayland virtual-keyboard)."@@ -3624,7 +3616,10 @@ impl Tool for PressKeyTool {// click restores the prior top-level before returning.let deliver_fg = delivery.is_foreground();let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {- if mods.is_empty() && key_for_task.eq_ignore_ascii_case("enter") {+ if resolved_element_idx.is_none()+ && mods.is_empty()+ && key_for_task.eq_ignore_ascii_case("enter")+ {if inject_terminal_input(pid, xid, "\n")? {return Ok(());}@@ -3637,9 +3632,23 @@ impl Tool for PressKeyTool {// XSendEvent (no focus steal) for apps that accept it.if deliver_fg {return crate::input::with_x11_foreground(xid, 80, || {+ if let Some(element_index) = resolved_element_idx {+ if !crate::atspi::focus_element(pid, element_index)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {element_index}"+ );+ }+ }crate::input::send_key_xtest(&key_for_task, &m)});}+ if let Some(element_index) = resolved_element_idx {+ if !crate::atspi::focus_element(pid, element_index)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {element_index}"+ );+ }+ }if let Some((x, y)) = px_target {crate::input::send_key_at(xid, x, y, &key_for_task, &m)} else {diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rsindex d379737538..8982c0eda9 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs@@ -1217,6 +1217,33 @@ pub fn activate_window_for_input_target()}+/// Run a focus-bound keyboard transaction only after a compositor adapter has+/// confirmed the exact PID/window pair, and restore the previously focused+/// toplevel afterward. Global virtual-keyboard/libei input is refused when the+/// compositor cannot provide this read-back contract.+pub fn with_target_foreground<T>(+ pid: u32,+ window_id: u64,+ body: impl FnOnce() -> anyhow::Result<T>,+) -> anyhow::Result<T> {+ if let Some(window) = sway_ipc::window_for_id(window_id) {+ if window.pid != pid {+ anyhow::bail!(+ "foreground_unavailable: Sway window {window_id} belongs to pid {}, not pid {pid}",+ window.pid+ );+ }+ return sway_ipc::with_focused_container(window_id, body);+ }+ if shell_helper::trusted_window_for_id(pid, window_id).is_some() {+ return shell_helper::with_focused_window(pid, window_id, body);+ }+ anyhow::bail!(+ "foreground_unavailable: no trusted Wayland compositor adapter can confirm exact \+ target window {window_id} for pid {pid}; no global keyboard input was sent"+ )+}+/// Query the first `wl_output`'s pixel dimensions via a short Wayland/// roundtrip, independent of the virtual-pointer protocol. Used by the libei/// fallback (which never opens a `VptrSession`) to reproduce the vptr path'sdiff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rsindex b32e4aa984..1a791db5fc 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs@@ -417,8 +417,16 @@ pub fn activate_window(window_id: u64) -> bool {if !accepted {return false;}- std::thread::sleep(Duration::from_millis(60));- window_is_focused(window_id)+ let deadline = std::time::Instant::now() + Duration::from_millis(500);+ loop {+ if window_is_focused(window_id) {+ return true;+ }+ if std::time::Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(Duration::from_millis(10));+ }}fn trusted_activate_window(window_id: u64) -> bool {@@ -430,12 +438,20 @@ fn trusted_activate_window(window_id: u64) -> bool {if !accepted {return false;}- std::thread::sleep(Duration::from_millis(60));- trusted_shell_windows(None).is_some_and(|windows| {- windows- .into_iter()- .any(|window| window.info.xid == u64::from(window_id) && window.focused)- })+ let deadline = std::time::Instant::now() + Duration::from_millis(500);+ loop {+ if trusted_shell_windows(None).is_some_and(|windows| {+ windows+ .into_iter()+ .any(|window| window.info.xid == u64::from(window_id) && window.focused)+ }) {+ return true;+ }+ if std::time::Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(Duration::from_millis(10));+ }}fn window_is_focused(window_id: u32) -> bool {diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rsindex 971b846166..bc090b94cb 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs@@ -182,6 +182,19 @@ fn focus_container(id: u64) -> bool {.is_ok_and(|status| status.success())}+fn wait_for_container_focus(id: u64, timeout: std::time::Duration) -> bool {+ let deadline = std::time::Instant::now() + timeout;+ loop {+ if window_for_id(id).is_some_and(|window| window.focused) {+ return true;+ }+ if std::time::Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(std::time::Duration::from_millis(10));+ }+}+/// Briefly focus one compositor-attested container, run `body`, then restore/// the previously focused container. The caller must already have verified the/// target container's PID and id through [`window_for_id`].@@ -197,8 +210,7 @@ pub fn with_focused_container<T>(if !focus_container(id) {anyhow::bail!("Sway refused to focus exact container {id}");}- std::thread::sleep(std::time::Duration::from_millis(80));- if !window_for_id(id).is_some_and(|window| window.focused) {+ if !wait_for_container_focus(id, std::time::Duration::from_millis(500)) {anyhow::bail!("Sway did not confirm focus on exact container {id}");}let result = body();@@ -208,8 +220,7 @@ pub fn with_focused_container<T>(if !focus_container(prior) {anyhow::bail!("Sway could not restore prior container {prior}");}- std::thread::sleep(std::time::Duration::from_millis(80));- if !window_for_id(prior).is_some_and(|window| window.focused) {+ if !wait_for_container_focus(prior, std::time::Duration::from_millis(500)) {anyhow::bail!("Sway did not confirm restored container {prior}");}Ok(())diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rsindex 724217369f..371773760d 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs@@ -284,6 +284,29 @@ pub fn focus_element(element_ptr: usize) -> anyhow::Result<()> {}}+/// Report whether `element_ptr` is the application's currently focused element.+///+/// This is a read-only confirmation for the foreground typing rung: an+/// `AXFocused` write can be accepted by the element and then immediately+/// clobbered when AppKit installs the window's remembered first responder, so+/// "the write returned success" is not evidence that focus stuck. Identity is+/// compared with `CFEqual` because the app hands back a fresh `AXUIElementRef`+/// for the same underlying element.+///+/// A `false` return is deliberately conservative: an app whose+/// `AXFocusedUIElement` is unreadable reports not-focused, which at worst costs+/// one extra re-apply.+pub fn is_element_focused(pid: i32, element_ptr: usize) -> bool {+ unsafe {+ let Some(focused) = crate::ax::bindings::focused_element_of_pid(pid) else {+ return false;+ };+ let same = CFEqual(focused as CFTypeRef, element_ptr as CFTypeRef) != 0;+ CFRelease(focused as CFTypeRef);+ same+ }+}+/// Set the AXValue of an element (for dropdowns, text fields, etc.).pub fn set_ax_value(element_ptr: usize, value: &str) -> anyhow::Result<()> {let err = unsafe { set_string_attr(element_ptr as AXUIElementRef, "AXValue", value) };diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rsindex 87a2e026f0..a657ced2b2 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs@@ -660,15 +660,34 @@ pub fn make_exact_window_key(target_pid: libc::pid_t, target_wid: u32) -> bool {true}-/// Tool-agnostic foreground-assist: briefly front `window_id`, run `body` (which-/// posts the synthetic input), then restore the prior frontmost process.+/// Tool-agnostic foreground-assist: briefly front `window_id`, wait for the+/// activation to actually land, run `body` (which posts the synthetic input),+/// then restore the prior frontmost process.////// This is the `delivery_mode:"foreground"` rung of the best-effort-background-/// ladder, shared by `type_text` and `click`. It is the same brief front →-/// act → restore primitive `press_key`/`hotkey` use for NSMenu key dispatch —-/// see [`with_menu_shortcut_activation`], which this delegates to. Reached only-/// when the agent has seen the background rungs fail (clicks) or the field is-/// unverifiable + focus-sensitive (Catalyst typing).+/// ladder, shared by `type_text` and `click`. Reached only when the agent has+/// seen the background rungs fail (clicks) or the field is unverifiable ++/// focus-sensitive (Catalyst typing).+///+/// ## Why this does not delegate to [`with_menu_shortcut_activation`]+///+/// It used to. That helper posts `set_front` and calls `action` immediately,+/// which is correct for its own purpose: NSMenu key dispatch only needs the key+/// event *enqueued* in the target's run-loop queue, so first-responder identity+/// is irrelevant and the sub-millisecond front → act → restore is a feature.+///+/// Input delivery has the opposite requirement. `set_front` is asynchronous, so+/// running the body straight away means the body's `AXFocused` write races+/// AppKit's own activation. AppKit wins: when activation completes it installs+/// the window's remembered first responder and clobbers the write. The+/// keystrokes then land wherever the app chose — for a Catalyst app such as+/// WhatsApp, the message list rather than the composer, which silently scrolls+/// the transcript instead of typing.+///+/// So this waits for the activation to be observable before running `body`, the+/// same ordering [`with_foreground_hid_activation`] already relies on. The wait+/// is a bounded poll rather than a fixed sleep: an app that activates in 10 ms+/// pays 10 ms, and a slow Catalyst/RDP surface still gets its full budget.////// Returns `Ok(true)` when the brief activation happened, `Ok(false)` when the/// fronting SPIs are unavailable (the body still ran, just without a front).@@ -677,7 +696,81 @@ pub fn with_foreground_assist(target_wid: u32,body: impl FnOnce() -> anyhow::Result<()>,) -> anyhow::Result<bool> {- with_menu_shortcut_activation(target_pid, target_wid, body)+ let set_front = match set_front_process_fn() {+ Some(f) => f,+ None => {+ // SPIs unavailable — run body anyway without activation.+ body()?;+ return Ok(false);+ }+ };++ let mut prev_psn = [0u8; 8];+ let prev_ok = get_front_process_fn()+ .map(|f| unsafe { f(prev_psn.as_mut_ptr() as *mut c_void) } == 0)+ .unwrap_or(false);++ let mut target_psn = [0u8; 8];+ if !get_process_psn_for_window(target_wid, target_pid, &mut target_psn) {+ body()?;+ return Ok(false);+ }++ unsafe { set_front(target_psn.as_ptr() as *const c_void, target_wid, 0x400) };+ // `set_front` moves WindowServer's front process but does not make the+ // target's NSWindow key, and AppKit installs a first responder only for a+ // key window. Without this the app is "frontmost" to WindowServer while+ // remaining, from AppKit's point of view, unfocused — so the AXFocused+ // write in the body has no responder chain to attach to.+ make_exact_window_key(target_pid, target_wid);+ await_window_focused(target_pid, target_wid);++ let result = body();++ if prev_ok {+ unsafe { set_front(prev_psn.as_ptr() as *const c_void, 0, 0x400) };+ }++ result?;+ Ok(true)+}++/// Upper bound on how long [`with_foreground_assist`] waits for a requested+/// activation to become observable. Chosen to cover a Catalyst app's activation+/// plus key-window install; past this the caller proceeds anyway so a stubborn+/// target degrades to the old behavior instead of hanging.+const ACTIVATION_WAIT_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(400);++/// Poll interval for [`await_window_focused`]. Short enough that a fast native+/// app pays roughly one tick, long enough not to spin on the WindowServer.+const ACTIVATION_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);++/// Block until `target_wid` is the application's focused AX window, or the+/// timeout expires. Returns whether that state was observed.+///+/// The predicate is deliberately `AXFocusedWindow` and not+/// `NSWorkspace.frontmostApplication`. The latter does not observe a+/// SkyLight-level front-process change at all: polling it every 15ms across a+/// full foreground `type_text` against WhatsApp showed zero transitions while+/// the target was demonstrably being fronted, so a frontmost-based wait always+/// burns its whole timeout and never actually gates on anything. `AXFocusedWindow`+/// is the same proof [`preserves_exact_existing_focus`] already trusts to decide+/// whether a window is focused.+///+/// A `false` return is not fatal: the caller proceeds with delivery regardless,+/// because a target that never reports focus is exactly the case the pre-existing+/// best-effort contract already covered.+fn await_window_focused(pid: libc::pid_t, window_id: u32) -> bool {+ let deadline = std::time::Instant::now() + ACTIVATION_WAIT_TIMEOUT;+ loop {+ if crate::ax::bindings::focused_window_id_of_pid(pid) == Some(window_id) {+ return true;+ }+ if std::time::Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(ACTIVATION_POLL_INTERVAL);+ }}/// Activate an exact target window for a global HID keyboard action.@@ -721,7 +814,14 @@ pub fn with_foreground_hid_activation(anyhow::bail!("WindowServer rejected foreground HID activation");}- std::thread::sleep(std::time::Duration::from_millis(40));+ make_exact_window_key(target_pid, target_wid);+ if !await_window_focused(target_pid, target_wid) {+ if prev_ok {+ unsafe { set_front(prev_psn.as_ptr() as *const c_void, 0, 0x400) };+ }+ anyhow::bail!("exact target window did not become focused for foreground HID delivery");+ }+let result = action();std::thread::sleep(std::time::Duration::from_millis(40));diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rsindex 685bf9e31c..699a8e8f12 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs@@ -37,14 +37,15 @@ fn def() -> &'static ToolDef {window by default:\n\• `background` (default): post the combo to the target pid WITHOUT \fronting or raising it — uses the macOS 14+ auth-message envelope so \- Chromium/Electron accept it as trusted live input. No focus steal. \+ Chromium/Electron accept it as trusted live input. With an AX target, \+ focus that exact element first. No top-level focus steal. \`window_id` here only targets the combo; it does not raise.\n\• `foreground`: briefly front the window (NSMenu path, < 1 ms via \SLPSSetFrontProcessWithOptions) so native menu key-equivalents \(Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the \explicit escalation for menu-bar shortcuts on non-Chromium apps that \- ignore a background combo. With x,y, the focused field receives \- the chord through the foreground HID queue (needed by native \+ ignore a background combo. With an AX target or x,y, the focused field \+ receives the chord through the foreground HID queue (needed by native \Chromium fields such as the omnibox). Requires window_id.\n\n\A combo is never driver-verifiable (no read-back) → effect:\"unverifiable\"; \confirm via screenshot. NOTE: a keyboard combo does NOT focus a text \@@ -75,6 +76,9 @@ fn def() -> &'static ToolDef {"type": "integer","description": "Target window. Required for delivery_mode:\"foreground\" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode."},+ "element_index": cua_driver_core::tool_schema::element_index_schema(),+ "element_token": cua_driver_core::tool_schema::element_token_schema(),+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the chord to the frontmost application." },"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()},@@ -95,6 +99,23 @@ fn is_modifier(k: &str) -> bool {)}+const HOTKEY_FOCUS_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(200);+const HOTKEY_FOCUS_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);++fn focus_hotkey_element(pid: i32, element_ptr: usize) -> anyhow::Result<()> {+ let deadline = std::time::Instant::now() + HOTKEY_FOCUS_TIMEOUT;+ loop {+ crate::input::ax_actions::focus_element(element_ptr)?;+ if crate::input::ax_actions::is_element_focused(pid, element_ptr) {+ return Ok(());+ }+ if std::time::Instant::now() >= deadline {+ anyhow::bail!("requested hotkey element did not become focused");+ }+ std::thread::sleep(HOTKEY_FOCUS_POLL_INTERVAL);+ }+}+fn screen_sharing_modifier_delivery_error(is_screen_sharing: bool,has_modifiers: bool,@@ -130,8 +151,6 @@ impl Tool for HotkeyTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- let _ = &self.state;-if args.opt_str("scope").as_deref() == Some("desktop")&& args.get("pid").is_none()&& args.get("window_id").is_none()@@ -209,13 +228,60 @@ impl Tool for HotkeyTool {// Use the last non-modifier key; if there are multiple, treat earlier ones as extra keys.let key = non_modifiers.last().unwrap().clone();let key_display = raw_keys.join("+");- let window_id = args.opt_u64("window_id").map(|v| v as u32);+ let element_token_arg = args.opt_str("element_token");+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);+ let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);+ let resolved = match cua_driver_core::element_token::resolve_element_args(+ pid,+ element_index_arg,+ element_token_arg.as_deref(),+ args.opt_str("snapshot_id").as_deref(),+ window_id_arg,+ "hotkey",+ ) {+ Ok(resolved) => resolved,+ Err(error) => return error,+ };+ let (element_index, window_id) = match resolved {+ cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),+ cua_driver_core::element_token::ResolvedElement::Element {+ window_id,+ element_index,+ via_token: _,+ } => (Some(element_index), window_id),+ };// delivery_mode gates whether we raise: background (default) never fronts// the window — passing window_id only targets the combo. foreground is the// explicit NSMenu-activation rung for menu shortcuts that ignore a// background combo (matches click/type_text).let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());let fg = delivery_mode.is_foreground();+ let px = args.get("x").and_then(|value| value.as_f64());+ let py = args.get("y").and_then(|value| value.as_f64());+ if px.is_some() && py.is_some() && element_index.is_some() {+ return ToolResult::error(+ "Pass either element_index (ax) or x,y (px) to hotkey, not both.",+ );+ }++ let element_guard = if let (Some(index), Some(window_id)) = (element_index, window_id) {+ match self+ .state+ .element_cache+ .get_element_retained(pid, window_id, index)+ {+ Some(guard) => Some(guard),+ None => {+ return ToolResult::error(format!(+ "Element index {index} not found. Call get_window_state first."+ ));+ }+ }+ } else {+ None+ };+ let element_ptr = element_guard.as_ref().map(|guard| guard.as_ptr());+let screen_sharing_target = crate::input::keyboard::is_screen_sharing_pid(pid);if let Some(error) = screen_sharing_modifier_delivery_error(screen_sharing_target,@@ -238,7 +304,7 @@ impl Tool for HotkeyTool {match super::gate_background_window_action(pid,wid,- None,+ element_ptr,cua_driver_core::background_input::BackgroundAction::GenericKey,).await@@ -253,13 +319,52 @@ impl Tool for HotkeyTool {None};- // px form: focus the field before sending the combo. Foreground delivery- // still needs to front the target for the chord itself: the focus helper- // restores the previous app before returning.- let px_focus = {- let px = args.get("x").and_then(|v| v.as_f64());- let py = args.get("y").and_then(|v| v.as_f64());- if let (Some(cx), Some(cy)) = (px, py) {+ // Web-content AX nodes can acknowledge AXFocused without moving the+ // renderer's real first responder. That makes a direct focus write an+ // unsafe oracle for Chromium/WebKit hotkeys: the following PID/HID+ // chord can still land on the renderer's remembered control. Resolve+ // the requested AX node's exact center and reuse the proven PX focus+ // ladder for web areas. The request remains snapshot-bound AX+ // targeting; only the focus transport falls back through a hit-test+ // and, on the explicit foreground rung, a real click when required.+ let web_ax_focus_xy =+ if let (Some(ptr), Some(wid), Some(index)) = (element_ptr, window_id, element_index) {+ let is_web = tokio::task::spawn_blocking(move || {+ super::type_text::target_in_web_area(pid, Some((ptr, Some(index))), Some(wid))+ })+ .await+ .unwrap_or(true);+ if is_web {+ tokio::task::spawn_blocking(move || unsafe {+ let (screen_x, screen_y) = crate::ax::bindings::element_screen_center(+ ptr as crate::ax::bindings::AXUIElementRef,+ )?;+ let frame = super::px_frame::resolve_window_px_frame(wid).ok()?;+ Some((+ (screen_x - frame.bounds.x) * frame.scale,+ (screen_y - frame.bounds.y) * frame.scale,+ ))+ })+ .await+ .unwrap_or(None)+ } else {+ None+ }+ } else {+ None+ };++ // PX form, plus the web-content AX fallback above: focus the field+ // before sending the combo. Foreground delivery still needs to front+ // the target for the chord itself because the focus helper restores+ // the previous app before returning.+ let coordinate_focus = {+ let focus_xy = match ((px, py), web_ax_focus_xy) {+ ((Some(cx), Some(cy)), _) => Some((cx, cy)),+ ((None, None), Some(center)) => Some(center),+ _ => None,+ };+ if let Some((cx, cy)) = focus_xy {let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool())@@ -302,12 +407,12 @@ impl Tool for HotkeyTool {|| async move {tokio::task::spawn_blocking(move || {let m: Vec<&str> = modifiers.iter().map(String::as_str).collect();- match (fg, px_focus, window_id) {+ match (fg, coordinate_focus, window_id, element_ptr) {// Chrome's native omnibox and Chromium/Electron inputs// require a genuine foreground HID chord. Keep the exact// target frontmost until both key events are consumed;// otherwise Cmd+A/Cmd+V can be silently ignored.- (true, true, Some(wid)) => {+ (true, true, Some(wid), _) => {crate::input::skylight::with_foreground_hid_activation(pid as libc::pid_t,wid,@@ -321,11 +426,26 @@ impl Tool for HotkeyTool {)?;Ok(())}+ // An AX-addressed chord has the same renderer-focus+ // requirement as the px form. Activate the exact window,+ // establish and confirm the requested child focus after+ // activation, then use the guarded global HID queue.+ (true, false, Some(wid), Some(ptr)) => {+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ || {+ focus_hotkey_element(pid, ptr)?;+ crate::input::keyboard::press_key_bare_global(&key, &m)+ },+ )?;+ Ok(())+ }// Screen Sharing is an input forwarder: modifier flags// on a PID-routed base-key event are not relayed to the// guest. Emit the physical modifier down/base/up// sequence through the guarded foreground HID path.- (true, false, Some(wid))+ (true, false, Some(wid), None)if crate::input::keyboard::is_screen_sharing_pid(pid) =>{crate::input::skylight::with_foreground_hid_activation(@@ -337,7 +457,7 @@ impl Tool for HotkeyTool {}// foreground rung: briefly front the window so NSMenu key// equivalents dispatch, then restore prior frontmost.- (true, false, Some(wid)) => {+ (true, false, Some(wid), None) => {crate::input::skylight::with_menu_shortcut_activation(pid as libc::pid_t,wid,@@ -347,6 +467,10 @@ impl Tool for HotkeyTool {}// background (default): auth-envelope post to the pid, no// raise — even when window_id was supplied for targeting.+ (false, false, _, Some(ptr)) => {+ focus_hotkey_element(pid, ptr)?;+ crate::input::keyboard::hotkey(pid, &key, &m)+ }_ => crate::input::keyboard::hotkey(pid, &key, &m),}})@@ -397,6 +521,16 @@ impl Tool for HotkeyTool {mod tests {use super::*;+ #[test]+ fn hotkey_contract_accepts_snapshot_bound_ax_targets() {+ let properties = def().input_schema["properties"]+ .as_object()+ .expect("hotkey properties");+ for field in ["element_index", "element_token", "snapshot_id"] {+ assert!(properties.contains_key(field), "missing {field} schema");+ }+ }+#[test]fn screen_sharing_modifier_hotkeys_fail_closed_without_foreground_window() {for (foreground, window_id) in [(false, None), (false, Some(7)), (true, None)] {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex fe7269f154..68287e0c93 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -374,10 +374,16 @@ pub(crate) async fn focus_by_pixel(// AXFocused is non-destructive: unlike a second real click, it keeps a// Cmd+A selection intact before a follow-up type_text or Cmd+V.tokio::time::sleep(std::time::Duration::from_millis(120)).await;- return Ok(());- }-- if !foreground {+ // A background focus-click is `effect:"unverifiable"` by construction:+ // it cannot prove the renderer moved its first responder. Returning on+ // "it didn't error" therefore skipped the real-click fallback below+ // whenever the click was a silent no-op — advancing on transport+ // success alone, which is exactly what the ladder forbids. Confirm the+ // focus actually moved before claiming this rung worked.+ if !foreground || pixel_focus_landed(pid, window_id, x, y).await {+ return Ok(());+ }+ } else if !foreground {return Err(cua_driver_core::protocol::ToolResult::error(format!("focus pixel-click at ({x:.0},{y:.0}) failed.")));@@ -415,6 +421,51 @@ pub(crate) async fn focus_by_pixel(Ok(())}+/// Confirm that a focus pixel-click actually moved the application's focused+/// element onto the clicked point.+///+/// The background focus-click reports `effect:"unverifiable"`, so this is the+/// read-back that lets [`focus_by_pixel`] decide whether the cheap rung worked+/// or the real-click fallback is still required. It reuses the same+/// window-local-pixels → screen translation the click itself used, so the+/// comparison is in one coordinate space.+///+/// Returns `false` whenever the answer cannot be established (no window id,+/// untranslatable frame, unreadable focused element or rect). That is the+/// conservative direction: an unprovable focus escalates to the stronger rung+/// rather than being reported as success.+async fn pixel_focus_landed(pid: i32, window_id: Option<u32>, x: f64, y: f64) -> bool {+ let Some(wid) = window_id else {+ return false;+ };+ tokio::task::spawn_blocking(move || {+ let Ok(frame) = px_frame::resolve_window_px_frame(wid) else {+ return false;+ };+ let (screen_x, screen_y, _, _) = frame.to_screen(x, y);+ unsafe {+ let Some(focused) = crate::ax::bindings::focused_element_of_pid(pid) else {+ return false;+ };+ let rect = crate::ax::bindings::element_screen_rect(focused);+ core_foundation::base::CFRelease(focused as core_foundation::base::CFTypeRef);+ let Some(rect) = rect else {+ return false;+ };+ point_within_rect(rect, screen_x, screen_y)+ }+ })+ .await+ .unwrap_or(false)+}++/// Whether `[x, y, width, height]` (screen coordinates, top-left origin)+/// contains the point. A degenerate rect never contains anything, so an app+/// reporting a zero-sized focused element escalates rather than false-confirms.+fn point_within_rect([rx, ry, rw, rh]: [f64; 4], x: f64, y: f64) -> bool {+ rw > 0.0 && rh > 0.0 && x >= rx && x < rx + rw && y >= ry && y < ry + rh+}+/// Thread-safe per-pid zoom context registry.pub struct ZoomRegistry {inner: std::sync::Mutex<HashMap<i32, ZoomContext>>,@@ -1118,6 +1169,43 @@ mod cursor_overlay_facility_tests {// macOS cursor sampler (CoreGraphics), so the start-guard test runs here in// platform-macos where build.rs links the frameworks — the core crate's test// binary has no CoreGraphics linkage.+#[cfg(test)]+mod pixel_focus_readback_tests {+ use super::point_within_rect;++ #[test]+ fn confirms_a_point_inside_the_focused_element() {+ let composer = [100.0, 800.0, 250.0, 24.0];+ assert!(point_within_rect(composer, 220.0, 812.0));+ assert!(+ point_within_rect(composer, 100.0, 800.0),+ "top-left is inside"+ );+ }++ #[test]+ fn rejects_a_point_outside_the_focused_element() {+ // The WhatsApp case: the click targeted the composer but focus stayed on+ // the transcript above it, so the clicked point is not inside the+ // focused element's rect and the caller must escalate.+ let transcript = [100.0, 100.0, 640.0, 690.0];+ assert!(!point_within_rect(transcript, 220.0, 812.0));+ }++ #[test]+ fn excludes_the_far_edges() {+ let rect = [0.0, 0.0, 10.0, 10.0];+ assert!(!point_within_rect(rect, 10.0, 5.0));+ assert!(!point_within_rect(rect, 5.0, 10.0));+ }++ #[test]+ fn a_degenerate_rect_never_confirms() {+ assert!(!point_within_rect([5.0, 5.0, 0.0, 0.0], 5.0, 5.0));+ assert!(!point_within_rect([5.0, 5.0, -3.0, 10.0], 5.0, 6.0));+ }+}+#[cfg(test)]mod recording_start_guard_tests {use cua_driver_core::recording::RecordingSession;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rsindex 90c8ac4a53..49f8eca442 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs@@ -751,6 +751,11 @@ fn web_readback_next_rung(is_electron: bool, used_pixel_focus: bool) -> Option<&const DELIVERY_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);const DELIVERY_DRAIN_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);+/// Pause before the single `AXFocused` re-apply in the foreground rung. Covers+/// an app that installs its own first responder just after activation is+/// observable, which would otherwise clobber the first write.+const FOCUS_REAPPLY_DELAY: std::time::Duration = std::time::Duration::from_millis(30);+/// Keyboard-rung policy for one window-addressed background insert, decided/// once by the pure exact-target core before any input is posted.#[derive(Clone, Debug)]@@ -1019,20 +1024,29 @@ fn cgevent_type_verified(settle_ms: u64,window_id: Option<u32>,) -> anyhow::Result<(bool, Option<usize>)> {- // Focus the target element first so the keystrokes land in IT. Critical in+ // Focus the target element so the keystrokes land in IT. Critical in// foreground mode: a freshly-fronted window's keyboard focus may be on the// search box or nowhere, so without this the text goes into the void (or the// wrong field). AXFocused is best-effort — harmless when unsupported.+ //+ // Ordering matters as much as the write itself. `with_foreground_assist` has+ // already waited for the activation to land, so AppKit has installed the+ // window's remembered first responder by now and this write lands *after*+ // it rather than being clobbered by it. Re-applying once on a failed+ // read-back covers apps that install their responder slightly late.if let Some((ptr, _)) = element_ptr_and_idx {let _ = crate::input::ax_actions::focus_element(ptr);+ if settle_ms > 0 && !crate::input::ax_actions::is_element_focused(pid, ptr) {+ std::thread::sleep(FOCUS_REAPPLY_DELAY);+ let _ = crate::input::ax_actions::focus_element(ptr);+ }}// First-keystroke settle (foreground rung only — caller passes `settle_ms > 0`).- // After a window is fronted (with_foreground_assist) and the element focused,- // the surface isn't ready to accept input for a few tens of ms, so the FIRST- // synthesized character gets eaten: typing "i love u" rendered "love u" (the- // leading "i " was dropped). A short sleep here lets focus settle before the- // first key event. Background/terminal call sites pass 0 — they have no front- // transition and must not pay this latency.+ // Even once the window is front and the element focused, the surface isn't+ // ready to accept input for a few tens of ms, so the FIRST synthesized+ // character gets eaten: typing "i love u" rendered "love u" (the leading+ // "i " was dropped). A short sleep here covers that. Background/terminal call+ // sites pass 0 — they have no front transition and must not pay this latency.if settle_ms > 0 {std::thread::sleep(std::time::Duration::from_millis(settle_ms));}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rsindex bddf0502e2..1a1d2e5163 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs@@ -427,6 +427,22 @@ pub fn post_key(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<()> {/// worker, the foreground swap may silently fail and SendInput land on the/// wrong window. Callers should funnel hotkey calls through the uia worker.pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<()> {+ send_key_synthesized_after_focus(hwnd, key, modifiers, || Ok(()))+}++/// Foreground key delivery with a target-specific focus step performed only+/// after Windows has confirmed the exact top-level HWND as foreground.+///+/// Element-addressed callers use this to avoid the activation/focus race:+/// UIA `SetFocus` before `SetForegroundWindow` can be overwritten by the+/// ensuing activation. The callback may establish and verify child focus; no+/// input is inserted when either foreground or child-focus confirmation fails.+pub fn send_key_synthesized_after_focus(+ hwnd: u64,+ key: &str,+ modifiers: &[&str],+ focus: impl FnOnce() -> Result<()>,+) -> Result<()> {let target = HWND(hwnd as *mut _);if target.0.is_null() {bail!("invalid target hwnd");@@ -455,50 +471,9 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<events.push(key_input(*mvk, true));}- unsafe {- // Save & set foreground so SendInput lands on `target`.- let prev_fg = GetForegroundWindow();- // Robust auto bring-to-front (AttachThreadInput, same engine as- // bring_to_front / macOS with_foreground_assist) — a bare- // SetForegroundWindow is denied by the foreground-lock without UIAccess.- // Restored to prev_fg below.- let _ = crate::input::force_foreground_assisted(target);- // Brief settle so the foreground swap is processed before we send.- sleep(Duration::from_millis(8));-- // Verify the swap actually happened. `SetForegroundWindow` returns a- // BOOL but Windows treats foreground-lock violations as a silent- // no-op in most cases (the call returns success-ish, the foreground- // doesn't change). The reliable check is observing the foreground- // after the settle. If we didn't get it, abort BEFORE SendInput —- // otherwise the events land on whatever app actually held foreground- // (typically the terminal hosting this process), which causes spooky- // side effects like Alt+Enter toggling the terminal into fullscreen.- let actual_fg = GetForegroundWindow();- if actual_fg != target {- // Don't restore — prev_fg is presumably still foreground anyway.- bail!(- "foreground_unavailable: foreground swap to exact target HWND {:?} was rejected by Windows \- (actual foreground is HWND {:?}). This daemon is not at \- UIAccess integrity, so SetForegroundWindow is subject to the \- foreground-lock and the swap silently fails. Without the \- swap, SendInput would land on the wrong window. Fix: install \- / spawn the cua-driver-uia worker (UIAccess-manifested PE) \- and route hotkey calls through it. Until then, the calling \- app must already be foreground for delivery_mode:\"foreground\" \- to be safe.",- target.0,- actual_fg.0- );- }-+ with_confirmed_foreground(target, "key delivery", focus, || unsafe {let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);if sent as usize != events.len() {- // SendInput returns the number of events successfully inserted.- // Anything less is a partial insertion (blocked by another input- // injector, foreground UIPI denial, etc.).- let restored = SetForegroundWindow(prev_fg);- let _ = restored;bail!("SendInput inserted only {sent} of {} events. Likely cause: \the daemon is not at UIAccess integrity, so SetForegroundWindow \@@ -507,16 +482,8 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<events.len());}-- // Brief settle to let the target process the keystrokes before we- // restore the previous foreground (otherwise the target might not- // get a chance to handle the accelerator before losing focus).- sleep(Duration::from_millis(40));- if !prev_fg.0.is_null() && prev_fg != target {- let _ = SetForegroundWindow(prev_fg);- }- }- Ok(())+ Ok(())+ })}/// Foreground-delivery text entry: the `delivery_mode:"foreground"` rung for@@ -531,6 +498,16 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result</// of a false success. Required for VCL/LibreOffice document grids and other/// targets where PostMessage WM_CHAR is silently dropped.pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {+ send_text_synthesized_after_focus(hwnd, text, || Ok(()))+}++/// Foreground Unicode delivery with child focus established after exact+/// top-level activation and before `SendInput`.+pub fn send_text_synthesized_after_focus(+ hwnd: u64,+ text: &str,+ focus: impl FnOnce() -> Result<()>,+) -> Result<()> {let target = HWND(hwnd as *mut _);if target.0.is_null() {bail!("invalid target hwnd");@@ -571,37 +548,9 @@ pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {return Ok(());}- unsafe {- // Save & set foreground so SendInput lands on `target`. Same verify-- // before-send guard as send_key_synthesized: a foreground-lock no-op- // would otherwise spray the text into whatever window actually held- // focus (typically the terminal hosting this daemon).- let prev_fg = GetForegroundWindow();- // Robust auto bring-to-front (AttachThreadInput, same engine as- // bring_to_front / macOS with_foreground_assist) — a bare- // SetForegroundWindow is denied by the foreground-lock without UIAccess.- // Restored to prev_fg below.- let _ = crate::input::force_foreground_assisted(target);- sleep(Duration::from_millis(8));- let actual_fg = GetForegroundWindow();- if actual_fg != target {- bail!(- "foreground_unavailable: foreground swap to exact target HWND {:?} was rejected by Windows \- (actual foreground is HWND {:?}). This daemon is not at \- UIAccess integrity, so SetForegroundWindow is subject to the \- foreground-lock and the swap silently fails. Without the swap, \- SendInput would land on the wrong window. Fix: install / spawn \- the cua-driver-uia worker (UIAccess-manifested PE) and route \- type_text through it. Until then, the calling app must already \- be foreground for delivery_mode:\"foreground\" to be safe.",- target.0,- actual_fg.0- );- }-+ with_confirmed_foreground(target, "text delivery", focus, || unsafe {let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);if sent as usize != events.len() {- let _ = SetForegroundWindow(prev_fg);bail!("SendInput inserted only {sent} of {} key events. Likely cause: \the daemon is not at UIAccess integrity, so SetForegroundWindow \@@ -609,14 +558,103 @@ pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {events.len());}+ Ok(())+ })+}- // Settle so the target processes the text before we yield focus back.- sleep(Duration::from_millis(40));- if !prev_fg.0.is_null() && prev_fg != target {- let _ = SetForegroundWindow(prev_fg);+fn wait_for_exact_foreground(target: HWND, timeout: Duration) -> bool {+ let deadline = Instant::now() + timeout;+ loop {+ if unsafe { GetForegroundWindow() } == target {+ return true;+ }+ if Instant::now() >= deadline {+ return false;}+ sleep(Duration::from_millis(10));}- Ok(())+}++/// Run one system-queue input transaction while `target`, or a live visible+/// same-process window whose owner chain reaches `target`, is foreground.+/// Foreground and optional child focus are checked against+/// external Win32/UIA state instead of inferred from API return values or a+/// fixed settle delay. The prior foreground is restored on every body/focus+/// result after activation succeeds.+fn with_confirmed_foreground<T>(+ target: HWND,+ operation: &str,+ focus: impl FnOnce() -> Result<()>,+ body: impl FnOnce() -> Result<T>,+) -> Result<T> {+ let previous = unsafe { GetForegroundWindow() };+ let _ = unsafe { crate::input::force_foreground_assisted(target) };+ if !wait_for_exact_foreground(target, Duration::from_millis(500)) {+ let actual = unsafe { GetForegroundWindow() };+ if !previous.0.is_null() && previous != target {+ let _ = unsafe { SetForegroundWindow(previous) };+ }+ bail!(+ "foreground_unavailable: Windows did not confirm exact target HWND {:?} for {operation} \+ within 500 ms (actual foreground HWND {:?}). Route the request through the \+ UIAccess-manifested cua-driver-uia worker; no input was sent.",+ target.0,+ actual.0+ );+ }++ let Some(foreground_target) = crate::win32::capture_foreground_target(target.0 as usize as u64)+ else {+ if !previous.0.is_null() && previous != target {+ let _ = unsafe { SetForegroundWindow(previous) };+ }+ bail!(+ "foreground_unavailable: exact target HWND {:?} disappeared before {operation}; \+ no input was sent",+ target.0+ );+ };++ let result = (|| {+ focus()?;+ let deadline = Instant::now() + Duration::from_millis(250);+ let actual = loop {+ let actual = unsafe { GetForegroundWindow() };+ if crate::win32::foreground_matches_target_or_owned_window(+ foreground_target,+ actual.0 as usize as u64,+ ) {+ break actual;+ }+ if Instant::now() >= deadline {+ break actual;+ }+ sleep(Duration::from_millis(10));+ };+ if !crate::win32::foreground_matches_target_or_owned_window(+ foreground_target,+ actual.0 as usize as u64,+ ) {+ bail!(+ "foreground_unavailable: exact target HWND {:?} or a verified same-process \+ owned window was not foreground while preparing {operation} \+ (actual foreground HWND {:?}); no input was sent",+ target.0,+ actual.0+ );+ }+ body()+ })();++ // Give the target message loop a bounded opportunity to consume the+ // inserted sequence before restoring the user's prior foreground.+ if result.is_ok() {+ sleep(Duration::from_millis(40));+ }+ if !previous.0.is_null() && previous != target {+ let _ = unsafe { SetForegroundWindow(previous) };+ }+ result}/// Build a single Unicode keyboard INPUT struct for one UTF-16 code unit,diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rsindex 9d9c05321c..970660cd60 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs@@ -21,7 +21,8 @@ pub use inject::{};pub use keyboard::{is_xaml_host_hwnd, post_char, post_key, post_type_text, post_type_text_with_delay,- send_key_synthesized, send_text_synthesized, wait_for_focused_descendant,+ send_key_synthesized, send_key_synthesized_after_focus, send_text_synthesized,+ send_text_synthesized_after_focus, wait_for_focused_descendant,};pub use mouse::{has_chromium_descendant, is_chromium_target_window, post_click, post_click_screen,diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rsindex f5b4af1785..30c9c92182 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs@@ -586,7 +586,7 @@ fn send_click_synthesized_mods_impl();}let foreground_target = if activate {- match crate::win32::capture_post_action_foreground_target(target.0 as usize as u64) {+ match crate::win32::capture_foreground_target(target.0 as usize as u64) {Some(target) => Some(target),None => bail!("foreground_unavailable: exact target HWND {:?} disappeared before mouse \@@ -696,7 +696,7 @@ fn send_click_synthesized_mods_impl(}if activate {let actual = GetForegroundWindow();- if !crate::win32::post_action_foreground_matches(+ if !crate::win32::foreground_matches_target_or_owned_window(foreground_target.expect("foreground target captured before input"),actual.0 as usize as u64,) {diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex e2b91730d1..3e0326ac05 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -3912,6 +3912,83 @@ pub struct TypeTextTool {state: Arc<ToolState>,}+fn wait_for_cached_element_keyboard_focus(+ state: &ToolState,+ pid: u32,+ hwnd: u64,+ element_index: usize,+ timeout: std::time::Duration,+) -> bool {+ let deadline = std::time::Instant::now() + timeout;+ loop {+ if state+ .element_cache+ .element_has_keyboard_focus(pid, hwnd, element_index)+ == Some(true)+ {+ return true;+ }+ if std::time::Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(std::time::Duration::from_millis(10));+ }+}++/// Establish exact UIA child focus after the owning top-level HWND is already+/// confirmed foreground. Chromium providers sometimes reject UIA `SetFocus`;+/// in that case a real click at the accessibility-derived center is the+/// bounded fallback. Both routes require `CurrentHasKeyboardFocus` read-back+/// before any keyboard input is allowed to leave the driver.+fn focus_cached_element_for_foreground(+ state: &ToolState,+ pid: u32,+ hwnd: u64,+ element_index: usize,+ click_point: Option<(i32, i32)>,+) -> anyhow::Result<()> {+ // `fg_bypass` is a background-only shield: for Chromium it disables the+ // top-level HWND while UIA runs so the provider cannot self-activate. A+ // disabled foreground window immediately loses foreground on Windows,+ // which would make the enclosing verify-before-SendInput transaction+ // fail closed. The foreground route has already activated the exact HWND,+ // so focus the cached element directly and verify it below.+ let set_focus = state.element_cache.focus_element(pid, hwnd, element_index);+ if set_focus.is_ok()+ && wait_for_cached_element_keyboard_focus(+ state,+ pid,+ hwnd,+ element_index,+ std::time::Duration::from_millis(350),+ )+ {+ return Ok(());+ }++ if let Some((x, y)) = click_point {+ crate::input::send_click_synthesized_active_mods(hwnd, x, y, 1, "left", &[])?;+ if wait_for_cached_element_keyboard_focus(+ state,+ pid,+ hwnd,+ element_index,+ std::time::Duration::from_millis(500),+ ) {+ return Ok(());+ }+ }++ let detail = set_focus+ .err()+ .map(|error| format!("; UIA SetFocus failed: {error}"))+ .unwrap_or_default();+ anyhow::bail!(+ "foreground_unavailable: Windows did not confirm keyboard focus on exact UIA element \+ [{element_index}] in HWND 0x{hwnd:x}{detail}; no keyboard input was sent"+ )+}+static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();#[async_trait]@@ -4170,12 +4247,10 @@ impl Tool for TypeTextTool {// rejected (daemon not at UIAccess integrity), it returns an error// rather than a false success.if delivery == DeliveryMode::Foreground {- // An indexed foreground type targets that element, not whichever- // child happened to retain focus in the top-level window. UIA- // SetFocus is not sufficient for Chromium renderer controls, so- // establish real system focus with the same foreground coordinate- // actuator used by an indexed click before sending Unicode input.- if let Some(idx) = elem_idx {+ // Resolve the optional click fallback before entering the atomic+ // activation/focus/input transaction. The focus itself happens+ // only after exact top-level foreground is confirmed.+ let focus_target = if let Some(idx) = elem_idx {let (cx, cy) =match self.state@@ -4201,25 +4276,19 @@ impl Tool for TypeTextTool {Ok(point) => point,Err(result) => return result,};- let focus_result = tokio::task::spawn_blocking(move || {- crate::input::send_click_synthesized_active_mods(hwnd, cx, cy, 1, "left", &[])- })- .await;- match focus_result {- Ok(Ok(())) => {- tokio::time::sleep(std::time::Duration::from_millis(120)).await;- }- Ok(Err(error)) => return ToolResult::error(error.to_string()),- Err(error) => {- return ToolResult::error(format!(- "foreground element-focus task failed: {error}"- ))- }- }- }+ Some((idx as usize, (cx, cy)))+ } else {+ None+ };let text_fg = text.clone();+ let state = self.state.clone();let r = tokio::task::spawn_blocking(move || {- crate::input::send_text_synthesized(hwnd, &text_fg)+ crate::input::send_text_synthesized_after_focus(hwnd, &text_fg, || {+ if let Some((idx, point)) = focus_target {+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, Some(point))?;+ }+ Ok(())+ })}).await;return match r {@@ -4949,7 +5018,7 @@ impl Tool for PressKeyTool {{return error;}- } else if let Some(idx) = elem_idx {+ } else if let Some(idx) = elem_idx.filter(|_| delivery != DeliveryMode::Foreground) {let state = self.state.clone();let focused = tokio::task::spawn_blocking(move || {crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || {@@ -4975,9 +5044,22 @@ impl Tool for PressKeyTool {// Skipped when px-focus already fronted/clicked the target — the key then// goes via the plain background post path below.if !px_focus && delivery == DeliveryMode::Foreground {+ let focus_target = elem_idx.map(|idx| {+ let point = self+ .state+ .element_cache+ .get_element_center(pid, hwnd, idx as usize);+ (idx as usize, point)+ });+ let state = self.state.clone();let send_result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods.iter().map(String::as_str).collect();- crate::input::send_key_synthesized(hwnd, &key, &m)+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {+ if let Some((idx, point)) = focus_target {+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, point)?;+ }+ Ok(())+ })}).await;return match send_result {@@ -5073,6 +5155,9 @@ impl Tool for HotkeyTool {"pid":{"type":"integer","description":"Target process ID."},"keys":{"type":"array","items":{"type":"string"},"minItems":2,"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},+ "element_index": cua_driver_core::tool_schema::element_index_schema(),+ "element_token": cua_driver_core::tool_schema::element_token_schema(),+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),"x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},"y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},"window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},@@ -5172,12 +5257,41 @@ impl Tool for HotkeyTool {} else {return ToolResult::error("Missing required array field keys.");};- let hwnd_opt = args.opt_u64("window_id");let raw_pid = match args.require_i64("pid") {Ok(v) => v,Err(e) => return e,};let pid = raw_pid as u32;+ let resolved = match cua_driver_core::element_token::resolve_element_args(+ pid as i32,+ args.opt_u64("element_index").map(|value| value as usize),+ args.opt_str("element_token").as_deref(),+ args.opt_str("snapshot_id").as_deref(),+ args.opt_u64("window_id").map(|value| value as u32),+ "hotkey",+ ) {+ Ok(resolved) => resolved,+ Err(error) => return error,+ };+ let elem_idx = match &resolved {+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {+ Some(*element_index)+ }+ cua_driver_core::element_token::ResolvedElement::None => None,+ };+ let hwnd_opt = match &resolved {+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => window_id+ .map(|value| value as u64)+ .or_else(|| args.opt_u64("window_id")),+ cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),+ };+ if elem_idx.is_some() && hwnd_opt.is_none() {+ return ToolResult::error(+ "window_id is required when element_index is used — the element_index cache \+ is scoped per (pid, window_id). Pass the same window_id you used in \+ `get_window_state`.",+ );+ }let delivery = DeliveryMode::from_args(&args);// Resolve HWND: use window_id if given, else pick first window for pid.@@ -5213,6 +5327,11 @@ impl Tool for HotkeyTool {let px = args.get("x").and_then(|v| v.as_f64());let py = args.get("y").and_then(|v| v.as_f64());if let (Some(cx), Some(cy)) = (px, py) {+ if elem_idx.is_some() {+ return ToolResult::error(+ "Pass either element_index (ax) or x,y (px) to hotkey, not both.",+ );+ }let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool())@@ -5334,10 +5453,20 @@ impl Tool for HotkeyTool {// global modifier state, so Chromium never observes Ctrl+Shift+H as a// chord even though the renderer control is focused.let use_send_input = delivery == DeliveryMode::Foreground;+ let focus_target = elem_idx.map(|idx| {+ let point = self.state.element_cache.get_element_center(pid, hwnd, idx);+ (idx, point)+ });+ let state = self.state.clone();let result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods.iter().map(String::as_str).collect();if use_send_input {- crate::input::send_key_synthesized(hwnd, &key, &m)+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {+ if let Some((idx, point)) = focus_target {+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, point)?;+ }+ Ok(())+ })} else {crate::input::post_key(hwnd, &key, &m)}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rsindex 58a6694d96..38f98f89d7 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs@@ -7,7 +7,7 @@ pub mod windows;pub use apps::{list_descendants, list_processes, related_processes, ProcessInfo};pub use installed_apps::{list_installed_apps, InstalledApp};pub(crate) use windows::{- capture_post_action_foreground_target, find_window_by_pid_and_handle, list_windows_via_win32,- post_action_foreground_matches, window_owner_pid,+ capture_foreground_target, find_window_by_pid_and_handle,+ foreground_matches_target_or_owned_window, list_windows_via_win32, window_owner_pid,};pub use windows::{list_windows, resolve_uwp_host_window, WindowInfo};diff --git a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rsindex e200d7aab8..840b11ba5e 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs@@ -154,29 +154,27 @@ fn owner_chain_reaches_target(}#[derive(Clone, Copy, Debug)]-pub(crate) struct PostActionForegroundTarget {+pub(crate) struct ForegroundTarget {hwnd: u64,pid: u32,owner: Option<u64>,}-/// Snapshot the exact target identity and owner before pointer input is sent.-pub(crate) fn capture_post_action_foreground_target(- target: u64,-) -> Option<PostActionForegroundTarget> {+/// Snapshot the exact target identity and owner before global input is sent.+pub(crate) fn capture_foreground_target(target: u64) -> Option<ForegroundTarget> {let pid = window_owner_pid(target)?;let owner = unsafe { GetWindow(HWND(target as *mut _), GW_OWNER) }.ok().filter(|owner| !owner.0.is_null()).map(|owner| owner.0 as usize as u64);- Some(PostActionForegroundTarget {+ Some(ForegroundTarget {hwnd: target,pid,owner,})}-/// Verify the foreground after an exact-target pointer action.+/// Verify the foreground before or after an exact-target global input action.////// The requested HWND must become foreground before input is sent. The action/// may then legitimately open a same-process owned popup or modal, so the@@ -185,8 +183,8 @@ pub(crate) fn capture_post_action_foreground_target(/// A dismiss action may instead destroy the requested owned modal; that is/// accepted only when foreground returns to its snapshotted same-process owner./// Unrelated same-process siblings and foreign foreground windows fail closed.-pub(crate) fn post_action_foreground_matches(- target: PostActionForegroundTarget,+pub(crate) fn foreground_matches_target_or_owned_window(+ target: ForegroundTarget,actual: u64,) -> bool {let current_target_pid = window_owner_pid(target.hwnd);diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.csindex 8799710807..4f7eb410a7 100644--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs@@ -55,6 +55,10 @@ public partial class MainWindow : Window// PostMessage actually arrived and was actionable.var source = HwndSource.FromHwnd(new WindowInteropHelper(this).Handle);source?.AddHook(OnWindowMessage);+ // Start with the deliberate decoy focused. Foreground element-addressed+ // keyboard tests must move focus to their exact UIA target after the+ // top-level activation; inheriting startup focus would hide that race.+ Keyboard.Focus(TxtDeferredInput);PublishFixtureState();}diff --git a/libs/cua-driver/tests/fixtures/shared/web/index.html b/libs/cua-driver/tests/fixtures/shared/web/index.htmlindex 6439929efb..23a860cece 100644--- a/libs/cua-driver/tests/fixtures/shared/web/index.html+++ b/libs/cua-driver/tests/fixtures/shared/web/index.html@@ -73,14 +73,14 @@autocapitalize="off" autocorrect="off" spellcheck="false" /><span class="mirror" id="lbl-input-mirror" data-cua-id="lbl-input-mirror">mirror=</span><input type="number" id="number-input" data-cua-id="number-input"- aria-label="number-input" value="42" />+ aria-label="number-input" value="42" autofocus /></div></fieldset><fieldset><legend>keyboard_task</legend><div class="row">- <input type="text" id="keyboard-input" data-cua-id="keyboard-input" autofocus+ <input type="text" id="keyboard-input" data-cua-id="keyboard-input"aria-label="keyboard-input" placeholder="keyboard input"autocapitalize="off" autocorrect="off" spellcheck="false" /><span class="mirror" id="keyboard-status" data-cua-id="keyboard-status">key_state=none</span>diff --git a/libs/cua-driver/tests/runners/macos-lume/README.md b/libs/cua-driver/tests/runners/macos-lume/README.mdindex d755f7e051..747a0ee90f 100644--- a/libs/cua-driver/tests/runners/macos-lume/README.md+++ b/libs/cua-driver/tests/runners/macos-lume/README.md@@ -40,6 +40,29 @@ repeatable while the private seed carries grants obtained through the normal`CuaDriverLocal.app` prompt flow. The SIP-on check below owns the separate claim thatthe supported permission flow still works with normal platform protection.+## Reuse a prepared private seed++Check the host-local inventory before building from the public base. Reuse a+seed only when its versioned maintainer log records the expected public base,+macOS build, toolchain versions, signing-certificate hash, TCC grants, and+Chrome consent state. The seed and both backups must be stopped:++```bash+SEED=cua-driver-macos-e2e-seed-26.5.2-YYYYMMDD+BACKUP_A="${SEED}-backup-a"+BACKUP_B="${SEED}-backup-b"++for VM in "$SEED" "$BACKUP_A" "$BACKUP_B"; do+ lume get "$VM" --format json | jq -e '.[0].status == "stopped"'+done+```++If those checks and the maintainer log match, skip image creation and clone a+worker from the seed under [Run the acceptance gate](#run-the-acceptance-gate).+Never boot a seed to inspect or update it. Clone a disposable worker and let the+acceptance preflight verify the recorded state there. Build a new versioned seed+when the inventory is missing, incomplete, or stale.+## Create the private seedPull the versioned public base into a mutable local builder. The initial guest@@ -87,6 +110,10 @@ printf '\n%s\n' 'eval "$(/opt/homebrew/bin/brew shellenv)"' \>> "$HOME/.zprofile"```+The pinned Homebrew installer pauses for a Return confirmation after its sudo+check. Keep this step in the VM display and confirm the prompt there. A detached+SSH run can look stalled while it is waiting for that input.+Open a new Terminal window and require each command to succeed:```bash@@ -164,6 +191,19 @@ bash libs/cua-driver/scripts/install-local.sh \~/.local/bin/cua-driver-local permissions grant```+Record the certificate hash after the successful install and keep that identity+for the life of the seed:++```bash+security find-certificate \+ -c 'CuaDriver Local Signing (cua-driver-rs)' -Z "$SIGNING_KEYCHAIN"+codesign -d -r- /Applications/CuaDriverLocal.app 2>&1 \+ | grep 'certificate leaf'+```++Recreating the certificate changes the app identity and invalidates inherited+TCC grants, even when the bundle identifier stays the same.+Complete the Accessibility and Screen Recording prompts. Tahoe 26 also asksseparately for Automation and direct ScreenCaptureKit access. Trigger allremaining consent paths before freezing the seed:@@ -177,19 +217,9 @@ osascript -e \# must not require Automation access to System Events.~/.local/bin/cua-driver-local list_apps '{}'-# A desktop screenshot triggers Tahoe's direct-capture/private-window prompt.-~/.local/bin/cua-driver-local call start_session \- '{"session":"seed-desktop-consent","capture_scope":"desktop"}'-~/.local/bin/cua-driver-local call get_desktop_state \- '{"session":"seed-desktop-consent"}' \- > /tmp/cua-driver-seed-desktop-state.json-~/.local/bin/cua-driver-local call end_session '{"session":"seed-desktop-consent"}'-jq -e '- .screenshot_mime_type == "image/png"- and .screenshot_width > 0- and .screenshot_height > 0- and (.screenshot_png_b64 | length) > 0-' /tmp/cua-driver-seed-desktop-state.json >/dev/null+# The app-owned grant flow probes a desktop capture and triggers Tahoe's+# direct-capture/private-window prompt.+~/.local/bin/cua-driver-local permissions grant```Choose Allow for `Terminal` -> `System Events` and on the CuaDriverLocal@@ -198,8 +228,18 @@ Events. Target-specific Automation prompts may still appear later when a userexplicitly requests an Apple Events-backed browser or app operation; do notpre-grant those in the seed. These are normal macOS consent flows; do not edit`TCC.db`. Rerun the commands and require them to finish without another prompt.-Then verify the daemon's own-identity and the read-only status contract before running the explicit+Notification banners can cover Tahoe's consent controls; swipe the banners away+before acting instead of clicking through them. After enabling Screen Recording,+restart the app-owned daemon once before checking status so the live process+observes the new grant:++```bash+~/.local/bin/cua-driver-local stop+open -a CuaDriverLocal+```++Then verify the daemon's own identity and the read-only status contract before+running the explicitLaunchServices-hosted grant flow. The first command must not raise a dialog;the second is intentionally prompt-capable and must be run by the human:@@ -301,12 +341,14 @@ standard autostart daemon even when a browser row fails.On macOS Tahoe, first-use Chrome can present a native local-network discoveryprompt over `chrome://inspect/#remote-debugging`. The standalone-browser laneuses loopback DevTools and does not need LAN discovery. Before freezing a seed-that will run this optional lane, launch Chrome on that exact page in the VM-display, choose **Don't Allow**, quit Chrome, then relaunch the page and require-that the prompt does not return. Do not answer or dismiss OS consent UI while-the behavior matrix is running. If an existing immutable seed lacks this-decision, clone it to a new versioned seed, complete this setup there, stop it,-and use that new seed for workers; never update the original seed in place.+that will run this optional lane, complete Chrome's welcome screen without+signing in and leave the default-browser and usage-reporting choices disabled.+Launch Chrome on that exact page in the VM display, choose **Don't Allow**, quit+Chrome, then relaunch the page and require that the prompt does not return. Do+not answer or dismiss OS consent UI while the behavior matrix is running. If an+existing immutable seed lacks these decisions, clone it to a new versioned seed,+complete this setup there, stop it, and use that new seed for workers; never+update the original seed in place.The entrypoint refuses the wrong OS, user session, SIP state, dirty orunidentified source, missing dependencies, ad-hoc signature, stale installed