Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): verify foreground focus before input (#3068)

Francesco Bonaccif9192dde69dd
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index ef3a8f7d38..37641828c0 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -413,8 +413,8 @@ than silently changing coordinate spaces.
### `hotkey`
Press a key combination — e.g. `["cmd", "c"]` for Copy, `["cmd", "shift", "4"]` for screenshot selection. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:
-• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. No focus steal. `window_id` here only targets the combo; it does not raise.
-• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.
+• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. `window_id` here only targets the combo; it does not raise.
+• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.
A combo is never driver-verifiable (no read-back) → effect:"unverifiable"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field — to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then `type_text`. If an app only accepts paste, call `clipboard_write`, then `clipboard_read` and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V.
@@ -423,10 +423,13 @@ Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modi
**Arguments:**
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
+- `element_index` (integer, optional): Element index from get_window_state. Cua Driver 0.17 requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.
+- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.
- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"]. items: 2–unbounded
- `pid` (integer, optional): Target process ID.
- `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application. default: `"window"`
- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.
+- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.
- `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action.
`display_id="primary"` is the portable desktop target in this release.
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
index 6746c4fbb3..dc82e03304 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
@@ -795,6 +795,7 @@ fn run_text_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> O
}
thread::sleep(Duration::from_millis(250));
assert_fixture_contains(fixture, &format!("mirror={text}"));
+ assert_fixture_value(fixture, "number-input", "42");
Observation::delivered(passed, Evidence::default())
}
@@ -818,6 +819,7 @@ fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &st
type_response.text()
);
assert_fixture_value(fixture, "keyboard-input", &text);
+ assert_fixture_value(fixture, "number-input", "42");
let post_type = snapshot(fixture);
let mut enter_args =
@@ -837,6 +839,7 @@ fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &st
enter_response.text()
);
assert_fixture_contains(fixture, &format!("key_state=submitted:{text}"));
+ assert_fixture_value(fixture, "number-input", "42");
let mut passed = vec![OracleKind::FixtureState];
if addressing == "px" {
@@ -873,6 +876,7 @@ fn run_press_key_action(fixture: &mut Fixture, addressing: &str, delivery: &str)
let mut passed = vec![OracleKind::FixtureState];
passed.extend(unverified_background_protocol_oracle(&response, delivery));
assert_fixture_contains(fixture, "key_state=enter");
+ assert_fixture_value(fixture, "number-input", "42");
Observation::delivered(passed, Evidence::default())
}
@@ -896,6 +900,7 @@ fn run_hotkey_action(fixture: &mut Fixture, addressing: &str, delivery: &str) ->
response.text()
);
assert_fixture_contains(fixture, "key_state=hotkey");
+ assert_fixture_value(fixture, "number-input", "42");
#[cfg(target_os = "macos")]
if fixture.name == "electron" && addressing == "px" && delivery == "foreground" {
run_macos_selection_hotkeys(fixture);
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
index 3726992c75..31e9da2cd0 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/harness_wpf_test.rs
@@ -718,53 +718,24 @@ fn harness_wpf_type_text() {
let snap = snapshot(driver, pid, wid);
let idx = ax::element_index_by_id(snap.text(), "txt-input")
.expect("txt-input not in snapshot");
-
- // WPF's TextBox needs *keyboard focus* for WM_CHAR delivery — and
- // PostMessage(WM_LBUTTONDOWN) doesn't reliably transfer keyboard
- // focus (WPF's input system treats posted events differently from
- // real ones). Use delivery_mode:"foreground" → SendInput synthesizes
- // an OS-level click that WPF treats identically to a user mouse,
- // landing actual keyboard focus on the TextBox.
- let _ = driver.call(
- "bring_to_front",
- serde_json::json!({
- "pid": pid as i64, "window_id": wid
- }),
- );
- std::thread::sleep(Duration::from_millis(300));
driver.start_behavior_recording();
- let _ = driver.call(
- "click",
- serde_json::json!({
- "pid": pid as i64, "window_id": wid, "element_index": idx,
- "snapshot_id": snap.snapshot_id(),
- "delivery_mode": "foreground"
- }),
- );
- std::thread::sleep(Duration::from_millis(400));
-
- // SendInput's restore_foreground_polling_best_effort may yank
- // foreground back from the harness window between click and
- // type_text. Re-assert foreground so PostMessage WM_CHAR finds
- // the TextBox with keyboard focus.
- let _ = driver.call(
- "bring_to_front",
- serde_json::json!({
- "pid": pid as i64, "window_id": wid
- }),
- );
- std::thread::sleep(Duration::from_millis(300));
-
+ // The fixture deliberately starts with txt-deferred-input focused.
+ // A single indexed type_text call must atomically activate the WPF
+ // window, focus txt-input, confirm that focus, and only then inject.
let resp = driver.call(
"type_text",
serde_json::json!({
"pid": pid as i64,
+ "window_id": wid,
+ "element_index": idx,
+ "snapshot_id": snap.snapshot_id(),
"text": "harness-typed",
"delivery_mode": "foreground"
}),
);
println!("type_text: {}", resp.text());
+ assert!(!resp.is_error(), "type_text failed: {}", resp.text());
std::thread::sleep(Duration::from_millis(700));
let post = snapshot(driver, pid, wid);
@@ -778,6 +749,20 @@ fn harness_wpf_type_text() {
"TextBox mirror did not reflect typed text. Mirror/input lines: {:?}",
mirror_lines
);
+ let decoy_idx = ax::element_index_by_id(text, "txt-deferred-input")
+ .expect("txt-deferred-input not in post-action snapshot");
+ let decoy = post.structured()["elements"]
+ .as_array()
+ .and_then(|elements| {
+ elements
+ .iter()
+ .find(|element| element["element_index"].as_u64() == Some(decoy_idx))
+ })
+ .expect("txt-deferred-input missing from structured elements");
+ assert!(
+ decoy["value"].as_str().unwrap_or_default().is_empty(),
+ "foreground type_text leaked into the pre-focused decoy: {decoy}"
+ );
println!("✅ harness_wpf_type_text: TextBox mirror advanced to 'harness-typed'");
Vec::new()
},
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
index 1310c9e200..01dfc99795 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs
@@ -2029,8 +2029,7 @@ pub fn scroll_element(pid: u32, idx: usize, direction: &str, amount: usize) -> R
/// updates its renderer-owned focused control. Sending key events immediately
/// after the acknowledgement can therefore split one string between the old
/// and new controls. Wait for the target's Focused state to become observable;
-/// if a toolkit does not publish that state, retain the historical successful
-/// result after a bounded settling interval.
+/// an acknowledgement without read-back is not sufficient for global input.
pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {
bounded(
async {
@@ -2078,7 +2077,7 @@ pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {
}
}
}
- Ok(true)
+ Ok(false)
},
|| Err(anyhow!("focus_element timed out for pid {pid}")),
)
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
index f408864f1f..4983a721bb 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
@@ -967,10 +967,11 @@ fn ewmh_activate_window(
/// primitives (proper `x_server_time` stamping beats the WM's focus-stealing
/// prevention).
///
-/// Best-effort: if no X display can be opened the body still runs (without
-/// activation) so a headless/Wayland path degrades rather than hard-fails.
-/// `settle_ms` is the pause after activation before the first injected event —
-/// the WM needs a moment to complete the focus swap (mirrors the macOS settle).
+/// The transition is confirmed from both EWMH active-window state and the X11
+/// core input-focus tree before `body` runs. A fixed delay or a successful
+/// `XSetInputFocus` return is not evidence that global XTest input is safe.
+/// `settle_ms` is retained as a compatibility hint and folded into the bounded
+/// confirmation timeout; it is no longer an unconditional sleep.
pub fn with_x11_foreground<T>(
xid: u64,
settle_ms: u64,
@@ -978,16 +979,18 @@ pub fn with_x11_foreground<T>(
) -> Result<T> {
let display = unsafe { x11::xlib::XOpenDisplay(ptr::null()) };
if display.is_null() {
- return body();
+ bail!("foreground_unavailable: cannot open DISPLAY to verify exact X11 input focus");
}
let prior = ewmh_active_window(display);
+ let mut prior_core_focus: x11::xlib::Window = 0;
+ let mut prior_revert = 0;
+ unsafe {
+ x11::xlib::XGetInputFocus(display, &mut prior_core_focus, &mut prior_revert);
+ }
ewmh_activate_window(display, xid as x11::xlib::Window, prior.unwrap_or(0));
unsafe {
x11::xlib::XSync(display, 0);
}
- if settle_ms > 0 {
- std::thread::sleep(std::time::Duration::from_millis(settle_ms));
- }
// EWMH `_NET_ACTIVE_WINDOW` is honored as *raise-only* by WMs with
// focus-stealing prevention (e.g. KWin): the window reaches the top of the
// stack — enough for a coordinate click, which lands by stacking — but the X
@@ -1008,12 +1011,45 @@ pub fn with_x11_foreground<T>(
x11::xlib::XSync(display, 0);
x11::xlib::XSetErrorHandler(prev_handler);
}
- let result = body();
- // Restore the prior active window (brief swap, like macOS/Windows).
+ let timeout = std::time::Duration::from_millis(settle_ms.max(400));
+ let deadline = std::time::Instant::now() + timeout;
+ let target = xid as x11::xlib::Window;
+ let focused = loop {
+ let active = ewmh_active_window(display) == Some(target);
+ if active && x11_focus_is_within(display, target) {
+ break true;
+ }
+ if std::time::Instant::now() >= deadline {
+ break false;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(10));
+ };
+ let result = if focused {
+ body()
+ } else {
+ let active = ewmh_active_window(display).unwrap_or(0);
+ Err(anyhow::anyhow!(
+ "foreground_unavailable: X11 did not confirm active window and input focus within \
+ exact target 0x{xid:x} before the {:?} deadline (active=0x{active:x}); no input was sent",
+ timeout
+ ))
+ };
+
+ // Restore both the EWMH active toplevel and the exact prior core focus.
if let Some(p) = prior {
ewmh_activate_window(display, p, xid as x11::xlib::Window);
+ }
+ if prior_core_focus != 0 {
unsafe {
+ let previous_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));
+ x11::xlib::XSetInputFocus(
+ display,
+ prior_core_focus,
+ prior_revert,
+ x11::xlib::CurrentTime,
+ );
x11::xlib::XSync(display, 0);
+ x11::xlib::XSetErrorHandler(previous_handler);
}
}
unsafe {
@@ -1022,6 +1058,47 @@ pub fn with_x11_foreground<T>(
result
}
+fn x11_focus_is_within(display: *mut x11::xlib::Display, target: x11::xlib::Window) -> bool {
+ let mut focused: x11::xlib::Window = 0;
+ let mut revert_to = 0;
+ unsafe {
+ x11::xlib::XGetInputFocus(display, &mut focused, &mut revert_to);
+ }
+ if focused == target {
+ return true;
+ }
+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };
+ while focused != 0 && focused != root {
+ let mut query_root = 0;
+ let mut parent = 0;
+ let mut children: *mut x11::xlib::Window = ptr::null_mut();
+ let mut child_count = 0;
+ let status = unsafe {
+ x11::xlib::XQueryTree(
+ display,
+ focused,
+ &mut query_root,
+ &mut parent,
+ &mut children,
+ &mut child_count,
+ )
+ };
+ if !children.is_null() {
+ unsafe {
+ x11::xlib::XFree(children.cast());
+ }
+ }
+ if status == 0 || parent == 0 || parent == focused {
+ return false;
+ }
+ if parent == target {
+ return true;
+ }
+ focused = parent;
+ }
+ false
+}
+
/// Activate `xid` and LEAVE it active (no restore) — the persistent foreground
/// swap behind `bring_to_front`. Returns the window that was active before, so
/// the caller can report/inspect it. Best-effort; returns `None` prior on a
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 046592ba68..514b0d1812 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -2995,24 +2995,18 @@ impl Tool for TypeTextTool {
&& (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid))
{
if let Some(idx) = resolved_elem_idx {
- let focused =
- tokio::task::spawn_blocking(move || crate::atspi::focus_element(pid, idx))
- .await;
- match focused {
- Ok(Ok(true)) => {}
- Ok(Ok(false)) => {
- return ToolResult::error(format!(
- "AT-SPI Component.GrabFocus returned false for element {idx}"
- ))
- }
- Ok(Err(error)) => return ToolResult::error(error.to_string()),
- Err(error) => return ToolResult::error(format!("Task error: {error}")),
- }
-
let text_w = text.clone();
- let result =
- tokio::task::spawn_blocking(move || crate::wayland::type_text(xid, &text_w))
- .await;
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::with_target_foreground(pid, xid, || {
+ if !crate::atspi::focus_element(pid, idx)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {idx}"
+ );
+ }
+ crate::wayland::type_text_focused(&text_w)
+ })
+ })
+ .await;
return match result {
Ok(Ok(())) => ToolResult::text(format!(
"Typed {text_len} character(s) (via Wayland virtual-keyboard)."
@@ -3062,8 +3056,20 @@ impl Tool for TypeTextTool {
);
}
let text_w = text.clone();
- let result =
- tokio::task::spawn_blocking(move || crate::wayland::type_text(xid, &text_w)).await;
+ let idx = resolved_elem_idx;
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::with_target_foreground(pid, xid, || {
+ if let Some(idx) = idx {
+ if !crate::atspi::focus_element(pid, idx)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {idx}"
+ );
+ }
+ }
+ crate::wayland::type_text_focused(&text_w)
+ })
+ })
+ .await;
return match result {
Ok(Ok(())) => ToolResult::text(format!(
"Typed {text_len} character(s) (via Wayland virtual-keyboard)."
@@ -3131,24 +3137,17 @@ impl Tool for TypeTextTool {
// producing the renderer input event, so web embedders use real XTest
// key events. Native toolkits keep their verifiable AT-SPI path below.
if delivery.is_foreground() && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid)) {
- if let Some(idx) = resolved_elem_idx {
- let focused =
- tokio::task::spawn_blocking(move || crate::atspi::focus_element(pid, idx))
- .await;
- match focused {
- Ok(Ok(true)) => {}
- Ok(Ok(false)) => {
- return ToolResult::error(format!(
- "AT-SPI Component.GrabFocus returned false for element {idx}"
- ))
- }
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
let text_f = text.clone();
+ let idx = resolved_elem_idx;
let result = tokio::task::spawn_blocking(move || {
crate::input::with_x11_foreground(xid, 80, || {
+ if let Some(idx) = idx {
+ if !crate::atspi::focus_element(pid, idx)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {idx}"
+ );
+ }
+ }
crate::input::send_type_text_xtest(&text_f)
})
})
@@ -3445,10 +3444,10 @@ impl Tool for PressKeyTool {
};
let mods: Vec<String> = args.str_array("modifiers");
- // Surface 6: resolve element_token / element_index for the window-id
- // hint. press_key targets a window via XSendEvent, so we only need
- // the resolved window_id — element_index itself is not used to
- // address an AX node here (no focus-grab path).
+ // Surface 6: resolve the element token/index into both its owning
+ // window and exact child. Foreground delivery establishes child focus
+ // inside the verified top-level activation transaction; background
+ // XSendEvent retains the historical direct-window path.
let element_token_arg = args.opt_str("element_token");
let window_id_arg = args.opt_u64("window_id");
let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);
@@ -3469,6 +3468,12 @@ impl Tool for PressKeyTool {
}
cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
};
+ let resolved_element_idx = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
let xid = match xid_opt {
Some(x) => x,
None => {
@@ -3511,7 +3516,7 @@ impl Tool for PressKeyTool {
if px.is_some() != py.is_some() {
return ToolResult::error("Pass both x and y to press_key, or neither.");
}
- if px.is_some() && element_index_arg.is_some() {
+ if px.is_some() && resolved_element_idx.is_some() {
return ToolResult::error(
"Pass either element_index (ax) or x,y (px) to press_key, not both.",
);
@@ -3521,7 +3526,7 @@ impl Tool for PressKeyTool {
// Preserve legacy modifiers by promoting the request to a chord.
if crate::wayland::is_inject_mode() {
if let Err(error) =
- focus_nested_inject_target(pid, xid, element_index_arg, px.zip(py)).await
+ focus_nested_inject_target(pid, xid, resolved_element_idx, px.zip(py)).await
{
return error;
}
@@ -3549,28 +3554,6 @@ impl Tool for PressKeyTool {
};
}
- // An element-addressed keypress needs to establish the target's
- // focus before the window-level X11 key event is sent. AT-SPI's
- // primary action is the focus-free route for editable web controls;
- // resolving the element only for its window ID silently loses the key
- // on Chromium inputs whose window is backgrounded.
- if let Some(element_index) = element_index_arg {
- let focused = tokio::task::spawn_blocking(move || {
- crate::atspi::focus_element(pid, element_index)
- })
- .await;
- match focused {
- Ok(Ok(true)) => {}
- Ok(Ok(false)) => {
- return ToolResult::error(format!(
- "AT-SPI Component.GrabFocus returned false for element {element_index}"
- ))
- }
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
-
let px_target = {
if let (Some(cx), Some(cy)) = (px, py) {
let from_zoom = args.bool_or("from_zoom", false);
@@ -3599,16 +3582,25 @@ impl Tool for PressKeyTool {
// to become a chord here — otherwise the modifiers are dropped and the
// bare key is typed as a character (Ctrl+S inserts a literal "s").
if crate::wayland::wayland_input_enabled() {
- let result = match press_key_chord(&mods, &key) {
- None => {
- let key_w = key.clone();
- tokio::task::spawn_blocking(move || crate::wayland::press_key(xid, &key_w))
- .await
- }
- Some(chord) => {
- tokio::task::spawn_blocking(move || crate::wayland::hotkey(xid, &chord)).await
- }
- };
+ let key_w = key.clone();
+ let chord = press_key_chord(&mods, &key);
+ let idx = resolved_element_idx;
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::with_target_foreground(pid, xid, || {
+ if let Some(idx) = idx {
+ if !crate::atspi::focus_element(pid, idx)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {idx}"
+ );
+ }
+ }
+ match chord {
+ Some(keys) => crate::wayland::hotkey_focused(&keys),
+ None => crate::wayland::press_key_focused(&key_w),
+ }
+ })
+ })
+ .await;
return match result {
Ok(Ok(())) => ToolResult::text(format!(
"Pressed key '{key}' (via Wayland virtual-keyboard)."
@@ -3624,7 +3616,10 @@ impl Tool for PressKeyTool {
// click restores the prior top-level before returning.
let deliver_fg = delivery.is_foreground();
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
- if mods.is_empty() && key_for_task.eq_ignore_ascii_case("enter") {
+ if resolved_element_idx.is_none()
+ && mods.is_empty()
+ && key_for_task.eq_ignore_ascii_case("enter")
+ {
if inject_terminal_input(pid, xid, "\n")? {
return Ok(());
}
@@ -3637,9 +3632,23 @@ impl Tool for PressKeyTool {
// XSendEvent (no focus steal) for apps that accept it.
if deliver_fg {
return crate::input::with_x11_foreground(xid, 80, || {
+ if let Some(element_index) = resolved_element_idx {
+ if !crate::atspi::focus_element(pid, element_index)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {element_index}"
+ );
+ }
+ }
crate::input::send_key_xtest(&key_for_task, &m)
});
}
+ if let Some(element_index) = resolved_element_idx {
+ if !crate::atspi::focus_element(pid, element_index)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {element_index}"
+ );
+ }
+ }
if let Some((x, y)) = px_target {
crate::input::send_key_at(xid, x, y, &key_for_task, &m)
} else {
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
index d379737538..8982c0eda9 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
@@ -1217,6 +1217,33 @@ pub fn activate_window_for_input_target(
)
}
+/// Run a focus-bound keyboard transaction only after a compositor adapter has
+/// confirmed the exact PID/window pair, and restore the previously focused
+/// toplevel afterward. Global virtual-keyboard/libei input is refused when the
+/// compositor cannot provide this read-back contract.
+pub fn with_target_foreground<T>(
+ pid: u32,
+ window_id: u64,
+ body: impl FnOnce() -> anyhow::Result<T>,
+) -> anyhow::Result<T> {
+ if let Some(window) = sway_ipc::window_for_id(window_id) {
+ if window.pid != pid {
+ anyhow::bail!(
+ "foreground_unavailable: Sway window {window_id} belongs to pid {}, not pid {pid}",
+ window.pid
+ );
+ }
+ return sway_ipc::with_focused_container(window_id, body);
+ }
+ if shell_helper::trusted_window_for_id(pid, window_id).is_some() {
+ return shell_helper::with_focused_window(pid, window_id, body);
+ }
+ anyhow::bail!(
+ "foreground_unavailable: no trusted Wayland compositor adapter can confirm exact \
+ target window {window_id} for pid {pid}; no global keyboard input was sent"
+ )
+}
+
/// Query the first `wl_output`'s pixel dimensions via a short Wayland
/// roundtrip, independent of the virtual-pointer protocol. Used by the libei
/// fallback (which never opens a `VptrSession`) to reproduce the vptr path's
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
index b32e4aa984..1a791db5fc 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
@@ -417,8 +417,16 @@ pub fn activate_window(window_id: u64) -> bool {
if !accepted {
return false;
}
- std::thread::sleep(Duration::from_millis(60));
- window_is_focused(window_id)
+ let deadline = std::time::Instant::now() + Duration::from_millis(500);
+ loop {
+ if window_is_focused(window_id) {
+ return true;
+ }
+ if std::time::Instant::now() >= deadline {
+ return false;
+ }
+ std::thread::sleep(Duration::from_millis(10));
+ }
}
fn trusted_activate_window(window_id: u64) -> bool {
@@ -430,12 +438,20 @@ fn trusted_activate_window(window_id: u64) -> bool {
if !accepted {
return false;
}
- std::thread::sleep(Duration::from_millis(60));
- trusted_shell_windows(None).is_some_and(|windows| {
- windows
- .into_iter()
- .any(|window| window.info.xid == u64::from(window_id) && window.focused)
- })
+ let deadline = std::time::Instant::now() + Duration::from_millis(500);
+ loop {
+ if trusted_shell_windows(None).is_some_and(|windows| {
+ windows
+ .into_iter()
+ .any(|window| window.info.xid == u64::from(window_id) && window.focused)
+ }) {
+ return true;
+ }
+ if std::time::Instant::now() >= deadline {
+ return false;
+ }
+ std::thread::sleep(Duration::from_millis(10));
+ }
}
fn window_is_focused(window_id: u32) -> bool {
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs
index 971b846166..bc090b94cb 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/sway_ipc.rs
@@ -182,6 +182,19 @@ fn focus_container(id: u64) -> bool {
.is_ok_and(|status| status.success())
}
+fn wait_for_container_focus(id: u64, timeout: std::time::Duration) -> bool {
+ let deadline = std::time::Instant::now() + timeout;
+ loop {
+ if window_for_id(id).is_some_and(|window| window.focused) {
+ return true;
+ }
+ if std::time::Instant::now() >= deadline {
+ return false;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(10));
+ }
+}
+
/// Briefly focus one compositor-attested container, run `body`, then restore
/// the previously focused container. The caller must already have verified the
/// target container's PID and id through [`window_for_id`].
@@ -197,8 +210,7 @@ pub fn with_focused_container<T>(
if !focus_container(id) {
anyhow::bail!("Sway refused to focus exact container {id}");
}
- std::thread::sleep(std::time::Duration::from_millis(80));
- if !window_for_id(id).is_some_and(|window| window.focused) {
+ if !wait_for_container_focus(id, std::time::Duration::from_millis(500)) {
anyhow::bail!("Sway did not confirm focus on exact container {id}");
}
let result = body();
@@ -208,8 +220,7 @@ pub fn with_focused_container<T>(
if !focus_container(prior) {
anyhow::bail!("Sway could not restore prior container {prior}");
}
- std::thread::sleep(std::time::Duration::from_millis(80));
- if !window_for_id(prior).is_some_and(|window| window.focused) {
+ if !wait_for_container_focus(prior, std::time::Duration::from_millis(500)) {
anyhow::bail!("Sway did not confirm restored container {prior}");
}
Ok(())
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
index 724217369f..371773760d 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/ax_actions.rs
@@ -284,6 +284,29 @@ pub fn focus_element(element_ptr: usize) -> anyhow::Result<()> {
}
}
+/// Report whether `element_ptr` is the application's currently focused element.
+///
+/// This is a read-only confirmation for the foreground typing rung: an
+/// `AXFocused` write can be accepted by the element and then immediately
+/// clobbered when AppKit installs the window's remembered first responder, so
+/// "the write returned success" is not evidence that focus stuck. Identity is
+/// compared with `CFEqual` because the app hands back a fresh `AXUIElementRef`
+/// for the same underlying element.
+///
+/// A `false` return is deliberately conservative: an app whose
+/// `AXFocusedUIElement` is unreadable reports not-focused, which at worst costs
+/// one extra re-apply.
+pub fn is_element_focused(pid: i32, element_ptr: usize) -> bool {
+ unsafe {
+ let Some(focused) = crate::ax::bindings::focused_element_of_pid(pid) else {
+ return false;
+ };
+ let same = CFEqual(focused as CFTypeRef, element_ptr as CFTypeRef) != 0;
+ CFRelease(focused as CFTypeRef);
+ same
+ }
+}
+
/// Set the AXValue of an element (for dropdowns, text fields, etc.).
pub fn set_ax_value(element_ptr: usize, value: &str) -> anyhow::Result<()> {
let err = unsafe { set_string_attr(element_ptr as AXUIElementRef, "AXValue", value) };
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
index 87a2e026f0..a657ced2b2 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
@@ -660,15 +660,34 @@ pub fn make_exact_window_key(target_pid: libc::pid_t, target_wid: u32) -> bool {
true
}
-/// Tool-agnostic foreground-assist: briefly front `window_id`, run `body` (which
-/// posts the synthetic input), then restore the prior frontmost process.
+/// Tool-agnostic foreground-assist: briefly front `window_id`, wait for the
+/// activation to actually land, run `body` (which posts the synthetic input),
+/// then restore the prior frontmost process.
///
/// This is the `delivery_mode:"foreground"` rung of the best-effort-background
-/// ladder, shared by `type_text` and `click`. It is the same brief front →
-/// act → restore primitive `press_key`/`hotkey` use for NSMenu key dispatch —
-/// see [`with_menu_shortcut_activation`], which this delegates to. Reached only
-/// when the agent has seen the background rungs fail (clicks) or the field is
-/// unverifiable + focus-sensitive (Catalyst typing).
+/// ladder, shared by `type_text` and `click`. Reached only when the agent has
+/// seen the background rungs fail (clicks) or the field is unverifiable +
+/// focus-sensitive (Catalyst typing).
+///
+/// ## Why this does not delegate to [`with_menu_shortcut_activation`]
+///
+/// It used to. That helper posts `set_front` and calls `action` immediately,
+/// which is correct for its own purpose: NSMenu key dispatch only needs the key
+/// event *enqueued* in the target's run-loop queue, so first-responder identity
+/// is irrelevant and the sub-millisecond front → act → restore is a feature.
+///
+/// Input delivery has the opposite requirement. `set_front` is asynchronous, so
+/// running the body straight away means the body's `AXFocused` write races
+/// AppKit's own activation. AppKit wins: when activation completes it installs
+/// the window's remembered first responder and clobbers the write. The
+/// keystrokes then land wherever the app chose — for a Catalyst app such as
+/// WhatsApp, the message list rather than the composer, which silently scrolls
+/// the transcript instead of typing.
+///
+/// So this waits for the activation to be observable before running `body`, the
+/// same ordering [`with_foreground_hid_activation`] already relies on. The wait
+/// is a bounded poll rather than a fixed sleep: an app that activates in 10 ms
+/// pays 10 ms, and a slow Catalyst/RDP surface still gets its full budget.
///
/// Returns `Ok(true)` when the brief activation happened, `Ok(false)` when the
/// fronting SPIs are unavailable (the body still ran, just without a front).
@@ -677,7 +696,81 @@ pub fn with_foreground_assist(
target_wid: u32,
body: impl FnOnce() -> anyhow::Result<()>,
) -> anyhow::Result<bool> {
- with_menu_shortcut_activation(target_pid, target_wid, body)
+ let set_front = match set_front_process_fn() {
+ Some(f) => f,
+ None => {
+ // SPIs unavailable — run body anyway without activation.
+ body()?;
+ return Ok(false);
+ }
+ };
+
+ let mut prev_psn = [0u8; 8];
+ let prev_ok = get_front_process_fn()
+ .map(|f| unsafe { f(prev_psn.as_mut_ptr() as *mut c_void) } == 0)
+ .unwrap_or(false);
+
+ let mut target_psn = [0u8; 8];
+ if !get_process_psn_for_window(target_wid, target_pid, &mut target_psn) {
+ body()?;
+ return Ok(false);
+ }
+
+ unsafe { set_front(target_psn.as_ptr() as *const c_void, target_wid, 0x400) };
+ // `set_front` moves WindowServer's front process but does not make the
+ // target's NSWindow key, and AppKit installs a first responder only for a
+ // key window. Without this the app is "frontmost" to WindowServer while
+ // remaining, from AppKit's point of view, unfocused — so the AXFocused
+ // write in the body has no responder chain to attach to.
+ make_exact_window_key(target_pid, target_wid);
+ await_window_focused(target_pid, target_wid);
+
+ let result = body();
+
+ if prev_ok {
+ unsafe { set_front(prev_psn.as_ptr() as *const c_void, 0, 0x400) };
+ }
+
+ result?;
+ Ok(true)
+}
+
+/// Upper bound on how long [`with_foreground_assist`] waits for a requested
+/// activation to become observable. Chosen to cover a Catalyst app's activation
+/// plus key-window install; past this the caller proceeds anyway so a stubborn
+/// target degrades to the old behavior instead of hanging.
+const ACTIVATION_WAIT_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(400);
+
+/// Poll interval for [`await_window_focused`]. Short enough that a fast native
+/// app pays roughly one tick, long enough not to spin on the WindowServer.
+const ACTIVATION_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
+
+/// Block until `target_wid` is the application's focused AX window, or the
+/// timeout expires. Returns whether that state was observed.
+///
+/// The predicate is deliberately `AXFocusedWindow` and not
+/// `NSWorkspace.frontmostApplication`. The latter does not observe a
+/// SkyLight-level front-process change at all: polling it every 15ms across a
+/// full foreground `type_text` against WhatsApp showed zero transitions while
+/// the target was demonstrably being fronted, so a frontmost-based wait always
+/// burns its whole timeout and never actually gates on anything. `AXFocusedWindow`
+/// is the same proof [`preserves_exact_existing_focus`] already trusts to decide
+/// whether a window is focused.
+///
+/// A `false` return is not fatal: the caller proceeds with delivery regardless,
+/// because a target that never reports focus is exactly the case the pre-existing
+/// best-effort contract already covered.
+fn await_window_focused(pid: libc::pid_t, window_id: u32) -> bool {
+ let deadline = std::time::Instant::now() + ACTIVATION_WAIT_TIMEOUT;
+ loop {
+ if crate::ax::bindings::focused_window_id_of_pid(pid) == Some(window_id) {
+ return true;
+ }
+ if std::time::Instant::now() >= deadline {
+ return false;
+ }
+ std::thread::sleep(ACTIVATION_POLL_INTERVAL);
+ }
}
/// Activate an exact target window for a global HID keyboard action.
@@ -721,7 +814,14 @@ pub fn with_foreground_hid_activation(
anyhow::bail!("WindowServer rejected foreground HID activation");
}
- std::thread::sleep(std::time::Duration::from_millis(40));
+ make_exact_window_key(target_pid, target_wid);
+ if !await_window_focused(target_pid, target_wid) {
+ if prev_ok {
+ unsafe { set_front(prev_psn.as_ptr() as *const c_void, 0, 0x400) };
+ }
+ anyhow::bail!("exact target window did not become focused for foreground HID delivery");
+ }
+
let result = action();
std::thread::sleep(std::time::Duration::from_millis(40));
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
index 685bf9e31c..699a8e8f12 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
@@ -37,14 +37,15 @@ fn def() -> &'static ToolDef {
window by default:\n\
• `background` (default): post the combo to the target pid WITHOUT \
fronting or raising it — uses the macOS 14+ auth-message envelope so \
- Chromium/Electron accept it as trusted live input. No focus steal. \
+ Chromium/Electron accept it as trusted live input. With an AX target, \
+ focus that exact element first. No top-level focus steal. \
`window_id` here only targets the combo; it does not raise.\n\
• `foreground`: briefly front the window (NSMenu path, < 1 ms via \
SLPSSetFrontProcessWithOptions) so native menu key-equivalents \
(Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the \
explicit escalation for menu-bar shortcuts on non-Chromium apps that \
- ignore a background combo. With x,y, the focused field receives \
- the chord through the foreground HID queue (needed by native \
+ ignore a background combo. With an AX target or x,y, the focused field \
+ receives the chord through the foreground HID queue (needed by native \
Chromium fields such as the omnibox). Requires window_id.\n\n\
A combo is never driver-verifiable (no read-back) → effect:\"unverifiable\"; \
confirm via screenshot. NOTE: a keyboard combo does NOT focus a text \
@@ -75,6 +76,9 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "Target window. Required for delivery_mode:\"foreground\" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode."
},
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the chord to the frontmost application." },
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
@@ -95,6 +99,23 @@ fn is_modifier(k: &str) -> bool {
)
}
+const HOTKEY_FOCUS_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(200);
+const HOTKEY_FOCUS_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
+
+fn focus_hotkey_element(pid: i32, element_ptr: usize) -> anyhow::Result<()> {
+ let deadline = std::time::Instant::now() + HOTKEY_FOCUS_TIMEOUT;
+ loop {
+ crate::input::ax_actions::focus_element(element_ptr)?;
+ if crate::input::ax_actions::is_element_focused(pid, element_ptr) {
+ return Ok(());
+ }
+ if std::time::Instant::now() >= deadline {
+ anyhow::bail!("requested hotkey element did not become focused");
+ }
+ std::thread::sleep(HOTKEY_FOCUS_POLL_INTERVAL);
+ }
+}
+
fn screen_sharing_modifier_delivery_error(
is_screen_sharing: bool,
has_modifiers: bool,
@@ -130,8 +151,6 @@ impl Tool for HotkeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- let _ = &self.state;
-
if args.opt_str("scope").as_deref() == Some("desktop")
&& args.get("pid").is_none()
&& args.get("window_id").is_none()
@@ -209,13 +228,60 @@ impl Tool for HotkeyTool {
// Use the last non-modifier key; if there are multiple, treat earlier ones as extra keys.
let key = non_modifiers.last().unwrap().clone();
let key_display = raw_keys.join("+");
- let window_id = args.opt_u64("window_id").map(|v| v as u32);
+ let element_token_arg = args.opt_str("element_token");
+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);
+ let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);
+ let resolved = match cua_driver_core::element_token::resolve_element_args(
+ pid,
+ element_index_arg,
+ element_token_arg.as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
+ window_id_arg,
+ "hotkey",
+ ) {
+ Ok(resolved) => resolved,
+ Err(error) => return error,
+ };
+ let (element_index, window_id) = match resolved {
+ cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),
+ cua_driver_core::element_token::ResolvedElement::Element {
+ window_id,
+ element_index,
+ via_token: _,
+ } => (Some(element_index), window_id),
+ };
// delivery_mode gates whether we raise: background (default) never fronts
// the window — passing window_id only targets the combo. foreground is the
// explicit NSMenu-activation rung for menu shortcuts that ignore a
// background combo (matches click/type_text).
let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());
let fg = delivery_mode.is_foreground();
+ let px = args.get("x").and_then(|value| value.as_f64());
+ let py = args.get("y").and_then(|value| value.as_f64());
+ if px.is_some() && py.is_some() && element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_index (ax) or x,y (px) to hotkey, not both.",
+ );
+ }
+
+ let element_guard = if let (Some(index), Some(window_id)) = (element_index, window_id) {
+ match self
+ .state
+ .element_cache
+ .get_element_retained(pid, window_id, index)
+ {
+ Some(guard) => Some(guard),
+ None => {
+ return ToolResult::error(format!(
+ "Element index {index} not found. Call get_window_state first."
+ ));
+ }
+ }
+ } else {
+ None
+ };
+ let element_ptr = element_guard.as_ref().map(|guard| guard.as_ptr());
+
let screen_sharing_target = crate::input::keyboard::is_screen_sharing_pid(pid);
if let Some(error) = screen_sharing_modifier_delivery_error(
screen_sharing_target,
@@ -238,7 +304,7 @@ impl Tool for HotkeyTool {
match super::gate_background_window_action(
pid,
wid,
- None,
+ element_ptr,
cua_driver_core::background_input::BackgroundAction::GenericKey,
)
.await
@@ -253,13 +319,52 @@ impl Tool for HotkeyTool {
None
};
- // px form: focus the field before sending the combo. Foreground delivery
- // still needs to front the target for the chord itself: the focus helper
- // restores the previous app before returning.
- let px_focus = {
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
- if let (Some(cx), Some(cy)) = (px, py) {
+ // Web-content AX nodes can acknowledge AXFocused without moving the
+ // renderer's real first responder. That makes a direct focus write an
+ // unsafe oracle for Chromium/WebKit hotkeys: the following PID/HID
+ // chord can still land on the renderer's remembered control. Resolve
+ // the requested AX node's exact center and reuse the proven PX focus
+ // ladder for web areas. The request remains snapshot-bound AX
+ // targeting; only the focus transport falls back through a hit-test
+ // and, on the explicit foreground rung, a real click when required.
+ let web_ax_focus_xy =
+ if let (Some(ptr), Some(wid), Some(index)) = (element_ptr, window_id, element_index) {
+ let is_web = tokio::task::spawn_blocking(move || {
+ super::type_text::target_in_web_area(pid, Some((ptr, Some(index))), Some(wid))
+ })
+ .await
+ .unwrap_or(true);
+ if is_web {
+ tokio::task::spawn_blocking(move || unsafe {
+ let (screen_x, screen_y) = crate::ax::bindings::element_screen_center(
+ ptr as crate::ax::bindings::AXUIElementRef,
+ )?;
+ let frame = super::px_frame::resolve_window_px_frame(wid).ok()?;
+ Some((
+ (screen_x - frame.bounds.x) * frame.scale,
+ (screen_y - frame.bounds.y) * frame.scale,
+ ))
+ })
+ .await
+ .unwrap_or(None)
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
+ // PX form, plus the web-content AX fallback above: focus the field
+ // before sending the combo. Foreground delivery still needs to front
+ // the target for the chord itself because the focus helper restores
+ // the previous app before returning.
+ let coordinate_focus = {
+ let focus_xy = match ((px, py), web_ax_focus_xy) {
+ ((Some(cx), Some(cy)), _) => Some((cx, cy)),
+ ((None, None), Some(center)) => Some(center),
+ _ => None,
+ };
+ if let Some((cx, cy)) = focus_xy {
let from_zoom = args
.get("from_zoom")
.and_then(|v| v.as_bool())
@@ -302,12 +407,12 @@ impl Tool for HotkeyTool {
|| async move {
tokio::task::spawn_blocking(move || {
let m: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- match (fg, px_focus, window_id) {
+ match (fg, coordinate_focus, window_id, element_ptr) {
// Chrome's native omnibox and Chromium/Electron inputs
// require a genuine foreground HID chord. Keep the exact
// target frontmost until both key events are consumed;
// otherwise Cmd+A/Cmd+V can be silently ignored.
- (true, true, Some(wid)) => {
+ (true, true, Some(wid), _) => {
crate::input::skylight::with_foreground_hid_activation(
pid as libc::pid_t,
wid,
@@ -321,11 +426,26 @@ impl Tool for HotkeyTool {
)?;
Ok(())
}
+ // An AX-addressed chord has the same renderer-focus
+ // requirement as the px form. Activate the exact window,
+ // establish and confirm the requested child focus after
+ // activation, then use the guarded global HID queue.
+ (true, false, Some(wid), Some(ptr)) => {
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ || {
+ focus_hotkey_element(pid, ptr)?;
+ crate::input::keyboard::press_key_bare_global(&key, &m)
+ },
+ )?;
+ Ok(())
+ }
// Screen Sharing is an input forwarder: modifier flags
// on a PID-routed base-key event are not relayed to the
// guest. Emit the physical modifier down/base/up
// sequence through the guarded foreground HID path.
- (true, false, Some(wid))
+ (true, false, Some(wid), None)
if crate::input::keyboard::is_screen_sharing_pid(pid) =>
{
crate::input::skylight::with_foreground_hid_activation(
@@ -337,7 +457,7 @@ impl Tool for HotkeyTool {
}
// foreground rung: briefly front the window so NSMenu key
// equivalents dispatch, then restore prior frontmost.
- (true, false, Some(wid)) => {
+ (true, false, Some(wid), None) => {
crate::input::skylight::with_menu_shortcut_activation(
pid as libc::pid_t,
wid,
@@ -347,6 +467,10 @@ impl Tool for HotkeyTool {
}
// background (default): auth-envelope post to the pid, no
// raise — even when window_id was supplied for targeting.
+ (false, false, _, Some(ptr)) => {
+ focus_hotkey_element(pid, ptr)?;
+ crate::input::keyboard::hotkey(pid, &key, &m)
+ }
_ => crate::input::keyboard::hotkey(pid, &key, &m),
}
})
@@ -397,6 +521,16 @@ impl Tool for HotkeyTool {
mod tests {
use super::*;
+ #[test]
+ fn hotkey_contract_accepts_snapshot_bound_ax_targets() {
+ let properties = def().input_schema["properties"]
+ .as_object()
+ .expect("hotkey properties");
+ for field in ["element_index", "element_token", "snapshot_id"] {
+ assert!(properties.contains_key(field), "missing {field} schema");
+ }
+ }
+
#[test]
fn screen_sharing_modifier_hotkeys_fail_closed_without_foreground_window() {
for (foreground, window_id) in [(false, None), (false, Some(7)), (true, None)] {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index fe7269f154..68287e0c93 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -374,10 +374,16 @@ pub(crate) async fn focus_by_pixel(
// AXFocused is non-destructive: unlike a second real click, it keeps a
// Cmd+A selection intact before a follow-up type_text or Cmd+V.
tokio::time::sleep(std::time::Duration::from_millis(120)).await;
- return Ok(());
- }
-
- if !foreground {
+ // A background focus-click is `effect:"unverifiable"` by construction:
+ // it cannot prove the renderer moved its first responder. Returning on
+ // "it didn't error" therefore skipped the real-click fallback below
+ // whenever the click was a silent no-op — advancing on transport
+ // success alone, which is exactly what the ladder forbids. Confirm the
+ // focus actually moved before claiming this rung worked.
+ if !foreground || pixel_focus_landed(pid, window_id, x, y).await {
+ return Ok(());
+ }
+ } else if !foreground {
return Err(cua_driver_core::protocol::ToolResult::error(format!(
"focus pixel-click at ({x:.0},{y:.0}) failed."
)));
@@ -415,6 +421,51 @@ pub(crate) async fn focus_by_pixel(
Ok(())
}
+/// Confirm that a focus pixel-click actually moved the application's focused
+/// element onto the clicked point.
+///
+/// The background focus-click reports `effect:"unverifiable"`, so this is the
+/// read-back that lets [`focus_by_pixel`] decide whether the cheap rung worked
+/// or the real-click fallback is still required. It reuses the same
+/// window-local-pixels → screen translation the click itself used, so the
+/// comparison is in one coordinate space.
+///
+/// Returns `false` whenever the answer cannot be established (no window id,
+/// untranslatable frame, unreadable focused element or rect). That is the
+/// conservative direction: an unprovable focus escalates to the stronger rung
+/// rather than being reported as success.
+async fn pixel_focus_landed(pid: i32, window_id: Option<u32>, x: f64, y: f64) -> bool {
+ let Some(wid) = window_id else {
+ return false;
+ };
+ tokio::task::spawn_blocking(move || {
+ let Ok(frame) = px_frame::resolve_window_px_frame(wid) else {
+ return false;
+ };
+ let (screen_x, screen_y, _, _) = frame.to_screen(x, y);
+ unsafe {
+ let Some(focused) = crate::ax::bindings::focused_element_of_pid(pid) else {
+ return false;
+ };
+ let rect = crate::ax::bindings::element_screen_rect(focused);
+ core_foundation::base::CFRelease(focused as core_foundation::base::CFTypeRef);
+ let Some(rect) = rect else {
+ return false;
+ };
+ point_within_rect(rect, screen_x, screen_y)
+ }
+ })
+ .await
+ .unwrap_or(false)
+}
+
+/// Whether `[x, y, width, height]` (screen coordinates, top-left origin)
+/// contains the point. A degenerate rect never contains anything, so an app
+/// reporting a zero-sized focused element escalates rather than false-confirms.
+fn point_within_rect([rx, ry, rw, rh]: [f64; 4], x: f64, y: f64) -> bool {
+ rw > 0.0 && rh > 0.0 && x >= rx && x < rx + rw && y >= ry && y < ry + rh
+}
+
/// Thread-safe per-pid zoom context registry.
pub struct ZoomRegistry {
inner: std::sync::Mutex<HashMap<i32, ZoomContext>>,
@@ -1118,6 +1169,43 @@ mod cursor_overlay_facility_tests {
// macOS cursor sampler (CoreGraphics), so the start-guard test runs here in
// platform-macos where build.rs links the frameworks — the core crate's test
// binary has no CoreGraphics linkage.
+#[cfg(test)]
+mod pixel_focus_readback_tests {
+ use super::point_within_rect;
+
+ #[test]
+ fn confirms_a_point_inside_the_focused_element() {
+ let composer = [100.0, 800.0, 250.0, 24.0];
+ assert!(point_within_rect(composer, 220.0, 812.0));
+ assert!(
+ point_within_rect(composer, 100.0, 800.0),
+ "top-left is inside"
+ );
+ }
+
+ #[test]
+ fn rejects_a_point_outside_the_focused_element() {
+ // The WhatsApp case: the click targeted the composer but focus stayed on
+ // the transcript above it, so the clicked point is not inside the
+ // focused element's rect and the caller must escalate.
+ let transcript = [100.0, 100.0, 640.0, 690.0];
+ assert!(!point_within_rect(transcript, 220.0, 812.0));
+ }
+
+ #[test]
+ fn excludes_the_far_edges() {
+ let rect = [0.0, 0.0, 10.0, 10.0];
+ assert!(!point_within_rect(rect, 10.0, 5.0));
+ assert!(!point_within_rect(rect, 5.0, 10.0));
+ }
+
+ #[test]
+ fn a_degenerate_rect_never_confirms() {
+ assert!(!point_within_rect([5.0, 5.0, 0.0, 0.0], 5.0, 5.0));
+ assert!(!point_within_rect([5.0, 5.0, -3.0, 10.0], 5.0, 6.0));
+ }
+}
+
#[cfg(test)]
mod recording_start_guard_tests {
use cua_driver_core::recording::RecordingSession;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 90c8ac4a53..49f8eca442 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -751,6 +751,11 @@ fn web_readback_next_rung(is_electron: bool, used_pixel_focus: bool) -> Option<&
const DELIVERY_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
const DELIVERY_DRAIN_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
+/// Pause before the single `AXFocused` re-apply in the foreground rung. Covers
+/// an app that installs its own first responder just after activation is
+/// observable, which would otherwise clobber the first write.
+const FOCUS_REAPPLY_DELAY: std::time::Duration = std::time::Duration::from_millis(30);
+
/// Keyboard-rung policy for one window-addressed background insert, decided
/// once by the pure exact-target core before any input is posted.
#[derive(Clone, Debug)]
@@ -1019,20 +1024,29 @@ fn cgevent_type_verified(
settle_ms: u64,
window_id: Option<u32>,
) -> anyhow::Result<(bool, Option<usize>)> {
- // Focus the target element first so the keystrokes land in IT. Critical in
+ // Focus the target element so the keystrokes land in IT. Critical in
// foreground mode: a freshly-fronted window's keyboard focus may be on the
// search box or nowhere, so without this the text goes into the void (or the
// wrong field). AXFocused is best-effort — harmless when unsupported.
+ //
+ // Ordering matters as much as the write itself. `with_foreground_assist` has
+ // already waited for the activation to land, so AppKit has installed the
+ // window's remembered first responder by now and this write lands *after*
+ // it rather than being clobbered by it. Re-applying once on a failed
+ // read-back covers apps that install their responder slightly late.
if let Some((ptr, _)) = element_ptr_and_idx {
let _ = crate::input::ax_actions::focus_element(ptr);
+ if settle_ms > 0 && !crate::input::ax_actions::is_element_focused(pid, ptr) {
+ std::thread::sleep(FOCUS_REAPPLY_DELAY);
+ let _ = crate::input::ax_actions::focus_element(ptr);
+ }
}
// First-keystroke settle (foreground rung only — caller passes `settle_ms > 0`).
- // After a window is fronted (with_foreground_assist) and the element focused,
- // the surface isn't ready to accept input for a few tens of ms, so the FIRST
- // synthesized character gets eaten: typing "i love u" rendered "love u" (the
- // leading "i " was dropped). A short sleep here lets focus settle before the
- // first key event. Background/terminal call sites pass 0 — they have no front
- // transition and must not pay this latency.
+ // Even once the window is front and the element focused, the surface isn't
+ // ready to accept input for a few tens of ms, so the FIRST synthesized
+ // character gets eaten: typing "i love u" rendered "love u" (the leading
+ // "i " was dropped). A short sleep here covers that. Background/terminal call
+ // sites pass 0 — they have no front transition and must not pay this latency.
if settle_ms > 0 {
std::thread::sleep(std::time::Duration::from_millis(settle_ms));
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
index bddf0502e2..1a1d2e5163 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
@@ -427,6 +427,22 @@ pub fn post_key(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<()> {
/// worker, the foreground swap may silently fail and SendInput land on the
/// wrong window. Callers should funnel hotkey calls through the uia worker.
pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<()> {
+ send_key_synthesized_after_focus(hwnd, key, modifiers, || Ok(()))
+}
+
+/// Foreground key delivery with a target-specific focus step performed only
+/// after Windows has confirmed the exact top-level HWND as foreground.
+///
+/// Element-addressed callers use this to avoid the activation/focus race:
+/// UIA `SetFocus` before `SetForegroundWindow` can be overwritten by the
+/// ensuing activation. The callback may establish and verify child focus; no
+/// input is inserted when either foreground or child-focus confirmation fails.
+pub fn send_key_synthesized_after_focus(
+ hwnd: u64,
+ key: &str,
+ modifiers: &[&str],
+ focus: impl FnOnce() -> Result<()>,
+) -> Result<()> {
let target = HWND(hwnd as *mut _);
if target.0.is_null() {
bail!("invalid target hwnd");
@@ -455,50 +471,9 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<
events.push(key_input(*mvk, true));
}
- unsafe {
- // Save & set foreground so SendInput lands on `target`.
- let prev_fg = GetForegroundWindow();
- // Robust auto bring-to-front (AttachThreadInput, same engine as
- // bring_to_front / macOS with_foreground_assist) — a bare
- // SetForegroundWindow is denied by the foreground-lock without UIAccess.
- // Restored to prev_fg below.
- let _ = crate::input::force_foreground_assisted(target);
- // Brief settle so the foreground swap is processed before we send.
- sleep(Duration::from_millis(8));
-
- // Verify the swap actually happened. `SetForegroundWindow` returns a
- // BOOL but Windows treats foreground-lock violations as a silent
- // no-op in most cases (the call returns success-ish, the foreground
- // doesn't change). The reliable check is observing the foreground
- // after the settle. If we didn't get it, abort BEFORE SendInput —
- // otherwise the events land on whatever app actually held foreground
- // (typically the terminal hosting this process), which causes spooky
- // side effects like Alt+Enter toggling the terminal into fullscreen.
- let actual_fg = GetForegroundWindow();
- if actual_fg != target {
- // Don't restore — prev_fg is presumably still foreground anyway.
- bail!(
- "foreground_unavailable: foreground swap to exact target HWND {:?} was rejected by Windows \
- (actual foreground is HWND {:?}). This daemon is not at \
- UIAccess integrity, so SetForegroundWindow is subject to the \
- foreground-lock and the swap silently fails. Without the \
- swap, SendInput would land on the wrong window. Fix: install \
- / spawn the cua-driver-uia worker (UIAccess-manifested PE) \
- and route hotkey calls through it. Until then, the calling \
- app must already be foreground for delivery_mode:\"foreground\" \
- to be safe.",
- target.0,
- actual_fg.0
- );
- }
-
+ with_confirmed_foreground(target, "key delivery", focus, || unsafe {
let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);
if sent as usize != events.len() {
- // SendInput returns the number of events successfully inserted.
- // Anything less is a partial insertion (blocked by another input
- // injector, foreground UIPI denial, etc.).
- let restored = SetForegroundWindow(prev_fg);
- let _ = restored;
bail!(
"SendInput inserted only {sent} of {} events. Likely cause: \
the daemon is not at UIAccess integrity, so SetForegroundWindow \
@@ -507,16 +482,8 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<
events.len()
);
}
-
- // Brief settle to let the target process the keystrokes before we
- // restore the previous foreground (otherwise the target might not
- // get a chance to handle the accelerator before losing focus).
- sleep(Duration::from_millis(40));
- if !prev_fg.0.is_null() && prev_fg != target {
- let _ = SetForegroundWindow(prev_fg);
- }
- }
- Ok(())
+ Ok(())
+ })
}
/// Foreground-delivery text entry: the `delivery_mode:"foreground"` rung for
@@ -531,6 +498,16 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<
/// of a false success. Required for VCL/LibreOffice document grids and other
/// targets where PostMessage WM_CHAR is silently dropped.
pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {
+ send_text_synthesized_after_focus(hwnd, text, || Ok(()))
+}
+
+/// Foreground Unicode delivery with child focus established after exact
+/// top-level activation and before `SendInput`.
+pub fn send_text_synthesized_after_focus(
+ hwnd: u64,
+ text: &str,
+ focus: impl FnOnce() -> Result<()>,
+) -> Result<()> {
let target = HWND(hwnd as *mut _);
if target.0.is_null() {
bail!("invalid target hwnd");
@@ -571,37 +548,9 @@ pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {
return Ok(());
}
- unsafe {
- // Save & set foreground so SendInput lands on `target`. Same verify-
- // before-send guard as send_key_synthesized: a foreground-lock no-op
- // would otherwise spray the text into whatever window actually held
- // focus (typically the terminal hosting this daemon).
- let prev_fg = GetForegroundWindow();
- // Robust auto bring-to-front (AttachThreadInput, same engine as
- // bring_to_front / macOS with_foreground_assist) — a bare
- // SetForegroundWindow is denied by the foreground-lock without UIAccess.
- // Restored to prev_fg below.
- let _ = crate::input::force_foreground_assisted(target);
- sleep(Duration::from_millis(8));
- let actual_fg = GetForegroundWindow();
- if actual_fg != target {
- bail!(
- "foreground_unavailable: foreground swap to exact target HWND {:?} was rejected by Windows \
- (actual foreground is HWND {:?}). This daemon is not at \
- UIAccess integrity, so SetForegroundWindow is subject to the \
- foreground-lock and the swap silently fails. Without the swap, \
- SendInput would land on the wrong window. Fix: install / spawn \
- the cua-driver-uia worker (UIAccess-manifested PE) and route \
- type_text through it. Until then, the calling app must already \
- be foreground for delivery_mode:\"foreground\" to be safe.",
- target.0,
- actual_fg.0
- );
- }
-
+ with_confirmed_foreground(target, "text delivery", focus, || unsafe {
let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);
if sent as usize != events.len() {
- let _ = SetForegroundWindow(prev_fg);
bail!(
"SendInput inserted only {sent} of {} key events. Likely cause: \
the daemon is not at UIAccess integrity, so SetForegroundWindow \
@@ -609,14 +558,103 @@ pub fn send_text_synthesized(hwnd: u64, text: &str) -> Result<()> {
events.len()
);
}
+ Ok(())
+ })
+}
- // Settle so the target processes the text before we yield focus back.
- sleep(Duration::from_millis(40));
- if !prev_fg.0.is_null() && prev_fg != target {
- let _ = SetForegroundWindow(prev_fg);
+fn wait_for_exact_foreground(target: HWND, timeout: Duration) -> bool {
+ let deadline = Instant::now() + timeout;
+ loop {
+ if unsafe { GetForegroundWindow() } == target {
+ return true;
+ }
+ if Instant::now() >= deadline {
+ return false;
}
+ sleep(Duration::from_millis(10));
}
- Ok(())
+}
+
+/// Run one system-queue input transaction while `target`, or a live visible
+/// same-process window whose owner chain reaches `target`, is foreground.
+/// Foreground and optional child focus are checked against
+/// external Win32/UIA state instead of inferred from API return values or a
+/// fixed settle delay. The prior foreground is restored on every body/focus
+/// result after activation succeeds.
+fn with_confirmed_foreground<T>(
+ target: HWND,
+ operation: &str,
+ focus: impl FnOnce() -> Result<()>,
+ body: impl FnOnce() -> Result<T>,
+) -> Result<T> {
+ let previous = unsafe { GetForegroundWindow() };
+ let _ = unsafe { crate::input::force_foreground_assisted(target) };
+ if !wait_for_exact_foreground(target, Duration::from_millis(500)) {
+ let actual = unsafe { GetForegroundWindow() };
+ if !previous.0.is_null() && previous != target {
+ let _ = unsafe { SetForegroundWindow(previous) };
+ }
+ bail!(
+ "foreground_unavailable: Windows did not confirm exact target HWND {:?} for {operation} \
+ within 500 ms (actual foreground HWND {:?}). Route the request through the \
+ UIAccess-manifested cua-driver-uia worker; no input was sent.",
+ target.0,
+ actual.0
+ );
+ }
+
+ let Some(foreground_target) = crate::win32::capture_foreground_target(target.0 as usize as u64)
+ else {
+ if !previous.0.is_null() && previous != target {
+ let _ = unsafe { SetForegroundWindow(previous) };
+ }
+ bail!(
+ "foreground_unavailable: exact target HWND {:?} disappeared before {operation}; \
+ no input was sent",
+ target.0
+ );
+ };
+
+ let result = (|| {
+ focus()?;
+ let deadline = Instant::now() + Duration::from_millis(250);
+ let actual = loop {
+ let actual = unsafe { GetForegroundWindow() };
+ if crate::win32::foreground_matches_target_or_owned_window(
+ foreground_target,
+ actual.0 as usize as u64,
+ ) {
+ break actual;
+ }
+ if Instant::now() >= deadline {
+ break actual;
+ }
+ sleep(Duration::from_millis(10));
+ };
+ if !crate::win32::foreground_matches_target_or_owned_window(
+ foreground_target,
+ actual.0 as usize as u64,
+ ) {
+ bail!(
+ "foreground_unavailable: exact target HWND {:?} or a verified same-process \
+ owned window was not foreground while preparing {operation} \
+ (actual foreground HWND {:?}); no input was sent",
+ target.0,
+ actual.0
+ );
+ }
+ body()
+ })();
+
+ // Give the target message loop a bounded opportunity to consume the
+ // inserted sequence before restoring the user's prior foreground.
+ if result.is_ok() {
+ sleep(Duration::from_millis(40));
+ }
+ if !previous.0.is_null() && previous != target {
+ let _ = unsafe { SetForegroundWindow(previous) };
+ }
+ result
}
/// Build a single Unicode keyboard INPUT struct for one UTF-16 code unit,
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
index 9d9c05321c..970660cd60 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
@@ -21,7 +21,8 @@ pub use inject::{
};
pub use keyboard::{
is_xaml_host_hwnd, post_char, post_key, post_type_text, post_type_text_with_delay,
- send_key_synthesized, send_text_synthesized, wait_for_focused_descendant,
+ send_key_synthesized, send_key_synthesized_after_focus, send_text_synthesized,
+ send_text_synthesized_after_focus, wait_for_focused_descendant,
};
pub use mouse::{
has_chromium_descendant, is_chromium_target_window, post_click, post_click_screen,
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
index f5b4af1785..30c9c92182 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
@@ -586,7 +586,7 @@ fn send_click_synthesized_mods_impl(
);
}
let foreground_target = if activate {
- match crate::win32::capture_post_action_foreground_target(target.0 as usize as u64) {
+ match crate::win32::capture_foreground_target(target.0 as usize as u64) {
Some(target) => Some(target),
None => bail!(
"foreground_unavailable: exact target HWND {:?} disappeared before mouse \
@@ -696,7 +696,7 @@ fn send_click_synthesized_mods_impl(
}
if activate {
let actual = GetForegroundWindow();
- if !crate::win32::post_action_foreground_matches(
+ if !crate::win32::foreground_matches_target_or_owned_window(
foreground_target.expect("foreground target captured before input"),
actual.0 as usize as u64,
) {
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index e2b91730d1..3e0326ac05 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -3912,6 +3912,83 @@ pub struct TypeTextTool {
state: Arc<ToolState>,
}
+fn wait_for_cached_element_keyboard_focus(
+ state: &ToolState,
+ pid: u32,
+ hwnd: u64,
+ element_index: usize,
+ timeout: std::time::Duration,
+) -> bool {
+ let deadline = std::time::Instant::now() + timeout;
+ loop {
+ if state
+ .element_cache
+ .element_has_keyboard_focus(pid, hwnd, element_index)
+ == Some(true)
+ {
+ return true;
+ }
+ if std::time::Instant::now() >= deadline {
+ return false;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(10));
+ }
+}
+
+/// Establish exact UIA child focus after the owning top-level HWND is already
+/// confirmed foreground. Chromium providers sometimes reject UIA `SetFocus`;
+/// in that case a real click at the accessibility-derived center is the
+/// bounded fallback. Both routes require `CurrentHasKeyboardFocus` read-back
+/// before any keyboard input is allowed to leave the driver.
+fn focus_cached_element_for_foreground(
+ state: &ToolState,
+ pid: u32,
+ hwnd: u64,
+ element_index: usize,
+ click_point: Option<(i32, i32)>,
+) -> anyhow::Result<()> {
+ // `fg_bypass` is a background-only shield: for Chromium it disables the
+ // top-level HWND while UIA runs so the provider cannot self-activate. A
+ // disabled foreground window immediately loses foreground on Windows,
+ // which would make the enclosing verify-before-SendInput transaction
+ // fail closed. The foreground route has already activated the exact HWND,
+ // so focus the cached element directly and verify it below.
+ let set_focus = state.element_cache.focus_element(pid, hwnd, element_index);
+ if set_focus.is_ok()
+ && wait_for_cached_element_keyboard_focus(
+ state,
+ pid,
+ hwnd,
+ element_index,
+ std::time::Duration::from_millis(350),
+ )
+ {
+ return Ok(());
+ }
+
+ if let Some((x, y)) = click_point {
+ crate::input::send_click_synthesized_active_mods(hwnd, x, y, 1, "left", &[])?;
+ if wait_for_cached_element_keyboard_focus(
+ state,
+ pid,
+ hwnd,
+ element_index,
+ std::time::Duration::from_millis(500),
+ ) {
+ return Ok(());
+ }
+ }
+
+ let detail = set_focus
+ .err()
+ .map(|error| format!("; UIA SetFocus failed: {error}"))
+ .unwrap_or_default();
+ anyhow::bail!(
+ "foreground_unavailable: Windows did not confirm keyboard focus on exact UIA element \
+ [{element_index}] in HWND 0x{hwnd:x}{detail}; no keyboard input was sent"
+ )
+}
+
static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
#[async_trait]
@@ -4170,12 +4247,10 @@ impl Tool for TypeTextTool {
// rejected (daemon not at UIAccess integrity), it returns an error
// rather than a false success.
if delivery == DeliveryMode::Foreground {
- // An indexed foreground type targets that element, not whichever
- // child happened to retain focus in the top-level window. UIA
- // SetFocus is not sufficient for Chromium renderer controls, so
- // establish real system focus with the same foreground coordinate
- // actuator used by an indexed click before sending Unicode input.
- if let Some(idx) = elem_idx {
+ // Resolve the optional click fallback before entering the atomic
+ // activation/focus/input transaction. The focus itself happens
+ // only after exact top-level foreground is confirmed.
+ let focus_target = if let Some(idx) = elem_idx {
let (cx, cy) =
match self
.state
@@ -4201,25 +4276,19 @@ impl Tool for TypeTextTool {
Ok(point) => point,
Err(result) => return result,
};
- let focus_result = tokio::task::spawn_blocking(move || {
- crate::input::send_click_synthesized_active_mods(hwnd, cx, cy, 1, "left", &[])
- })
- .await;
- match focus_result {
- Ok(Ok(())) => {
- tokio::time::sleep(std::time::Duration::from_millis(120)).await;
- }
- Ok(Err(error)) => return ToolResult::error(error.to_string()),
- Err(error) => {
- return ToolResult::error(format!(
- "foreground element-focus task failed: {error}"
- ))
- }
- }
- }
+ Some((idx as usize, (cx, cy)))
+ } else {
+ None
+ };
let text_fg = text.clone();
+ let state = self.state.clone();
let r = tokio::task::spawn_blocking(move || {
- crate::input::send_text_synthesized(hwnd, &text_fg)
+ crate::input::send_text_synthesized_after_focus(hwnd, &text_fg, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, Some(point))?;
+ }
+ Ok(())
+ })
})
.await;
return match r {
@@ -4949,7 +5018,7 @@ impl Tool for PressKeyTool {
{
return error;
}
- } else if let Some(idx) = elem_idx {
+ } else if let Some(idx) = elem_idx.filter(|_| delivery != DeliveryMode::Foreground) {
let state = self.state.clone();
let focused = tokio::task::spawn_blocking(move || {
crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || {
@@ -4975,9 +5044,22 @@ impl Tool for PressKeyTool {
// Skipped when px-focus already fronted/clicked the target — the key then
// goes via the plain background post path below.
if !px_focus && delivery == DeliveryMode::Foreground {
+ let focus_target = elem_idx.map(|idx| {
+ let point = self
+ .state
+ .element_cache
+ .get_element_center(pid, hwnd, idx as usize);
+ (idx as usize, point)
+ });
+ let state = self.state.clone();
let send_result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- crate::input::send_key_synthesized(hwnd, &key, &m)
+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, point)?;
+ }
+ Ok(())
+ })
})
.await;
return match send_result {
@@ -5073,6 +5155,9 @@ impl Tool for HotkeyTool {
"pid":{"type":"integer","description":"Target process ID."},
"keys":{"type":"array","items":{"type":"string"},"minItems":2,
"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},
+ "element_index": cua_driver_core::tool_schema::element_index_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),
"x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
"y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
"window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},
@@ -5172,12 +5257,41 @@ impl Tool for HotkeyTool {
} else {
return ToolResult::error("Missing required array field keys.");
};
- let hwnd_opt = args.opt_u64("window_id");
let raw_pid = match args.require_i64("pid") {
Ok(v) => v,
Err(e) => return e,
};
let pid = raw_pid as u32;
+ let resolved = match cua_driver_core::element_token::resolve_element_args(
+ pid as i32,
+ args.opt_u64("element_index").map(|value| value as usize),
+ args.opt_str("element_token").as_deref(),
+ args.opt_str("snapshot_id").as_deref(),
+ args.opt_u64("window_id").map(|value| value as u32),
+ "hotkey",
+ ) {
+ Ok(resolved) => resolved,
+ Err(error) => return error,
+ };
+ let elem_idx = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ let hwnd_opt = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => window_id
+ .map(|value| value as u64)
+ .or_else(|| args.opt_u64("window_id")),
+ cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
+ };
+ if elem_idx.is_some() && hwnd_opt.is_none() {
+ return ToolResult::error(
+ "window_id is required when element_index is used — the element_index cache \
+ is scoped per (pid, window_id). Pass the same window_id you used in \
+ `get_window_state`.",
+ );
+ }
let delivery = DeliveryMode::from_args(&args);
// Resolve HWND: use window_id if given, else pick first window for pid.
@@ -5213,6 +5327,11 @@ impl Tool for HotkeyTool {
let px = args.get("x").and_then(|v| v.as_f64());
let py = args.get("y").and_then(|v| v.as_f64());
if let (Some(cx), Some(cy)) = (px, py) {
+ if elem_idx.is_some() {
+ return ToolResult::error(
+ "Pass either element_index (ax) or x,y (px) to hotkey, not both.",
+ );
+ }
let from_zoom = args
.get("from_zoom")
.and_then(|v| v.as_bool())
@@ -5334,10 +5453,20 @@ impl Tool for HotkeyTool {
// global modifier state, so Chromium never observes Ctrl+Shift+H as a
// chord even though the renderer control is focused.
let use_send_input = delivery == DeliveryMode::Foreground;
+ let focus_target = elem_idx.map(|idx| {
+ let point = self.state.element_cache.get_element_center(pid, hwnd, idx);
+ (idx, point)
+ });
+ let state = self.state.clone();
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
if use_send_input {
- crate::input::send_key_synthesized(hwnd, &key, &m)
+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&state, pid, hwnd, idx, point)?;
+ }
+ Ok(())
+ })
} else {
crate::input::post_key(hwnd, &key, &m)
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs
index 58a6694d96..38f98f89d7 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/win32/mod.rs
@@ -7,7 +7,7 @@ pub mod windows;
pub use apps::{list_descendants, list_processes, related_processes, ProcessInfo};
pub use installed_apps::{list_installed_apps, InstalledApp};
pub(crate) use windows::{
- capture_post_action_foreground_target, find_window_by_pid_and_handle, list_windows_via_win32,
- post_action_foreground_matches, window_owner_pid,
+ capture_foreground_target, find_window_by_pid_and_handle,
+ foreground_matches_target_or_owned_window, list_windows_via_win32, window_owner_pid,
};
pub use windows::{list_windows, resolve_uwp_host_window, WindowInfo};
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
index e200d7aab8..840b11ba5e 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
@@ -154,29 +154,27 @@ fn owner_chain_reaches_target(
}
#[derive(Clone, Copy, Debug)]
-pub(crate) struct PostActionForegroundTarget {
+pub(crate) struct ForegroundTarget {
hwnd: u64,
pid: u32,
owner: Option<u64>,
}
-/// Snapshot the exact target identity and owner before pointer input is sent.
-pub(crate) fn capture_post_action_foreground_target(
- target: u64,
-) -> Option<PostActionForegroundTarget> {
+/// Snapshot the exact target identity and owner before global input is sent.
+pub(crate) fn capture_foreground_target(target: u64) -> Option<ForegroundTarget> {
let pid = window_owner_pid(target)?;
let owner = unsafe { GetWindow(HWND(target as *mut _), GW_OWNER) }
.ok()
.filter(|owner| !owner.0.is_null())
.map(|owner| owner.0 as usize as u64);
- Some(PostActionForegroundTarget {
+ Some(ForegroundTarget {
hwnd: target,
pid,
owner,
})
}
-/// Verify the foreground after an exact-target pointer action.
+/// Verify the foreground before or after an exact-target global input action.
///
/// The requested HWND must become foreground before input is sent. The action
/// may then legitimately open a same-process owned popup or modal, so the
@@ -185,8 +183,8 @@ pub(crate) fn capture_post_action_foreground_target(
/// A dismiss action may instead destroy the requested owned modal; that is
/// accepted only when foreground returns to its snapshotted same-process owner.
/// Unrelated same-process siblings and foreign foreground windows fail closed.
-pub(crate) fn post_action_foreground_matches(
- target: PostActionForegroundTarget,
+pub(crate) fn foreground_matches_target_or_owned_window(
+ target: ForegroundTarget,
actual: u64,
) -> bool {
let current_target_pid = window_owner_pid(target.hwnd);
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
index 8799710807..4f7eb410a7 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml.cs
@@ -55,6 +55,10 @@ public partial class MainWindow : Window
// PostMessage actually arrived and was actionable.
var source = HwndSource.FromHwnd(new WindowInteropHelper(this).Handle);
source?.AddHook(OnWindowMessage);
+ // Start with the deliberate decoy focused. Foreground element-addressed
+ // keyboard tests must move focus to their exact UIA target after the
+ // top-level activation; inheriting startup focus would hide that race.
+ Keyboard.Focus(TxtDeferredInput);
PublishFixtureState();
}
diff --git a/libs/cua-driver/tests/fixtures/shared/web/index.html b/libs/cua-driver/tests/fixtures/shared/web/index.html
index 6439929efb..23a860cece 100644
--- a/libs/cua-driver/tests/fixtures/shared/web/index.html
+++ b/libs/cua-driver/tests/fixtures/shared/web/index.html
@@ -73,14 +73,14 @@
autocapitalize="off" autocorrect="off" spellcheck="false" />
<span class="mirror" id="lbl-input-mirror" data-cua-id="lbl-input-mirror">mirror=</span>
<input type="number" id="number-input" data-cua-id="number-input"
- aria-label="number-input" value="42" />
+ aria-label="number-input" value="42" autofocus />
</div>
</fieldset>
<fieldset>
<legend>keyboard_task</legend>
<div class="row">
- <input type="text" id="keyboard-input" data-cua-id="keyboard-input" autofocus
+ <input type="text" id="keyboard-input" data-cua-id="keyboard-input"
aria-label="keyboard-input" placeholder="keyboard input"
autocapitalize="off" autocorrect="off" spellcheck="false" />
<span class="mirror" id="keyboard-status" data-cua-id="keyboard-status">key_state=none</span>
diff --git a/libs/cua-driver/tests/runners/macos-lume/README.md b/libs/cua-driver/tests/runners/macos-lume/README.md
index d755f7e051..747a0ee90f 100644
--- a/libs/cua-driver/tests/runners/macos-lume/README.md
+++ b/libs/cua-driver/tests/runners/macos-lume/README.md
@@ -40,6 +40,29 @@ repeatable while the private seed carries grants obtained through the normal
`CuaDriverLocal.app` prompt flow. The SIP-on check below owns the separate claim that
the supported permission flow still works with normal platform protection.
+## Reuse a prepared private seed
+
+Check the host-local inventory before building from the public base. Reuse a
+seed only when its versioned maintainer log records the expected public base,
+macOS build, toolchain versions, signing-certificate hash, TCC grants, and
+Chrome consent state. The seed and both backups must be stopped:
+
+```bash
+SEED=cua-driver-macos-e2e-seed-26.5.2-YYYYMMDD
+BACKUP_A="${SEED}-backup-a"
+BACKUP_B="${SEED}-backup-b"
+
+for VM in "$SEED" "$BACKUP_A" "$BACKUP_B"; do
+ lume get "$VM" --format json | jq -e '.[0].status == "stopped"'
+done
+```
+
+If those checks and the maintainer log match, skip image creation and clone a
+worker from the seed under [Run the acceptance gate](#run-the-acceptance-gate).
+Never boot a seed to inspect or update it. Clone a disposable worker and let the
+acceptance preflight verify the recorded state there. Build a new versioned seed
+when the inventory is missing, incomplete, or stale.
+
## Create the private seed
Pull the versioned public base into a mutable local builder. The initial guest
@@ -87,6 +110,10 @@ printf '\n%s\n' 'eval "$(/opt/homebrew/bin/brew shellenv)"' \
>> "$HOME/.zprofile"
```
+The pinned Homebrew installer pauses for a Return confirmation after its sudo
+check. Keep this step in the VM display and confirm the prompt there. A detached
+SSH run can look stalled while it is waiting for that input.
+
Open a new Terminal window and require each command to succeed:
```bash
@@ -164,6 +191,19 @@ bash libs/cua-driver/scripts/install-local.sh \
~/.local/bin/cua-driver-local permissions grant
```
+Record the certificate hash after the successful install and keep that identity
+for the life of the seed:
+
+```bash
+security find-certificate \
+ -c 'CuaDriver Local Signing (cua-driver-rs)' -Z "$SIGNING_KEYCHAIN"
+codesign -d -r- /Applications/CuaDriverLocal.app 2>&1 \
+ | grep 'certificate leaf'
+```
+
+Recreating the certificate changes the app identity and invalidates inherited
+TCC grants, even when the bundle identifier stays the same.
+
Complete the Accessibility and Screen Recording prompts. Tahoe 26 also asks
separately for Automation and direct ScreenCaptureKit access. Trigger all
remaining consent paths before freezing the seed:
@@ -177,19 +217,9 @@ osascript -e \
# must not require Automation access to System Events.
~/.local/bin/cua-driver-local list_apps '{}'
-# A desktop screenshot triggers Tahoe's direct-capture/private-window prompt.
-~/.local/bin/cua-driver-local call start_session \
- '{"session":"seed-desktop-consent","capture_scope":"desktop"}'
-~/.local/bin/cua-driver-local call get_desktop_state \
- '{"session":"seed-desktop-consent"}' \
- > /tmp/cua-driver-seed-desktop-state.json
-~/.local/bin/cua-driver-local call end_session '{"session":"seed-desktop-consent"}'
-jq -e '
- .screenshot_mime_type == "image/png"
- and .screenshot_width > 0
- and .screenshot_height > 0
- and (.screenshot_png_b64 | length) > 0
-' /tmp/cua-driver-seed-desktop-state.json >/dev/null
+# The app-owned grant flow probes a desktop capture and triggers Tahoe's
+# direct-capture/private-window prompt.
+~/.local/bin/cua-driver-local permissions grant
```
Choose Allow for `Terminal` -> `System Events` and on the CuaDriverLocal
@@ -198,8 +228,18 @@ Events. Target-specific Automation prompts may still appear later when a user
explicitly requests an Apple Events-backed browser or app operation; do not
pre-grant those in the seed. These are normal macOS consent flows; do not edit
`TCC.db`. Rerun the commands and require them to finish without another prompt.
-Then verify the daemon's own
-identity and the read-only status contract before running the explicit
+Notification banners can cover Tahoe's consent controls; swipe the banners away
+before acting instead of clicking through them. After enabling Screen Recording,
+restart the app-owned daemon once before checking status so the live process
+observes the new grant:
+
+```bash
+~/.local/bin/cua-driver-local stop
+open -a CuaDriverLocal
+```
+
+Then verify the daemon's own identity and the read-only status contract before
+running the explicit
LaunchServices-hosted grant flow. The first command must not raise a dialog;
the second is intentionally prompt-capable and must be run by the human:
@@ -301,12 +341,14 @@ standard autostart daemon even when a browser row fails.
On macOS Tahoe, first-use Chrome can present a native local-network discovery
prompt over `chrome://inspect/#remote-debugging`. The standalone-browser lane
uses loopback DevTools and does not need LAN discovery. Before freezing a seed
-that will run this optional lane, launch Chrome on that exact page in the VM
-display, choose **Don't Allow**, quit Chrome, then relaunch the page and require
-that the prompt does not return. Do not answer or dismiss OS consent UI while
-the behavior matrix is running. If an existing immutable seed lacks this
-decision, clone it to a new versioned seed, complete this setup there, stop it,
-and use that new seed for workers; never update the original seed in place.
+that will run this optional lane, complete Chrome's welcome screen without
+signing in and leave the default-browser and usage-reporting choices disabled.
+Launch Chrome on that exact page in the VM display, choose **Don't Allow**, quit
+Chrome, then relaunch the page and require that the prompt does not return. Do
+not answer or dismiss OS consent UI while the behavior matrix is running. If an
+existing immutable seed lacks these decisions, clone it to a new versioned seed,
+complete this setup there, stop it, and use that new seed for workers; never
+update the original seed in place.
The entrypoint refuses the wrong OS, user session, SIP state, dirty or
unidentified source, missing dependencies, ad-hoc signature, stale installed