Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-rs)(windows): no-foreground audit v2 — hotkey Chromium + Invoke restore + click polling (#1669)

Francesco Bonacci53ac662204e2
diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md
index bfa9cf9be3..6ef01f2414 100644
--- a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md
+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md
@@ -157,6 +157,32 @@ best-effort: `SetForegroundWindow` from non-UIAccess processes is subject
to Windows' foreground-lock and may silently no-op — failures are logged
at `tracing::trace!` and not surfaced as errors.
+**`hotkey` modifier dispatch on Chromium/Electron (Windows).** When
+the target is detected as a Chromium-family window (`Chrome_WidgetWin_*`
+class — covers every Chromium browser AND every Electron app: Chrome,
+Edge, Brave, Arc, Vivaldi, Slack, VS Code, Discord, Teams, Notion),
+`hotkey` with modifiers routes through `PostMessage(WM_KEYDOWN/UP)`
+instead of `SendInput` — no foreground swap. Chromium's
+`Browser::HandleKeyboardEvent` reads modifier state from the WM_KEYDOWN
+LPARAM bits, NOT from `GetKeyState`, so PostMessage delivery works for
+real accelerators (Ctrl+T, Ctrl+W, Ctrl+L, Ctrl+Shift+B, etc.). The
+SendInput-swap path (`send_key_synthesized`) remains the dispatch for
+**non-Chromium Win32 + modifiers** — classic apps (LibreOffice, FAR,
+classic Notepad) use `TranslateAccelerator` which requires the system
+modifier state updated, and PostMessage can't do that.
+
+**Chromium pixel-click foreground polling restore.** `click({pid, x, y})`
+on a Chromium target falls through to `send_click_synthesized` (SendInput
++ brief foreground swap) because Chromium's input thread filters by
+queue-origin and PostMessage-delivered clicks don't fire DOM events. The
+synchronous restore inside `send_click_synthesized` covers the
+immediate swap; an additional polling guard (same shape as `launch_app`'s
+`FocusRestoreGuard`) catches the **asynchronous** Chromium re-activation
+that can happen as the renderer's input handler processes the click
+(focus().activate() / WebContents::Activate() — 100-500 ms later). The
+guard is gated on `GetWindowThreadProcessId(fg_now) == pid` so user
+Alt-Tabs are respected.
+
## Defaults — always prefer cua-driver over shell shims
**Default transport is the `cua-driver` CLI** — `Bash` shelling out
@@ -654,28 +680,39 @@ browser instance identified by `(pid, window_id)`:
create it manually.
- **The Favorites bar must already be visible.** This is a
one-time setup: press `Ctrl+Shift+B` once inside the browser
- and the setting persists across sessions. cua-driver does
- **not** synthesize Ctrl+Shift+B for you any more, because the
- synthesizer routed via `SetForegroundWindow(target) →
- SendInput → SetForegroundWindow(prev_fg)` produces a visible
- foreground flicker and the restore can fail silently under
- UIPI / UIAccess constraints — both violations of the
- no-foreground contract. If the bar is hidden when
- `execute_javascript` is called, the bookmark path bails with a
- clear error and falls through to the CDP path (if configured).
+ and the setting persists across sessions. **If the bar is
+ hidden, cua-driver now synthesizes `Ctrl+Shift+B` via
+ `PostMessage(WM_KEYDOWN/UP)` with no foreground swap** —
+ Chromium's `Browser::HandleKeyboardEvent` dispatches
+ accelerators from the WM_KEYDOWN LPARAM bits without
+ consulting `GetKeyState`, so PostMessage works without the
+ `SetForegroundWindow` dance that previously violated the
+ no-foreground contract. After a brief settle the path
+ re-checks for the bar; if it's still hidden (locked-down
+ browser policy / non-Chromium target) the call bails with a
+ clear error and falls through to the CDP path.
- The user's expression should be a single statement or block;
`return` inside the IIFE is honored. Bookmarks strip line
breaks, so multi-line expressions are joined with spaces.
- **Known limitation — transient browser activation on Invoke.**
+ **Known limitation — Chromium's window activation on Invoke.**
When the bookmark is invoked via UIA `InvokePattern::Invoke`,
- Chromium briefly raises the browser window because clicking a
+ Chromium activates the browser window because clicking a
bookmark is a user-initiated navigation in Chromium's input
- model. This is a Chromium-side activation we cannot suppress
- from the UIA caller. We document it rather than mask it; the
- rest of the page-tool actions (`get_text`, `query_dom`) remain
- no-foreground. If this matters for your flow, use the CDP path
- instead — `Runtime.evaluate` does not trigger window activation.
+ model. The activation happens inside Chromium's window-aura
+ layer, not in our UIA call. **Mitigation in place**: a polling
+ foreground-restore guard runs immediately after the Invoke —
+ same pattern `launch_app` uses (PR #1668) — capturing the
+ user's foreground HWND beforehand and calling
+ `SetForegroundWindow(prev)` once Chromium grabs foreground.
+ The restore is gated on `GetWindowThreadProcessId(fg_now) ==
+ browser_pid` so we never yank focus from a window the user
+ legitimately Alt-Tabbed to. Without UIAccess (the daemon's
+ normal integrity) Windows' foreground lock may deny the
+ restore — in that case the browser dwell time is bounded to
+ the ~600 ms poll budget instead of "until next user action".
+ The `get_text` and `query_dom` actions don't share this issue
+ (no Invoke → no activation).
2. **CDP fallback** — `Runtime.evaluate` via raw WebSocket against
`--remote-debugging-port=N`. Requires the browser launched with
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
index 4a7e54d532..91ea6f7c7b 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
@@ -1692,10 +1692,27 @@ impl Tool for ClickTool {
.await
.unwrap_or(false);
if chromium {
+ // Belt-and-suspenders foreground restore. `send_click_synthesized`
+ // does its own SetForegroundWindow(prev_fg) ~40ms after the
+ // click, BUT Chromium's reaction to the click can include an
+ // async re-activation (focus().activate() in the renderer-side
+ // event handler, sometimes 100-500 ms later). The launch_app
+ // FocusRestoreGuard (PR #1668) already proved the polling
+ // pattern works; reuse the same helper to catch the async case.
+ //
+ // Captured here (async-runtime side) rather than inside the
+ // blocking task so we have a known-stable "before" snapshot.
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
let send_result = tokio::task::spawn_blocking(move || {
crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn)
})
.await;
+ // Kick off the polling restore regardless of click result
+ // — even a partially-inserted click might have started an
+ // async activation Chromium can't undo.
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
match send_result {
Ok(Ok(())) => {
let click_word = match count {
@@ -2120,28 +2137,48 @@ impl Tool for HotkeyTool {
};
}
- // Non-XAML Win32 path. Two routes available:
- // 1. SendInput synthesized hotkey — pushes the events onto the
- // *system input queue*. Updates GetKeyState's modifier state, so
- // TranslateAccelerator-based apps (LibreOffice, FAR, classic
- // Notepad, etc.) see Ctrl+S as a real accelerator. Trade-off:
- // brief focus theft to ensure SendInput lands on the right HWND.
- // 2. PostMessage WM_KEYDOWN/UP — no focus theft, but the
- // synthesized Ctrl/Shift/Alt modifier never updates
- // GetKeyState, so accelerators don't fire. Only useful for
- // non-accelerator key sequences.
- // We pick route 1 when modifiers are present (the accelerator case),
- // route 2 otherwise (plain non-modifier keys still post fine).
+ // Non-XAML target. Three routing decisions:
+ // 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus
+ // theft. Modifier state is encoded in the LPARAM bits, not
+ // in GetKeyState, so apps that read modifier state directly
+ // from the message (Chromium's `Browser::HandleKeyboardEvent`,
+ // Chromium renderer's input handler, every web app) see
+ // Ctrl+X correctly. Apps that consult `GetKeyState` via
+ // `TranslateAccelerator` (LibreOffice, FAR, classic Notepad)
+ // DON'T fire the accelerator — the modifier looks unset to
+ // them.
+ // 2. SendInput synthesized hotkey — pushes events onto the
+ // *system input queue*. Updates GetKeyState system-wide, so
+ // TranslateAccelerator-based apps see Ctrl+S as a real
+ // accelerator. Trade-off: brief foreground swap so SendInput
+ // lands on the right HWND. **This is the only path that
+ // violates the no-foreground contract** — we minimise its
+ // use to apps that genuinely require it.
+ //
+ // Routing:
+ // - No modifiers → PostMessage (no accelerator concern).
+ // - Modifiers + Chromium-family target → PostMessage. Chromium
+ // processes accelerators in `Browser::HandleKeyboardEvent`,
+ // which reads modifier state from the WM_KEYDOWN LPARAM
+ // directly (not via GetKeyState). Every Chromium browser
+ // (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app
+ // (Slack/VS Code/Discord/Teams/Notion) detects as Chromium
+ // via `is_chromium_target_window` and gets the no-foreground
+ // path.
+ // - Modifiers + non-Chromium Win32 → SendInput (the
+ // TranslateAccelerator path). The foreground swap stays.
let has_modifiers = !mods.is_empty();
+ let is_chromium = crate::input::is_chromium_target_window(hwnd);
+ let use_send_input = has_modifiers && !is_chromium;
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- if has_modifiers {
+ if use_send_input {
crate::input::send_key_synthesized(hwnd, &key, &m)
} else {
crate::input::post_key(hwnd, &key, &m)
}
}).await;
- let path = if has_modifiers { "SendInput" } else { "PostMessage" };
+ let path = if use_send_input { "SendInput" } else { "PostMessage" };
match result {
Ok(Ok(())) => ToolResult::text(format!(
"✅ Pressed {key_display} on pid {raw_pid} via {path} \
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs
index 11fe671828..57f91bdccf 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs
@@ -188,31 +188,37 @@ unsafe fn try_bookmark_exec_blocking(
let edge_window = root_for_window(&automation, hwnd_raw, pid)
.context("could not resolve Edge top-level window for (pid, window_id)")?;
- // 2) Ensure the favorites bar is visible. If not, we bail.
+ // 2) Ensure the favorites bar is visible. If not, send Ctrl+Shift+B
+ // via PostMessage (no foreground swap — Chromium's
+ // Browser::HandleKeyboardEvent dispatches accelerators from
+ // WM_KEYDOWN LPARAM modifier bits without consulting GetKeyState).
//
- // Historical note: we used to synthesize Ctrl+Shift+B here via
- // `send_key_synthesized`, which does a transient
- // `SetForegroundWindow(target) → SendInput → SetForegroundWindow(prev_fg)`
- // dance. That dance is visible to the user (foreground flicker) and the
- // restore can fail silently under UIPI / UIAccess constraints, both of
- // which violate the cua-driver no-foreground contract. Every
- // `page.execute_javascript` call where the user hadn't already shown
- // the favorites bar would trigger this — unacceptable.
- //
- // The favorites-bar setting persists across browser sessions, so the
- // one-time setup cost (Ctrl+Shift+B in the browser) is borne by the
- // user once, never by the agent.
- let fav_bar = find_favorites_bar(&automation, &edge_window).ok_or_else(|| {
- anyhow!(
- "Favorites bar not visible — cua-driver-rs's bookmark-URL JS \
- exec needs an always-on favorites bar so we can invoke the \
- `cua-driver-eval` bookmark without stealing the foreground. \
- Show the bar once via Ctrl+Shift+B and the setting persists \
- across sessions. The bookmark-exec path is now disabled for \
- this call; `execute_javascript` will fall through to the CDP \
- fallback if it's configured."
- )
- })?;
+ // Historical note: PR #1668 removed the auto-toggle here because it
+ // used `send_key_synthesized` (SendInput + SetForegroundWindow swap)
+ // which violated the no-foreground contract. With `post_key`
+ // (PostMessage) routing — verified safe on every Chromium browser +
+ // Electron app — the auto-toggle is restored. If the bar STILL isn't
+ // visible after a brief settle, we bail with the same actionable
+ // error (covers locked-down browser policies / classic Edge profiles).
+ let fav_bar = if let Some(bar) = find_favorites_bar(&automation, &edge_window) {
+ bar
+ } else {
+ // No SendInput, no SetForegroundWindow, no UIPI risk.
+ let _ = crate::input::post_key(hwnd_raw, "b", &["ctrl", "shift"]);
+ std::thread::sleep(std::time::Duration::from_millis(150));
+ find_favorites_bar(&automation, &edge_window).ok_or_else(|| {
+ anyhow!(
+ "Favorites bar not visible after Ctrl+Shift+B PostMessage \
+ — cua-driver-rs's bookmark-URL JS exec needs the bar so \
+ we can invoke the `cua-driver-eval` bookmark. The toggle \
+ keystroke didn't take (browser policy override, locked-down \
+ profile, or non-Chromium target). Show the bar manually via \
+ Ctrl+Shift+B in the browser — the setting persists across \
+ sessions. `execute_javascript` will fall through to the \
+ CDP fallback if it's configured."
+ )
+ })?
+ };
// 3) Find or create the cua-driver-eval bookmark.
let bookmark = find_bookmark(&automation, &fav_bar, BOOKMARK_NAME);
@@ -247,8 +253,40 @@ unsafe fn try_bookmark_exec_blocking(
"captured original tab title: {:?}", original_title
);
- // 6) Invoke the bookmark.
+ // 6) Invoke the bookmark with a no-foreground guard around it.
+ //
+ // Chromium activates the browser window when a bookmark is clicked
+ // because `WebContents::OpenURL` treats it as user-initiated
+ // navigation (the activation lives in Chromium's window-aura layer,
+ // not in our UIA call). Two-layer mitigation:
+ //
+ // (a) Capture `GetForegroundWindow()` immediately before the Invoke.
+ // (b) After the Invoke, briefly poll for "Chromium activated the
+ // browser" and `SetForegroundWindow(prev)` the user's window
+ // back. Same pattern `launch_app`'s `FocusRestoreGuard`
+ // successfully uses; the restore is gated on the new
+ // foreground being owned by Chromium so we never yank focus
+ // from a window the user legitimately tabbed to.
+ //
+ // Without UIAccess (the daemon's normal integrity), the restore
+ // SetForegroundWindow may be blocked by the system foreground lock;
+ // we log at trace level and continue — the brief activation is
+ // unavoidable without forking Chromium, but the dwell time on the
+ // browser is < ~150 ms instead of "until next user action".
+ let prev_fg_for_invoke = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow()
+ };
invoke_element(&bookmark).context("InvokePattern.Invoke on bookmark failed")?;
+ // Restore in a tight loop — at this point the bookmark URL is firing
+ // and Chromium is about to activate. Poll for ~600 ms; once we see
+ // Chromium grab foreground, swap back. We do this synchronously
+ // (not via tokio::spawn) because we WANT to block the poll-marker
+ // step below until the foreground is restored — otherwise a slow
+ // restore leaves the browser frontmost while we read the title.
+ restore_foreground_after_browser_activation(
+ prev_fg_for_invoke,
+ pid as u32,
+ );
// 7) Poll the active tab's title until it starts with CUA: / CUA_ERR:
let result = poll_for_marker(&automation, &edge_window, &original_title)?;
@@ -666,6 +704,104 @@ unsafe fn invoke_element(elem: &IUIAutomationElement) -> Result<()> {
Ok(())
}
+/// Belt-and-suspenders foreground restore after the bookmark Invoke.
+///
+/// Chromium's bookmark click triggers `WebContents::OpenURL` →
+/// `Browser::ActivateContents` → `SetForegroundWindow` on the browser
+/// frame. This happens whether we like it or not, in Chromium-side code
+/// we don't control. The mitigation: capture `prev_fg` immediately
+/// before our `InvokePattern.Invoke` (the caller does this), then poll
+/// for "the browser pid grabbed foreground" and call `SetForegroundWindow(prev_fg)`
+/// the user's window back. Mirrors `launch_app`'s `FocusRestoreGuard`
+/// (PR #1668, fixed by CodeRabbit feedback on PR #1668 to gate on PID
+/// ownership rather than any foreground change).
+///
+/// Synchronous (not tokio::spawn) because the caller's next step is
+/// `poll_for_marker`, which reads the active tab's title via UIA — that
+/// must happen against the now-restored foreground state so we don't
+/// see transient state.
+///
+/// Best-effort: when Windows' foreground-lock denies the restore (we're
+/// not at UIAccess), the dwell time on the browser is still bounded to
+/// the ~600 ms poll budget. Without this guard the browser stays
+/// frontmost until the next user action.
+unsafe fn restore_foreground_after_browser_activation(
+ prev_fg: windows::Win32::Foundation::HWND,
+ browser_pid: u32,
+) {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow, SetForegroundWindow,
+ };
+
+ if prev_fg.0.is_null() {
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "no prior foreground to restore — skipping"
+ );
+ return;
+ }
+ if browser_pid == 0 {
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "no browser pid — skipping restore (no ownership signal)"
+ );
+ return;
+ }
+
+ // Poll for ~600 ms in 50 ms steps. Chromium's activation typically
+ // lands within ~150 ms but the URL-load can stretch the window.
+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(600);
+ while std::time::Instant::now() < deadline {
+ let fg_now = GetForegroundWindow();
+ if fg_now == prev_fg {
+ // Chromium didn't (yet) grab foreground — keep polling
+ // briefly; if it never does we just exit cleanly.
+ std::thread::sleep(std::time::Duration::from_millis(50));
+ continue;
+ }
+ let mut fg_pid: u32 = 0;
+ let _ = GetWindowThreadProcessId(fg_now, Some(&mut fg_pid));
+ if fg_pid != browser_pid {
+ // Foreground changed but to something we didn't expect
+ // (user Alt-Tabbed mid-eval). Leave it alone.
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "foreground changed to non-browser pid {fg_pid} (expected {browser_pid}) — leaving as-is"
+ );
+ return;
+ }
+ // Chromium grabbed foreground — restore the user's window.
+ if !IsWindow(prev_fg).as_bool() {
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "prev foreground HWND no longer valid — skipping restore"
+ );
+ return;
+ }
+ let ok = SetForegroundWindow(prev_fg).as_bool();
+ if ok {
+ tracing::debug!(
+ target: "page.bookmark_exec.focus_restore",
+ "restored foreground to HWND 0x{:x} after browser pid {browser_pid} activation",
+ prev_fg.0 as usize
+ );
+ } else {
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "SetForegroundWindow returned FALSE (foreground-lock denial; need UIAccess) — \
+ browser will keep focus until next user action"
+ );
+ }
+ return;
+ }
+ tracing::trace!(
+ target: "page.bookmark_exec.focus_restore",
+ "browser pid {browser_pid} did not steal foreground within 600ms — no restore needed"
+ );
+ let _ = HWND::default(); // silence "unused import" possibility on cfg
+}
+
/// Find the TabItem that's currently selected — Chromium exposes the
/// active tab as the only TabItem whose `IsSelected` is true. We don't
/// strictly need IsSelected: a fallback is "the first TabItem in the tab