Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-rs)(windows): no-foreground audit v2 — hotkey Chromium + Invoke restore + click polling (#1669)
Francesco Bonacci53ac662204e2
diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.mdindex bfa9cf9be3..6ef01f2414 100644--- a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md@@ -157,6 +157,32 @@ best-effort: `SetForegroundWindow` from non-UIAccess processes is subjectto Windows' foreground-lock and may silently no-op — failures are loggedat `tracing::trace!` and not surfaced as errors.+**`hotkey` modifier dispatch on Chromium/Electron (Windows).** When+the target is detected as a Chromium-family window (`Chrome_WidgetWin_*`+class — covers every Chromium browser AND every Electron app: Chrome,+Edge, Brave, Arc, Vivaldi, Slack, VS Code, Discord, Teams, Notion),+`hotkey` with modifiers routes through `PostMessage(WM_KEYDOWN/UP)`+instead of `SendInput` — no foreground swap. Chromium's+`Browser::HandleKeyboardEvent` reads modifier state from the WM_KEYDOWN+LPARAM bits, NOT from `GetKeyState`, so PostMessage delivery works for+real accelerators (Ctrl+T, Ctrl+W, Ctrl+L, Ctrl+Shift+B, etc.). The+SendInput-swap path (`send_key_synthesized`) remains the dispatch for+**non-Chromium Win32 + modifiers** — classic apps (LibreOffice, FAR,+classic Notepad) use `TranslateAccelerator` which requires the system+modifier state updated, and PostMessage can't do that.++**Chromium pixel-click foreground polling restore.** `click({pid, x, y})`+on a Chromium target falls through to `send_click_synthesized` (SendInput++ brief foreground swap) because Chromium's input thread filters by+queue-origin and PostMessage-delivered clicks don't fire DOM events. The+synchronous restore inside `send_click_synthesized` covers the+immediate swap; an additional polling guard (same shape as `launch_app`'s+`FocusRestoreGuard`) catches the **asynchronous** Chromium re-activation+that can happen as the renderer's input handler processes the click+(focus().activate() / WebContents::Activate() — 100-500 ms later). The+guard is gated on `GetWindowThreadProcessId(fg_now) == pid` so user+Alt-Tabs are respected.+## Defaults — always prefer cua-driver over shell shims**Default transport is the `cua-driver` CLI** — `Bash` shelling out@@ -654,28 +680,39 @@ browser instance identified by `(pid, window_id)`:create it manually.- **The Favorites bar must already be visible.** This is aone-time setup: press `Ctrl+Shift+B` once inside the browser- and the setting persists across sessions. cua-driver does- **not** synthesize Ctrl+Shift+B for you any more, because the- synthesizer routed via `SetForegroundWindow(target) →- SendInput → SetForegroundWindow(prev_fg)` produces a visible- foreground flicker and the restore can fail silently under- UIPI / UIAccess constraints — both violations of the- no-foreground contract. If the bar is hidden when- `execute_javascript` is called, the bookmark path bails with a- clear error and falls through to the CDP path (if configured).+ and the setting persists across sessions. **If the bar is+ hidden, cua-driver now synthesizes `Ctrl+Shift+B` via+ `PostMessage(WM_KEYDOWN/UP)` with no foreground swap** —+ Chromium's `Browser::HandleKeyboardEvent` dispatches+ accelerators from the WM_KEYDOWN LPARAM bits without+ consulting `GetKeyState`, so PostMessage works without the+ `SetForegroundWindow` dance that previously violated the+ no-foreground contract. After a brief settle the path+ re-checks for the bar; if it's still hidden (locked-down+ browser policy / non-Chromium target) the call bails with a+ clear error and falls through to the CDP path.- The user's expression should be a single statement or block;`return` inside the IIFE is honored. Bookmarks strip linebreaks, so multi-line expressions are joined with spaces.- **Known limitation — transient browser activation on Invoke.**+ **Known limitation — Chromium's window activation on Invoke.**When the bookmark is invoked via UIA `InvokePattern::Invoke`,- Chromium briefly raises the browser window because clicking a+ Chromium activates the browser window because clicking abookmark is a user-initiated navigation in Chromium's input- model. This is a Chromium-side activation we cannot suppress- from the UIA caller. We document it rather than mask it; the- rest of the page-tool actions (`get_text`, `query_dom`) remain- no-foreground. If this matters for your flow, use the CDP path- instead — `Runtime.evaluate` does not trigger window activation.+ model. The activation happens inside Chromium's window-aura+ layer, not in our UIA call. **Mitigation in place**: a polling+ foreground-restore guard runs immediately after the Invoke —+ same pattern `launch_app` uses (PR #1668) — capturing the+ user's foreground HWND beforehand and calling+ `SetForegroundWindow(prev)` once Chromium grabs foreground.+ The restore is gated on `GetWindowThreadProcessId(fg_now) ==+ browser_pid` so we never yank focus from a window the user+ legitimately Alt-Tabbed to. Without UIAccess (the daemon's+ normal integrity) Windows' foreground lock may deny the+ restore — in that case the browser dwell time is bounded to+ the ~600 ms poll budget instead of "until next user action".+ The `get_text` and `query_dom` actions don't share this issue+ (no Invoke → no activation).2. **CDP fallback** — `Runtime.evaluate` via raw WebSocket against`--remote-debugging-port=N`. Requires the browser launched withdiff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rsindex 4a7e54d532..91ea6f7c7b 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs@@ -1692,10 +1692,27 @@ impl Tool for ClickTool {.await.unwrap_or(false);if chromium {+ // Belt-and-suspenders foreground restore. `send_click_synthesized`+ // does its own SetForegroundWindow(prev_fg) ~40ms after the+ // click, BUT Chromium's reaction to the click can include an+ // async re-activation (focus().activate() in the renderer-side+ // event handler, sometimes 100-500 ms later). The launch_app+ // FocusRestoreGuard (PR #1668) already proved the polling+ // pattern works; reuse the same helper to catch the async case.+ //+ // Captured here (async-runtime side) rather than inside the+ // blocking task so we have a known-stable "before" snapshot.+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };let send_result = tokio::task::spawn_blocking(move || {crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn)}).await;+ // Kick off the polling restore regardless of click result+ // — even a partially-inserted click might have started an+ // async activation Chromium can't undo.+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));match send_result {Ok(Ok(())) => {let click_word = match count {@@ -2120,28 +2137,48 @@ impl Tool for HotkeyTool {};}- // Non-XAML Win32 path. Two routes available:- // 1. SendInput synthesized hotkey — pushes the events onto the- // *system input queue*. Updates GetKeyState's modifier state, so- // TranslateAccelerator-based apps (LibreOffice, FAR, classic- // Notepad, etc.) see Ctrl+S as a real accelerator. Trade-off:- // brief focus theft to ensure SendInput lands on the right HWND.- // 2. PostMessage WM_KEYDOWN/UP — no focus theft, but the- // synthesized Ctrl/Shift/Alt modifier never updates- // GetKeyState, so accelerators don't fire. Only useful for- // non-accelerator key sequences.- // We pick route 1 when modifiers are present (the accelerator case),- // route 2 otherwise (plain non-modifier keys still post fine).+ // Non-XAML target. Three routing decisions:+ // 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus+ // theft. Modifier state is encoded in the LPARAM bits, not+ // in GetKeyState, so apps that read modifier state directly+ // from the message (Chromium's `Browser::HandleKeyboardEvent`,+ // Chromium renderer's input handler, every web app) see+ // Ctrl+X correctly. Apps that consult `GetKeyState` via+ // `TranslateAccelerator` (LibreOffice, FAR, classic Notepad)+ // DON'T fire the accelerator — the modifier looks unset to+ // them.+ // 2. SendInput synthesized hotkey — pushes events onto the+ // *system input queue*. Updates GetKeyState system-wide, so+ // TranslateAccelerator-based apps see Ctrl+S as a real+ // accelerator. Trade-off: brief foreground swap so SendInput+ // lands on the right HWND. **This is the only path that+ // violates the no-foreground contract** — we minimise its+ // use to apps that genuinely require it.+ //+ // Routing:+ // - No modifiers → PostMessage (no accelerator concern).+ // - Modifiers + Chromium-family target → PostMessage. Chromium+ // processes accelerators in `Browser::HandleKeyboardEvent`,+ // which reads modifier state from the WM_KEYDOWN LPARAM+ // directly (not via GetKeyState). Every Chromium browser+ // (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app+ // (Slack/VS Code/Discord/Teams/Notion) detects as Chromium+ // via `is_chromium_target_window` and gets the no-foreground+ // path.+ // - Modifiers + non-Chromium Win32 → SendInput (the+ // TranslateAccelerator path). The foreground swap stays.let has_modifiers = !mods.is_empty();+ let is_chromium = crate::input::is_chromium_target_window(hwnd);+ let use_send_input = has_modifiers && !is_chromium;let result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods.iter().map(String::as_str).collect();- if has_modifiers {+ if use_send_input {crate::input::send_key_synthesized(hwnd, &key, &m)} else {crate::input::post_key(hwnd, &key, &m)}}).await;- let path = if has_modifiers { "SendInput" } else { "PostMessage" };+ let path = if use_send_input { "SendInput" } else { "PostMessage" };match result {Ok(Ok(())) => ToolResult::text(format!("✅ Pressed {key_display} on pid {raw_pid} via {path} \diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rsindex 11fe671828..57f91bdccf 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs@@ -188,31 +188,37 @@ unsafe fn try_bookmark_exec_blocking(let edge_window = root_for_window(&automation, hwnd_raw, pid).context("could not resolve Edge top-level window for (pid, window_id)")?;- // 2) Ensure the favorites bar is visible. If not, we bail.+ // 2) Ensure the favorites bar is visible. If not, send Ctrl+Shift+B+ // via PostMessage (no foreground swap — Chromium's+ // Browser::HandleKeyboardEvent dispatches accelerators from+ // WM_KEYDOWN LPARAM modifier bits without consulting GetKeyState).//- // Historical note: we used to synthesize Ctrl+Shift+B here via- // `send_key_synthesized`, which does a transient- // `SetForegroundWindow(target) → SendInput → SetForegroundWindow(prev_fg)`- // dance. That dance is visible to the user (foreground flicker) and the- // restore can fail silently under UIPI / UIAccess constraints, both of- // which violate the cua-driver no-foreground contract. Every- // `page.execute_javascript` call where the user hadn't already shown- // the favorites bar would trigger this — unacceptable.- //- // The favorites-bar setting persists across browser sessions, so the- // one-time setup cost (Ctrl+Shift+B in the browser) is borne by the- // user once, never by the agent.- let fav_bar = find_favorites_bar(&automation, &edge_window).ok_or_else(|| {- anyhow!(- "Favorites bar not visible — cua-driver-rs's bookmark-URL JS \- exec needs an always-on favorites bar so we can invoke the \- `cua-driver-eval` bookmark without stealing the foreground. \- Show the bar once via Ctrl+Shift+B and the setting persists \- across sessions. The bookmark-exec path is now disabled for \- this call; `execute_javascript` will fall through to the CDP \- fallback if it's configured."- )- })?;+ // Historical note: PR #1668 removed the auto-toggle here because it+ // used `send_key_synthesized` (SendInput + SetForegroundWindow swap)+ // which violated the no-foreground contract. With `post_key`+ // (PostMessage) routing — verified safe on every Chromium browser ++ // Electron app — the auto-toggle is restored. If the bar STILL isn't+ // visible after a brief settle, we bail with the same actionable+ // error (covers locked-down browser policies / classic Edge profiles).+ let fav_bar = if let Some(bar) = find_favorites_bar(&automation, &edge_window) {+ bar+ } else {+ // No SendInput, no SetForegroundWindow, no UIPI risk.+ let _ = crate::input::post_key(hwnd_raw, "b", &["ctrl", "shift"]);+ std::thread::sleep(std::time::Duration::from_millis(150));+ find_favorites_bar(&automation, &edge_window).ok_or_else(|| {+ anyhow!(+ "Favorites bar not visible after Ctrl+Shift+B PostMessage \+ — cua-driver-rs's bookmark-URL JS exec needs the bar so \+ we can invoke the `cua-driver-eval` bookmark. The toggle \+ keystroke didn't take (browser policy override, locked-down \+ profile, or non-Chromium target). Show the bar manually via \+ Ctrl+Shift+B in the browser — the setting persists across \+ sessions. `execute_javascript` will fall through to the \+ CDP fallback if it's configured."+ )+ })?+ };// 3) Find or create the cua-driver-eval bookmark.let bookmark = find_bookmark(&automation, &fav_bar, BOOKMARK_NAME);@@ -247,8 +253,40 @@ unsafe fn try_bookmark_exec_blocking("captured original tab title: {:?}", original_title);- // 6) Invoke the bookmark.+ // 6) Invoke the bookmark with a no-foreground guard around it.+ //+ // Chromium activates the browser window when a bookmark is clicked+ // because `WebContents::OpenURL` treats it as user-initiated+ // navigation (the activation lives in Chromium's window-aura layer,+ // not in our UIA call). Two-layer mitigation:+ //+ // (a) Capture `GetForegroundWindow()` immediately before the Invoke.+ // (b) After the Invoke, briefly poll for "Chromium activated the+ // browser" and `SetForegroundWindow(prev)` the user's window+ // back. Same pattern `launch_app`'s `FocusRestoreGuard`+ // successfully uses; the restore is gated on the new+ // foreground being owned by Chromium so we never yank focus+ // from a window the user legitimately tabbed to.+ //+ // Without UIAccess (the daemon's normal integrity), the restore+ // SetForegroundWindow may be blocked by the system foreground lock;+ // we log at trace level and continue — the brief activation is+ // unavoidable without forking Chromium, but the dwell time on the+ // browser is < ~150 ms instead of "until next user action".+ let prev_fg_for_invoke = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow()+ };invoke_element(&bookmark).context("InvokePattern.Invoke on bookmark failed")?;+ // Restore in a tight loop — at this point the bookmark URL is firing+ // and Chromium is about to activate. Poll for ~600 ms; once we see+ // Chromium grab foreground, swap back. We do this synchronously+ // (not via tokio::spawn) because we WANT to block the poll-marker+ // step below until the foreground is restored — otherwise a slow+ // restore leaves the browser frontmost while we read the title.+ restore_foreground_after_browser_activation(+ prev_fg_for_invoke,+ pid as u32,+ );// 7) Poll the active tab's title until it starts with CUA: / CUA_ERR:let result = poll_for_marker(&automation, &edge_window, &original_title)?;@@ -666,6 +704,104 @@ unsafe fn invoke_element(elem: &IUIAutomationElement) -> Result<()> {Ok(())}+/// Belt-and-suspenders foreground restore after the bookmark Invoke.+///+/// Chromium's bookmark click triggers `WebContents::OpenURL` →+/// `Browser::ActivateContents` → `SetForegroundWindow` on the browser+/// frame. This happens whether we like it or not, in Chromium-side code+/// we don't control. The mitigation: capture `prev_fg` immediately+/// before our `InvokePattern.Invoke` (the caller does this), then poll+/// for "the browser pid grabbed foreground" and call `SetForegroundWindow(prev_fg)`+/// the user's window back. Mirrors `launch_app`'s `FocusRestoreGuard`+/// (PR #1668, fixed by CodeRabbit feedback on PR #1668 to gate on PID+/// ownership rather than any foreground change).+///+/// Synchronous (not tokio::spawn) because the caller's next step is+/// `poll_for_marker`, which reads the active tab's title via UIA — that+/// must happen against the now-restored foreground state so we don't+/// see transient state.+///+/// Best-effort: when Windows' foreground-lock denies the restore (we're+/// not at UIAccess), the dwell time on the browser is still bounded to+/// the ~600 ms poll budget. Without this guard the browser stays+/// frontmost until the next user action.+unsafe fn restore_foreground_after_browser_activation(+ prev_fg: windows::Win32::Foundation::HWND,+ browser_pid: u32,+) {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow, SetForegroundWindow,+ };++ if prev_fg.0.is_null() {+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "no prior foreground to restore — skipping"+ );+ return;+ }+ if browser_pid == 0 {+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "no browser pid — skipping restore (no ownership signal)"+ );+ return;+ }++ // Poll for ~600 ms in 50 ms steps. Chromium's activation typically+ // lands within ~150 ms but the URL-load can stretch the window.+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(600);+ while std::time::Instant::now() < deadline {+ let fg_now = GetForegroundWindow();+ if fg_now == prev_fg {+ // Chromium didn't (yet) grab foreground — keep polling+ // briefly; if it never does we just exit cleanly.+ std::thread::sleep(std::time::Duration::from_millis(50));+ continue;+ }+ let mut fg_pid: u32 = 0;+ let _ = GetWindowThreadProcessId(fg_now, Some(&mut fg_pid));+ if fg_pid != browser_pid {+ // Foreground changed but to something we didn't expect+ // (user Alt-Tabbed mid-eval). Leave it alone.+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "foreground changed to non-browser pid {fg_pid} (expected {browser_pid}) — leaving as-is"+ );+ return;+ }+ // Chromium grabbed foreground — restore the user's window.+ if !IsWindow(prev_fg).as_bool() {+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "prev foreground HWND no longer valid — skipping restore"+ );+ return;+ }+ let ok = SetForegroundWindow(prev_fg).as_bool();+ if ok {+ tracing::debug!(+ target: "page.bookmark_exec.focus_restore",+ "restored foreground to HWND 0x{:x} after browser pid {browser_pid} activation",+ prev_fg.0 as usize+ );+ } else {+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "SetForegroundWindow returned FALSE (foreground-lock denial; need UIAccess) — \+ browser will keep focus until next user action"+ );+ }+ return;+ }+ tracing::trace!(+ target: "page.bookmark_exec.focus_restore",+ "browser pid {browser_pid} did not steal foreground within 600ms — no restore needed"+ );+ let _ = HWND::default(); // silence "unused import" possibility on cfg+}+/// Find the TabItem that's currently selected — Chromium exposes the/// active tab as the only TabItem whose `IsSelected` is true. We don't/// strictly need IsSelected: a fallback is "the first TabItem in the tab