Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-rs)(windows): agent-cursor z-order + background-dispatch hardening (#1692)
Francesco Bonacci892edd475233
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex d99fde21b7..35734178ae 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -458,6 +458,7 @@ doesn't-survive-across-sessions caveat.| `Invalid element_index N for pid X window_id W` | Index is stale or out of range | Re-run `get_window_state` with the same window_id, pick a fresh index from the new tree || `window_id W belongs to pid P, not …` | Passed a window_id that's owned by a different process | Use `list_windows({pid: X})` to enumerate this pid's own windows || `AX action … failed with code …` / `UIA invoke failed` | Element doesn't support the default action | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center |+| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this *exact* string for BOTH a permission-prompt denial AND a manual interrupt (Esc / stop) of a running tool — they are indistinguishable from the tool result | Treat as "tool canceled, no result, await the user." Do NOT paraphrase ("you stopped me") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |Platform-specific errors (TCC dialogs on macOS, Session 0 / UACprompts on Windows, AT-SPI bus issues on Linux) live in theirdiff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.mdindex 6ef01f2414..695e8cf5f5 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md@@ -26,6 +26,66 @@ driver gives you (no cursor warp, no taskbar flash, no windowrestore-and-raise) stops mattering — you just shipped a `SendInput`wrapper with extra steps.+### Cursor feedback for JS-driven browser actions: prefer `click_element`++The `page` tool gained a fourth action: `click_element`. It takes a CSS+selector, animates the agent cursor to the element's on-screen center,+fires a click-pulse, then runs `el.click()`. Prefer this over+`execute_javascript('document.querySelector(...).click()')` whenever you+want the user to see what the agent is doing — raw `execute_javascript`+will perform the click but leaves the cursor frozen.++```json+{ "action": "click_element", "pid": <int>, "window_id": <int>, "selector": "button.submit" }+```++Returns the resolved screen coords in `structuredContent` so callers can+chain subsequent operations against the same point.++### How the contract is enforced per call: the `dispatch` field++Every Windows input tool (`click`, `double_click`, `right_click`,+`drag`, `scroll`, `press_key`, `hotkey`, `type_text`) accepts an+optional `dispatch` field. The default is `"background"` — strict+no-foreground:++| `dispatch` | Behavior on Windows |+|---|---|+| `"background"` (DEFAULT) | PostMessage / UIA path only. If the target's input stack is known to silently drop PostMessage for this event kind (Chromium DOM mouse + key-combos, GTK button widgets), the call returns a structured `background_unavailable` error. **No foreground swap, ever.** |+| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. Required to drive Chromium DOM content or GTK button widgets reliably. Flashes the target visible unless `bring_to_front` was called first. |+| `"auto"` | Historical heuristic: silently falls back to SendInput on known-problematic targets. Opt-in for callers that prefer the old "things just work, sometimes at the cost of focus" behavior. |++`background_unavailable` error shape:++```json+{+ "isError": true,+ "structuredContent": {+ "code": "background_unavailable",+ "target_class": "Chrome_WidgetWin_1",+ "event_kind": "mouse_click",+ "suggestion": "Either call bring_to_front then retry with dispatch:\"foreground\", or accept the foreground swap by setting dispatch:\"foreground\" directly."+ }+}+```++The recommended flow when an agent gets that error:++1. `bring_to_front(pid)` — activates the target ONCE (visible flicker).+2. Subsequent input calls with `dispatch:"foreground"` deliver via+ SendInput WITHOUT a per-call flash (the SetForegroundWindow swap+ inside SendInput is a no-op because the target is already frontmost).+3. When done, leave the target as the user's foreground or call+ `hotkey({pid: original_fg_pid, keys: ["alt","tab"]})` to put their+ prior window back. **There is no "restore" tool** — you brought+ the target forward deliberately; restoring is your responsibility.++The `bring_to_front` tool uses an `AttachThreadInput` trick so the+foreground swap succeeds even when the daemon isn't at UIAccess+integrity (the same trick that powers `send_key_synthesized`).+Returns `{previous_fg_hwnd, now_fg_hwnd, landed_on_target}`.++Before running any shell command, ask: **"does this raise, activate,foreground, or steal focus from any app?"** If yes, don't run it.Every one of the commands below activates the target on Windows anddiff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rsindex f27049025b..8c8328d773 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs@@ -903,7 +903,17 @@ pub fn run_call(}Err(e) => {// Daemon became unreachable mid-call — fall through to in-process.- tracing::debug!("Daemon forwarding failed ({e}), running in-process");+ // Promoted from `tracing::debug!` to `eprintln!` so callers see+ // the degradation: in-process execution gets a FRESH ToolState,+ // which means state-dependent tools (`click`, `type_text`,+ // `set_value` — anything that reads the element_index cache)+ // will fail with "Element N not in cache" even when a prior+ // `get_window_state` populated the daemon's cache, because the+ // daemon's cache and the in-process cache are different.+ eprintln!(+ "[cua-driver] WARNING: daemon proxy to {socket_path} failed ({e}); \+ running '{tool}' in-process. State-dependent tools may misbehave."+ );}}}diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rsindex 07d60e55f5..d4cccacb18 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs@@ -109,14 +109,52 @@ impl DaemonResponse {// ── Client ────────────────────────────────────────────────────────────────────-/// Probe whether a daemon is listening on `socket_path` by sending a-/// trivial `list` request and checking for a valid response.+/// Probe whether a daemon is listening on `socket_path`.+///+/// On Windows this uses `WaitNamedPipeW` with a tiny timeout, which checks+/// whether a pipe instance is available **without consuming one**. The+/// previous design (sending a `list` request) opened a pipe instance,+/// then the immediately-following real `send_request` had to wait for the+/// daemon to spin up its NEXT instance — that race caused real tool calls+/// (especially state-dependent ones like `click` that need the daemon's+/// element_index cache) to fall through to the in-process path with a+/// fresh empty cache. See the conversation around the+/// "Element 3 not in cache" bug for the diagnosis.+///+/// `WaitNamedPipeW(name, 1)`:+/// - Returns TRUE if an instance is currently available.+/// - Returns FALSE if not available within 1 ms.+/// - Does NOT consume the instance — that's reserved for the actual call.pub fn is_daemon_listening(socket_path: &str) -> bool {- let req = DaemonRequest { method: "list".into(), name: None, args: None };- send_request(socket_path, &req)- .ok()- .map(|r| r.ok)- .unwrap_or(false)+ #[cfg(target_os = "windows")]+ {+ use std::os::windows::ffi::OsStrExt;++ // Raw FFI to `WaitNamedPipeW` so this crate doesn't have to pull in+ // the `windows` crate as a direct dependency just for one probe.+ // `kernel32.dll` is implicitly linked on Windows targets.+ #[link(name = "kernel32")]+ extern "system" {+ fn WaitNamedPipeW(lpNamedPipeName: *const u16, nTimeOut: u32) -> i32;+ }++ let wide: Vec<u16> = std::ffi::OsStr::new(socket_path)+ .encode_wide()+ .chain(std::iter::once(0))+ .collect();+ // NMPWAIT_NOWAIT == 1: "do not wait at all"; returns nonzero only if an+ // instance is immediately available. We don't want to block here —+ // this is a one-shot existence probe.+ unsafe { WaitNamedPipeW(wide.as_ptr(), 1) != 0 }+ }+ #[cfg(not(target_os = "windows"))]+ {+ let req = DaemonRequest { method: "list".into(), name: None, args: None };+ send_request(socket_path, &req)+ .ok()+ .map(|r| r.ok)+ .unwrap_or(false)+ }}/// Read the PID stored in `pid_file_path`, if any.@@ -505,6 +543,69 @@ fn is_self_at_high_il() -> bool {}}+/// Build a SECURITY_ATTRIBUTES that lets any Medium-IL (or higher) process+/// open the named pipe, even though the daemon itself is running at+/// High IL (via the autostart Scheduled Task's `RunLevel=Highest`, per+/// #1630). Without this, the default UIPI rule "no write-up across IL+/// boundaries" makes the High-IL daemon's pipe unreachable from a normal+/// Medium-IL user shell — every `cua-driver <tool>` call from the CLI+/// silently falls through to in-process execution with a fresh, empty+/// `ToolState`, which breaks the element_index cache invariant+/// (`get_window_state` → `click(element_index)` stops working because+/// the two calls land in different ToolState instances).+///+/// SDDL: `D:(A;OICI;GA;;;WD)S:(ML;;NW;;;LW)`+/// - DACL grants `GENERIC_ALL` to `WD` (Everyone).+/// - SACL sets the mandatory label to LOW with `NW` (NoWriteUp), so+/// processes at Low-IL and above can write the pipe — i.e. no+/// IL-based write restriction in practice.+///+/// Returns the SECURITY_ATTRIBUTES struct AND the raw security-descriptor+/// pointer (which the caller must keep alive for the lifetime of the+/// pipe-server, then free via `LocalFree`).+#[cfg(target_os = "windows")]+unsafe fn build_open_pipe_security_attrs()+ -> Option<(SecurityAttributesRaw, *mut std::ffi::c_void)>+{+ #[link(name = "advapi32")]+ extern "system" {+ fn ConvertStringSecurityDescriptorToSecurityDescriptorW(+ string_security_descriptor: *const u16,+ string_sd_revision: u32,+ security_descriptor: *mut *mut std::ffi::c_void,+ security_descriptor_size: *mut u32,+ ) -> i32;+ }+ let sddl: Vec<u16> = "D:(A;OICI;GA;;;WD)S:(ML;;NW;;;LW)\0"+ .encode_utf16()+ .collect();+ let mut sd_ptr: *mut std::ffi::c_void = std::ptr::null_mut();+ let mut sd_size: u32 = 0;+ let ok = ConvertStringSecurityDescriptorToSecurityDescriptorW(+ sddl.as_ptr(),+ 1, // SDDL_REVISION_1+ &mut sd_ptr,+ &mut sd_size,+ );+ if ok == 0 || sd_ptr.is_null() {+ return None;+ }+ let attrs = SecurityAttributesRaw {+ n_length: std::mem::size_of::<SecurityAttributesRaw>() as u32,+ lp_security_descriptor: sd_ptr,+ b_inherit_handle: 0,+ };+ Some((attrs, sd_ptr))+}++#[cfg(target_os = "windows")]+#[repr(C)]+struct SecurityAttributesRaw {+ n_length: u32,+ lp_security_descriptor: *mut std::ffi::c_void,+ b_inherit_handle: i32,+}+#[cfg(target_os = "windows")]pub async fn run_serve(registry: std::sync::Arc<mcp_server::tool::ToolRegistry>,@@ -516,6 +617,25 @@ pub async fn run_serve(eprintln!("cua-driver daemon listening on {socket_path}");+ // Build the cross-IL security descriptor once, reuse on every pipe+ // instance. If this fails we fall back to the default-ACL `create`+ // (which is High-IL exclusive when the daemon is at High IL — i.e.+ // the bug we're trying to fix). Log the failure so it's diagnosable.+ let security_attrs = unsafe { build_open_pipe_security_attrs() };+ if security_attrs.is_none() {+ eprintln!(+ "cua-driver: failed to build cross-IL SECURITY_ATTRIBUTES; pipe will be \+ High-IL exclusive. CLI calls from Medium-IL shells will fall through \+ to in-process and break state-dependent tool sequences."+ );+ }+ // Hold the SD pointer alive for the lifetime of run_serve. We never+ // free it — the daemon process exit reclaims it.+ let sec_attrs_ptr: *mut std::ffi::c_void = match &security_attrs {+ Some((attrs, _sd_ptr)) => attrs as *const _ as *mut _,+ None => std::ptr::null_mut(),+ };+// Spawn the sibling uiAccess'd worker if it's installed. Best-effort —// the main daemon still serves requests even if the worker fails to start.maybe_spawn_uia_worker();@@ -533,10 +653,23 @@ pub async fn run_serve(loop {// Create a new pipe server instance to accept the next client.- let server = ServerOptions::new()- .first_pipe_instance(false)- .create(socket_path)- .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?;+ // Use create_with_security_attributes_raw so Medium-IL clients can+ // open the pipe even though we're at High IL (see comment on+ // build_open_pipe_security_attrs). Fall back to default ACL if+ // SD construction failed at startup.+ let server = if sec_attrs_ptr.is_null() {+ ServerOptions::new()+ .first_pipe_instance(false)+ .create(socket_path)+ .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?+ } else {+ unsafe {+ ServerOptions::new()+ .first_pipe_instance(false)+ .create_with_security_attributes_raw(socket_path, sec_attrs_ptr)+ .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?+ }+ };tokio::select! {result = server.connect() => {diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rsindex 2669ef248d..d19078fdc4 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs@@ -16,6 +16,7 @@ pub mod shape;pub mod capture_utils;pub mod util;pub mod render_state;+pub mod z_order;pub use palette::Palette;pub use motion::{MotionConfig, Spring};@@ -23,6 +24,7 @@ pub use bezier::CubicBezier;pub use path_planner::{PathPlanner, PlannedPath, PathState};pub use shape::CursorShape;pub use render_state::{RenderStateCore, FocusRect, render_frame, draw_default_arrow};+pub use z_order::ZOrderEnforcer;/// Configuration assembled from CLI arguments and passed to every/// platform backend when it initialises the overlay window.diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rsindex 3011569eee..fa6f4dfaae 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs@@ -123,10 +123,15 @@ impl RenderStateCore {/// speed switches from `min_start_speed` to `min_end_speed` at the/// midpoint so the cursor decelerates as it approaches the target./// Spring overshoot is `0.5` (Windows/Linux convention).- pub fn tick_motion(&mut self, dt: f64) {+ ///+ /// Returns `true` when the planned path just ended (so the caller can+ /// fire an arrival oneshot to unblock `animate_cursor_to`).+ pub fn tick_motion(&mut self, dt: f64) -> bool {let spring_k = self.motion.spring * 400.0;let spring_c = self.motion.spring * 20.0;+ let mut fire_arrival = false;+if let Some(ref p) = self.path {let path_frac = (self.dist / p.length.max(1.0)).clamp(0.0, 1.0);let profile =@@ -155,6 +160,7 @@ impl RenderStateCore {self.heading = end_heading;self.path = None;self.dist = 0.0;+ fire_arrival = true;} else {let s: PathState = p.sample(self.dist);self.pos = (s.x, s.y);@@ -189,6 +195,8 @@ impl RenderStateCore {}self.tick_idle(dt);++ fire_arrival}/// Advance the animation by `dt` seconds using the hardcoded Swiftdiff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rsnew file mode 100644index 0000000000..d44eb3d02f--- /dev/null+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rs@@ -0,0 +1,47 @@+//! Cross-platform contract for keeping the agent-cursor overlay at z+1 of+//! the application under test.+//!+//! Each platform crate owns a concrete [`ZOrderEnforcer`] that holds the+//! overlay's native window handle and is driven by the platform's existing+//! tick cadence (Win32 `WM_TIMER`, macOS render frame, X11 render thread).+//! `reassert` is the single named seam where a future OS-event-driven+//! observer (`SetWinEventHook(EVENT_OBJECT_REORDER)`, AppKit+//! `NSWindowDidChangeOcclusionState`, X11 `SubstructureNotifyMask`) can+//! replace the polling implementation without touching the render loops.+//!+//! ## Contract+//!+//! - `target = Some(wid)` — the overlay must sit **one stacking position+//! above** the native window identified by `wid` (HWND root on Windows,+//! `CGWindowID` on macOS, X11 XID on Linux). If a third window is+//! currently above the target, the overlay must appear **below** it so+//! the user's foreground keeps rendering on top of the synthetic cursor.+//!+//! - `target = None` — the overlay must sit at the top of the **non-topmost**+//! band. It must never be promoted into the OS "always-on-top" band+//! (`HWND_TOPMOST`, `NSStatusWindowLevel`, override-redirect raise).+//!+//! - `wid` no longer maps to a live window — fall back to the `None`+//! behaviour for this tick. Do not error; the next tick may have a+//! new target.+//!+//! - Implementations must be cheap and idempotent. `reassert` is called+//! on every render tick; doing redundant work is acceptable, but+//! side effects (focus changes, activations, animations) are not.++/// Keeps the overlay window at z+1 of `target`.+///+/// See the [module-level docs](self) for the full behavioural contract.+///+/// The trait is deliberately not `Send + 'static` — each platform's+/// enforcer is owned by exactly one thread (the overlay render thread)+/// and can borrow platform handles (e.g. the X11 `Connection`) that+/// aren't easily made `'static`. Cross-thread setup is handled by each+/// platform's own static / `OnceLock`.+pub trait ZOrderEnforcer {+ /// Re-pin the overlay so it sits just above `target` (or at the top+ /// of the non-topmost band when `target` is `None`).+ ///+ /// Called from the platform render thread on every tick.+ fn reassert(&self, target: Option<u64>);+}diff --git a/libs/cua-driver/rust/crates/mcp-server/src/page.rs b/libs/cua-driver/rust/crates/mcp-server/src/page.rsindex d064ea3dbb..af86773527 100644--- a/libs/cua-driver/rust/crates/mcp-server/src/page.rs+++ b/libs/cua-driver/rust/crates/mcp-server/src/page.rs@@ -16,6 +16,7 @@//! get a clear error instead of a panic.use async_trait::async_trait;+use serde::{Deserialize, Serialize};use serde_json::Value;use std::sync::Arc;@@ -24,6 +25,23 @@ use crate::{tool::{Tool, ToolDef},};+/// Result of a `click_element` call. The backend is responsible for driving+/// the cursor overlay before returning; `screen_x` / `screen_y` are reported+/// back to the caller for diagnostics + downstream chaining.+#[derive(Debug, Clone, Serialize, Deserialize)]+pub struct ClickElementResult {+ /// Screen-space x of the element center, after cursor animation landed.+ pub screen_x: f64,+ /// Screen-space y of the element center.+ pub screen_y: f64,+ /// Viewport-space x (page-relative, before scroll offset compensation).+ pub viewport_x: f64,+ /// Viewport-space y.+ pub viewport_y: f64,+ /// Free-form message for the agent (e.g. "Clicked button.submit").+ pub message: String,+}+/// Platform-specific backend for the `page` tool.////// All methods take `(pid, window_id)` for consistency across actions.@@ -69,6 +87,29 @@ pub trait PageBackend: Send + Sync {(Chrome/Brave/Edge use Apple Events as the JS backend there).")}++ /// Click an element identified by a CSS selector, animating the agent+ /// cursor to the element's on-screen center before firing the click.+ ///+ /// The backend is responsible for: (1) running JS to derive the+ /// element's screen coordinates; (2) driving the cursor overlay+ /// (`pin_above` + `animate_cursor_to` + `ClickPulse`); (3) firing+ /// `element.click()` via JS; (4) returning the coords for diagnostics.+ ///+ /// Default impl returns an actionable "not implemented" error so+ /// non-Windows backends keep compiling until they're wired up.+ async fn click_element(+ &self,+ _pid: i32,+ _window_id: u32,+ _selector: &str,+ ) -> anyhow::Result<ClickElementResult> {+ anyhow::bail!(+ "click_element is not yet implemented on this platform's page \+ backend. Use execute_javascript with `el.click()` directly \+ (no visible cursor animation)."+ )+ }}/// The cross-platform `page` MCP tool. Holds a backend trait object the@@ -95,6 +136,10 @@ fn def() -> &'static ToolDef {- execute_javascript: Run JS and return the result.\n\- get_text: Extract visible text from the page.\n\- query_dom: Find elements matching a CSS selector.\n\+ - click_element: Click a CSS-selected element AND animate the agent cursor \+ to its on-screen center first (so the user sees what the agent is doing). \+ Prefer over `execute_javascript('el.click()')` whenever you want visible \+ cursor feedback.\n\- enable_javascript_apple_events: macOS-only — patch the browser's \Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user \confirmation and a browser restart).".into(),@@ -111,9 +156,13 @@ fn def() -> &'static ToolDef {"window_id": { "type": "integer", "description": "Target window ID from list_windows." },"action": {"type": "string",- "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],+ "enum": ["execute_javascript", "get_text", "query_dom", "click_element", "enable_javascript_apple_events"],"description": "Action to perform."},+ "selector": {+ "type": "string",+ "description": "CSS selector for click_element (e.g. 'button.submit', '#login a')."+ },"javascript": {"type": "string","description": "JavaScript to execute. Required for execute_javascript."@@ -242,6 +291,36 @@ impl Tool for PageTool {Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),},+ "click_element" => {+ // Trim before emptiness check so whitespace-only selectors+ // fail fast at the input boundary rather than blowing up+ // inside the backend's JS payload (`querySelector(" ")`+ // returns null and the error there is much less specific).+ let selector = match args+ .get("selector")+ .and_then(|v| v.as_str())+ .map(str::trim)+ {+ Some(s) if !s.is_empty() => s.to_owned(),+ _ => return ToolResult::error(+ "Missing required parameter: selector (CSS selector for click_element)"+ ),+ };+ match self.backend.click_element(pid, window_id, &selector).await {+ Ok(res) => {+ let structured = serde_json::json!({+ "screen_x": res.screen_x,+ "screen_y": res.screen_y,+ "viewport_x": res.viewport_x,+ "viewport_y": res.viewport_y,+ "selector": selector,+ });+ ToolResult::text(res.message).with_structured(structured)+ }+ Err(e) => ToolResult::error(format!("click_element failed: {e}")),+ }+ }+"query_dom" => {let selector = args.get("css_selector")diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rsindex 2d5ab55d40..7ffd0d16d3 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs@@ -21,6 +21,8 @@ use std::sync::{Mutex, OnceLock};use std::time::{Duration, Instant};use cursor_overlay::{CursorConfig, OverlayCommand, RenderStateCore};+#[cfg(target_os = "linux")]+use cursor_overlay::ZOrderEnforcer;// ── Global channel ────────────────────────────────────────────────────────@@ -205,6 +207,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayColet frame_dur = Duration::from_millis(16);let mut last_tick = Instant::now();let mut last_ztick = Instant::now();+ let z_enforcer = X11ZOrderEnforcer { conn: &conn, win };loop {let now = Instant::now();@@ -240,23 +243,16 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCopaint_x11(&conn, win, scr_w, scr_h, depth, visual_id, &pm);}- // Z-order maintenance every 80ms.+ // Z-order maintenance every 80ms — delegate to the cross-platform+ // ZOrderEnforcer so the contract for "z+1 of the application under+ // test" is documented once in `cursor_overlay::z_order`.if last_ztick.elapsed() >= Duration::from_millis(80) {last_ztick = Instant::now();let pinned_wid = {let guard = RENDER.lock().unwrap();guard.as_ref().and_then(|rs| rs.core.pinned_wid)};- let aux = if let Some(target_xid) = pinned_wid {- // Place overlay just above the pinned X11 window.- ConfigureWindowAux::new()- .sibling(target_xid as u32)- .stack_mode(StackMode::ABOVE)- } else {- ConfigureWindowAux::new().stack_mode(StackMode::ABOVE)- };- conn.configure_window(win, &aux).ok();- conn.flush().ok();+ z_enforcer.reassert(pinned_wid);}// Drain any X events (needed to avoid blocking).@@ -269,6 +265,57 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCo}}+// ── Z-order enforcer (Linux impl of cursor_overlay::ZOrderEnforcer) ──────++/// X11 implementation of [`cursor_overlay::ZOrderEnforcer`].+///+/// Borrows the X11 connection and overlay window id; lives only inside+/// `run_overlay_thread` (the X11 connection is not `'static`). Called+/// every 80 ms from the render loop.+#[cfg(target_os = "linux")]+struct X11ZOrderEnforcer<'a, C: x11rb::connection::Connection> {+ conn: &'a C,+ win: u32,+}++#[cfg(target_os = "linux")]+impl<'a, C: x11rb::connection::Connection> ZOrderEnforcer for X11ZOrderEnforcer<'a, C> {+ fn reassert(&self, target: Option<u64>) {+ use x11rb::protocol::xproto::*;+ use x11rb::protocol::xproto::ConnectionExt as _;++ // Per the ZOrderEnforcer trait contract, a stale `target` (window+ // gone) should fall back to the `None` behavior — top of the+ // normal stack, no sibling. Using a stale XID as a `sibling` here+ // triggers BadWindow on every tick of the overlay-enforcer loop+ // (~125 Hz), spamming the X server and silently skipping the+ // intended z-reassertion. Probe liveness via get_window_attributes+ // before committing to the sibling path.+ let target_live = target.and_then(|xid| {+ self.conn+ .get_window_attributes(xid as u32)+ .ok()+ .and_then(|c| c.reply().ok())+ .map(|_| xid)+ });+ let aux = if let Some(target_xid) = target_live {+ // Place overlay just above the pinned X11 window.+ ConfigureWindowAux::new()+ .sibling(target_xid as u32)+ .stack_mode(StackMode::ABOVE)+ } else {+ // No pin (or stale target XID) → raise to the top of the+ // normal stack. (X11 has no OS-level "always-on-top" band like+ // Windows / NSStatusWindowLevel, so a plain ABOVE here cannot+ // accidentally float over a focused foreground app the way+ // HWND_TOPMOST would on Windows.)+ ConfigureWindowAux::new().stack_mode(StackMode::ABOVE)+ };+ self.conn.configure_window(self.win, &aux).ok();+ self.conn.flush().ok();+ }+}+#[cfg(target_os = "linux")]fn find_argb_visual(conn: &impl x11rb::connection::Connection,diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex 718ebf7d55..f7be2ef946 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -2051,6 +2051,59 @@ impl Tool for KillAppTool {}}+// ── bring_to_front (Linux stub) ──────────────────────────────────────────────++pub struct BringToFrontTool;++static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();++#[async_trait]+impl Tool for BringToFrontTool {+ fn def(&self) -> &ToolDef {+ BTF_DEF.get_or_init(|| ToolDef {+ name: "bring_to_front".into(),+ description:+ "Activate a window so subsequent input tools land on it. **Windows-only \+ today:** on Linux this stub returns an error; the X11/Wayland equivalents \+ (`wmctrl -a`, `xdotool windowactivate`) aren't wired up because the Linux \+ input tools deliver via AT-SPI / X11 input injection which already reaches \+ backgrounded windows without needing activation."+ .into(),+ input_schema: serde_json::json!({+ "type":"object","required":["pid"],"properties":{+ "pid":{"type":"integer"},+ "window_id":{"type":"integer"}+ },"additionalProperties":false+ }),+ read_only: false, destructive: false, idempotent: true, open_world: false,+ })+ }++ async fn invoke(&self, _args: Value) -> ToolResult {+ ToolResult::error(+ "bring_to_front is Windows-only today. On Linux the input tools deliver via \+ AT-SPI / X11 input injection which already reaches backgrounded windows. If \+ you need explicit activation for your own UX reasons, shell out to \+ `wmctrl -a` or `xdotool windowactivate` from outside cua-driver."+ .to_string(),+ )+ .with_structured(serde_json::json!({+ "code": "bring_to_front_unsupported_on_platform",+ "platform": "linux",+ // Machine-readable remediation hint — mirrors the macOS+ // bring_to_front stub's structured `suggestion` field so+ // cross-platform clients can dispatch on a uniform key.+ "suggestion":+ "Linux input tools (click / type_text / press_key / hotkey) already \+ reach backgrounded windows via AT-SPI / X11 input injection — \+ there is no equivalent need to bring a window to the foreground. \+ If you need explicit window activation for UX reasons, shell out \+ to `wmctrl -a <title>` or `xdotool windowactivate <wid>` from \+ outside cua-driver.",+ }))+ }+}+// ── registry ─────────────────────────────────────────────────────────────────pub fn build_registry(compat: bool) -> ToolRegistry {@@ -2061,6 +2114,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry {r.register(Box::new(GetWindowStateTool { state: state.clone() }));r.register(Box::new(LaunchAppTool));r.register(Box::new(KillAppTool));+ r.register(Box::new(BringToFrontTool));r.register(Box::new(ClickTool { state: state.clone() }));r.register(Box::new(DoubleClickTool { state: state.clone() }));r.register(Box::new(RightClickTool { state: state.clone() }));@@ -2070,11 +2124,10 @@ pub fn build_registry(compat: bool) -> ToolRegistry {r.register(Box::new(HotkeyTool));r.register(Box::new(SetValueTool));r.register(Box::new(ScrollTool));- if compat {- r.register(Box::new(ScreenshotCompatTool { state: state.clone() }));- } else {- r.register(Box::new(ScreenshotTool { state: state.clone() }));- }+ // `screenshot` removed - see the matching comment in+ // platform-windows/src/tools/impl_.rs::build_registry. Canonical+ // screenshot path is `get_window_state` with `capture_mode:"vision"`.+ let _ = compat;r.register(Box::new(GetScreenSizeTool));r.register(Box::new(GetCursorPositionTool));r.register(Box::new(MoveCursorTool { state: state.clone() }));diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rsindex febfc2d6aa..626567da11 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs@@ -34,7 +34,7 @@ use std::sync::{Mutex, OnceLock};use std::time::{Duration, Instant};use cursor_overlay::{- CursorConfig, FocusRect, MotionConfig, OverlayCommand, RenderStateCore,+ CursorConfig, FocusRect, MotionConfig, OverlayCommand, RenderStateCore, ZOrderEnforcer,};// ── Arrival-signal channel ────────────────────────────────────────────────@@ -358,14 +358,15 @@ fn render_loop(}// Repin: immediately on target change, then defensive every ~1 s.+ // Delegate to the cross-platform ZOrderEnforcer so the contract for+ // "z+1 of the application under test" is documented once in+ // `cursor_overlay::z_order`.repin_frames += 1;let pin_changed = pinned_wid != last_pinned;last_pinned = pinned_wid;- if let Some(wid) = pinned_wid {- if pin_changed || repin_frames >= 60 {- dispatch_pin_above(win_ptr, wid);- repin_frames = 0;- }+ if pinned_wid.is_some() && (pin_changed || repin_frames >= 60) {+ MacZOrderEnforcer { win_ptr }.reassert(pinned_wid);+ repin_frames = 0;} else if repin_frames >= 60 {repin_frames = 0;}@@ -487,6 +488,33 @@ fn dispatch_pin_above(win_ptr: usize, target_wid: u64) {}}+// ── Z-order enforcer (macOS impl of cursor_overlay::ZOrderEnforcer) ──────++/// macOS implementation of [`cursor_overlay::ZOrderEnforcer`].+///+/// Holds the NSWindow pointer as a `usize` and dispatches the+/// `orderWindow:relativeTo:` call to the main queue (AppKit must run on+/// the main thread).+///+/// `target = None` is treated as a no-op here rather than raising to the+/// front: macOS creates the overlay at `NSNormalWindowLevel` and never+/// promotes it into an "always-on-top" level the way Windows does with+/// `HWND_TOPMOST`, so there is no topmost band to escape. Letting the+/// overlay stay where the user's activations have left it keeps it out+/// of the way when no agent action is in flight.+struct MacZOrderEnforcer {+ win_ptr: usize,+}++impl ZOrderEnforcer for MacZOrderEnforcer {+ fn reassert(&self, target: Option<u64>) {+ if let Some(wid) = target {+ dispatch_pin_above(self.win_ptr, wid);+ }+ // target = None → no-op; see struct doc comment.+ }+}+/// Create a `CGImage` from a `tiny_skia::Pixmap` (premultiplied RGBA)./// Returns a `+1` retained pointer that the caller must release.fn pixmap_to_cgimage(pixmap: &tiny_skia::Pixmap) -> Option<usize> {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rsnew file mode 100644index 0000000000..5cba0a8b26--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rs@@ -0,0 +1,71 @@+//! macOS `bring_to_front` stub.+//!+//! `bring_to_front` exists to let agents pay a one-shot foreground swap on+//! Windows before driving Chromium-content or GTK-button targets through+//! the SendInput-required code path. On macOS, equivalent functionality is+//! offered by `NSRunningApplication.activate` / Cocoa's window-ordering+//! APIs, which the macOS input tools never need internally — every+//! `CGEvent.postToPid` based dispatch reaches the target without+//! foreground manipulation. So the macOS stub returns an actionable+//! "Windows-only" error.++use async_trait::async_trait;+use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef}};+use serde_json::Value;++pub struct BringToFrontTool;++static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();++fn def() -> &'static ToolDef {+ DEF.get_or_init(|| ToolDef {+ name: "bring_to_front".into(),+ description:+ "Activate a window so subsequent input tools with `dispatch:\"foreground\"` \+ land on it without a per-call SetForegroundWindow flash. **Windows-only:** \+ on macOS this tool returns an error pointing to the platform-native \+ `NSRunningApplication.activate` (which the macOS input tools don't need \+ because CGEvent.postToPid reaches backgrounded windows). On Linux this \+ tool also stubs out; use `wmctrl -a` or `xdotool windowactivate` if \+ you need explicit activation."+ .into(),+ input_schema: serde_json::json!({+ "type": "object",+ "required": ["pid"],+ "properties": {+ "pid": { "type": "integer" },+ "window_id": { "type": "integer" }+ },+ "additionalProperties": false,+ }),+ read_only: false,+ destructive: false,+ idempotent: true,+ open_world: false,+ })+}++#[async_trait]+impl Tool for BringToFrontTool {+ fn def(&self) -> &ToolDef { def() }++ async fn invoke(&self, _args: Value) -> ToolResult {+ ToolResult::error(+ "bring_to_front is Windows-only. On macOS, input tools do not need \+ explicit foreground activation: every CGEvent.postToPid dispatch \+ reaches backgrounded windows. If you need to bring the app \+ visually forward for your own UX reasons, use \+ NSRunningApplication.activate via your own AppleScript / shell \+ call — cua-driver intentionally does not expose that path because \+ it violates the no-foreground contract."+ .to_string(),+ )+ .with_structured(serde_json::json!({+ "code": "bring_to_front_unsupported_on_platform",+ "platform": "macos",+ "suggestion":+ "macOS input tools deliver via CGEvent.postToPid which is \+ inherently background-safe; no bring_to_front equivalent needed.",+ }))+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex ea19ad7ddc..0b2df500f4 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -5,6 +5,7 @@ mod list_windows;mod get_window_state;mod launch_app;mod kill_app;+mod bring_to_front;mod click;mod double_click;mod right_click;@@ -211,6 +212,7 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) {registry.register(Box::new(get_window_state::GetWindowStateTool::new(state.clone())));registry.register(Box::new(launch_app::LaunchAppTool));registry.register(Box::new(kill_app::KillAppTool));+ registry.register(Box::new(bring_to_front::BringToFrontTool));registry.register(Box::new(click::ClickTool::new(state.clone())));registry.register(Box::new(double_click::DoubleClickTool::new(state.clone())));registry.register(Box::new(right_click::RightClickTool::new(state.clone())));@@ -220,11 +222,10 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) {registry.register(Box::new(hotkey::HotkeyTool::new(state.clone())));registry.register(Box::new(set_value::SetValueTool::new(state.clone())));registry.register(Box::new(scroll::ScrollTool::new(state.clone())));- if compat {- registry.register(Box::new(screenshot_compat::ClaudeCodeCompatScreenshotTool::new(state.clone())));- } else {- registry.register(Box::new(screenshot::ScreenshotTool { state: state.clone() }));- }+ // `screenshot` removed - see the matching comment in+ // platform-windows/src/tools/impl_.rs::build_registry. Canonical+ // screenshot path is `get_window_state` with `capture_mode:"vision"`.+ let _ = compat;registry.register(Box::new(get_screen_size::GetScreenSizeTool));registry.register(Box::new(get_cursor_position::GetCursorPositionTool));registry.register(Box::new(move_cursor::MoveCursorTool::new(state.clone())));diff --git a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml b/libs/cua-driver/rust/crates/platform-windows/Cargo.tomlindex 2d79fba608..85b9a94091 100644--- a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml+++ b/libs/cua-driver/rust/crates/platform-windows/Cargo.toml@@ -51,5 +51,19 @@ windows = { version = "0.58", features = ["Management_Deployment","ApplicationModel","Foundation_Collections",+ # Windows.Graphics.Capture (WGC) - the supported way to capture+ # UWP / DirectComposition windows even when they're occluded by+ # another window. PrintWindow returns black for these surfaces,+ # and the screen-region BitBlt fallback returns the occluding+ # window's pixels. Min OS: Win10 1903 (~100% of users today).+ "Graphics_Capture",+ "Graphics_DirectX",+ "Graphics_DirectX_Direct3D11",+ "Win32_Graphics_Direct3D11",+ "Win32_Graphics_Direct3D",+ "Win32_Graphics_Dxgi",+ "Win32_Graphics_Dxgi_Common",+ "Win32_System_WinRT_Direct3D11",+ "Win32_System_WinRT_Graphics_Capture",] }base64 = { workspace = true }diff --git a/libs/cua-driver/rust/crates/platform-windows/src/capture.rs b/libs/cua-driver/rust/crates/platform-windows/src/capture.rsindex cc3ffffa67..c689cec765 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/capture.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/capture.rs@@ -68,6 +68,62 @@ fn is_mostly_black_bgra(bgra: &[u8]) -> bool {sampled > 0 && (black * 200) >= (sampled * 199)}+/// Probe whether `target` is currently obscured by some other window.+/// Samples `WindowFromPoint` at the target's center + four corners (insetting+/// 2 px from each corner so we don't catch invisible 1-px frame slop). If any+/// sample returns a window whose root ancestor isn't `target` itself, we+/// consider the target occluded. Used by the screen-region BitBlt path to+/// warn callers that the bitmap they're about to receive does NOT actually+/// show the target's pixels — it shows whatever's covering it. Without this+/// signal, the BitBlt result is silently misleading (see the regression+/// captured during the autonomous-test journal: a backgrounded Notepad+/// screenshot returned LibreOffice's pixels).+unsafe fn target_is_obscured(target: HWND) -> bool {+ use windows::Win32::Foundation::{POINT, RECT};+ use windows::Win32::UI::WindowsAndMessaging::{+ GetAncestor, GetWindowRect, WindowFromPoint, GA_ROOT,+ };++ if target.0.is_null() {+ return false;+ }+ let mut rect = RECT::default();+ if GetWindowRect(target, &mut rect).is_err() {+ return false;+ }+ let w = rect.right - rect.left;+ let h = rect.bottom - rect.top;+ if w <= 4 || h <= 4 {+ return false;+ }+ // 5 sample points: 4 corners (inset 2 px) + center.+ let pts: [(i32, i32); 5] = [+ (rect.left + 2, rect.top + 2),+ (rect.right - 3, rect.top + 2),+ (rect.left + 2, rect.bottom - 3),+ (rect.right - 3, rect.bottom - 3),+ ((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2),+ ];+ let target_root = GetAncestor(target, GA_ROOT);+ let mut covered_samples = 0;+ for (x, y) in &pts {+ let p = POINT { x: *x, y: *y };+ let owner = WindowFromPoint(p);+ if owner.0.is_null() {+ continue;+ }+ let owner_root = GetAncestor(owner, GA_ROOT);+ if owner_root != target_root {+ covered_samples += 1;+ }+ }+ // Threshold of 2 of 5: a single corner being "covered" can be a layered+ // overlay (e.g. the cua-driver agent cursor) that's not actually opaque+ // over the content; we don't want to false-positive on that. Two or more+ // sample points missing means real content is being covered.+ covered_samples >= 2+}+/// Fallback capture path: BitBlt the desktop DC over the rectangle covered/// by `hwnd`'s on-screen bounds. Works for UWP / WinUI3 / DirectComposition/// surfaces that PrintWindow can't reach, as long as the window is on-screen@@ -128,7 +184,18 @@ unsafe fn screenshot_via_screen_region(hwnd: HWND) -> Result<(Vec<u8>, i32, i32)/// Capture a window by HWND, returning raw PNG bytes.pub fn screenshot_window_bytes(hwnd: u64) -> Result<Vec<u8>> {- unsafe { screenshot_window_bytes_unsafe(hwnd) }+ screenshot_window_bytes_with_occlusion(hwnd).map(|(b, _)| b)+}++/// Capture a window by HWND. Returns `(png_bytes, occluded_flag)` — the+/// boolean is `true` when the capture had to fall through to the+/// screen-region BitBlt path AND another window was visibly covering the+/// target at sample time. In that case the bitmap reflects the COVERING+/// window's pixels, not the target's. Callers that surface the image to a+/// user / LLM should attach an explicit warning. See `target_is_obscured`+/// for the sampling heuristic.+pub fn screenshot_window_bytes_with_occlusion(hwnd: u64) -> Result<(Vec<u8>, bool)> {+ unsafe { screenshot_window_bytes_with_occlusion_unsafe(hwnd) }}/// Capture a window by HWND, returning (base64_png, width, height).@@ -143,24 +210,45 @@ pub fn screenshot_window(hwnd: u64) -> Result<(String, u32, u32)> {Ok((BASE64.encode(&png_bytes), w, h))}-unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {+unsafe fn screenshot_window_bytes_with_occlusion_unsafe(hwnd: u64) -> Result<(Vec<u8>, bool)> {use windows::Win32::UI::WindowsAndMessaging::GetClientRect;use windows::Win32::Foundation::RECT;let hwnd_raw = hwnd;let hwnd = HWND(hwnd as *mut _);- // CUA-542 routing: for known XAML / WinUI3 / UWP targets, skip- // PrintWindow entirely and go straight to the screen-region BitBlt- // path. PrintWindow either returns all-black bitmaps for those- // surfaces or — as observed for backgrounded Calculator — a tiny- // clipped capture of the window's collapsed client rect. The- // screen-region path reads from the live desktop DC, which has the- // real composited image.+ // CUA-542 routing: for known XAML / WinUI3 / UWP targets, the+ // PrintWindow GDI path returns black (DirectComposition isn't in+ // the GDI back-buffer). We try Windows.Graphics.Capture (WGC)+ // first — that's the only API that returns the target's actual+ // composited pixels even when it's occluded by another window+ // (the Calculator-behind-terminal regression captured in the+ // autonomous-test journal). On older Windows / GPU stalls / cloaked+ // windows WGC may fail; fall back to the screen-region BitBlt+ // path, then to PrintWindow as a last resort.if crate::input::is_xaml_host_hwnd(hwnd_raw) {+ match crate::wgc::screenshot_window_via_wgc(hwnd_raw) {+ Ok((pixels, w, h)) => {+ return Ok((+ mcp_server::image_utils::encode_bgra_to_png(&pixels, w, h)?,+ false, // WGC reads target's own pixels — never occluded by definition+ ));+ }+ Err(e) => {+ tracing::warn!(+ target: "cua-driver",+ "screenshot: WGC failed for XAML target hwnd 0x{hwnd_raw:x}: {e}; \+ falling back to screen-region BitBlt (may be occluded)."+ );+ }+ }+ let occluded = target_is_obscured(hwnd);match screenshot_via_screen_region(hwnd) {Ok((pixels, w, h)) => {- return mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32);+ return Ok((+ mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)?,+ occluded,+ ));}Err(e) => {// Screen-region failed — fall through and try PrintWindow as a@@ -218,12 +306,35 @@ unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {if ok == 0 { bail!("GetDIBits returned 0"); }// CUA-542: detect the all-black bitmap PrintWindow returns for- // DirectComposition-backed UWP / WinUI3 surfaces and retry via- // screen-region BitBlt. See module docs for the rationale.+ // DirectComposition-backed UWP / WinUI3 surfaces. Recovery order:+ // 1. WGC (occlusion-immune; works for UWP).+ // 2. Screen-region BitBlt (works when target is on-screen and+ // not covered).+ // The WGC-first ordering covers backgrounded UWP targets the+ // screen-region path mishandles (returns covering window's pixels).if is_mostly_black_bgra(&pixels) {+ match crate::wgc::screenshot_window_via_wgc(hwnd_raw) {+ Ok((alt_pixels, w, h)) => {+ return Ok((+ mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, w, h)?,+ false,+ ));+ }+ Err(e) => {+ tracing::warn!(+ target: "cua-driver",+ "screenshot: PrintWindow returned black AND WGC failed for hwnd 0x{hwnd_raw:x}: {e}; \+ trying screen-region BitBlt next."+ );+ }+ }+ let occluded = target_is_obscured(hwnd);match screenshot_via_screen_region(hwnd) {Ok((alt_pixels, alt_w, alt_h)) => {- return mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, alt_w as u32, alt_h as u32);+ return Ok((+ mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, alt_w as u32, alt_h as u32)?,+ occluded,+ ));}Err(e) => {// Screen-region path failed too — return the (black) PrintWindow@@ -244,7 +355,10 @@ unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {// uncompressed-PNG path that produced ~5x larger output. The// `image` crate's encoder is already a workspace dep so the// smaller output is free).- mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)+ // PrintWindow itself reads from the target's own DC, so the bitmap+ // we return here is the target's pixels even when occluded — no+ // occluded warning needed on this path.+ Ok((mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)?, false))}// NOTE: previously this module carried a hand-rolleddiff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rsnew file mode 100644index 0000000000..c1c0364e51--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs@@ -0,0 +1,258 @@+//! Shared dispatch-mode logic for cua-driver-rs Windows input tools.+//!+//! Each input tool accepts an optional `dispatch` field with three modes:+//!+//! - `background` (DEFAULT) — PostMessage / UIA path only. If+//! `would_be_silently_dropped(target, event_kind)` returns true, the tool+//! returns a structured `background_unavailable` error so the caller can+//! `bring_to_front` then retry with `dispatch:"foreground"`. This is the+//! default because cua-driver's value proposition is that input never+//! steals foreground — making that the silent guarantee, not a hidden+//! trade-off, is worth a breaking change to existing callers that+//! previously got auto-fallback to SendInput.+//!+//! - `foreground` — SendInput with a brief `SetForegroundWindow(target)`+//! swap, restoring the prior foreground after the events flush. Brief+//! flash unless the target was already foreground. Required to drive+//! Chromium-content or GTK-button widget targets reliably.+//!+//! - `auto` — current heuristics: each tool's existing PostMessage /+//! SendInput routing is preserved bit-for-bit. Opt-in for callers that+//! want the historical silent-fallback behavior. Not the default.+//!+//! The matrix of "which (window class, event kind) pairs are silently+//! dropped by PostMessage" lives here as `would_be_silently_dropped`.+//! Tools call into this helper instead of inlining the class checks.++use serde_json::Value;++/// Family of synthetic input events. Used together with the target HWND+/// to decide whether PostMessage will be silently dropped.+#[derive(Copy, Clone, Debug)]+pub enum EventKind {+ /// `WM_LBUTTONDOWN/UP`, `WM_RBUTTONDOWN/UP`, `WM_MBUTTONDOWN/UP`.+ MouseClick,+ /// `WM_MOUSEMOVE` (also covers drag intermediate steps).+ MouseMove,+ /// `WM_VSCROLL` / `WM_HSCROLL`.+ MouseScroll,+ /// `WM_KEYDOWN/UP` with no modifiers — plain key tap.+ Keystroke,+ /// `WM_KEYDOWN/UP` with modifiers — accelerator candidate (Ctrl+S etc).+ KeyCombo,+ /// `WM_CHAR` text input.+ TextInput,+}++impl EventKind {+ pub fn name(self) -> &'static str {+ match self {+ Self::MouseClick => "mouse_click",+ Self::MouseMove => "mouse_move",+ Self::MouseScroll => "mouse_scroll",+ Self::Keystroke => "keystroke",+ Self::KeyCombo => "key_combo",+ Self::TextInput => "text_input",+ }+ }+}++/// Client-controlled dispatch policy.+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum DispatchMode {+ /// Current heuristics. Each tool keeps its existing routing.+ Auto,+ /// PostMessage / UIA only. Error if dispatch would be silently dropped.+ Background,+ /// SendInput with brief foreground swap.+ Foreground,+}++impl DispatchMode {+ /// Parse from a tool's `dispatch` JSON arg.+ ///+ /// Missing / unknown values resolve to `Background` — cua-driver's+ /// default is now strict no-foreground. Callers that want the+ /// historical silent-fallback heuristics must explicitly pass+ /// `dispatch:"auto"`.+ pub fn from_args(args: &Value) -> Self {+ match args.get("dispatch").and_then(|v| v.as_str()) {+ Some("auto") => Self::Auto,+ Some("foreground") => Self::Foreground,+ _ => Self::Background,+ }+ }+}++/// JSON-schema fragment for the `dispatch` field. Include this in every+/// input tool's `input_schema.properties.dispatch`.+pub fn dispatch_schema() -> Value {+ serde_json::json!({+ "type": "string",+ "enum": ["background", "foreground", "auto"],+ "default": "background",+ "description":+ "Dispatch mode. 'background' (default) refuses to swap \+ foreground; returns a structured background_unavailable error \+ if PostMessage would be silently dropped for this event kind \+ on this target. 'foreground' explicitly accepts a brief \+ SetForegroundWindow swap (SendInput path) — required to drive \+ Chromium-content or GTK-button widget targets reliably. \+ 'auto' uses cua-driver's internal heuristics (silent fallback \+ to SendInput on known-problematic targets); opt-in for \+ callers that prefer the historical behavior."+ })+}++/// Returns true if PostMessage on `hwnd` for this `event_kind` is empirically+/// known to be silently dropped by the target's input stack.+///+/// Combines the existing `is_chromium_target_window` detector with new GTK+/// (`gdkWindowToplevel` / `gdkSurfaceToplevel`) detection.+pub fn would_be_silently_dropped(hwnd: u64, kind: EventKind) -> bool {+ use EventKind::*;+ if crate::input::is_chromium_target_window(hwnd) {+ // Chromium's input thread architecture requires SendInput-queue+ // origin for mouse + key-combo events (#1623). Plain keystrokes and+ // text input via WM_CHAR still work because they go through+ // Chromium's IME path, which DOES consume Win32 messages.+ return matches!(kind, MouseClick | MouseMove | MouseScroll | KeyCombo);+ }+ if is_gtk_target_window(hwnd) {+ // Conservative flag for GTK: button widgets ignore PostMessage+ // clicks, drawing-area widgets accept them. We cannot distinguish+ // at the HWND level (single HWND for the whole GTK window) so we+ // flag mouse clicks broadly. Canvas-style drag works in practice;+ // caller can still opt to retry with dispatch:"background" on the+ // drag path if the click error wasn't actually load-bearing.+ return matches!(kind, MouseClick);+ }+ if is_vcl_target_window(hwnd) {+ // VCL (LibreOffice / OpenOffice family) routes accelerator keys+ // through TranslateAccelerator, which reads `GetKeyState`.+ // PostMessage(WM_KEYDOWN) delivers the key but does NOT update+ // GetKeyState, so single-key dialog accelerators (Y/N for+ // confirmations, Esc / Enter / Tab) and modifier combos (Ctrl+S,+ // Alt+F4) silently fail. Plain WM_CHAR text input through the+ // document widgets still works (verified end-to-end against+ // Writer's main editing area). Flag the keystroke-class events+ // so dispatch:"background" surfaces a structured error instead+ // of pretending to succeed.+ return matches!(kind, Keystroke | KeyCombo);+ }+ false+}++/// Detect LibreOffice / OpenOffice (VCL framework) windows.+///+/// VCL on Windows registers window classes with a `SAL` prefix (StarOffice's+/// "Service Abstraction Layer"): `SALFRAME` (top-level), `SALSUBFRAME`+/// (dialogs / popups), `SALOBJECT` (embedded), `SALMENU` (menu host),+/// `SALTMPSUBFRAME` (transient). The `SAL` prefix is unique enough to+/// match by leading characters.+pub fn is_vcl_target_window(hwnd: u64) -> bool {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;+ if hwnd == 0 {+ return false;+ }+ let mut buf = [0u16; 64];+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };+ if n <= 0 {+ return false;+ }+ let class = String::from_utf16_lossy(&buf[..n as usize]);+ class.starts_with("SAL")+}++/// Detect GTK/GDK top-level windows.+///+/// GTK 3 on Windows uses class `gdkWindowToplevel`; GTK 4 uses+/// `gdkSurfaceToplevel`. Match by prefix in case future GTK versions+/// append a suffix the way Chromium does (`Chrome_WidgetWin_0` /+/// `Chrome_WidgetWin_1`).+pub fn is_gtk_target_window(hwnd: u64) -> bool {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;+ if hwnd == 0 {+ return false;+ }+ let mut buf = [0u16; 64];+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };+ if n <= 0 {+ return false;+ }+ let class = String::from_utf16_lossy(&buf[..n as usize]);+ class.starts_with("gdkWindow") || class.starts_with("gdkSurface")+}++/// Read a window's class name into an owned String. Best-effort; returns+/// `"<unknown>"` on any failure so it can drop straight into a diagnostic.+pub fn read_class_name(hwnd: u64) -> String {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;+ if hwnd == 0 {+ return "<unknown>".into();+ }+ let mut buf = [0u16; 256];+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };+ if n <= 0 {+ return "<unknown>".into();+ }+ String::from_utf16_lossy(&buf[..n as usize])+}++/// Build the structured `background_unavailable` error returned when+/// `dispatch:"background"` would silently drop.+pub fn background_unavailable_error(+ hwnd: u64,+ kind: EventKind,+) -> mcp_server::protocol::ToolResult {+ let class = read_class_name(hwnd);+ let text = format!(+ "Background dispatch is not available for target window class \+ '{class}' on this event kind ({}). Either call bring_to_front \+ then retry with dispatch:\"foreground\", or accept the foreground \+ swap directly by setting dispatch:\"foreground\".",+ kind.name()+ );+ mcp_server::protocol::ToolResult::error(text)+ .with_structured(serde_json::json!({+ "code": "background_unavailable",+ "target_class": class,+ "event_kind": kind.name(),+ "suggestion":+ "Either call bring_to_front then retry with dispatch:\"foreground\", \+ or accept the foreground swap by setting dispatch:\"foreground\" directly.",+ }))+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn dispatch_mode_parses_known_values() {+ let j = |s: &str| serde_json::json!({"dispatch": s});+ assert_eq!(DispatchMode::from_args(&j("auto")), DispatchMode::Auto);+ assert_eq!(DispatchMode::from_args(&j("background")), DispatchMode::Background);+ assert_eq!(DispatchMode::from_args(&j("foreground")), DispatchMode::Foreground);+ }++ #[test]+ fn dispatch_mode_defaults_to_background() {+ // Missing field, garbage value, null — all resolve to Background.+ // This is the cua-driver no-foreground-by-default contract;+ // breaking change to callers who used to rely on silent SendInput+ // fallback against Chromium / GTK-button targets.+ assert_eq!(DispatchMode::from_args(&serde_json::json!({})), DispatchMode::Background);+ assert_eq!(+ DispatchMode::from_args(&serde_json::json!({"dispatch": "garbage"})),+ DispatchMode::Background+ );+ assert_eq!(+ DispatchMode::from_args(&serde_json::json!({"dispatch": null})),+ DispatchMode::Background+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rsindex 416dfd409e..ad7aee477a 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs@@ -176,35 +176,87 @@ pub fn post_char(hwnd: u64, ch: char) -> Result<()> {Ok(())}+/// Post `\n` (or `\r`, or `\r\n`) as a real Enter keystroke pair —+/// WM_KEYDOWN/WM_KEYUP(VK_RETURN) — to the focused child `h`.+///+/// Why a keystroke and not WM_CHAR(0x0A) or WM_CHAR(0x0D): standard Win32+/// edit controls accept `WM_CHAR(0x0D)` (carriage return) and insert a+/// newline; richer controls (LibreOffice's VCL edit, modern WPF/WinForms,+/// Scintilla) ignore `0x0A` outright and only sometimes accept `0x0D`.+/// A real `WM_KEYDOWN(VK_RETURN)` is the universally-honoured "Enter+/// pressed" signal — it produces a paragraph break in word processors,+/// activates the default button in dialogs, and submits forms in browsers.+/// The previous `WM_CHAR(0x0A)`-as-newline path silently joined every+/// line of multi-line `type_text` input into a single run (visible in the+/// LibreOffice Writer ode-to-a-background-cursor screenshot).+unsafe fn post_enter_keystroke(h: HWND) -> Result<()> {+ let vk = windows::Win32::UI::Input::KeyboardAndMouse::VK_RETURN;+ let scan = MapVirtualKeyW(vk.0 as u32, MAPVK_VK_TO_VSC);+ let lp_down = 1u32 | (scan << 16);+ let lp_up = lp_down | (1u32 << 30) | (1u32 << 31);+ PostMessageW(h, WM_KEYDOWN, WPARAM(vk.0 as usize), LPARAM(lp_down as isize))?;+ // Hold time between KEYDOWN and KEYUP — matches `post_key`'s pattern and+ // gives the target's message loop time to TranslateMessage the KEYDOWN+ // (which synthesizes WM_CHAR(0x0D)) and DispatchMessage the paragraph+ // break before the KEYUP arrives. Without this gap, LibreOffice Writer+ // intermittently produced a double paragraph break AND dropped the+ // next character — visible in the "ABC\nDEF\nGHI" repro as "ABC / DEF /+ // (gap) / HI".+ sleep(Duration::from_millis(KEY_DELAY_MS));+ PostMessageW(h, WM_KEYUP, WPARAM(vk.0 as usize), LPARAM(lp_up as isize))?;+ Ok(())+}+/// Post all characters in a string as WM_CHAR messages with inter-key delay.+///+/// Line breaks (`\n`, `\r`, or `\r\n`) are emitted as Enter keystrokes+/// (see [`post_enter_keystroke`]) instead of literal `WM_CHAR(0x0A/0x0D)`,+/// which most rich-text Win32 controls drop.pub fn post_type_text(hwnd: u64, text: &str) -> Result<()> {- if let Some(msg) = crate::input::post_message_blocked_by_uipi(hwnd) {- anyhow::bail!(msg);- }- let h_parent = HWND(hwnd as *mut _);- // Resolve focused-child once at entry — re-querying per-character would- // race with text-insertion side effects on the focus.- let h = focused_descendant(h_parent).unwrap_or(h_parent);- for ch in text.chars() {- let code = ch as u32 as usize;- unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }- sleep(Duration::from_millis(KEY_DELAY_MS));- }- Ok(())+ post_type_text_with_delay(hwnd, text, 0)}/// Post all characters in `text` as WM_CHAR messages with a configurable/// inter-character delay (on top of the baseline KEY_DELAY_MS gap).+///+/// Line breaks (`\n`, `\r`, or `\r\n`) are emitted as Enter keystrokes+/// (see [`post_enter_keystroke`]) — see the LibreOffice screenshot in+/// the PR description for the bug this fixes.pub fn post_type_text_with_delay(hwnd: u64, text: &str, inter_char_ms: u64) -> Result<()> {if let Some(msg) = crate::input::post_message_blocked_by_uipi(hwnd) {anyhow::bail!(msg);}let h_parent = HWND(hwnd as *mut _);+ // Resolve focused-child once at entry — re-querying per-character would+ // race with text-insertion side effects on the focus.let h = focused_descendant(h_parent).unwrap_or(h_parent);++ let mut prev_was_cr = false;for ch in text.chars() {- let code = ch as u32 as usize;- unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }- sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms));+ match ch {+ // `\r\n` (Windows-style line ending) → single Enter. The `\r`+ // does the Enter; the following `\n` is consumed silently.+ '\n' if prev_was_cr => {+ prev_was_cr = false;+ }+ '\n' | '\r' => {+ unsafe { post_enter_keystroke(h)?; }+ prev_was_cr = ch == '\r';+ // Extra settle after Enter — paragraph creation in rich+ // editors (VCL Writer, Scintilla, RichEdit) is heavier than+ // a single-character insert, so the baseline KEY_DELAY_MS ++ // inter_char_ms is not enough on slower hosts. The extra+ // 20 ms covers the queue drain without noticeably slowing+ // multi-line typing.+ sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms + 20));+ }+ _ => {+ prev_was_cr = false;+ let code = ch as u32 as usize;+ unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }+ sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms));+ }+ }}Ok(())}@@ -317,6 +369,31 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<// Brief settle so the foreground swap is processed before we send.sleep(Duration::from_millis(8));+ // Verify the swap actually happened. `SetForegroundWindow` returns a+ // BOOL but Windows treats foreground-lock violations as a silent+ // no-op in most cases (the call returns success-ish, the foreground+ // doesn't change). The reliable check is observing the foreground+ // after the settle. If we didn't get it, abort BEFORE SendInput —+ // otherwise the events land on whatever app actually held foreground+ // (typically the terminal hosting this process), which causes spooky+ // side effects like Alt+Enter toggling the terminal into fullscreen.+ let actual_fg = GetForegroundWindow();+ if actual_fg != target {+ // Don't restore — prev_fg is presumably still foreground anyway.+ bail!(+ "Foreground swap to target HWND {:?} was rejected by Windows \+ (actual foreground is HWND {:?}). This daemon is not at \+ UIAccess integrity, so SetForegroundWindow is subject to the \+ foreground-lock and the swap silently fails. Without the \+ swap, SendInput would land on the wrong window. Fix: install \+ / spawn the cua-driver-uia worker (UIAccess-manifested PE) \+ and route hotkey calls through it. Until then, the calling \+ app must already be foreground for dispatch:\"foreground\" \+ to be safe.",+ target.0, actual_fg.0+ );+ }+let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);if sent as usize != events.len() {// SendInput returns the number of events successfully inserted.diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rsindex 114ac9bd5f..bf7b1c3f49 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs@@ -11,6 +11,7 @@pub mod mouse;pub mod keyboard;+pub mod dispatch;pub use mouse::{is_chromium_target_window, post_click, post_click_screen, send_click_synthesized};pub use keyboard::{diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rsindex 317d6a415b..d42f29430c 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs@@ -323,6 +323,25 @@ pub fn send_click_synthesized(let _ = SetForegroundWindow(target);sleep(Duration::from_millis(8));+ // Verify the swap actually happened. Without UIAccess the call returns+ // success but the foreground stays put — and SendInput then lands on+ // whatever window IS foreground (typically the terminal hosting this+ // process). Abort before injecting so we don't click random apps.+ let actual_fg = GetForegroundWindow();+ if actual_fg != target {+ bail!(+ "Foreground swap to target HWND {:?} was rejected by Windows \+ (actual foreground is HWND {:?}). This daemon is not at \+ UIAccess integrity, so SetForegroundWindow is subject to the \+ foreground-lock and the swap silently fails. Without the \+ swap, SendInput-based mouse events would land on the wrong \+ window. Fix: install / spawn the cua-driver-uia worker \+ (UIAccess-manifested PE) and route Chromium coord clicks \+ through it.",+ target.0, actual_fg.0+ );+ }+// Move the cursor first so the OS hover state matches before the click.// `SetCursorPos` is the visible cursor move; the MOUSEEVENTF_MOVE input// ensures Chromium's input filter sees a coordinated move event.diff --git a/libs/cua-driver/rust/crates/platform-windows/src/lib.rs b/libs/cua-driver/rust/crates/platform-windows/src/lib.rsindex 663b801b80..e56c19188e 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/lib.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/lib.rs@@ -27,6 +27,9 @@ pub mod input;#[cfg(target_os = "windows")]pub mod capture;+#[cfg(target_os = "windows")]+pub mod wgc;+#[cfg(target_os = "windows")]pub mod launch_uwp;diff --git a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rsindex d1729abab0..4650c5107b 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs@@ -22,7 +22,7 @@ use std::sync::{Mutex, OnceLock};use std::time::Instant;use cursor_overlay::{- CursorConfig, MotionConfig, OverlayCommand, RenderStateCore,+ CursorConfig, MotionConfig, OverlayCommand, RenderStateCore, ZOrderEnforcer,};// ── Global channel ────────────────────────────────────────────────────────@@ -31,6 +31,14 @@ static CMD_TX: OnceLock<std::sync::mpsc::SyncSender<OverlayCommand>> = OnceLock:static CMD_RX_CELL: Mutex<Option<std::sync::mpsc::Receiver<OverlayCommand>>> = Mutex::new(None);static RENDER: Mutex<Option<RenderState>> = Mutex::new(None);+// ── Arrival-signal channel ────────────────────────────────────────────────+//+// `animate_cursor_to` installs a oneshot sender here, the render thread's+// `WM_TIMER` handler fires it the tick the planned path ends. Mirrors+// macOS so click handlers can `.await` until the cursor visually lands+// before dispatching the actual UIA / PostMessage action.+static ARRIVAL_TX: Mutex<Option<tokio::sync::oneshot::Sender<()>>> = Mutex::new(None);+pub fn init(cfg: CursorConfig) {let (tx, rx) = std::sync::mpsc::sync_channel(4096);let _ = CMD_TX.set(tx);@@ -84,6 +92,52 @@ pub fn is_at_initial_position() -> bool {.unwrap_or(true)}+/// Animate the overlay cursor to `(x, y)` and suspend until the planned+/// path completes (the spring-settle phase that follows is allowed to keep+/// running — we only wait for the visible glide to land).+///+/// Mirrors `platform_macos::cursor::overlay::animate_cursor_to`. Returns+/// immediately (no animation, no wait) when:+/// - the overlay is disabled, or+/// - the cursor is still at the off-screen sentinel `(-200, -200)` — in+/// that case the caller should rely on `ClickPulse` to snap the cursor.+pub async fn animate_cursor_to(x: f64, y: f64) {+ let should_animate = {+ let guard = RENDER.lock().unwrap();+ match guard.as_ref() {+ Some(rs) if rs.core.cfg.enabled && rs.core.visible && rs.core.pos.0 > -50.0 => true,+ _ => false,+ }+ };+ if !should_animate {+ return;+ }++ // Install the oneshot sender BEFORE issuing MoveTo, so the render+ // thread's arrival-fire can never lose a race against an immediate+ // path-end (e.g. zero-length glide).+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();+ {+ let mut guard = ARRIVAL_TX.lock().unwrap();+ // A previous in-flight animation gets superseded by this one —+ // unblock its waiter so it doesn't hang forever.+ if let Some(old_tx) = guard.take() {+ let _ = old_tx.send(());+ }+ *guard = Some(tx);+ }++ send_command(OverlayCommand::MoveTo {+ x,+ y,+ // Arrive pointing upper-left (45°) — same convention as macOS /+ // Swift reference (`endAngleDegrees: 45`).+ end_heading_radians: std::f64::consts::FRAC_PI_4,+ });++ let _ = rx.await;+}+/// Spin up the overlay on a dedicated thread (STA for Win32 message loop)./// This is a non-blocking call — the overlay runs on its own thread.pub fn run_on_thread() {@@ -144,8 +198,11 @@ impl RenderState {}}- fn tick(&mut self, dt: f64) {- self.core.tick_motion(dt);+ /// Advance the motion state by `dt`. Returns `true` the tick the+ /// planned path completes, so the WM_TIMER handler can fire the+ /// arrival oneshot that unblocks `animate_cursor_to`.+ fn tick(&mut self, dt: f64) -> bool {+ self.core.tick_motion(dt)}fn apply_command(&mut self, cmd: OverlayCommand) {@@ -244,6 +301,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCoOVERLAY_HWND.store(hwnd.0 as isize, std::sync::atomic::Ordering::Relaxed);*CMD_RX_WIN.lock().unwrap() = Some(rx);LAST_ZTICK.store(0, std::sync::atomic::Ordering::Relaxed);+ let _ = Z_ORDER.set(WinZOrderEnforcer { hwnd_isize: hwnd.0 as isize });// Standard Win32 message loop.let mut msg = MSG::default();@@ -267,6 +325,7 @@ static OVERLAY_HWND: std::sync::atomic::AtomicIsize =static CMD_RX_WIN: Mutex<Option<std::sync::mpsc::Receiver<OverlayCommand>>> = Mutex::new(None);static LAST_ZTICK: std::sync::atomic::AtomicU64 =std::sync::atomic::AtomicU64::new(0);+static Z_ORDER: OnceLock<WinZOrderEnforcer> = OnceLock::new();// ── Window procedure ──────────────────────────────────────────────────────@@ -290,7 +349,7 @@ unsafe extern "system" fn wnd_proc(// Drain commands and tick animation. Measure real dt from last// tick — Windows timer resolution defaults to 15ms so the// hardcoded 8ms was running the animation at half speed.- let pixmap = {+ let (pixmap, fire_arrival) = {let mut guard = RENDER.lock().unwrap();if let Some(rs) = guard.as_mut() {// Drain the channel.@@ -304,17 +363,17 @@ unsafe extern "system" fn wnd_proc(let now = std::time::Instant::now();let dt = now.duration_since(rs.last_tick).as_secs_f64().clamp(0.0, 0.05);rs.last_tick = now;- rs.tick(dt);- Some(cursor_overlay::render_frame(+ let arrived = rs.tick(dt);+ (Some(cursor_overlay::render_frame(&rs.core,rs.virt_w.max(1) as u32,rs.virt_h.max(1) as u32,rs.virt_x as f64,rs.virt_y as f64,None, // focus-rect is macOS-only- ))+ )), arrived)} else {- None+ (None, false)}};@@ -322,11 +381,28 @@ unsafe extern "system" fn wnd_proc(update_layered_window(hwnd, &pm);}- // Z-order maintenance every 80ms.+ // Fire arrival oneshot the tick the path just ended — unblocks+ // any `animate_cursor_to(...).await` so the click action only+ // dispatches once the cursor has visually landed.+ if fire_arrival {+ if let Ok(mut guard) = ARRIVAL_TX.lock() {+ if let Some(tx) = guard.take() {+ let _ = tx.send(());+ }+ }+ }++ // Z-order maintenance every 80ms — delegate to the cross-platform+ // ZOrderEnforcer so the contract for "z+1 of the application under+ // test" is documented once in `cursor_overlay::z_order`.let last = LAST_ZTICK.load(std::sync::atomic::Ordering::Relaxed);if now_ms.wrapping_sub(last) >= 80 {LAST_ZTICK.store(now_ms, std::sync::atomic::Ordering::Relaxed);- reapply_z_order(hwnd);+ let pinned_wid = RENDER.lock().ok()+ .and_then(|g| g.as_ref().and_then(|rs| rs.core.pinned_wid));+ if let Some(enforcer) = Z_ORDER.get() {+ enforcer.reassert(pinned_wid);+ }}LRESULT(0)@@ -434,54 +510,81 @@ unsafe fn update_layered_window(ReleaseDC(None, hdc_screen);}-#[cfg(target_os = "windows")]-unsafe fn reapply_z_order(hwnd: windows::Win32::Foundation::HWND) {- use windows::Win32::Foundation::HWND;- use windows::Win32::UI::WindowsAndMessaging::*;+// ── Z-order enforcer (Windows impl of cursor_overlay::ZOrderEnforcer) ────- // Read pinned_wid from render state (RENDER lock released before this is called).- let pinned_wid = {- let guard = RENDER.lock().unwrap();- guard.as_ref().and_then(|rs| rs.core.pinned_wid)- };-- let pinned_target = pinned_wid.and_then(|wid| {- let h = HWND(wid as *mut _);- if IsWindow(h).as_bool() { Some(h) } else { None }- });-- // The overlay must sit JUST above the pinned target window so the- // user's foreground app (a different non-topmost window — say their- // terminal) renders on top of the overlay. Two pitfalls:- //- // 1. HWND_TOPMOST was the previous fallback. Once Windows promotes- // a window into the topmost band (sets WS_EX_TOPMOST), a later- // SetWindowPos with a normal target_hwnd does NOT drop it back- // out — the overlay stays above EVERYTHING non-topmost (incl.- // the user's foreground). That was the symptom in #1688-style- // reports: overlay sticks visible over a terminal even when- // the pinned window (Calculator) is behind it.- // 2. To drop out of the topmost band, we need an explicit- // SetWindowPos(hwnd, HWND_NOTOPMOST, …) call. Then we can- // issue a second SetWindowPos with the real target_hwnd so- // the overlay lands ABOVE the pin but BELOW any window- // currently above the pin (the user's foreground).- //- // Fallback when there's no live pin: HWND_TOP — top of non-topmost- // band, NOT the topmost band. So a "no pin" overlay still respects- // the user's foreground stack.- let _ = SetWindowPos(- hwnd,- HWND_NOTOPMOST,- 0, 0, 0, 0,- SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOOWNERZORDER,- );+/// Win32 implementation of [`cursor_overlay::ZOrderEnforcer`].+///+/// Stores the overlay HWND as an `isize` (HWND is `*mut c_void` and not+/// `Send`/`Sync`) and rehydrates it inside `reassert`. Driven by the+/// `WM_TIMER` branch in `wnd_proc` on the overlay STA thread.+struct WinZOrderEnforcer {+ hwnd_isize: isize,+}- let insert_after = pinned_target.unwrap_or(HWND_TOP);- let _ = SetWindowPos(- hwnd,- insert_after,- 0, 0, 0, 0,- SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_SHOWWINDOW | SWP_NOOWNERZORDER,- );+impl ZOrderEnforcer for WinZOrderEnforcer {+ fn reassert(&self, target: Option<u64>) {+ #[cfg(target_os = "windows")]+ unsafe {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::*;++ let hwnd = HWND(self.hwnd_isize as *mut _);++ let pinned_target = target.and_then(|wid| {+ let h = HWND(wid as *mut _);+ if IsWindow(h).as_bool() { Some(h) } else { None }+ });++ // The overlay must sit JUST above the pinned target window so the+ // user's foreground app (a different non-topmost window — say their+ // terminal) renders on top of the overlay. Three Win32 pitfalls:+ //+ // 1. HWND_TOPMOST was the previous fallback. Once Windows promotes+ // a window into the topmost band (sets WS_EX_TOPMOST), a later+ // SetWindowPos with a normal target_hwnd does NOT drop it back+ // out — the overlay stays above EVERYTHING non-topmost (incl.+ // the user's foreground). That was the symptom in #1688-style+ // reports.+ // 2. To drop out of the topmost band, we need an explicit+ // SetWindowPos(hwnd, HWND_NOTOPMOST, …) call before the real+ // z-order placement.+ // 3. `SetWindowPos(hwnd, target_hwnd, …)` does NOT mean "put hwnd+ // above target_hwnd" — Win32 semantics are "insert hwnd+ // *after* target_hwnd in z-order" (i.e. one slot BELOW+ // target). To land overlay *above* target we have to insert+ // it after target's previous sibling instead — the window+ // currently just above target. `GetWindow(target, GW_HWNDPREV)`+ // returns that (or null when target is already the topmost+ // non-topmost window, in which case HWND_TOP raises overlay+ // to the top and pushes target one slot down). This is the+ // pitfall macOS / Linux dodge by virtue of their explicit+ // `orderWindow:above:` / `StackMode::ABOVE` APIs.+ //+ // Fallback when there's no live pin: HWND_TOP — top of non-topmost+ // band, NOT the topmost band. So a "no pin" overlay still respects+ // the user's foreground stack.+ let _ = SetWindowPos(+ hwnd,+ HWND_NOTOPMOST,+ 0, 0, 0, 0,+ SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOOWNERZORDER,+ );++ let insert_after = match pinned_target {+ Some(target) => {+ let prev = GetWindow(target, GW_HWNDPREV).unwrap_or(HWND(std::ptr::null_mut()));+ if !prev.0.is_null() { prev } else { HWND_TOP }+ }+ None => HWND_TOP,+ };+ let _ = SetWindowPos(+ hwnd,+ insert_after,+ 0, 0, 0, 0,+ SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_SHOWWINDOW | SWP_NOOWNERZORDER,+ );++ let _ = target;+ }+ }}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 1515781681..92b80a6eb3 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -2,12 +2,37 @@use async_trait::async_trait;+/// Pin the agent-cursor overlay above `hwnd` in the z-order, normalising to+/// the **root** ancestor so the pin lands on the window that actually appears+/// in the global z-stack.+///+/// For UWP / packaged apps the inner `Windows.UI.Core.CoreWindow` is hosted+/// inside `ApplicationFrameHost.exe` — UIA targets the inner window but only+/// the host appears in the z-order. `GA_ROOT` normalises both inputs to the+/// host so the overlay sits at z+1 of whatever is actually painted on screen.+///+/// No-op when the overlay is disabled; the command is just dropped by the+/// render thread in that case.+fn pin_overlay_above(hwnd: u64) {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT};+ let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };+ let wid = if !root.0.is_null() { root.0 as u64 } else { hwnd };+ crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(wid));+}+/// Animate the agent cursor to (sx, sy) in screen coordinates and wait for the/// glide to finish before returning. No-op when the overlay is not enabled.////// On the very first call the cursor is at the off-screen initial position/// (-200, -200). Animating from there would cause a jarring off-screen fly-in,/// so we snap to the target with a ClickPulse first and skip the glide wait.+///+/// For all subsequent calls this defers to+/// [`crate::overlay::animate_cursor_to`], which sends `MoveTo` and awaits the+/// render thread's arrival oneshot — that's how we keep the click action+/// from firing before the cursor has visually landed (the old heuristic+/// `tokio::sleep(80..600 ms)` was racing the spring-physics glide).async fn overlay_glide_to(sx: f64, sy: f64) {if !crate::overlay::is_enabled() { return; }let pos = crate::overlay::current_position();@@ -16,15 +41,7 @@ async fn overlay_glide_to(sx: f64, sy: f64) {crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });return;}- // Compute travel time from distance and speed (matches Swift: peak=900, avg~467 pts/sec).- let dist = ((sx - pos.0).powi(2) + (sy - pos.1).powi(2)).sqrt();- let avg_speed = 467.0_f64; // (300 + 900 + 200) / 3 ≈ Swift avg- let ms = (dist / avg_speed * 1000.0).clamp(80.0, 600.0) as u64;- // Always arrive at 45° (upper-left) — matches Swift's endAngleDegrees: 45.0.- crate::overlay::send_command(cursor_overlay::OverlayCommand::MoveTo {- x: sx, y: sy, end_heading_radians: std::f64::consts::FRAC_PI_4,- });- tokio::time::sleep(std::time::Duration::from_millis(ms)).await;+ crate::overlay::animate_cursor_to(sx, sy).await;}use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef, ToolRegistry}};use serde_json::{json, Value};@@ -571,7 +588,25 @@ impl Tool for GetWindowStateTool {std::time::Duration::from_secs(4), blocking).await{Ok(join_result) => join_result.map_err(|e| anyhow::anyhow!("task panic: {e}")),- Err(_elapsed) => Err(anyhow::anyhow!("get_window_state timed out after 4s (UIA provider unresponsive)")),+ Err(_elapsed) => {+ // Surface the target's window class + an actionable hint+ // instead of just "UIA provider unresponsive". The class+ // points the caller at the right workaround (e.g. SALFRAME+ // → screenshot + pixel coords + dispatch:"foreground"; UWP+ // class → retry with capture_mode:"vision" to skip the+ // UIA walk and just get a screenshot).+ let class = crate::input::dispatch::read_class_name(hwnd);+ Err(anyhow::anyhow!(+ "get_window_state timed out after 4s (UIA provider unresponsive on \+ hwnd 0x{hwnd:x}, class '{class}'). Fallback options: \+ (a) re-call this tool with `capture_mode:\"vision\"` to skip the \+ UIA walk and just capture the screenshot — bypasses UIA entirely, \+ then drive via pixel `click(x, y)` against the returned image; \+ (b) if the target is a transient VCL / message-box dialog, send \+ `press_key` with `dispatch:\"foreground\"` (SendInput) to fire the \+ default accelerator (Esc / Enter / Y / N) without needing the tree."+ ))+ }};let result = result.and_then(|r| r);@@ -991,7 +1026,8 @@ impl Tool for LaunchAppTool {"additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process (or activation arguments for packaged apps)."},"electron_debugging_port":{"type":"integer","description":"Accepted for cross-platform parity; currently no-op on Windows."},"webkit_inspector_port":{"type":"integer","description":"Accepted for cross-platform parity; no-op on Windows."},- "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."}+ "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."},+ "start_minimized":{"type":"boolean","description":"When true, launch the app's window minimized to the taskbar instead of restored-but-not-activated. Use this when the agent wants to drive the app entirely in the background — the user's previously-frontmost window (e.g. terminal) stays visually on top. Implementation uses SW_SHOWMINNOACTIVE for the ShellExecuteEx path and a follow-up ShowWindow(SW_MINIMIZE) on the AUMID path. UIA / background dispatch still work on a minimized window; only `screenshot` and `dispatch:\"foreground\"` need it restored."}},"additionalProperties":false}),read_only: false, destructive: false, idempotent: true, open_world: true,})@@ -1006,6 +1042,41 @@ impl Tool for LaunchAppTool {let urls: Vec<String> = args.get("urls").and_then(|v| v.as_array()).map(|a| a.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect()).unwrap_or_default();+ let start_minimized = args.get("start_minimized")+ .and_then(|v| v.as_bool())+ .unwrap_or(false);+ // `nShow` for the ShellExecuteEx-based launches. SW_SHOWNOACTIVATE+ // keeps the new window from grabbing focus but still renders it on+ // top of the previous frontmost window. SW_SHOWMINNOACTIVE both+ // skips activation AND drops the window to the taskbar, which is+ // what an agent doing background work wants — the user's terminal+ // stays visually on top. UIA / PostMessage / set_value all keep+ // working against a minimized window; only screenshot and+ // dispatch:"foreground" pay a cost.+ let n_show: i32 = if start_minimized {+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWMINNOACTIVE.0+ } else {+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE.0+ };+ // Snapshot the set of currently-running pids BEFORE we launch. The+ // detached start_minimized polling task uses this to detect "any+ // process that came into existence as a result of our launch", so+ // it can minimize their windows too — covers the launcher-stub+ // case (LibreOffice swriter.exe → soffice.bin, GIMP gimp-3.exe →+ // gimp-3.2.exe, etc.) where the launched pid exits and a child+ // process with a different name owns the actual window. Captured+ // here (before the launch) so the new soffice.bin pid won't be+ // in the snapshot.+ let pre_launch_pids: std::sync::Arc<std::collections::HashSet<u32>> =+ if start_minimized {+ let pids: std::collections::HashSet<u32> = crate::win32::list_processes()+ .into_iter()+ .map(|p| p.pid)+ .collect();+ std::sync::Arc::new(pids)+ } else {+ std::sync::Arc::new(std::collections::HashSet::new())+ };// Capture the foreground window BEFORE any launch path runs so the// post-spawn polling restore (below) has a target HWND to flip back@@ -1181,11 +1252,11 @@ impl Tool for LaunchAppTool {let urls_clone = urls.clone();let target_for_shell = target_file_opt.clone();let extra_for_shell = extra_joined.clone();+ let n_show_for_shell = n_show;let result = tokio::task::spawn_blocking(move || -> anyhow::Result<u32> {use windows::Win32::UI::Shell::{ShellExecuteExW, SHELLEXECUTEINFOW, SEE_MASK_NOCLOSEPROCESS,};- use windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE;use windows::Win32::System::Threading::GetProcessId;use windows::Win32::Foundation::CloseHandle;use windows::core::PCWSTR;@@ -1211,7 +1282,7 @@ impl Tool for LaunchAppTool {} else {PCWSTR(args_w.as_ptr())},- nShow: SW_SHOWNOACTIVATE.0,+ nShow: n_show_for_shell,..Default::default()};unsafe { ShellExecuteExW(&mut info)?; }@@ -1230,7 +1301,7 @@ impl Tool for LaunchAppTool {cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,lpVerb: PCWSTR(op_w.as_ptr()),lpFile: PCWSTR(file.as_ptr()),- nShow: SW_SHOWNOACTIVATE.0,+ nShow: n_show_for_shell,..Default::default()};unsafe { let _ = ShellExecuteExW(&mut url_info); }@@ -1275,9 +1346,9 @@ impl Tool for LaunchAppTool {// capture, mirroring Swift's NSWorkspace secondary-URL flow).if aumid_for_uwp.is_some() && !urls.is_empty() {let urls_clone = urls.clone();+ let n_show_for_urls = n_show;let _ = tokio::task::spawn_blocking(move || {use windows::Win32::UI::Shell::{ShellExecuteExW, SHELLEXECUTEINFOW};- use windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE;use windows::core::PCWSTR;fn to_wide(s: &str) -> Vec<u16> {s.encode_utf16().chain(std::iter::once(0)).collect()@@ -1289,7 +1360,7 @@ impl Tool for LaunchAppTool {cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,lpVerb: PCWSTR(op_w.as_ptr()),lpFile: PCWSTR(file.as_ptr()),- nShow: SW_SHOWNOACTIVATE.0,+ nShow: n_show_for_urls,..Default::default()};unsafe { let _ = ShellExecuteExW(&mut url_info); }@@ -1412,6 +1483,139 @@ impl Tool for LaunchAppTool {}let pid = resolved_pid;+ // start_minimized post-pass: drop every resolved window to the+ // taskbar. The desktop ShellExecuteEx path already minimizes via+ // nShow=SW_SHOWMINNOACTIVE, but three cases still need this+ // fallback:+ // 1. AUMID / UWP activations come up un-minimized —+ // ActivateApplication has no nShow.+ // 2. Apps that ignore the nShow hint and decide their own geometry+ // (notable offenders: LibreOffice, some Electron apps).+ // 3. Launcher-stub flows (LO swriter.exe → soffice.bin, GIMP, etc.)+ // where the launched pid exits before the real window appears,+ // so `windows_json` is empty at this point.+ //+ // To cover (3) we spawn a background polling task that keeps+ // looking for windows under the launched pid family for up to 5 s+ // and minimizes anything that materializes. The task runs detached+ // so the launch_app response isn't held up by it.+ //+ // SW_MINIMIZE itself activates "the next top-level window in z-order"+ // which would shift the user's focus, but the foreground-restore+ // polling task (spawned earlier in this method via+ // `restore_foreground_polling_best_effort`) flips foreground back to+ // the pre-launch window, so the net effect is "minimize and leave+ // the user's window where it was".+ if start_minimized {+ // First, minimize anything already resolved (covers the common+ // single-process path where windows_json was populated).+ let immediate_hwnds: Vec<u64> = windows_json.iter()+ .filter_map(|w| w["window_id"].as_u64())+ .collect();+ // Capture pid + basename for the post-launch polling task to+ // pick up launcher-stub child processes.+ let stub_basename = target_file_opt+ .as_deref()+ .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())+ .unwrap_or("")+ .to_owned();+ // parent_pid is the launched pid family root — used below to+ // constrain the post-launch minimize sweep to descendants /+ // name-relatives of the launched app, so we don't accidentally+ // minimize a user's unrelated app that started during the 5 s+ // poll window.+ let parent_pid = pid;+ let _ = tokio::task::spawn_blocking(move || {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_MINIMIZE};+ for h in immediate_hwnds {+ unsafe { let _ = ShowWindow(HWND(h as *mut _), SW_MINIMIZE); }+ }+ }).await;+ // Detached polling for launcher-stub late-window cases.+ // Strategy: every 200 ms for 5 s, find pids that+ // (a) weren't in the pre-launch snapshot, AND+ // (b) are part of the launched app's family — either a+ // descendant of the launched root pid, OR a process+ // sharing the launched binary's name prefix (covers+ // renamed-binary chains like swriter.exe → soffice.bin+ // where the child's name doesn't match but it's a+ // direct descendant).+ // Minimize THEIR visible windows only. Without the family+ // constraint, anything the user opens during the 5 s window+ // (Chrome, terminal, IDE) would also get minimized — that's+ // the regression CodeRabbit flagged.+ //+ // Loop ends early once we've minimized the first set of+ // windows AND remained idle for one tick — the typical+ // app has its main window up within ~2 s of launch.+ let pre_pids = pre_launch_pids.clone();+ let basename_for_poll = stub_basename.clone();+ tokio::spawn(async move {+ use std::collections::HashSet;+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_MINIMIZE};+ let mut minimized: HashSet<u64> = HashSet::new();+ let mut idle_ticks_after_any_hit: u8 = 0;+ let mut hit_count_total: usize = 0;+ for _ in 0..25 {+ let pre_pids_clone = pre_pids.clone();+ let basename_clone = basename_for_poll.clone();+ // Build the family pid set fresh each tick — both+ // descendant graph and name-relatives can grow as+ // launcher-stub chains spawn deeper children.+ let family_new_pids: Vec<u32> = tokio::task::spawn_blocking(move || {+ let related: std::collections::HashSet<u32> =+ crate::win32::related_processes(parent_pid, &basename_clone)+ .into_iter()+ .chain(std::iter::once(parent_pid))+ .collect();+ crate::win32::list_processes()+ .into_iter()+ .filter(|p| {+ !pre_pids_clone.contains(&p.pid) && related.contains(&p.pid)+ })+ .map(|p| p.pid)+ .collect()+ })+ .await+ .unwrap_or_default();+ let mut tick_hits: usize = 0;+ for cpid in family_new_pids {+ let wins = tokio::task::spawn_blocking(move || {+ crate::win32::list_windows(Some(cpid))+ })+ .await+ .unwrap_or_default();+ for w in wins {+ if minimized.insert(w.hwnd) {+ tick_hits += 1;+ hit_count_total += 1;+ let hwnd_iso = w.hwnd as usize;+ let _ = tokio::task::spawn_blocking(move || unsafe {+ let _ = ShowWindow(+ HWND(hwnd_iso as *mut _),+ SW_MINIMIZE,+ );+ }).await;+ }+ }+ }+ if tick_hits == 0 {+ idle_ticks_after_any_hit += 1;+ if hit_count_total > 0 && idle_ticks_after_any_hit >= 3 {+ // Stable: had hits, then 600 ms of nothing new.+ // Done.+ break;+ }+ } else {+ idle_ticks_after_any_hit = 0;+ }+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;+ }+ });+ }+// Match Swift text format 1:1.let mut summary = format!("✅ Launched {display} (pid {pid}) in background.");if !windows_json.is_empty() {@@ -1499,7 +1703,8 @@ impl Tool for ClickTool {"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Windows-only convenience; Swift exposes right-click as a separate `right_click` tool."},"count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},- "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."}+ "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -1508,6 +1713,7 @@ impl Tool for ClickTool {async fn invoke(&self, args: Value) -> ToolResult {use mcp_server::tool_args::ArgsExt;+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};let pid = match args.require_u32("pid") { Ok(v) => v, Err(e) => return e };let hwnd_opt = args.opt_u64("window_id");let elem_idx = args.opt_u64("element_index").map(|v| v as usize);@@ -1515,6 +1721,7 @@ impl Tool for ClickTool {let y = args.opt_f64("y");let button = args.str_or("button", "left");let count = args.u64_or("count", 1) as usize;+ let dispatch = DispatchMode::from_args(&args);// Resolve HWND: explicit, or auto from pid.let hwnd = match hwnd_opt {@@ -1534,13 +1741,37 @@ impl Tool for ClickTool {Some(v) => v,None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),};- // Step 2: animate cursor to screen coords (awaits glide_duration_ms).+ // Step 2: pin overlay to target window, then animate to screen coords.+ pin_overlay_above(hwnd);overlay_glide_to(cx as f64, cy as f64).await;// Step 3: click pulse + actual click.crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {x: cx as f64, y: cy as f64,});let btn = button.clone();++ // dispatch:"foreground" — skip UIA Invoke and use SendInput at the+ // cached element center. The caller explicitly chose foreground+ // delivery; UIA Invoke would be background-safe (which they+ // rejected) and might fail on elements that don't support+ // InvokePattern anyway.+ if dispatch == DispatchMode::Foreground {+ let btn_fg = btn.clone();+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Performed SendInput click on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }// Try UIA Invoke first (it works for UWP / modern XAML / web// content where PostMessage(WM_LBUTTONDOWN) hits the outer// HWND but never reaches the inner XAML/composition element).@@ -1594,6 +1825,15 @@ impl Tool for ClickTool {}}// PostMessage fallback (legacy Win32 + non-Invokable elements).+ // dispatch:"background" refuses the fallback on targets known+ // to silently drop PostMessage clicks (Chromium content, GTK+ // buttons). We surface a tagged error here so the outer match+ // can convert to the structured background_unavailable result.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ anyhow::bail!("__CUA_BG_UNAVAILABLE_CLICK__");+ }crate::input::post_click_screen(hwnd, cx, cy, count, &btn)?;let action_name = match btn.as_str() {"right" => "ShowMenu",@@ -1603,6 +1843,9 @@ impl Tool for ClickTool {}).await;match result {Ok(Ok(msg)) => ToolResult::text(msg),+ Ok(Err(e)) if e.to_string().contains("__CUA_BG_UNAVAILABLE_CLICK__") => {+ background_unavailable_error(hwnd, EventKind::MouseClick)+ }Ok(Err(e)) => ToolResult::error(e.to_string()),Err(e) => ToolResult::error(format!("Task error: {e}")),}@@ -1627,6 +1870,7 @@ impl Tool for ClickTool {let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);(pt.x as f64, pt.y as f64)};+ pin_overlay_above(hwnd);overlay_glide_to(sx, sy).await;crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });let btn = button.clone();@@ -1660,6 +1904,30 @@ impl Tool for ClickTool {// right-click (UIA has no by-coord `ShowContextMenu`) and// multi-click (`Invoke()` is single-fire by definition)// skip directly to PostMessage.+ // dispatch:"foreground" short-circuit — caller explicitly chose+ // foreground delivery. Skip the UIA hit-test (UIA Invoke is+ // background-safe and would deliver the click without the+ // foreground swap the caller asked for).+ if dispatch == DispatchMode::Foreground {+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn)+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => {+ let click_word = match count { 2 => "double-click", 3 => "triple-click", _ => "click" };+ ToolResult::text(format!(+ "✅ Sent {click_word} via SendInput to pid {pid} at ({sx},{sy}) (dispatch:foreground)."+ ))+ }+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }+let use_uia = (btn == "left" || btn == "middle") && count == 1;if use_uia {let invoked = tokio::task::spawn_blocking(move || {@@ -1676,6 +1944,25 @@ impl Tool for ClickTool {}}+ // UIA hit-test didn't land. Decide between PostMessage / SendInput+ // based on dispatch mode.+ //+ // dispatch:"background" — refuse to swap foreground. If the target+ // is known to silently drop PostMessage mouse events (Chromium+ // DOM content, GTK button widgets), surface a structured+ // background_unavailable error so the caller can bring_to_front+ // then retry with dispatch:"foreground".+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }++ // dispatch:"auto" — historical heuristic: Chromium targets get+ // SendInput with foreground swap (#1623); everything else takes+ // the PostMessage path. This branch is ONLY reached on Auto mode+ // (Background returned above; Foreground was handled at the top).+// #1623: PostMessage(WM_LBUTTONDOWN) to Chromium frame HWNDs doesn't// reach the DOM input pipeline — Chromium's input thread only accepts// events with SendInput-queue origin. For Chromium targets, take the@@ -1807,7 +2094,8 @@ impl Tool for TypeTextTool {"text":{"type":"string","description":"Text to insert at the focused element's cursor."},"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used."},"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},- "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."}+ "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -1816,11 +2104,13 @@ impl Tool for TypeTextTool {async fn invoke(&self, args: Value) -> ToolResult {use mcp_server::tool_args::ArgsExt;+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };let pid = raw_pid as u32;let text = match args.require_str("text") { Ok(v) => v, Err(e) => return e };let hwnd_opt = args.opt_u64("window_id");let elem_idx = args.opt_u64("element_index");+ let dispatch = DispatchMode::from_args(&args);if elem_idx.is_some() && hwnd_opt.is_none() {return ToolResult::error("window_id is required when element_index is used — the element_index cache \@@ -1840,6 +2130,36 @@ impl Tool for TypeTextTool {};let text_len = text.chars().count();+ // dispatch:"background" — TextInput is currently never flagged as+ // silently dropped (Chromium accepts WM_CHAR through its IME path),+ // but call the helper so the policy stays centralised in dispatch.rs+ // and future targets can be added without touching this site.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::TextInput)+ {+ return background_unavailable_error(hwnd, EventKind::TextInput);+ }+ // dispatch:"foreground" — no SendInput-based text-input helper yet.+ // Caller can build a foreground-style "type" by hotkey/press_key per+ // character; or use bring_to_front then dispatch:"auto" which keeps+ // PostMessage WM_CHAR (works for the targets we know about).+ if dispatch == DispatchMode::Foreground {+ return ToolResult::error(+ "dispatch:\"foreground\" is not yet implemented for type_text. \+ Use bring_to_front and retry with dispatch:\"auto\" — \+ PostMessage WM_CHAR works against an already-foreground \+ window for legacy Win32 targets, and UIA ValuePattern.SetValue \+ handles XAML/WinUI3 hosts when element_index is supplied. \+ TODO: add a send_text_synthesized helper that drives per-char \+ SendInput keystrokes.".to_string(),+ );+ }++ // Pin the agent-cursor overlay above the target window so the synthetic+ // cursor stays sandwiched at z+1 of the type target for the full+ // duration of the keystrokes (both XAML/UIA and PostMessage paths).+ pin_overlay_above(hwnd);+// CUA-543 routing: PostMessage WM_CHAR doesn't reach modern// XAML / WinUI3 hosts. When the target is one of those AND the// caller has supplied an element_index, route through UIA@@ -1936,7 +2256,8 @@ impl Tool for PressKeyTool {"key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},"modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},- "window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."}+ "window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -1945,11 +2266,13 @@ impl Tool for PressKeyTool {async fn invoke(&self, args: Value) -> ToolResult {use mcp_server::tool_args::ArgsExt;+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };let pid = raw_pid as u32;let key = match args.require_str("key") { Ok(v) => v, Err(e) => return e };let mods: Vec<String> = args.str_array("modifiers");let hwnd_opt = args.opt_u64("window_id");+ let dispatch = DispatchMode::from_args(&args);// Swift requires window_id when element_index is supplied — ports the// same validation.let elem_idx = args.opt_u64("element_index");@@ -1970,6 +2293,30 @@ impl Tool for PressKeyTool {}};let key_display = key.clone();+ // Background mode: plain keystrokes (no modifiers) go through Chromium+ // and GTK fine — would_be_silently_dropped returns false for the+ // Keystroke variant by design. KeyCombo (modifiers) on Chromium IS+ // dropped, so check that when modifiers are present.+ let event_kind = if mods.is_empty() { EventKind::Keystroke } else { EventKind::KeyCombo };+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, event_kind)+ {+ return background_unavailable_error(hwnd, event_kind);+ }+ // Foreground: send_key_synthesized takes the SetForegroundWindow path.+ if dispatch == DispatchMode::Foreground {+ let send_result = tokio::task::spawn_blocking(move || {+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();+ crate::input::send_key_synthesized(hwnd, &key, &m)+ }).await;+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent {key_display} via SendInput on pid {raw_pid} (dispatch:foreground)."+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }let result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods.iter().map(String::as_str).collect();crate::input::post_key(hwnd, &key, &m)@@ -2034,7 +2381,8 @@ impl Tool for HotkeyTool {"pid":{"type":"integer","description":"Target process ID."},"keys":{"type":"array","items":{"type":"string"},"minItems":2,"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},- "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."}+ "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -2043,9 +2391,11 @@ impl Tool for HotkeyTool {async fn invoke(&self, args: Value) -> ToolResult {use mcp_server::tool_args::ArgsExt;+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};let hwnd_opt = args.opt_u64("window_id");let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };let pid = raw_pid as u32;+ let dispatch = DispatchMode::from_args(&args);// Parse keys array (Swift's only path). Legacy key+modifiers shape// still accepted as a Rust-side convenience.@@ -2175,8 +2525,24 @@ impl Tool for HotkeyTool {// - Modifiers + non-Chromium Win32 → SendInput (the// TranslateAccelerator path). The foreground swap stays.let has_modifiers = !mods.is_empty();+ // dispatch:"background" — refuse if PostMessage of the key combo+ // would be silently dropped on this target (Chromium with modifiers).+ let event_kind = if has_modifiers { EventKind::KeyCombo } else { EventKind::Keystroke };+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, event_kind)+ {+ return background_unavailable_error(hwnd, event_kind);+ }+ // dispatch:"foreground" — explicit SendInput swap (the path that+ // unblocks TranslateAccelerator-style apps). Auto mode preserves the+ // historical "Chromium → PostMessage, non-Chromium-with-modifiers →+ // SendInput" heuristic.let is_chromium = crate::input::is_chromium_target_window(hwnd);- let use_send_input = has_modifiers && !is_chromium;+ let use_send_input = match dispatch {+ DispatchMode::Foreground => true,+ DispatchMode::Background => false,+ DispatchMode::Auto => has_modifiers && !is_chromium,+ };let result = tokio::task::spawn_blocking(move || {let m: Vec<&str> = mods.iter().map(String::as_str).collect();if use_send_input {@@ -2310,7 +2676,8 @@ impl Tool for ScrollTool {"amount":{"type":"integer","minimum":1,"maximum":50,"description":"Number of scroll ticks. Default 3."},"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},- "element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."}+ "element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: false, idempotent: false, open_world: true,@@ -2318,12 +2685,14 @@ impl Tool for ScrollTool {}async fn invoke(&self, args: Value) -> ToolResult {+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};// Swift error wording 1:1.let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(p) => p,None => return ToolResult::error("Missing required integer field pid."),};let pid = raw_pid as u32;+ let dispatch = DispatchMode::from_args(&args);let direction = match args.get("direction").and_then(|v| v.as_str()) {Some(d) => d.to_owned(),None => return ToolResult::error("Missing required string field direction."),@@ -2354,6 +2723,52 @@ impl Tool for ScrollTool {}};+ // dispatch:"background" — WM_VSCROLL/HSCROLL is silently dropped by+ // Chromium and may be by GTK. Don't use the generic+ // `background_unavailable_error` helper here: its standard+ // remediation suggests retrying with `dispatch:"foreground"`, but+ // scroll's foreground path is unimplemented (see the explicit+ // foreground rejection below), so the advertised recovery would+ // dead-end at another error. Surface a scroll-specific structured+ // error that points at `bring_to_front` + `dispatch:"auto"` (the+ // PostMessage path against an already-foreground window works for+ // most non-Chromium targets) as the actual viable recovery.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseScroll)+ {+ let class = crate::input::dispatch::read_class_name(hwnd);+ return ToolResult::error(format!(+ "Background scroll dispatch is not available for target window class \+ '{class}' on this event kind (mouse_scroll). The Chromium / GTK input \+ stack silently drops PostMessage WM_VSCROLL/HSCROLL events. \+ dispatch:\"foreground\" is also unimplemented for scroll — there is no \+ SendInput-based wheel-synthesis helper yet. Workaround: call \+ bring_to_front(pid, window_id) to make the target frontmost, then \+ retry with dispatch:\"auto\" — PostMessage scroll against an already-\+ foreground window works for most non-Chromium targets."+ ))+ .with_structured(serde_json::json!({+ "code": "scroll_unavailable",+ "target_class": class,+ "event_kind": "mouse_scroll",+ "suggestion":+ "bring_to_front + dispatch:\"auto\". dispatch:\"foreground\" is \+ NOT a valid recovery for scroll — that path returns an explicit \+ not-implemented error.",+ }));+ }+ // dispatch:"foreground" — TODO: route through SendInput MOUSEEVENTF_WHEEL+ // with brief SetForegroundWindow swap. No helper yet.+ if dispatch == DispatchMode::Foreground {+ return ToolResult::error(+ "dispatch:\"foreground\" is not yet implemented for the scroll tool. \+ Use bring_to_front to activate the target, then retry with \+ dispatch:\"auto\" (PostMessage scroll works against an already-\+ foreground window for most non-Chromium targets). TODO: add a \+ send_wheel_synthesized helper.".to_string(),+ );+ }+let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {use windows::Win32::Foundation::{HWND, LPARAM, WPARAM};use windows::Win32::UI::WindowsAndMessaging::{@@ -2459,24 +2874,31 @@ impl Tool for ScreenshotTool {let is_jpeg = format == "jpeg";let max_dim = self.state.config.read().unwrap().max_image_dimension;- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(String, u32, u32)> {- let raw = match hwnd_opt {- Some(hwnd) => crate::capture::screenshot_window_bytes(hwnd)?,- None => crate::capture::screenshot_display_bytes()?,+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(String, u32, u32, bool)> {+ // Use the occlusion-aware variant so we can flag captures that+ // went through the screen-region BitBlt path while another+ // window was visibly covering the target — those captures show+ // the OBSCURING window's pixels, not the target's, and silent+ // delivery would mislead the agent (regression captured in+ // the autonomous-test journal: a backgrounded Notepad+ // screenshot returned LibreOffice's pixels).+ let (raw, occluded) = match hwnd_opt {+ Some(hwnd) => crate::capture::screenshot_window_bytes_with_occlusion(hwnd)?,+ None => (crate::capture::screenshot_display_bytes()?, false),};let png_bytes = crate::capture::resize_png_if_needed(&raw, max_dim)?;let (w, h) = crate::capture::png_dimensions_pub(&png_bytes)?;use base64::{engine::general_purpose::STANDARD as B64, Engine as _};if is_jpeg {let jpeg = crate::capture::png_bytes_to_jpeg(&png_bytes, quality)?;- Ok((B64.encode(&jpeg), w, h))+ Ok((B64.encode(&jpeg), w, h, occluded))} else {- Ok((B64.encode(&png_bytes), w, h))+ Ok((B64.encode(&png_bytes), w, h, occluded))}}).await;match result {- Ok(Ok((b64, w, h))) => {+ Ok(Ok((b64, w, h, occluded))) => {let label = if is_jpeg { "jpeg" } else { "png" };// Match Swift `ScreenshotTool.swift` text format 1:1:// "✅ Window screenshot — WxH png [window_id: ID]"@@ -2493,7 +2915,33 @@ impl Tool for ScreenshotTool {mcp_server::protocol::Content::image_png(b64)};tr.content.push(img_content);- tr.structured_content = Some(json!({ "width": w, "height": h, "format": label }));+ // Occlusion warning: when the screen-region BitBlt path was+ // used AND another window was on top, the bitmap shows the+ // covering window. Tell the caller so they don't reason from+ // the wrong pixels. Always before the image in the content+ // vec so it's visible regardless of how the consumer reads.+ if occluded {+ tr.content.insert(0, mcp_server::protocol::Content::text(+ format!(+ "⚠️ Screenshot may not show the target window — HWND \+ {} appears to be obscured by another window. The \+ returned bitmap reflects whatever pixels are on \+ screen at the target's coordinates, not the \+ target's own backing store. Call \+ `bring_to_front(pid, window_id)` first if you need \+ an accurate capture, or use `get_window_state` \+ with `capture_mode:\"ax\"` for occlusion-immune \+ UIA tree access.",+ hwnd_opt.unwrap_or(0)+ )+ ));+ }+ tr.structured_content = Some(json!({+ "width": w,+ "height": h,+ "format": label,+ "occluded": occluded,+ }));tr}Ok(Err(e)) => ToolResult::error(e.to_string()),@@ -2640,12 +3088,14 @@ impl Tool for DoubleClickTool {"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys held during the gesture. Accepted for parity; currently no-op on Windows."},- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."}+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,})}async fn invoke(&self, args: Value) -> ToolResult {+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};// Swift error wording 1:1.let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(p) => p,@@ -2657,6 +3107,7 @@ impl Tool for DoubleClickTool {let elem_idx = args.opt_u64("element_index").map(|v| v as usize);let x = args.opt_f64("x");let y = args.opt_f64("y");+ let dispatch = DispatchMode::from_args(&args);// Swift validates "both x and y or neither" and "no element_index without window_id".let has_xy = x.is_some() && y.is_some();let partial_xy = x.is_some() != y.is_some();@@ -2693,8 +3144,32 @@ impl Tool for DoubleClickTool {Some(v) => v,None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),};+ pin_overlay_above(hwnd);overlay_glide_to(cx as f64, cy as f64).await;crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 });+ // dispatch:"background" — reject if PostMessage would be silently dropped.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }+ // dispatch:"foreground" — route through SendInput at the cached coords.+ if dispatch == DispatchMode::Foreground {+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, cx, cy, 2, "left")+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent double-click via SendInput on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }let result = tokio::task::spawn_blocking(move || -> anyhow::Result<String> {crate::input::post_click_screen(hwnd, cx, cy, 2, "left")?;// Swift text format 1:1: `"✅ Posted double-click to [N] role \"title\" at screen-point (X, Y)."`.@@ -2726,8 +3201,33 @@ impl Tool for DoubleClickTool {let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);(pt.x as f64, pt.y as f64)};+ pin_overlay_above(hwnd);overlay_glide_to(sx, sy).await;crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });+ // dispatch:"background" — reject if PostMessage would be silently dropped.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }+ // dispatch:"foreground" — SendInput at screen coords with FG swap.+ if dispatch == DispatchMode::Foreground {+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, 2, "left")+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent double-click via SendInput to pid {pid} at screen ({},{}) (dispatch:foreground).",+ sx as i32, sy as i32+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }let (xi, yi) = (px as i32, py as i32);let result = tokio::task::spawn_blocking(move || crate::input::post_click(hwnd, xi, yi, 2, "left")).await;match result {@@ -2784,12 +3284,14 @@ impl Tool for RightClickTool {"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys held during the right-click. Accepted for parity; currently no-op on Windows."},- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."}+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,})}async fn invoke(&self, args: Value) -> ToolResult {+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};// Swift error wording 1:1.let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(p) => p,@@ -2801,6 +3303,7 @@ impl Tool for RightClickTool {let elem_idx = args.opt_u64("element_index").map(|v| v as usize);let x = args.opt_f64("x");let y = args.opt_f64("y");+ let dispatch = DispatchMode::from_args(&args);// Port Swift's full validation set.let has_xy = x.is_some() && y.is_some();let partial_xy = x.is_some() != y.is_some();@@ -2837,8 +3340,30 @@ impl Tool for RightClickTool {Some(v) => v,None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),};+ pin_overlay_above(hwnd);overlay_glide_to(cx as f64, cy as f64).await;crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 });+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }+ if dispatch == DispatchMode::Foreground {+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, cx, cy, 1, "right")+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent right-click via SendInput on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }let result = tokio::task::spawn_blocking(move || -> anyhow::Result<String> {crate::input::post_click_screen(hwnd, cx, cy, 1, "right")?;// Match Swift's element-path text 1:1@@ -2871,8 +3396,31 @@ impl Tool for RightClickTool {let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);(pt.x as f64, pt.y as f64)};+ pin_overlay_above(hwnd);overlay_glide_to(sx, sy).await;crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }+ if dispatch == DispatchMode::Foreground {+ let prev_fg_addr = unsafe {+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize+ };+ let send_result = tokio::task::spawn_blocking(move || {+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, 1, "right")+ }).await;+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));+ return match send_result {+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent right-click via SendInput to pid {pid} at screen ({},{}) (dispatch:foreground).",+ sx as i32, sy as i32+ )),+ Ok(Err(e)) => ToolResult::error(e.to_string()),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }let (xi, yi) = (px as i32, py as i32);let result = tokio::task::spawn_blocking(move || crate::input::post_click(hwnd, xi, yi, 1, "right")).await;match result {@@ -2919,18 +3467,21 @@ impl Tool for DragTool {"steps":{"type":"integer","minimum":1,"maximum":200,"description":"Number of intermediate WM_MOUSEMOVE events. Default: 20."},"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys: cmd/shift/option/ctrl (held via keyboard)."},"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: left."},- "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."}+ "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."},+ "dispatch": crate::input::dispatch::dispatch_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,})}async fn invoke(&self, args: Value) -> ToolResult {+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};// Swift error wording 1:1.let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(p) => p,None => return ToolResult::error("Missing required integer field pid."),};let pid = raw_pid as u32;+ let dispatch = DispatchMode::from_args(&args);use mcp_server::tool_args::ArgsExt;// Accepts numeric JSON as either float or integer — coerce both to f64.@@ -2990,6 +3541,45 @@ impl Tool for DragTool {(p1.x, p1.y, p2.x, p2.y)};+ // dispatch:"background" — refuse if PostMessage drag would silently drop.+ if dispatch == DispatchMode::Background+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)+ {+ return background_unavailable_error(hwnd, EventKind::MouseClick);+ }+ // dispatch:"foreground" — no SendInput-based drag helper yet. The+ // PostMessage drag works for the GTK-canvas + plain-Win32 cases we+ // care about today; Chromium DOM dragstart will need a real+ // SendInput-MOUSE-MOVE sequence, tracked as TODO.+ if dispatch == DispatchMode::Foreground {+ return ToolResult::error(+ "dispatch:\"foreground\" is not yet implemented for the drag tool. \+ Use bring_to_front to activate the target first, then retry with \+ dispatch:\"auto\" (PostMessage WM_MOUSEMOVE works against an \+ already-foreground window in most cases). TODO: add a \+ send_drag_synthesized helper analogous to send_click_synthesized."+ .to_string(),+ );+ }++ // Pin the agent-cursor overlay above the drag target so the synthetic+ // cursor stays sandwiched at z+1 of the dragged window for the full+ // path. Drag stays within a single HWND, so one pin at the start is+ // sufficient — the 80 ms z-order tick keeps it asserted thereafter.+ pin_overlay_above(hwnd);++ // Animate the agent cursor to the drag-start, fire a press pulse,+ // run the actual drag synthesis, then glide to the drag-end and+ // fire a release pulse. Mirrors macOS `tools/drag.rs:191-242`.+ // Cursor doesn't follow the interpolated PostMessage path during+ // the drag itself (the timing coordination would be invasive);+ // pre- and post-glides plus the press/release pulses are enough+ // signal for a user watching the agent operate.+ overlay_glide_to(sx_from as f64, sy_from as f64).await;+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {+ x: sx_from as f64, y: sy_from as f64,+ });+let button_c = button.clone();let result = tokio::task::spawn_blocking(move || {crate::input::mouse::post_drag(@@ -3000,6 +3590,16 @@ impl Tool for DragTool {)}).await;+ // Drag finished — glide the visual cursor to the drag-end and+ // pulse the release. Skipped on error so the cursor doesn't lie+ // about a successful endpoint.+ if matches!(&result, Ok(Ok(()))) {+ overlay_glide_to(sx_to as f64, sy_to as f64).await;+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {+ x: sx_to as f64, y: sy_to as f64,+ });+ }+match result {Ok(Ok(())) => {// Match Swift's text format verbatim.@@ -4017,6 +4617,135 @@ impl Tool for TypeTextCharsTool {// keyed on pid. Marked `destructive: true` so MCP clients with permission// gating prompt before invoking.+// ── bring_to_front ────────────────────────────────────────────────────────++pub struct BringToFrontTool;+static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();++#[async_trait]+impl Tool for BringToFrontTool {+ fn def(&self) -> &ToolDef {+ BTF_DEF.get_or_init(|| ToolDef {+ name: "bring_to_front".into(),+ description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \+ the OS foreground. \n\n\+ **This deliberately breaks the no-foreground contract.** Use only when an \+ agent is about to drive a sequence of operations against a target whose input \+ stack silently drops PostMessage (Chromium DOM content, GTK button widgets) \+ and the agent wants to pay the foreground cost once instead of per call. \n\n\+ Pairs with the `dispatch` field on input tools:\n\+ 1. Try `click(..., dispatch:\"background\")`. If it returns \+ `background_unavailable`, the target needs foreground delivery.\n\+ 2. Call `bring_to_front(pid)` so the target is foreground.\n\+ 3. Subsequent input calls with `dispatch:\"foreground\"` deliver via \+ SendInput WITHOUT visible flashing -- the SetForegroundWindow swap \+ inside SendInput is a no-op since target is already foreground.\n\n\+ Implementation uses the `AttachThreadInput` trick to bypass Windows' \+ foreground-lock when the daemon is not at UIAccess integrity. Returns \+ structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \+ Windows only; macOS / Linux return an error pointing at platform-native \+ alternatives.".into(),+ input_schema: json!({"type":"object","required":["pid"],"properties":{+ "pid":{"type":"integer","description":"Target process ID."},+ "window_id":{"type":"integer","description":"Optional HWND. Defaults to pid's first visible top-level window."}+ },"additionalProperties":false}),+ read_only: false,+ destructive: false,+ idempotent: true,+ open_world: false,+ })+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ use mcp_server::tool_args::ArgsExt;+ let pid = match args.require_u32("pid") { Ok(v) => v, Err(e) => return e };+ let hwnd_opt = args.opt_u64("window_id");++ // Resolve hwnd: explicit, or auto from pid.+ let hwnd = match hwnd_opt {+ Some(h) => h,+ None => {+ let windows = tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid))).await.unwrap_or_default();+ match windows.first() {+ Some(w) => w.hwnd,+ None => return ToolResult::error(format!("bring_to_front: no windows found for pid {pid}. Provide window_id.")),+ }+ }+ };++ // Run the foreground swap on a blocking thread. The AttachThreadInput+ // trick mirrors `send_key_synthesized` (input/keyboard.rs:313-345)+ // and is validated by `flash-repro/16-edge-launch-fg.ps1` for the+ // Edge launch focus-steal recovery case.+ let outcome = tokio::task::spawn_blocking(move || -> Result<(u64, u64), String> {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow,+ SetForegroundWindow,+ };+ use windows::Win32::System::Threading::{AttachThreadInput, GetCurrentThreadId};++ let target = HWND(hwnd as *mut _);+ if !unsafe { IsWindow(target) }.as_bool() {+ return Err(format!("hwnd 0x{hwnd:x} is not a valid window"));+ }++ let prev_fg = unsafe { GetForegroundWindow() };+ let prev_fg_addr = prev_fg.0 as u64;++ // Attach our thread to the current FG thread so we inherit its+ // FG-lock token; SetForegroundWindow then succeeds even when+ // the daemon is at Medium IL without UIAccess.+ let my_tid = unsafe { GetCurrentThreadId() };+ let mut fg_pid = 0u32;+ let fg_tid = unsafe { GetWindowThreadProcessId(prev_fg, Some(&mut fg_pid)) };+ let attached = fg_tid != 0 && fg_tid != my_tid;+ if attached {+ let _ = unsafe { AttachThreadInput(my_tid, fg_tid, true) };+ }+ let set_ok = unsafe { SetForegroundWindow(target) }.as_bool();+ if attached {+ let _ = unsafe { AttachThreadInput(my_tid, fg_tid, false) };+ }++ // SetForegroundWindow can silently return FALSE under UIPI /+ // foreground-lock denial. We still report what we tried.+ let _ = set_ok;+ let now_fg = unsafe { GetForegroundWindow() };+ Ok((prev_fg_addr, now_fg.0 as u64))+ }).await;++ match outcome {+ Ok(Ok((prev, now))) => {+ let landed_on_target = now == hwnd;+ let msg = if landed_on_target {+ format!("✅ bring_to_front: pid {pid} hwnd 0x{hwnd:x} is now foreground (was 0x{prev:x}).")+ } else {+ format!(+ "bring_to_front returned but the OS did not honor the swap: target 0x{hwnd:x}, \+ current foreground 0x{now:x}. Common cause: Windows foreground-lock denied \+ SetForegroundWindow from a non-UIAccess process. Route through the \+ cua-driver-uia worker if you have one running."+ )+ };+ let r = if landed_on_target {+ ToolResult::text(msg)+ } else {+ ToolResult::error(msg)+ };+ r.with_structured(json!({+ "previous_fg_hwnd": format!("0x{prev:x}"),+ "now_fg_hwnd": format!("0x{now:x}"),+ "target_hwnd": format!("0x{hwnd:x}"),+ "landed_on_target": landed_on_target,+ }))+ }+ Ok(Err(e)) => ToolResult::error(format!("bring_to_front: {e}")),+ Err(join) => ToolResult::error(format!("bring_to_front: blocking-task join error: {join}")),+ }+ }+}+pub struct KillAppTool;static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -4367,6 +5096,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry {r.register(Box::new(GetWindowStateTool { state: state.clone() }));r.register(Box::new(LaunchAppTool));r.register(Box::new(KillAppTool));+ r.register(Box::new(BringToFrontTool));r.register(Box::new(DebugWindowInfoTool));r.register(Box::new(ClickTool { state: state.clone() }));r.register(Box::new(DoubleClickTool { state: state.clone() }));@@ -4377,11 +5107,24 @@ pub fn build_registry(compat: bool) -> ToolRegistry {r.register(Box::new(HotkeyTool));r.register(Box::new(SetValueTool { state: state.clone() }));r.register(Box::new(ScrollTool));- if compat {- r.register(Box::new(ScreenshotCompatTool { state: state.clone() }));- } else {- r.register(Box::new(ScreenshotTool { state: state.clone() }));- }+ // `screenshot` / `ScreenshotCompatTool` removed from the tool surface+ // — `get_window_state` with `capture_mode:"vision"` is the single+ // canonical path for getting a window screenshot. Reasons:+ // 1. One entry point means callers always have a window_id context,+ // which `screenshot` made ambiguous (whole-display fallback+ // always picked the wrong thing for headless agent workflows).+ // 2. `get_window_state` already returns a JPEG in vision/som mode+ // — there's no information `screenshot` exposed that's not in+ // `get_window_state`.+ // 3. Claude Code's vision pipeline can be retargeted at+ // `get_window_state` via its tool-anchor config — no need for+ // a dedicated "screenshot" name.+ // The `ScreenshotTool` and `ScreenshotCompatTool` structs are kept in+ // this file for now as private impl helpers — the `capture::*`+ // functions they wrap are still the actual screenshot machinery.+ // Delete the structs in a follow-up after confirming no callers+ // depend on them via `Tool` trait reflection.+ let _ = compat; // formerly drove the ScreenshotCompatTool branchr.register(Box::new(GetScreenSizeTool));r.register(Box::new(GetCursorPositionTool));r.register(Box::new(MoveCursorTool { state: state.clone() }));diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rsindex 0c958d87bd..120d6999d3 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs@@ -28,7 +28,7 @@//! Returns an actionable error if neither path works.use async_trait::async_trait;-use mcp_server::page::PageBackend;+use mcp_server::page::{ClickElementResult, PageBackend};use windows::core::Interface;use windows::Win32::Foundation::HWND;@@ -128,6 +128,146 @@ impl PageBackend for WindowsPageBackend {let result = mcp_server::cdp::evaluate(port, javascript, true).await?;Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))}++ async fn click_element(+ &self,+ pid: i32,+ window_id: u32,+ selector: &str,+ ) -> anyhow::Result<ClickElementResult> {+ // Two-step:+ // (1) JS to derive element center + viewport screen position.+ // (2) Animate cursor overlay to those screen coords + ClickPulse.+ // (3) JS to fire element.click().+ //+ // Steps 1 and 3 reuse self.execute_javascript so they take the same+ // bookmark / CDP routing the agent already knows. Step 2 uses the+ // existing overlay primitives (pin + animate_cursor_to + ClickPulse).++ let selector_js = escape_js_string_literal(selector);++ // Step 1 — query coords.+ let probe_js = format!(+ "(function(){{\+ var el = document.querySelector('{selector_js}');\+ if (!el) throw new Error('element_not_found:{selector_js}');\+ var r = el.getBoundingClientRect();\+ return JSON.stringify({{\+ vx: r.left + r.width/2,\+ vy: r.top + r.height/2,\+ sx: window.screenX + (window.outerWidth - window.innerWidth)/2,\+ sy: window.screenY + (window.outerHeight - window.innerHeight),\+ dpr: window.devicePixelRatio || 1\+ }});\+ }})()"+ );+ let probe_raw = self.execute_javascript(pid, window_id, &probe_js).await?;+ let probe_json = strip_dispatch_prefix(&probe_raw);++ // The wrapper paths return the result JSON-stringified. The bookmark+ // wrapper wraps the user expression's return in JSON.stringify(),+ // which means the inner string itself is JSON-encoded — we need to+ // parse twice to get the actual coord object.+ let parsed: serde_json::Value = serde_json::from_str(&probe_json)+ .or_else(|_| {+ // Bookmark wrapper double-encoded the string — decode the+ // outer JSON-string, then parse the inner JSON.+ let inner: String = serde_json::from_str(&probe_json)?;+ serde_json::from_str::<serde_json::Value>(&inner)+ })+ .map_err(|e| anyhow::anyhow!(+ "click_element: could not parse coord JSON from probe (raw: {probe_raw:?}): {e}"+ ))?;++ // Required-field validation. The previous version defaulted any+ // missing field to 0.0, which silently animated the visible cursor+ // to (0,0) AND still fired element.click() — the click would still+ // hit the DOM element correctly but the visual cursor would lie.+ // For a fast-fail loud-error contract, require all four screen-+ // coord inputs from the JS probe and reject anything missing/NaN.+ // `dpr` retains its 1.0 default because devicePixelRatio is+ // genuinely optional on the JS side (older webviews / unusual+ // contexts may not expose it).+ let require = |key: &str| -> Result<f64, anyhow::Error> {+ parsed.get(key).and_then(|v| v.as_f64()).filter(|f| f.is_finite())+ .ok_or_else(|| anyhow::anyhow!(+ "click_element: probe JSON missing/invalid required field '{key}' \+ (raw: {probe_raw:?}). All four of vx/vy/sx/sy must be finite numbers."+ ))+ };+ let vx = require("vx")?;+ let vy = require("vy")?;+ let sx = require("sx")?;+ let sy = require("sy")?;+ let dpr = parsed.get("dpr").and_then(|v| v.as_f64())+ .filter(|f| f.is_finite() && *f > 0.0)+ .unwrap_or(1.0);++ let screen_x = sx + vx * dpr;+ let screen_y = sy + vy * dpr;++ // Step 2 — drive the cursor overlay. Pin above the browser's root+ // HWND so the overlay sits at z+1 of the page, glide, click-pulse.+ // Inlined GA_ROOT lookup mirrors the helper in tools/impl_.rs but+ // keeps page.rs from depending on a private symbol there.+ let hwnd = window_id as u64;+ {+ use windows::Win32::Foundation::HWND;+ use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT};+ let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };+ let pin_wid = if !root.0.is_null() { root.0 as u64 } else { hwnd };+ crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(pin_wid));+ }+ crate::overlay::animate_cursor_to(screen_x, screen_y).await;+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {+ x: screen_x,+ y: screen_y,+ });++ // Step 3 — fire the real click.+ let click_js = format!(+ "(function(){{\+ var el = document.querySelector('{selector_js}');\+ if (!el) throw new Error('element_not_found_on_click:{selector_js}');\+ el.click();\+ return 'clicked:{selector_js}';\+ }})()"+ );+ let _ = self.execute_javascript(pid, window_id, &click_js).await?;++ Ok(ClickElementResult {+ screen_x,+ screen_y,+ viewport_x: vx,+ viewport_y: vy,+ message: format!(+ "✅ click_element('{selector}') at screen ({screen_x:.0},{screen_y:.0}) on pid {pid}."+ ),+ })+ }+}++/// Escape `s` for safe interpolation inside a single-quoted JS string+/// literal. Backslash and apostrophe are the only characters that can+/// break out of `'...'` after the opening quote; newlines also confuse+/// the bookmark wrapper which strips them, so we replace with space.+fn escape_js_string_literal(s: &str) -> String {+ s.replace('\\', "\\\\")+ .replace('\'', "\\'")+ .replace('\r', " ")+ .replace('\n', " ")+}++/// `execute_javascript` returns the raw JS result prefixed with the+/// dispatch route (e.g. `"uia.bookmark_exec: ..."`, `"cdp.runtime.evaluate.user_gesture: ..."`).+/// Strip that prefix so coord parsing sees the bare value.+fn strip_dispatch_prefix(s: &str) -> String {+ for prefix in &["uia.bookmark_exec: ", "cdp.runtime.evaluate.user_gesture: "] {+ if let Some(rest) = s.strip_prefix(prefix) {+ return rest.to_owned();+ }+ }+ s.to_owned()}// ── implementation ────────────────────────────────────────────────────────diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rsindex 1a58886f7c..39cb9fb9e9 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs@@ -115,10 +115,74 @@ unsafe fn walk_tree_unsafe(hwnd: u64, query: Option<&str>) -> UiaTreeResult {}let hwnd_win = windows::Win32::Foundation::HWND(hwnd as *mut _);- let root_elem = match automation.ElementFromHandleBuildCache(hwnd_win, &cache_req) {++ // SAL (LibreOffice / OpenOffice) fast-path: VCL's UIA provider hangs+ // on `BuildUpdatedCache(TreeScope.Subtree)` for transient dialogs+ // (SALSUBFRAME class — Confirmation, Warning, modal Yes/No). Both+ // the atomic `ElementFromHandleBuildCache` and the split+ // `ElementFromHandle + BuildUpdatedCache(Subtree)` paths block+ // indefinitely under the daemon's MTA thread pool (the same code+ // runs fine via CLI in-process, suggesting a COM apartment /+ // concurrent-access interaction with SAL's provider). The 4 s+ // outer timeout fires and the caller gets the structured+ // diagnostic, but that's a wasted 4 s every call.+ //+ // Short-circuit by class detection: if the target's class starts+ // with "SAL", skip the bulk-cache walk entirely and return the+ // same synthetic stub the post-walk SAL-skip would emit, with the+ // three actionable fallback options. Saves the 4 s wait and the+ // ambiguity of "did the walk run? did it find nothing? did it+ // hang?" The caller sees the stub immediately and routes to the+ // appropriate workaround.+ // Class lookup once — used both for the fast-path skip decision+ // and the diagnostic stub. SALSUBFRAME is the empirically-confirmed+ // hang class (modal Confirmation / Warning dialogs). SALFRAME+ // (the document window, the Recovery dialog) walks fine, so don't+ // include it in the skip list — pixel + element_index against+ // Recovery's Discard All button worked end-to-end through the+ // primary path. SALMENU / SALTMPSUBFRAME are transient menu hosts+ // that the user typically doesn't get_window_state on; default to+ // skipping out of caution. If a real use-case emerges, narrow.+ let sal_class: Option<String> = {+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;+ let mut buf = [0u16; 64];+ let n = GetClassNameW(hwnd_win, &mut buf);+ if n > 0 {+ let s = String::from_utf16_lossy(&buf[..n as usize]);+ if s == "SALSUBFRAME" || s == "SALMENU" || s == "SALTMPSUBFRAME" {+ Some(s)+ } else { None }+ } else { None }+ };+ if let Some(class) = sal_class {+ let stub = format!(+ "- Window <{class} — SAL/VCL target, UIA walk skipped>\n\+ (SAL's UIA provider hangs on TreeScope.Subtree BuildUpdatedCache \+ when the daemon is running in MTA; the cua-driver fast-path \+ returns immediately rather than wait 4 s. Use one of: \+ (a) `screenshot(pid, window_id)` + pixel `click(x, y)`; \+ (b) `press_key` with `dispatch:\"foreground\"` (Esc / Enter / Y / N); \+ (c) `get_window_state` with `capture_mode:\"vision\"`.)\n"+ );+ return UiaTreeResult { tree_markdown: stub, nodes: Vec::new() };+ }++ // Two-call sequence (ElementFromHandle + BuildUpdatedCache) instead of+ // the atomic ElementFromHandleBuildCache. Empirically, the single-call+ // batched variant hangs against SAL even in CLI in-process tests;+ // the two-call path works in-process. Kept for non-SAL targets where+ // the difference matters for performance-sensitive providers.+ let uncached = match automation.ElementFromHandle(hwnd_win) {Ok(e) => e,Err(e) => return UiaTreeResult {- tree_markdown: format!("ElementFromHandleBuildCache failed: {e}"),+ tree_markdown: format!("ElementFromHandle failed: {e}"),+ nodes: Vec::new(),+ },+ };+ let root_elem = match uncached.BuildUpdatedCache(&cache_req) {+ Ok(e) => e,+ Err(e) => return UiaTreeResult {+ tree_markdown: format!("BuildUpdatedCache failed: {e}"),nodes: Vec::new(),},};@@ -147,34 +211,86 @@ unsafe fn walk_tree_unsafe(hwnd: u64, query: Option<&str>) -> UiaTreeResult {// erasing it AND leaving the consumed `MAX_TOTAL_ELEMENTS` budget intact// for the fallback (which would then truncate large trees prematurely).if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {- if let Some(target_pid) = pid_from_hwnd(hwnd_win) {- let mut fallback_nodes: Vec<UiaNode> = Vec::new();- let mut fallback_lines: Vec<(usize, String)> = Vec::new();- let mut fallback_counter = 0usize;- let mut fallback_total = 0usize;-+ // Skip the desktop-root walk-by-pid fallback for VCL / SAL+ // targets (LibreOffice, OpenOffice). The fallback does its own+ // `BuildUpdatedCache(TreeScope.Subtree)` per matched top-level+ // window — which is exactly the bulk-cache RPC shape that SAL+ // hangs on. The primary two-call path (ElementFromHandle ++ // BuildUpdatedCache on the dialog's own HWND) ALREADY dodged the+ // hang on that one specific call, but the fallback re-introduces+ // it. Returning the empty tree here lets the outer+ // get_window_state timeout fire its structured diagnostic+ // promptly instead of stalling 4 s on the fallback's hang.+ //+ // The diagnostic tells callers exactly how to drive the SAL+ // dialog without the tree: pixel click via screenshot,+ // capture_mode:"vision", or press_key with dispatch:"foreground"+ // for accelerator-style dismissal. That's enough for the+ // common modal-dismissal case (Yes/No/Esc on a Confirmation),+ // which is what SAL dialogs almost always need.+ let is_sal = {+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;+ let mut buf = [0u16; 64];+ let n = GetClassNameW(hwnd_win, &mut buf);+ n > 0 && {+ let class = String::from_utf16_lossy(&buf[..n as usize]);+ class.starts_with("SAL")+ }+ };+ if !is_sal {+ if let Some(target_pid) = pid_from_hwnd(hwnd_win) {+ let mut fallback_nodes: Vec<UiaNode> = Vec::new();+ let mut fallback_lines: Vec<(usize, String)> = Vec::new();+ let mut fallback_counter = 0usize;+ let mut fallback_total = 0usize;++ tracing::debug!(+ target: "uia",+ "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \+ falling back to GetRootElement + filter ProcessId={target_pid}"+ );+ walk_root_by_pid(+ &automation,+ &cache_req,+ target_pid,+ &mut fallback_nodes,+ &mut fallback_lines,+ &mut fallback_counter,+ &mut fallback_total,+ );++ if fallback_nodes.iter().any(|n| n.element_index.is_some()) {+ nodes = fallback_nodes;+ lines = fallback_lines;+ // counter/total aren't read after this point — they're+ // only used by walk_cached's &mut params for element+ // indexing inside that call.+ }+ }+ } else {tracing::debug!(target: "uia",- "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \- falling back to GetRootElement + filter ProcessId={target_pid}"+ "SAL target hwnd 0x{hwnd:x} returned empty primary tree; \+ skipping walk_root_by_pid fallback (known to re-hang on SAL Subtree fetch). \+ Caller should use press_key/screenshot fallbacks per the get_window_state diagnostic.");- walk_root_by_pid(- &automation,- &cache_req,- target_pid,- &mut fallback_nodes,- &mut fallback_lines,- &mut fallback_counter,- &mut fallback_total,+ // Return a tree_markdown that mirrors the get_window_state+ // timeout diagnostic so callers get the same actionable+ // fallback options even though the walk itself didn't hit+ // the 4 s outer timeout (because we skipped the hang-prone+ // fallback). Without this the caller sees an empty tree+ // and no error, which is less actionable.+ let stub = format!(+ "- Window <SAL class — empty primary UIA tree>\n\+ (SAL providers don't expose modal-dialog children via \+ ElementFromHandle, and the desktop-root fallback walk that \+ would normally find them is known to hang on SAL Subtree \+ BuildUpdatedCache. Use one of: \+ (a) `screenshot(pid, window_id)` + pixel `click(x, y)`; \+ (b) `press_key` with `dispatch:\"foreground\"` (Esc / Enter / Y / N); \+ (c) `get_window_state` with `capture_mode:\"vision\"`.)\n");-- if fallback_nodes.iter().any(|n| n.element_index.is_some()) {- nodes = fallback_nodes;- lines = fallback_lines;- // counter/total aren't read after this point — they're- // only used by walk_cached's &mut params for element- // indexing inside that call.- }+ return UiaTreeResult { tree_markdown: stub, nodes: Vec::new() };}}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/wgc.rs b/libs/cua-driver/rust/crates/platform-windows/src/wgc.rsnew file mode 100644index 0000000000..c8e8f38f06--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-windows/src/wgc.rs@@ -0,0 +1,226 @@+//! Windows.Graphics.Capture (WGC) backend for window screenshots.+//!+//! Used when PrintWindow returns black (UWP / DirectComposition) AND/OR+//! when the target window is occluded by another window. WGC captures+//! the window's own composited frames out of DWM regardless of z-order,+//! which is the only correct way to screenshot UWP apps like Calculator+//! when they're behind the user's terminal.+//!+//! Constraints:+//! - Minimum OS: Windows 10 version 1903 (~April 2019). ~100% of users+//! today; we surface a structured error on older builds.+//! - Cannot capture **minimized** windows. They have no rendered content.+//! For minimized targets, the caller should use UIA via+//! `get_window_state` `capture_mode:"ax"`. We surface this as an+//! error with the recommended fallback.+//! - First call per process pays ~50ms for D3D11 device creation ++//! COM activation factory lookup. Subsequent calls don't cache the+//! device — could be optimized later, but a single-shot screenshot+//! tool isn't the place to fight that.++use anyhow::{bail, Context, Result};+use std::time::Duration;++use windows::{+ Graphics::{+ Capture::{Direct3D11CaptureFramePool, GraphicsCaptureItem},+ DirectX::{Direct3D11::IDirect3DDevice, DirectXPixelFormat},+ },+ Win32::{+ Foundation::HWND,+ Graphics::{+ Direct3D::{D3D_DRIVER_TYPE_HARDWARE, D3D_FEATURE_LEVEL_11_0},+ Direct3D11::{+ D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D,+ D3D11_CPU_ACCESS_READ, D3D11_CREATE_DEVICE_BGRA_SUPPORT,+ D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_READ, D3D11_SDK_VERSION,+ D3D11_TEXTURE2D_DESC, D3D11_USAGE_STAGING,+ },+ Dxgi::IDXGIDevice,+ },+ System::WinRT::{+ Direct3D11::{CreateDirect3D11DeviceFromDXGIDevice, IDirect3DDxgiInterfaceAccess},+ Graphics::Capture::IGraphicsCaptureItemInterop,+ },+ UI::WindowsAndMessaging::IsIconic,+ },+ core::Interface,+};++/// Capture a window via WGC, returning BGRA pixels + (width, height).+/// Caller is responsible for encoding to PNG (or whatever format) via+/// `mcp_server::image_utils::encode_bgra_to_png`.+pub fn screenshot_window_via_wgc(hwnd: u64) -> Result<(Vec<u8>, u32, u32)> {+ unsafe { wgc_capture_impl(HWND(hwnd as *mut _)) }+}++unsafe fn wgc_capture_impl(hwnd: HWND) -> Result<(Vec<u8>, u32, u32)> {+ if IsIconic(hwnd).as_bool() {+ bail!(+ "WGC cannot capture a minimized window (no rendered content). \+ Restore the window first, or use `get_window_state` with \+ `capture_mode:\"ax\"` for UIA tree access — that works on \+ minimized windows."+ );+ }++ // 1. D3D11 device — feature level 11.0 + BGRA support (required by WGC).+ let mut d3d_device: Option<ID3D11Device> = None;+ let mut d3d_context: Option<ID3D11DeviceContext> = None;+ D3D11CreateDevice(+ None,+ D3D_DRIVER_TYPE_HARDWARE,+ windows::Win32::Foundation::HMODULE(std::ptr::null_mut()),+ D3D11_CREATE_DEVICE_BGRA_SUPPORT,+ Some(&[D3D_FEATURE_LEVEL_11_0]),+ D3D11_SDK_VERSION,+ Some(&mut d3d_device),+ None,+ Some(&mut d3d_context),+ )+ .context("D3D11CreateDevice failed (hardware feature-level 11.0 + BGRA)")?;+ let d3d_device = d3d_device.context("D3D11CreateDevice returned a None device")?;+ let d3d_context = d3d_context.context("D3D11CreateDevice returned a None context")?;++ // 2. DXGI device interface for the D3D11 device, then the WinRT+ // IDirect3DDevice wrapper that GraphicsCaptureFramePool needs.+ let dxgi_device: IDXGIDevice = d3d_device.cast().context("ID3D11Device → IDXGIDevice cast")?;+ let inspectable = CreateDirect3D11DeviceFromDXGIDevice(&dxgi_device)+ .context("CreateDirect3D11DeviceFromDXGIDevice failed")?;+ let direct3d_device: IDirect3DDevice = inspectable+ .cast()+ .context("CreateDirect3D11DeviceFromDXGIDevice → IDirect3DDevice cast")?;++ // 3. GraphicsCaptureItem for our target HWND.+ let interop_factory: IGraphicsCaptureItemInterop =+ windows::core::factory::<GraphicsCaptureItem, IGraphicsCaptureItemInterop>()+ .context("IGraphicsCaptureItemInterop activation factory")?;+ let item: GraphicsCaptureItem = interop_factory+ .CreateForWindow(hwnd)+ .context(+ "IGraphicsCaptureItemInterop::CreateForWindow failed — \+ requires Win10 1903+ and the target window must exist",+ )?;+ let item_size = item.Size().context("GraphicsCaptureItem::Size")?;+ if item_size.Width <= 0 || item_size.Height <= 0 {+ bail!(+ "WGC item size is {}x{} — target may be hidden/cloaked. \+ Try restoring the window or use UIA.",+ item_size.Width,+ item_size.Height+ );+ }++ // 4. Frame pool + capture session.+ // `CreateFreeThreaded` (not `Create`) — `Create` requires the+ // calling thread to have a CoreDispatcher/event loop running+ // for FrameArrived to fire, which we don't have inside a+ // daemon's spawn_blocking thread. `CreateFreeThreaded`+ // dispatches FrameArrived on a system thread-pool thread, which+ // is exactly what our mpsc channel handler expects. Without+ // this, FrameArrived silently never fires and our 1500 ms wait+ // always times out.+ //+ // Frame count 2 — WGC docs recommend ≥2 to avoid GPU stalls+ // while the consumer reads the previous frame. We only consume+ // one but the GPU side is happier with breathing room.+ let pool = Direct3D11CaptureFramePool::CreateFreeThreaded(+ &direct3d_device,+ DirectXPixelFormat::B8G8R8A8UIntNormalized,+ 2,+ item_size,+ )+ .context("Direct3D11CaptureFramePool::CreateFreeThreaded")?;+ let session = pool+ .CreateCaptureSession(&item)+ .context("Direct3D11CaptureFramePool::CreateCaptureSession")?;++ // 5. Hide the WGC capture indicator (yellow border) if the build+ // supports it. Available since Win11 (IsBorderRequired property).+ // Best-effort — older Win10 won't have the setter; ignore failure.+ let _ = session.SetIsBorderRequired(false);+ let _ = session.SetIsCursorCaptureEnabled(false);++ // 6. Start capture, then poll TryGetNextFrame until a frame is+ // available or we time out. We tried FrameArrived events first+ // (both `Create` and `CreateFreeThreaded` pool variants) but+ // the event never fired in our daemon's thread environment —+ // likely an STA/COM-apartment mismatch we couldn't isolate.+ // Polling is what windows-capture and OBS-style consumers do+ // for single-shot capture anyway: WGC drops frames in the pool+ // as DWM composes them, and TryGetNextFrame returns Ok(None) /+ // NULL until one's available. 50 ms polling interval keeps the+ // "frame finally arrived" detection latency low without burning+ // CPU on a tight spin.+ session.StartCapture().context("GraphicsCaptureSession::StartCapture")?;++ let deadline = std::time::Instant::now() + Duration::from_millis(1500);+ let mut frame_opt = None;+ while std::time::Instant::now() < deadline {+ match pool.TryGetNextFrame() {+ Ok(f) => { frame_opt = Some(f); break; }+ Err(_) => {+ // TryGetNextFrame returns an error when no frame is+ // available yet. Wait briefly, retry.+ std::thread::sleep(Duration::from_millis(50));+ }+ }+ }+ let frame = frame_opt.ok_or_else(|| anyhow::anyhow!(+ "WGC TryGetNextFrame returned no frame within 1500 ms — \+ DWM may not be compositing this window (cloaked / not actually \+ rendered). Fall through to screen-region BitBlt."+ ))?;++ // 8. Pull the underlying ID3D11Texture2D out of the WinRT frame+ // surface via the IDirect3DDxgiInterfaceAccess shim.+ let surface = frame.Surface().context("frame.Surface")?;+ let access: IDirect3DDxgiInterfaceAccess = surface+ .cast()+ .context("IDirect3DSurface → IDirect3DDxgiInterfaceAccess cast")?;+ let frame_texture: ID3D11Texture2D = access+ .GetInterface()+ .context("IDirect3DDxgiInterfaceAccess::GetInterface(ID3D11Texture2D)")?;++ // 9. Create a staging texture we can map for CPU read, copy frame in.+ let mut desc = D3D11_TEXTURE2D_DESC::default();+ frame_texture.GetDesc(&mut desc);+ desc.Usage = D3D11_USAGE_STAGING;+ desc.BindFlags = 0;+ desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ.0 as u32;+ desc.MiscFlags = 0;+ let mut staging: Option<ID3D11Texture2D> = None;+ d3d_device+ .CreateTexture2D(&desc, None, Some(&mut staging))+ .context("CreateTexture2D(staging)")?;+ let staging = staging.context("CreateTexture2D returned a None texture")?;+ d3d_context.CopyResource(&staging, &frame_texture);++ // 10. Map the staging texture, copy BGRA out row-by-row (RowPitch may+ // be padded — can't just copy `Width * 4 * Height` straight).+ let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();+ d3d_context+ .Map(&staging, 0, D3D11_MAP_READ, 0, Some(&mut mapped))+ .context("ID3D11DeviceContext::Map")?;+ let width = desc.Width as usize;+ let height = desc.Height as usize;+ let stride = mapped.RowPitch as usize;+ let mut bgra = vec![0u8; width * height * 4];+ let src_base = mapped.pData as *const u8;+ for row in 0..height {+ std::ptr::copy_nonoverlapping(+ src_base.add(row * stride),+ bgra.as_mut_ptr().add(row * width * 4),+ width * 4,+ );+ }+ d3d_context.Unmap(&staging, 0);++ // 11. Cleanup: close the session and pool explicitly. RAII would+ // handle this but being explicit makes the lifetimes easier to+ // audit. (No FrameArrived to unhook — we polled instead.)+ let _ = session.Close();+ let _ = pool.Close();++ Ok((bgra, desc.Width, desc.Height))+}diff --git a/libs/cua-driver/scripts/_install-common.psm1 b/libs/cua-driver/scripts/_install-common.psm1index c5ee0b5dbd..d781ff5d08 100644--- a/libs/cua-driver/scripts/_install-common.psm1+++ b/libs/cua-driver/scripts/_install-common.psm1@@ -52,20 +52,95 @@ function Stop-CuaDriverDaemons {return @(Get-Process -Name "cua-driver","cua-driver-uia" -ErrorAction SilentlyContinue)}+# Probe whether `\\.\pipe\cua-driver` is currently accepting connections.+# Distinguishes a *healthy* High-IL daemon (process alive AND pipe+# responsive - install can proceed by deferring to the user to restart+# from elevated PS) from a *stale* daemon (process alive but pipe gone+# - daemon process is hung, no new daemon can bind, MCP is broken until+# someone kills the zombie).+#+# Returns $true iff a real serve daemon is listening. Uses a short+# 200 ms timeout so install latency stays bounded.+function Test-CuaDriverPipeAlive {+ [CmdletBinding()]+ param()+ try {+ $fs = [System.IO.File]::Open(+ '\\.\pipe\cua-driver',+ [System.IO.FileMode]::Open,+ [System.IO.FileAccess]::ReadWrite,+ [System.IO.FileShare]::ReadWrite+ )+ $fs.Close()+ return $true+ } catch {+ return $false+ }+}++# Stop-CuaDriverDaemons + stale-daemon detection in one go. Returns a+# PSCustomObject with both the survivor list AND a `Stale` boolean+# flagging "processes are alive but the named pipe is dead". Used by+# install/install-local to swap the user-facing message from "you need+# to kill these from elevated PS" to the stronger "your daemon is+# WEDGED - kill it now or reboot, MCP is currently broken".+function Stop-CuaDriverDaemonsWithHealth {+ [CmdletBinding()]+ param()+ $survivors = Stop-CuaDriverDaemons+ $pipeAlive = Test-CuaDriverPipeAlive+ # @() forces array context so .Count works even when PowerShell+ # collapsed a 1-element array to a single PSObject (strict-mode+ # otherwise errors on .Count missing on the bare object).+ $count = @($survivors).Count+ $stale = ($count -gt 0 -and -not $pipeAlive)+ return [pscustomobject]@{+ Survivors = $survivors+ PipeAlive = $pipeAlive+ Stale = $stale+ }+}+# Prints a clear hint when Stop-CuaDriverDaemons leaves something# running (almost always a High-IL daemon spawned by the# RunLevel=Highest autostart task - Medium-IL kill returns Access# Denied for those).function Show-CuaDriverDaemonSurvivors {[CmdletBinding()]- param([Parameter(Mandatory = $true)][array]$Survivors)+ # AllowNull + non-mandatory: PowerShell collapses `return @()` from+ # Stop-CuaDriverDaemons to $null at the call site, which would fail+ # a `Mandatory = $true [array]` bind. The body below already treats+ # null and empty array as the no-survivors case, so accept both.+ #+ # `-Stale` toggles the user-facing wording from "the old binary is+ # still running" (mild - install completed, OLD binary in memory+ # until the daemon restarts) to "MCP is currently broken because+ # the daemon process is alive but its named pipe is dead" (loud -+ # nothing will work until the zombie process is killed).+ param(+ [Parameter()][AllowNull()][array]$Survivors,+ [switch]$Stale+ )if (-not $Survivors -or $Survivors.Count -eq 0) { return }$pids = ($Survivors | ForEach-Object { $_.Id }) -join ', '- Write-Host "Note: $($Survivors.Count) cua-driver process(es) still running after best-effort kill (pid: $pids)." -ForegroundColor Yellow- Write-Host " They are likely High-IL (spawned by RunLevel=Highest autostart task)." -ForegroundColor Yellow- Write-Host " From an elevated PowerShell:" -ForegroundColor Yellow- Write-Host " taskkill /IM cua-driver.exe /F" -ForegroundColor Yellow- Write-Host " Or just reboot. Until they exit, the OLD binary keeps running." -ForegroundColor Yellow+ if ($Stale) {+ Write-Host "" -ForegroundColor Red+ Write-Host "ERROR: cua-driver daemon is STALE - process alive (pid: $pids) but \\.\pipe\cua-driver" -ForegroundColor Red+ Write-Host " is not accepting connections. The daemon is wedged; MCP / CLI calls will fail" -ForegroundColor Red+ Write-Host " with 'cannot find the file specified' until this process is killed." -ForegroundColor Red+ Write-Host "" -ForegroundColor Red+ Write-Host " From an ELEVATED PowerShell (right-click PowerShell, 'Run as Administrator'):" -ForegroundColor Yellow+ Write-Host " Stop-Process -Id $pids -Force" -ForegroundColor Yellow+ Write-Host " schtasks /Run /TN 'cua-driver-serve'" -ForegroundColor Yellow+ Write-Host "" -ForegroundColor Yellow+ Write-Host " Reboot also clears it. After the kill+restart, MCP recovers automatically." -ForegroundColor Yellow+ } else {+ Write-Host "Note: $($Survivors.Count) cua-driver process(es) still running after best-effort kill (pid: $pids)." -ForegroundColor Yellow+ Write-Host " They are likely High-IL (spawned by RunLevel=Highest autostart task)." -ForegroundColor Yellow+ Write-Host " From an elevated PowerShell:" -ForegroundColor Yellow+ Write-Host " taskkill /IM cua-driver.exe /F" -ForegroundColor Yellow+ Write-Host " Or just reboot. Until they exit, the OLD binary keeps running." -ForegroundColor Yellow+ }}# Load this module from disk if `$LocalDir/_install-common.psm1` exists@@ -107,7 +182,139 @@ function Import-CuaDriverInstallModule {}}+# Trigger UAC and spawn a brief elevated PowerShell that kills the stale+# (High-IL) cua-driver pids and restarts the scheduled task. Returns+# $true iff the elevated helper actually killed everything AND the+# named pipe is alive again afterward. Returns $false on UAC cancel,+# elevation failure, or post-recovery health-check still failing.+#+# Why a helper rather than self-elevating the whole install: the+# install only needs admin to break the wedge. Binary copy, PATH+# munging, scheduled-task re-registration (which has its own UAC+# prompt anyway) all run fine at Medium IL. Asking for admin just+# for those steps would needlessly broaden the prompt's blast+# radius.+#+# The helper is a single short -Command string so we don't need a+# temp file. UAC users see "Windows PowerShell" as the requesting+# app, with the command in the elevation prompt's "Show details"+# panel.+function Invoke-CuaDriverStaleDaemonKill {+ [CmdletBinding()]+ param(+ [Parameter(Mandatory = $true)][int[]]$Pids+ )+ if (-not $Pids -or $Pids.Count -eq 0) { return $true }+ $pidList = ($Pids -join ',')+ # Build the elevated payload:+ # 1. Stop-Process each pid, force, swallow errors (process already gone is OK).+ # 2. Re-run the scheduled task so a fresh daemon binds the pipe.+ # 3. Brief wait + verify the pipe is reachable.+ # The exit code from the helper signals success (0) / pipe still+ # dead post-restart (3) so the install-side caller can tell the+ # user whether to expect MCP to work.+ $payload = @"+`$ErrorActionPreference = 'Continue'+foreach (`$p in @($pidList)) {+ try { Stop-Process -Id `$p -Force -ErrorAction Stop } catch {}+}+& schtasks.exe /Run /TN 'cua-driver-serve' | Out-Null+Start-Sleep -Milliseconds 1500+try {+ `$fs = [System.IO.File]::Open('\\.\pipe\cua-driver','Open','ReadWrite','ReadWrite')+ `$fs.Close()+ exit 0+} catch {+ exit 3+}+"@+ try {+ $proc = Start-Process `+ -FilePath powershell.exe `+ -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', $payload) `+ -Verb RunAs `+ -PassThru `+ -Wait `+ -WindowStyle Hidden `+ -ErrorAction Stop+ return ($proc.ExitCode -eq 0)+ } catch {+ # UAC cancel surfaces as System.ComponentModel.Win32Exception ("The+ # operation was canceled by the user"). Other failures (no+ # interactive desktop, etc.) take the same path. Return false so+ # the caller falls back to the printed instructions.+ return $false+ }+}++# Convenience wrapper: detect stale state, if stale prompt the user, on+# yes invoke the elevated kill, re-probe, report. Either way return the+# post-recovery state so the install script can decide whether to+# proceed quietly or print the manual-recovery instructions.+#+# Designed to be a drop-in replacement for the+# `Stop-CuaDriverDaemonsWithHealth + Show-CuaDriverDaemonSurvivors`+# pair at the call site.+function Repair-CuaDriverStaleDaemon {+ [CmdletBinding()]+ param(+ # Default-true so install scripts auto-prompt. Pass `-AutoConfirm`+ # to skip the y/n prompt (CI / automated runs) — UAC still prompts+ # for elevation itself.+ [switch]$AutoConfirm+ )+ $result = Stop-CuaDriverDaemonsWithHealth+ if (-not $result.Stale) {+ Show-CuaDriverDaemonSurvivors -Survivors $result.Survivors+ return $result+ }+ # Stale: tell user what we see, offer to elevate.+ $survivorPids = @($result.Survivors | ForEach-Object { $_.Id })+ Write-Host ""+ Write-Host "Detected STALE cua-driver daemon (pid: $($survivorPids -join ', '))." -ForegroundColor Yellow+ Write-Host " Process is alive but \\.\pipe\cua-driver is not accepting connections." -ForegroundColor Yellow+ Write-Host " Fixing this requires admin to terminate the High-IL daemon process." -ForegroundColor Yellow+ Write-Host ""++ $proceed = $AutoConfirm+ if (-not $proceed) {+ # Single-key prompt; default Y on Enter. Y/yes/[Enter] -> elevate,+ # anything else -> skip elevation and fall through to manual-recovery+ # instructions.+ $ans = Read-Host " Trigger UAC prompt to kill the stale daemon now? [Y/n]"+ $proceed = ($ans -eq '' -or $ans -match '^[Yy]')+ }++ if (-not $proceed) {+ Show-CuaDriverDaemonSurvivors -Survivors $result.Survivors -Stale+ return $result+ }++ Write-Host " Triggering UAC prompt (accept to kill the stale daemon)..." -ForegroundColor Cyan+ $ok = Invoke-CuaDriverStaleDaemonKill -Pids $survivorPids+ if ($ok) {+ Write-Host " Stale daemon killed; new cua-driver-serve started; pipe is healthy." -ForegroundColor Green+ return [pscustomobject]@{+ Survivors = @()+ PipeAlive = $true+ Stale = $false+ }+ } else {+ Write-Host " Elevated kill did not complete (UAC cancelled or pipe still dead)." -ForegroundColor Red+ # Re-probe; if the user accepted UAC but the daemon restart+ # failed, the survivors list may have shrunk to just the+ # un-killable processes — re-show with current state.+ $rep = Stop-CuaDriverDaemonsWithHealth+ Show-CuaDriverDaemonSurvivors -Survivors $rep.Survivors -Stale:$rep.Stale+ return $rep+ }+}+Export-ModuleMember -Function `Stop-CuaDriverDaemons, `+ Stop-CuaDriverDaemonsWithHealth, `+ Test-CuaDriverPipeAlive, `Show-CuaDriverDaemonSurvivors, `+ Invoke-CuaDriverStaleDaemonKill, `+ Repair-CuaDriverStaleDaemon, `Import-CuaDriverInstallModulediff --git a/libs/cua-driver/scripts/install-local.ps1 b/libs/cua-driver/scripts/install-local.ps1index 6e4d73499d..2e45fde4a6 100644--- a/libs/cua-driver/scripts/install-local.ps1+++ b/libs/cua-driver/scripts/install-local.ps1@@ -34,9 +34,19 @@[CmdletBinding()]param(- [switch]$AutoStart,+ # Default-on: most users want the daemon to come back at every+ # logon. Opt out with `-AutoStart:$false` (or the dedicated+ # `-NoAutoStart`) when running install-local.ps1 from CI / a+ # container build / a sandbox where you specifically don't want+ # a scheduled task registered.+ [switch]$AutoStart = $true,+ [switch]$NoAutoStart,[switch]$NoPathUpdate)+# `-NoAutoStart` is the explicit opt-out; takes precedence over the+# default-true `-AutoStart` for callers who'd rather read negative+# than `-AutoStart:$false`.+if ($NoAutoStart) { $AutoStart = $false }Set-StrictMode -Version Latest$ErrorActionPreference = "Stop"@@ -198,8 +208,16 @@ if (Test-Path -LiteralPath $DestBinary) {ForEach-Object { try { Remove-Item -LiteralPath $_.FullName -Force -ErrorAction SilentlyContinue } catch {} }Write-Step "killing previous cua-driver processes (best-effort; High-IL needs admin)"- $survivors = Stop-CuaDriverDaemons- Show-CuaDriverDaemonSurvivors -Survivors $survivors+ # Repair- variant does Stop-CuaDriverDaemonsWithHealth + stale+ # detection + UAC self-elevation when wedged: if survivors are+ # present AND the pipe is dead, prompts the user (y/n), and on+ # yes triggers a UAC prompt to spawn a brief elevated+ # PowerShell that kills the High-IL pids and re-runs the+ # scheduled task. On UAC accept + healthy pipe afterward, the+ # install proceeds at Medium IL like nothing happened. On UAC+ # cancel / failure, falls back to the same printed manual+ # recovery instructions the previous flow used.+ $null = Repair-CuaDriverStaleDaemon}Write-Step "staging into $VersionedDir"@@ -335,13 +353,23 @@ if (Test-Path -LiteralPath $HintsTxt) {}# Windows-specific autostart hint (kept inline; per-shell natural location).-if (-not $AutoStart) {+if ($AutoStart) {+ # Default branch: autostart was enabled (either by default or explicitly).+ # Surface the management subcommands so the user knows how to inspect /+ # disable later without digging through Task Scheduler.+ Write-Host ""+ Write-Host "Auto-start: 'cua-driver-serve' is registered at RunLevel=Highest." -ForegroundColor Cyan+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan+ Write-Host ""+} else {+ # Opt-out branch (-NoAutoStart or -AutoStart:`$false`).Write-Host ""- Write-Host "Auto-start at logon (Windows-native equivalent of macOS LaunchAgent):" -ForegroundColor Cyan+ Write-Host "Auto-start at logon (NOT enabled - re-run without -NoAutoStart to register, or:):" -ForegroundColor CyanWrite-Host " cua-driver autostart enable (register Scheduled Task at RunLevel=Highest)" -ForegroundColor CyanWrite-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor CyanWrite-Host " cua-driver autostart status (inspect)" -ForegroundColor CyanWrite-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan- Write-Host " Or re-run install-local.ps1 with -AutoStart for the same result." -ForegroundColor CyanWrite-Host ""}diff --git a/libs/cua-driver/scripts/install.ps1 b/libs/cua-driver/scripts/install.ps1index 176c8fa820..60945bfd95 100644--- a/libs/cua-driver/scripts/install.ps1+++ b/libs/cua-driver/scripts/install.ps1@@ -78,9 +78,19 @@[CmdletBinding()]param([string]$Release = "latest",- [switch]$AutoStart,+ # Default-on: cua-driver-serve is what makes the agent flow work+ # across logon / reboot. Without the scheduled task the user has+ # to remember to run `cua-driver autostart kick` every time, and+ # MCP-style flows go silently in-process. Opt out with+ # `-AutoStart:$false` or `-NoAutoStart` for CI / sandbox installs+ # that specifically don't want a scheduled task registered.+ [switch]$AutoStart = $true,+ [switch]$NoAutoStart,[switch]$NoPathUpdate)+# `-NoAutoStart` is the explicit opt-out and takes precedence over+# the default-true `-AutoStart`.+if ($NoAutoStart) { $AutoStart = $false }Set-StrictMode -Version Latest$ErrorActionPreference = "Stop"@@ -1205,8 +1215,12 @@ else {# Medium-IL kill and get reported via Show-CuaDriverDaemonSurvivors.Write-Host ""Write-Host "Stopping any previous cua-driver processes (best-effort; High-IL needs admin)..." -ForegroundColor Cyan-$survivors = Stop-CuaDriverDaemons-Show-CuaDriverDaemonSurvivors -Survivors $survivors+# Repair- handles the wedged-daemon case: detect stale (process alive+# but pipe dead), prompt the user, and on consent self-elevate via+# UAC to kill the High-IL pids + restart the scheduled task. On UAC+# cancel / failure it falls back to printing the manual recovery+# instructions, same as the previous behavior.+$null = Repair-CuaDriverStaleDaemonif ($AutoStart) {Write-Host ""@@ -1268,12 +1282,18 @@ catch {# Windows-specific autostart hint (kept inline; OS-natural location).Write-Host ""-Write-Host "Auto-start at logon (Windows-native equivalent of macOS LaunchAgent):" -ForegroundColor Cyan-Write-Host " cua-driver autostart enable (Scheduled Task at RunLevel=Highest)" -ForegroundColor Cyan-Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan-Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan-Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan-Write-Host " Or re-run this installer with -AutoStart for the same result." -ForegroundColor Cyan+if ($AutoStart) {+ Write-Host "Auto-start: 'cua-driver-serve' is registered at RunLevel=Highest." -ForegroundColor Cyan+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan+} else {+ Write-Host "Auto-start at logon (NOT enabled - re-run without -NoAutoStart to register, or:):" -ForegroundColor Cyan+ Write-Host " cua-driver autostart enable (Scheduled Task at RunLevel=Highest)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan+}Write-Host ""Write-Host "WARNING -- BETA: cua-driver-rs is a cross-platform Rust port of the Swift" -ForegroundColor YellowWrite-Host " cua-driver. Windows and Linux support is feature-complete; macOS" -ForegroundColor Yellow