Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-rs)(windows): agent-cursor z-order + background-dispatch hardening (#1692)

Francesco Bonacci892edd475233
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index d99fde21b7..35734178ae 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -458,6 +458,7 @@ doesn't-survive-across-sessions caveat.
| `Invalid element_index N for pid X window_id W` | Index is stale or out of range | Re-run `get_window_state` with the same window_id, pick a fresh index from the new tree |
| `window_id W belongs to pid P, not …` | Passed a window_id that's owned by a different process | Use `list_windows({pid: X})` to enumerate this pid's own windows |
| `AX action … failed with code …` / `UIA invoke failed` | Element doesn't support the default action | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center |
+| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this *exact* string for BOTH a permission-prompt denial AND a manual interrupt (Esc / stop) of a running tool — they are indistinguishable from the tool result | Treat as "tool canceled, no result, await the user." Do NOT paraphrase ("you stopped me") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |
Platform-specific errors (TCC dialogs on macOS, Session 0 / UAC
prompts on Windows, AT-SPI bus issues on Linux) live in their
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index 6ef01f2414..695e8cf5f5 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -26,6 +26,66 @@ driver gives you (no cursor warp, no taskbar flash, no window
restore-and-raise) stops mattering — you just shipped a `SendInput`
wrapper with extra steps.
+### Cursor feedback for JS-driven browser actions: prefer `click_element`
+
+The `page` tool gained a fourth action: `click_element`. It takes a CSS
+selector, animates the agent cursor to the element's on-screen center,
+fires a click-pulse, then runs `el.click()`. Prefer this over
+`execute_javascript('document.querySelector(...).click()')` whenever you
+want the user to see what the agent is doing — raw `execute_javascript`
+will perform the click but leaves the cursor frozen.
+
+```json
+{ "action": "click_element", "pid": <int>, "window_id": <int>, "selector": "button.submit" }
+```
+
+Returns the resolved screen coords in `structuredContent` so callers can
+chain subsequent operations against the same point.
+
+### How the contract is enforced per call: the `dispatch` field
+
+Every Windows input tool (`click`, `double_click`, `right_click`,
+`drag`, `scroll`, `press_key`, `hotkey`, `type_text`) accepts an
+optional `dispatch` field. The default is `"background"` — strict
+no-foreground:
+
+| `dispatch` | Behavior on Windows |
+|---|---|
+| `"background"` (DEFAULT) | PostMessage / UIA path only. If the target's input stack is known to silently drop PostMessage for this event kind (Chromium DOM mouse + key-combos, GTK button widgets), the call returns a structured `background_unavailable` error. **No foreground swap, ever.** |
+| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. Required to drive Chromium DOM content or GTK button widgets reliably. Flashes the target visible unless `bring_to_front` was called first. |
+| `"auto"` | Historical heuristic: silently falls back to SendInput on known-problematic targets. Opt-in for callers that prefer the old "things just work, sometimes at the cost of focus" behavior. |
+
+`background_unavailable` error shape:
+
+```json
+{
+ "isError": true,
+ "structuredContent": {
+ "code": "background_unavailable",
+ "target_class": "Chrome_WidgetWin_1",
+ "event_kind": "mouse_click",
+ "suggestion": "Either call bring_to_front then retry with dispatch:\"foreground\", or accept the foreground swap by setting dispatch:\"foreground\" directly."
+ }
+}
+```
+
+The recommended flow when an agent gets that error:
+
+1. `bring_to_front(pid)` — activates the target ONCE (visible flicker).
+2. Subsequent input calls with `dispatch:"foreground"` deliver via
+ SendInput WITHOUT a per-call flash (the SetForegroundWindow swap
+ inside SendInput is a no-op because the target is already frontmost).
+3. When done, leave the target as the user's foreground or call
+ `hotkey({pid: original_fg_pid, keys: ["alt","tab"]})` to put their
+ prior window back. **There is no "restore" tool** — you brought
+ the target forward deliberately; restoring is your responsibility.
+
+The `bring_to_front` tool uses an `AttachThreadInput` trick so the
+foreground swap succeeds even when the daemon isn't at UIAccess
+integrity (the same trick that powers `send_key_synthesized`).
+Returns `{previous_fg_hwnd, now_fg_hwnd, landed_on_target}`.
+
+
Before running any shell command, ask: **"does this raise, activate,
foreground, or steal focus from any app?"** If yes, don't run it.
Every one of the commands below activates the target on Windows and
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
index f27049025b..8c8328d773 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
@@ -903,7 +903,17 @@ pub fn run_call(
}
Err(e) => {
// Daemon became unreachable mid-call — fall through to in-process.
- tracing::debug!("Daemon forwarding failed ({e}), running in-process");
+ // Promoted from `tracing::debug!` to `eprintln!` so callers see
+ // the degradation: in-process execution gets a FRESH ToolState,
+ // which means state-dependent tools (`click`, `type_text`,
+ // `set_value` — anything that reads the element_index cache)
+ // will fail with "Element N not in cache" even when a prior
+ // `get_window_state` populated the daemon's cache, because the
+ // daemon's cache and the in-process cache are different.
+ eprintln!(
+ "[cua-driver] WARNING: daemon proxy to {socket_path} failed ({e}); \
+ running '{tool}' in-process. State-dependent tools may misbehave."
+ );
}
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
index 07d60e55f5..d4cccacb18 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
@@ -109,14 +109,52 @@ impl DaemonResponse {
// ── Client ────────────────────────────────────────────────────────────────────
-/// Probe whether a daemon is listening on `socket_path` by sending a
-/// trivial `list` request and checking for a valid response.
+/// Probe whether a daemon is listening on `socket_path`.
+///
+/// On Windows this uses `WaitNamedPipeW` with a tiny timeout, which checks
+/// whether a pipe instance is available **without consuming one**. The
+/// previous design (sending a `list` request) opened a pipe instance,
+/// then the immediately-following real `send_request` had to wait for the
+/// daemon to spin up its NEXT instance — that race caused real tool calls
+/// (especially state-dependent ones like `click` that need the daemon's
+/// element_index cache) to fall through to the in-process path with a
+/// fresh empty cache. See the conversation around the
+/// "Element 3 not in cache" bug for the diagnosis.
+///
+/// `WaitNamedPipeW(name, 1)`:
+/// - Returns TRUE if an instance is currently available.
+/// - Returns FALSE if not available within 1 ms.
+/// - Does NOT consume the instance — that's reserved for the actual call.
pub fn is_daemon_listening(socket_path: &str) -> bool {
- let req = DaemonRequest { method: "list".into(), name: None, args: None };
- send_request(socket_path, &req)
- .ok()
- .map(|r| r.ok)
- .unwrap_or(false)
+ #[cfg(target_os = "windows")]
+ {
+ use std::os::windows::ffi::OsStrExt;
+
+ // Raw FFI to `WaitNamedPipeW` so this crate doesn't have to pull in
+ // the `windows` crate as a direct dependency just for one probe.
+ // `kernel32.dll` is implicitly linked on Windows targets.
+ #[link(name = "kernel32")]
+ extern "system" {
+ fn WaitNamedPipeW(lpNamedPipeName: *const u16, nTimeOut: u32) -> i32;
+ }
+
+ let wide: Vec<u16> = std::ffi::OsStr::new(socket_path)
+ .encode_wide()
+ .chain(std::iter::once(0))
+ .collect();
+ // NMPWAIT_NOWAIT == 1: "do not wait at all"; returns nonzero only if an
+ // instance is immediately available. We don't want to block here —
+ // this is a one-shot existence probe.
+ unsafe { WaitNamedPipeW(wide.as_ptr(), 1) != 0 }
+ }
+ #[cfg(not(target_os = "windows"))]
+ {
+ let req = DaemonRequest { method: "list".into(), name: None, args: None };
+ send_request(socket_path, &req)
+ .ok()
+ .map(|r| r.ok)
+ .unwrap_or(false)
+ }
}
/// Read the PID stored in `pid_file_path`, if any.
@@ -505,6 +543,69 @@ fn is_self_at_high_il() -> bool {
}
}
+/// Build a SECURITY_ATTRIBUTES that lets any Medium-IL (or higher) process
+/// open the named pipe, even though the daemon itself is running at
+/// High IL (via the autostart Scheduled Task's `RunLevel=Highest`, per
+/// #1630). Without this, the default UIPI rule "no write-up across IL
+/// boundaries" makes the High-IL daemon's pipe unreachable from a normal
+/// Medium-IL user shell — every `cua-driver <tool>` call from the CLI
+/// silently falls through to in-process execution with a fresh, empty
+/// `ToolState`, which breaks the element_index cache invariant
+/// (`get_window_state` → `click(element_index)` stops working because
+/// the two calls land in different ToolState instances).
+///
+/// SDDL: `D:(A;OICI;GA;;;WD)S:(ML;;NW;;;LW)`
+/// - DACL grants `GENERIC_ALL` to `WD` (Everyone).
+/// - SACL sets the mandatory label to LOW with `NW` (NoWriteUp), so
+/// processes at Low-IL and above can write the pipe — i.e. no
+/// IL-based write restriction in practice.
+///
+/// Returns the SECURITY_ATTRIBUTES struct AND the raw security-descriptor
+/// pointer (which the caller must keep alive for the lifetime of the
+/// pipe-server, then free via `LocalFree`).
+#[cfg(target_os = "windows")]
+unsafe fn build_open_pipe_security_attrs()
+ -> Option<(SecurityAttributesRaw, *mut std::ffi::c_void)>
+{
+ #[link(name = "advapi32")]
+ extern "system" {
+ fn ConvertStringSecurityDescriptorToSecurityDescriptorW(
+ string_security_descriptor: *const u16,
+ string_sd_revision: u32,
+ security_descriptor: *mut *mut std::ffi::c_void,
+ security_descriptor_size: *mut u32,
+ ) -> i32;
+ }
+ let sddl: Vec<u16> = "D:(A;OICI;GA;;;WD)S:(ML;;NW;;;LW)\0"
+ .encode_utf16()
+ .collect();
+ let mut sd_ptr: *mut std::ffi::c_void = std::ptr::null_mut();
+ let mut sd_size: u32 = 0;
+ let ok = ConvertStringSecurityDescriptorToSecurityDescriptorW(
+ sddl.as_ptr(),
+ 1, // SDDL_REVISION_1
+ &mut sd_ptr,
+ &mut sd_size,
+ );
+ if ok == 0 || sd_ptr.is_null() {
+ return None;
+ }
+ let attrs = SecurityAttributesRaw {
+ n_length: std::mem::size_of::<SecurityAttributesRaw>() as u32,
+ lp_security_descriptor: sd_ptr,
+ b_inherit_handle: 0,
+ };
+ Some((attrs, sd_ptr))
+}
+
+#[cfg(target_os = "windows")]
+#[repr(C)]
+struct SecurityAttributesRaw {
+ n_length: u32,
+ lp_security_descriptor: *mut std::ffi::c_void,
+ b_inherit_handle: i32,
+}
+
#[cfg(target_os = "windows")]
pub async fn run_serve(
registry: std::sync::Arc<mcp_server::tool::ToolRegistry>,
@@ -516,6 +617,25 @@ pub async fn run_serve(
eprintln!("cua-driver daemon listening on {socket_path}");
+ // Build the cross-IL security descriptor once, reuse on every pipe
+ // instance. If this fails we fall back to the default-ACL `create`
+ // (which is High-IL exclusive when the daemon is at High IL — i.e.
+ // the bug we're trying to fix). Log the failure so it's diagnosable.
+ let security_attrs = unsafe { build_open_pipe_security_attrs() };
+ if security_attrs.is_none() {
+ eprintln!(
+ "cua-driver: failed to build cross-IL SECURITY_ATTRIBUTES; pipe will be \
+ High-IL exclusive. CLI calls from Medium-IL shells will fall through \
+ to in-process and break state-dependent tool sequences."
+ );
+ }
+ // Hold the SD pointer alive for the lifetime of run_serve. We never
+ // free it — the daemon process exit reclaims it.
+ let sec_attrs_ptr: *mut std::ffi::c_void = match &security_attrs {
+ Some((attrs, _sd_ptr)) => attrs as *const _ as *mut _,
+ None => std::ptr::null_mut(),
+ };
+
// Spawn the sibling uiAccess'd worker if it's installed. Best-effort —
// the main daemon still serves requests even if the worker fails to start.
maybe_spawn_uia_worker();
@@ -533,10 +653,23 @@ pub async fn run_serve(
loop {
// Create a new pipe server instance to accept the next client.
- let server = ServerOptions::new()
- .first_pipe_instance(false)
- .create(socket_path)
- .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?;
+ // Use create_with_security_attributes_raw so Medium-IL clients can
+ // open the pipe even though we're at High IL (see comment on
+ // build_open_pipe_security_attrs). Fall back to default ACL if
+ // SD construction failed at startup.
+ let server = if sec_attrs_ptr.is_null() {
+ ServerOptions::new()
+ .first_pipe_instance(false)
+ .create(socket_path)
+ .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?
+ } else {
+ unsafe {
+ ServerOptions::new()
+ .first_pipe_instance(false)
+ .create_with_security_attributes_raw(socket_path, sec_attrs_ptr)
+ .map_err(|e| anyhow::anyhow!("create named pipe {socket_path}: {e}"))?
+ }
+ };
tokio::select! {
result = server.connect() => {
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
index 2669ef248d..d19078fdc4 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
@@ -16,6 +16,7 @@ pub mod shape;
pub mod capture_utils;
pub mod util;
pub mod render_state;
+pub mod z_order;
pub use palette::Palette;
pub use motion::{MotionConfig, Spring};
@@ -23,6 +24,7 @@ pub use bezier::CubicBezier;
pub use path_planner::{PathPlanner, PlannedPath, PathState};
pub use shape::CursorShape;
pub use render_state::{RenderStateCore, FocusRect, render_frame, draw_default_arrow};
+pub use z_order::ZOrderEnforcer;
/// Configuration assembled from CLI arguments and passed to every
/// platform backend when it initialises the overlay window.
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
index 3011569eee..fa6f4dfaae 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
@@ -123,10 +123,15 @@ impl RenderStateCore {
/// speed switches from `min_start_speed` to `min_end_speed` at the
/// midpoint so the cursor decelerates as it approaches the target.
/// Spring overshoot is `0.5` (Windows/Linux convention).
- pub fn tick_motion(&mut self, dt: f64) {
+ ///
+ /// Returns `true` when the planned path just ended (so the caller can
+ /// fire an arrival oneshot to unblock `animate_cursor_to`).
+ pub fn tick_motion(&mut self, dt: f64) -> bool {
let spring_k = self.motion.spring * 400.0;
let spring_c = self.motion.spring * 20.0;
+ let mut fire_arrival = false;
+
if let Some(ref p) = self.path {
let path_frac = (self.dist / p.length.max(1.0)).clamp(0.0, 1.0);
let profile =
@@ -155,6 +160,7 @@ impl RenderStateCore {
self.heading = end_heading;
self.path = None;
self.dist = 0.0;
+ fire_arrival = true;
} else {
let s: PathState = p.sample(self.dist);
self.pos = (s.x, s.y);
@@ -189,6 +195,8 @@ impl RenderStateCore {
}
self.tick_idle(dt);
+
+ fire_arrival
}
/// Advance the animation by `dt` seconds using the hardcoded Swift
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rs
new file mode 100644
index 0000000000..d44eb3d02f
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/z_order.rs
@@ -0,0 +1,47 @@
+//! Cross-platform contract for keeping the agent-cursor overlay at z+1 of
+//! the application under test.
+//!
+//! Each platform crate owns a concrete [`ZOrderEnforcer`] that holds the
+//! overlay's native window handle and is driven by the platform's existing
+//! tick cadence (Win32 `WM_TIMER`, macOS render frame, X11 render thread).
+//! `reassert` is the single named seam where a future OS-event-driven
+//! observer (`SetWinEventHook(EVENT_OBJECT_REORDER)`, AppKit
+//! `NSWindowDidChangeOcclusionState`, X11 `SubstructureNotifyMask`) can
+//! replace the polling implementation without touching the render loops.
+//!
+//! ## Contract
+//!
+//! - `target = Some(wid)` — the overlay must sit **one stacking position
+//! above** the native window identified by `wid` (HWND root on Windows,
+//! `CGWindowID` on macOS, X11 XID on Linux). If a third window is
+//! currently above the target, the overlay must appear **below** it so
+//! the user's foreground keeps rendering on top of the synthetic cursor.
+//!
+//! - `target = None` — the overlay must sit at the top of the **non-topmost**
+//! band. It must never be promoted into the OS "always-on-top" band
+//! (`HWND_TOPMOST`, `NSStatusWindowLevel`, override-redirect raise).
+//!
+//! - `wid` no longer maps to a live window — fall back to the `None`
+//! behaviour for this tick. Do not error; the next tick may have a
+//! new target.
+//!
+//! - Implementations must be cheap and idempotent. `reassert` is called
+//! on every render tick; doing redundant work is acceptable, but
+//! side effects (focus changes, activations, animations) are not.
+
+/// Keeps the overlay window at z+1 of `target`.
+///
+/// See the [module-level docs](self) for the full behavioural contract.
+///
+/// The trait is deliberately not `Send + 'static` — each platform's
+/// enforcer is owned by exactly one thread (the overlay render thread)
+/// and can borrow platform handles (e.g. the X11 `Connection`) that
+/// aren't easily made `'static`. Cross-thread setup is handled by each
+/// platform's own static / `OnceLock`.
+pub trait ZOrderEnforcer {
+ /// Re-pin the overlay so it sits just above `target` (or at the top
+ /// of the non-topmost band when `target` is `None`).
+ ///
+ /// Called from the platform render thread on every tick.
+ fn reassert(&self, target: Option<u64>);
+}
diff --git a/libs/cua-driver/rust/crates/mcp-server/src/page.rs b/libs/cua-driver/rust/crates/mcp-server/src/page.rs
index d064ea3dbb..af86773527 100644
--- a/libs/cua-driver/rust/crates/mcp-server/src/page.rs
+++ b/libs/cua-driver/rust/crates/mcp-server/src/page.rs
@@ -16,6 +16,7 @@
//! get a clear error instead of a panic.
use async_trait::async_trait;
+use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::sync::Arc;
@@ -24,6 +25,23 @@ use crate::{
tool::{Tool, ToolDef},
};
+/// Result of a `click_element` call. The backend is responsible for driving
+/// the cursor overlay before returning; `screen_x` / `screen_y` are reported
+/// back to the caller for diagnostics + downstream chaining.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ClickElementResult {
+ /// Screen-space x of the element center, after cursor animation landed.
+ pub screen_x: f64,
+ /// Screen-space y of the element center.
+ pub screen_y: f64,
+ /// Viewport-space x (page-relative, before scroll offset compensation).
+ pub viewport_x: f64,
+ /// Viewport-space y.
+ pub viewport_y: f64,
+ /// Free-form message for the agent (e.g. "Clicked button.submit").
+ pub message: String,
+}
+
/// Platform-specific backend for the `page` tool.
///
/// All methods take `(pid, window_id)` for consistency across actions.
@@ -69,6 +87,29 @@ pub trait PageBackend: Send + Sync {
(Chrome/Brave/Edge use Apple Events as the JS backend there)."
)
}
+
+ /// Click an element identified by a CSS selector, animating the agent
+ /// cursor to the element's on-screen center before firing the click.
+ ///
+ /// The backend is responsible for: (1) running JS to derive the
+ /// element's screen coordinates; (2) driving the cursor overlay
+ /// (`pin_above` + `animate_cursor_to` + `ClickPulse`); (3) firing
+ /// `element.click()` via JS; (4) returning the coords for diagnostics.
+ ///
+ /// Default impl returns an actionable "not implemented" error so
+ /// non-Windows backends keep compiling until they're wired up.
+ async fn click_element(
+ &self,
+ _pid: i32,
+ _window_id: u32,
+ _selector: &str,
+ ) -> anyhow::Result<ClickElementResult> {
+ anyhow::bail!(
+ "click_element is not yet implemented on this platform's page \
+ backend. Use execute_javascript with `el.click()` directly \
+ (no visible cursor animation)."
+ )
+ }
}
/// The cross-platform `page` MCP tool. Holds a backend trait object the
@@ -95,6 +136,10 @@ fn def() -> &'static ToolDef {
- execute_javascript: Run JS and return the result.\n\
- get_text: Extract visible text from the page.\n\
- query_dom: Find elements matching a CSS selector.\n\
+ - click_element: Click a CSS-selected element AND animate the agent cursor \
+ to its on-screen center first (so the user sees what the agent is doing). \
+ Prefer over `execute_javascript('el.click()')` whenever you want visible \
+ cursor feedback.\n\
- enable_javascript_apple_events: macOS-only — patch the browser's \
Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user \
confirmation and a browser restart).".into(),
@@ -111,9 +156,13 @@ fn def() -> &'static ToolDef {
"window_id": { "type": "integer", "description": "Target window ID from list_windows." },
"action": {
"type": "string",
- "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],
+ "enum": ["execute_javascript", "get_text", "query_dom", "click_element", "enable_javascript_apple_events"],
"description": "Action to perform."
},
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for click_element (e.g. 'button.submit', '#login a')."
+ },
"javascript": {
"type": "string",
"description": "JavaScript to execute. Required for execute_javascript."
@@ -242,6 +291,36 @@ impl Tool for PageTool {
Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),
},
+ "click_element" => {
+ // Trim before emptiness check so whitespace-only selectors
+ // fail fast at the input boundary rather than blowing up
+ // inside the backend's JS payload (`querySelector(" ")`
+ // returns null and the error there is much less specific).
+ let selector = match args
+ .get("selector")
+ .and_then(|v| v.as_str())
+ .map(str::trim)
+ {
+ Some(s) if !s.is_empty() => s.to_owned(),
+ _ => return ToolResult::error(
+ "Missing required parameter: selector (CSS selector for click_element)"
+ ),
+ };
+ match self.backend.click_element(pid, window_id, &selector).await {
+ Ok(res) => {
+ let structured = serde_json::json!({
+ "screen_x": res.screen_x,
+ "screen_y": res.screen_y,
+ "viewport_x": res.viewport_x,
+ "viewport_y": res.viewport_y,
+ "selector": selector,
+ });
+ ToolResult::text(res.message).with_structured(structured)
+ }
+ Err(e) => ToolResult::error(format!("click_element failed: {e}")),
+ }
+ }
+
"query_dom" => {
let selector = args
.get("css_selector")
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
index 2d5ab55d40..7ffd0d16d3 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
@@ -21,6 +21,8 @@ use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
use cursor_overlay::{CursorConfig, OverlayCommand, RenderStateCore};
+#[cfg(target_os = "linux")]
+use cursor_overlay::ZOrderEnforcer;
// ── Global channel ────────────────────────────────────────────────────────
@@ -205,6 +207,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCo
let frame_dur = Duration::from_millis(16);
let mut last_tick = Instant::now();
let mut last_ztick = Instant::now();
+ let z_enforcer = X11ZOrderEnforcer { conn: &conn, win };
loop {
let now = Instant::now();
@@ -240,23 +243,16 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCo
paint_x11(&conn, win, scr_w, scr_h, depth, visual_id, &pm);
}
- // Z-order maintenance every 80ms.
+ // Z-order maintenance every 80ms — delegate to the cross-platform
+ // ZOrderEnforcer so the contract for "z+1 of the application under
+ // test" is documented once in `cursor_overlay::z_order`.
if last_ztick.elapsed() >= Duration::from_millis(80) {
last_ztick = Instant::now();
let pinned_wid = {
let guard = RENDER.lock().unwrap();
guard.as_ref().and_then(|rs| rs.core.pinned_wid)
};
- let aux = if let Some(target_xid) = pinned_wid {
- // Place overlay just above the pinned X11 window.
- ConfigureWindowAux::new()
- .sibling(target_xid as u32)
- .stack_mode(StackMode::ABOVE)
- } else {
- ConfigureWindowAux::new().stack_mode(StackMode::ABOVE)
- };
- conn.configure_window(win, &aux).ok();
- conn.flush().ok();
+ z_enforcer.reassert(pinned_wid);
}
// Drain any X events (needed to avoid blocking).
@@ -269,6 +265,57 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCo
}
}
+// ── Z-order enforcer (Linux impl of cursor_overlay::ZOrderEnforcer) ──────
+
+/// X11 implementation of [`cursor_overlay::ZOrderEnforcer`].
+///
+/// Borrows the X11 connection and overlay window id; lives only inside
+/// `run_overlay_thread` (the X11 connection is not `'static`). Called
+/// every 80 ms from the render loop.
+#[cfg(target_os = "linux")]
+struct X11ZOrderEnforcer<'a, C: x11rb::connection::Connection> {
+ conn: &'a C,
+ win: u32,
+}
+
+#[cfg(target_os = "linux")]
+impl<'a, C: x11rb::connection::Connection> ZOrderEnforcer for X11ZOrderEnforcer<'a, C> {
+ fn reassert(&self, target: Option<u64>) {
+ use x11rb::protocol::xproto::*;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+
+ // Per the ZOrderEnforcer trait contract, a stale `target` (window
+ // gone) should fall back to the `None` behavior — top of the
+ // normal stack, no sibling. Using a stale XID as a `sibling` here
+ // triggers BadWindow on every tick of the overlay-enforcer loop
+ // (~125 Hz), spamming the X server and silently skipping the
+ // intended z-reassertion. Probe liveness via get_window_attributes
+ // before committing to the sibling path.
+ let target_live = target.and_then(|xid| {
+ self.conn
+ .get_window_attributes(xid as u32)
+ .ok()
+ .and_then(|c| c.reply().ok())
+ .map(|_| xid)
+ });
+ let aux = if let Some(target_xid) = target_live {
+ // Place overlay just above the pinned X11 window.
+ ConfigureWindowAux::new()
+ .sibling(target_xid as u32)
+ .stack_mode(StackMode::ABOVE)
+ } else {
+ // No pin (or stale target XID) → raise to the top of the
+ // normal stack. (X11 has no OS-level "always-on-top" band like
+ // Windows / NSStatusWindowLevel, so a plain ABOVE here cannot
+ // accidentally float over a focused foreground app the way
+ // HWND_TOPMOST would on Windows.)
+ ConfigureWindowAux::new().stack_mode(StackMode::ABOVE)
+ };
+ self.conn.configure_window(self.win, &aux).ok();
+ self.conn.flush().ok();
+ }
+}
+
#[cfg(target_os = "linux")]
fn find_argb_visual(
conn: &impl x11rb::connection::Connection,
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 718ebf7d55..f7be2ef946 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -2051,6 +2051,59 @@ impl Tool for KillAppTool {
}
}
+// ── bring_to_front (Linux stub) ──────────────────────────────────────────────
+
+pub struct BringToFrontTool;
+
+static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for BringToFrontTool {
+ fn def(&self) -> &ToolDef {
+ BTF_DEF.get_or_init(|| ToolDef {
+ name: "bring_to_front".into(),
+ description:
+ "Activate a window so subsequent input tools land on it. **Windows-only \
+ today:** on Linux this stub returns an error; the X11/Wayland equivalents \
+ (`wmctrl -a`, `xdotool windowactivate`) aren't wired up because the Linux \
+ input tools deliver via AT-SPI / X11 input injection which already reaches \
+ backgrounded windows without needing activation."
+ .into(),
+ input_schema: serde_json::json!({
+ "type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer"},
+ "window_id":{"type":"integer"}
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ ToolResult::error(
+ "bring_to_front is Windows-only today. On Linux the input tools deliver via \
+ AT-SPI / X11 input injection which already reaches backgrounded windows. If \
+ you need explicit activation for your own UX reasons, shell out to \
+ `wmctrl -a` or `xdotool windowactivate` from outside cua-driver."
+ .to_string(),
+ )
+ .with_structured(serde_json::json!({
+ "code": "bring_to_front_unsupported_on_platform",
+ "platform": "linux",
+ // Machine-readable remediation hint — mirrors the macOS
+ // bring_to_front stub's structured `suggestion` field so
+ // cross-platform clients can dispatch on a uniform key.
+ "suggestion":
+ "Linux input tools (click / type_text / press_key / hotkey) already \
+ reach backgrounded windows via AT-SPI / X11 input injection — \
+ there is no equivalent need to bring a window to the foreground. \
+ If you need explicit window activation for UX reasons, shell out \
+ to `wmctrl -a <title>` or `xdotool windowactivate <wid>` from \
+ outside cua-driver.",
+ }))
+ }
+}
+
// ── registry ─────────────────────────────────────────────────────────────────
pub fn build_registry(compat: bool) -> ToolRegistry {
@@ -2061,6 +2114,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry {
r.register(Box::new(GetWindowStateTool { state: state.clone() }));
r.register(Box::new(LaunchAppTool));
r.register(Box::new(KillAppTool));
+ r.register(Box::new(BringToFrontTool));
r.register(Box::new(ClickTool { state: state.clone() }));
r.register(Box::new(DoubleClickTool { state: state.clone() }));
r.register(Box::new(RightClickTool { state: state.clone() }));
@@ -2070,11 +2124,10 @@ pub fn build_registry(compat: bool) -> ToolRegistry {
r.register(Box::new(HotkeyTool));
r.register(Box::new(SetValueTool));
r.register(Box::new(ScrollTool));
- if compat {
- r.register(Box::new(ScreenshotCompatTool { state: state.clone() }));
- } else {
- r.register(Box::new(ScreenshotTool { state: state.clone() }));
- }
+ // `screenshot` removed - see the matching comment in
+ // platform-windows/src/tools/impl_.rs::build_registry. Canonical
+ // screenshot path is `get_window_state` with `capture_mode:"vision"`.
+ let _ = compat;
r.register(Box::new(GetScreenSizeTool));
r.register(Box::new(GetCursorPositionTool));
r.register(Box::new(MoveCursorTool { state: state.clone() }));
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
index febfc2d6aa..626567da11 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
@@ -34,7 +34,7 @@ use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
use cursor_overlay::{
- CursorConfig, FocusRect, MotionConfig, OverlayCommand, RenderStateCore,
+ CursorConfig, FocusRect, MotionConfig, OverlayCommand, RenderStateCore, ZOrderEnforcer,
};
// ── Arrival-signal channel ────────────────────────────────────────────────
@@ -358,14 +358,15 @@ fn render_loop(
}
// Repin: immediately on target change, then defensive every ~1 s.
+ // Delegate to the cross-platform ZOrderEnforcer so the contract for
+ // "z+1 of the application under test" is documented once in
+ // `cursor_overlay::z_order`.
repin_frames += 1;
let pin_changed = pinned_wid != last_pinned;
last_pinned = pinned_wid;
- if let Some(wid) = pinned_wid {
- if pin_changed || repin_frames >= 60 {
- dispatch_pin_above(win_ptr, wid);
- repin_frames = 0;
- }
+ if pinned_wid.is_some() && (pin_changed || repin_frames >= 60) {
+ MacZOrderEnforcer { win_ptr }.reassert(pinned_wid);
+ repin_frames = 0;
} else if repin_frames >= 60 {
repin_frames = 0;
}
@@ -487,6 +488,33 @@ fn dispatch_pin_above(win_ptr: usize, target_wid: u64) {
}
}
+// ── Z-order enforcer (macOS impl of cursor_overlay::ZOrderEnforcer) ──────
+
+/// macOS implementation of [`cursor_overlay::ZOrderEnforcer`].
+///
+/// Holds the NSWindow pointer as a `usize` and dispatches the
+/// `orderWindow:relativeTo:` call to the main queue (AppKit must run on
+/// the main thread).
+///
+/// `target = None` is treated as a no-op here rather than raising to the
+/// front: macOS creates the overlay at `NSNormalWindowLevel` and never
+/// promotes it into an "always-on-top" level the way Windows does with
+/// `HWND_TOPMOST`, so there is no topmost band to escape. Letting the
+/// overlay stay where the user's activations have left it keeps it out
+/// of the way when no agent action is in flight.
+struct MacZOrderEnforcer {
+ win_ptr: usize,
+}
+
+impl ZOrderEnforcer for MacZOrderEnforcer {
+ fn reassert(&self, target: Option<u64>) {
+ if let Some(wid) = target {
+ dispatch_pin_above(self.win_ptr, wid);
+ }
+ // target = None → no-op; see struct doc comment.
+ }
+}
+
/// Create a `CGImage` from a `tiny_skia::Pixmap` (premultiplied RGBA).
/// Returns a `+1` retained pointer that the caller must release.
fn pixmap_to_cgimage(pixmap: &tiny_skia::Pixmap) -> Option<usize> {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rs
new file mode 100644
index 0000000000..5cba0a8b26
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/bring_to_front.rs
@@ -0,0 +1,71 @@
+//! macOS `bring_to_front` stub.
+//!
+//! `bring_to_front` exists to let agents pay a one-shot foreground swap on
+//! Windows before driving Chromium-content or GTK-button targets through
+//! the SendInput-required code path. On macOS, equivalent functionality is
+//! offered by `NSRunningApplication.activate` / Cocoa's window-ordering
+//! APIs, which the macOS input tools never need internally — every
+//! `CGEvent.postToPid` based dispatch reaches the target without
+//! foreground manipulation. So the macOS stub returns an actionable
+//! "Windows-only" error.
+
+use async_trait::async_trait;
+use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef}};
+use serde_json::Value;
+
+pub struct BringToFrontTool;
+
+static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn def() -> &'static ToolDef {
+ DEF.get_or_init(|| ToolDef {
+ name: "bring_to_front".into(),
+ description:
+ "Activate a window so subsequent input tools with `dispatch:\"foreground\"` \
+ land on it without a per-call SetForegroundWindow flash. **Windows-only:** \
+ on macOS this tool returns an error pointing to the platform-native \
+ `NSRunningApplication.activate` (which the macOS input tools don't need \
+ because CGEvent.postToPid reaches backgrounded windows). On Linux this \
+ tool also stubs out; use `wmctrl -a` or `xdotool windowactivate` if \
+ you need explicit activation."
+ .into(),
+ input_schema: serde_json::json!({
+ "type": "object",
+ "required": ["pid"],
+ "properties": {
+ "pid": { "type": "integer" },
+ "window_id": { "type": "integer" }
+ },
+ "additionalProperties": false,
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+}
+
+#[async_trait]
+impl Tool for BringToFrontTool {
+ fn def(&self) -> &ToolDef { def() }
+
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ ToolResult::error(
+ "bring_to_front is Windows-only. On macOS, input tools do not need \
+ explicit foreground activation: every CGEvent.postToPid dispatch \
+ reaches backgrounded windows. If you need to bring the app \
+ visually forward for your own UX reasons, use \
+ NSRunningApplication.activate via your own AppleScript / shell \
+ call — cua-driver intentionally does not expose that path because \
+ it violates the no-foreground contract."
+ .to_string(),
+ )
+ .with_structured(serde_json::json!({
+ "code": "bring_to_front_unsupported_on_platform",
+ "platform": "macos",
+ "suggestion":
+ "macOS input tools deliver via CGEvent.postToPid which is \
+ inherently background-safe; no bring_to_front equivalent needed.",
+ }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index ea19ad7ddc..0b2df500f4 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -5,6 +5,7 @@ mod list_windows;
mod get_window_state;
mod launch_app;
mod kill_app;
+mod bring_to_front;
mod click;
mod double_click;
mod right_click;
@@ -211,6 +212,7 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) {
registry.register(Box::new(get_window_state::GetWindowStateTool::new(state.clone())));
registry.register(Box::new(launch_app::LaunchAppTool));
registry.register(Box::new(kill_app::KillAppTool));
+ registry.register(Box::new(bring_to_front::BringToFrontTool));
registry.register(Box::new(click::ClickTool::new(state.clone())));
registry.register(Box::new(double_click::DoubleClickTool::new(state.clone())));
registry.register(Box::new(right_click::RightClickTool::new(state.clone())));
@@ -220,11 +222,10 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) {
registry.register(Box::new(hotkey::HotkeyTool::new(state.clone())));
registry.register(Box::new(set_value::SetValueTool::new(state.clone())));
registry.register(Box::new(scroll::ScrollTool::new(state.clone())));
- if compat {
- registry.register(Box::new(screenshot_compat::ClaudeCodeCompatScreenshotTool::new(state.clone())));
- } else {
- registry.register(Box::new(screenshot::ScreenshotTool { state: state.clone() }));
- }
+ // `screenshot` removed - see the matching comment in
+ // platform-windows/src/tools/impl_.rs::build_registry. Canonical
+ // screenshot path is `get_window_state` with `capture_mode:"vision"`.
+ let _ = compat;
registry.register(Box::new(get_screen_size::GetScreenSizeTool));
registry.register(Box::new(get_cursor_position::GetCursorPositionTool));
registry.register(Box::new(move_cursor::MoveCursorTool::new(state.clone())));
diff --git a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml b/libs/cua-driver/rust/crates/platform-windows/Cargo.toml
index 2d79fba608..85b9a94091 100644
--- a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml
+++ b/libs/cua-driver/rust/crates/platform-windows/Cargo.toml
@@ -51,5 +51,19 @@ windows = { version = "0.58", features = [
"Management_Deployment",
"ApplicationModel",
"Foundation_Collections",
+ # Windows.Graphics.Capture (WGC) - the supported way to capture
+ # UWP / DirectComposition windows even when they're occluded by
+ # another window. PrintWindow returns black for these surfaces,
+ # and the screen-region BitBlt fallback returns the occluding
+ # window's pixels. Min OS: Win10 1903 (~100% of users today).
+ "Graphics_Capture",
+ "Graphics_DirectX",
+ "Graphics_DirectX_Direct3D11",
+ "Win32_Graphics_Direct3D11",
+ "Win32_Graphics_Direct3D",
+ "Win32_Graphics_Dxgi",
+ "Win32_Graphics_Dxgi_Common",
+ "Win32_System_WinRT_Direct3D11",
+ "Win32_System_WinRT_Graphics_Capture",
] }
base64 = { workspace = true }
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/capture.rs b/libs/cua-driver/rust/crates/platform-windows/src/capture.rs
index cc3ffffa67..c689cec765 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/capture.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/capture.rs
@@ -68,6 +68,62 @@ fn is_mostly_black_bgra(bgra: &[u8]) -> bool {
sampled > 0 && (black * 200) >= (sampled * 199)
}
+/// Probe whether `target` is currently obscured by some other window.
+/// Samples `WindowFromPoint` at the target's center + four corners (insetting
+/// 2 px from each corner so we don't catch invisible 1-px frame slop). If any
+/// sample returns a window whose root ancestor isn't `target` itself, we
+/// consider the target occluded. Used by the screen-region BitBlt path to
+/// warn callers that the bitmap they're about to receive does NOT actually
+/// show the target's pixels — it shows whatever's covering it. Without this
+/// signal, the BitBlt result is silently misleading (see the regression
+/// captured during the autonomous-test journal: a backgrounded Notepad
+/// screenshot returned LibreOffice's pixels).
+unsafe fn target_is_obscured(target: HWND) -> bool {
+ use windows::Win32::Foundation::{POINT, RECT};
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetAncestor, GetWindowRect, WindowFromPoint, GA_ROOT,
+ };
+
+ if target.0.is_null() {
+ return false;
+ }
+ let mut rect = RECT::default();
+ if GetWindowRect(target, &mut rect).is_err() {
+ return false;
+ }
+ let w = rect.right - rect.left;
+ let h = rect.bottom - rect.top;
+ if w <= 4 || h <= 4 {
+ return false;
+ }
+ // 5 sample points: 4 corners (inset 2 px) + center.
+ let pts: [(i32, i32); 5] = [
+ (rect.left + 2, rect.top + 2),
+ (rect.right - 3, rect.top + 2),
+ (rect.left + 2, rect.bottom - 3),
+ (rect.right - 3, rect.bottom - 3),
+ ((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2),
+ ];
+ let target_root = GetAncestor(target, GA_ROOT);
+ let mut covered_samples = 0;
+ for (x, y) in &pts {
+ let p = POINT { x: *x, y: *y };
+ let owner = WindowFromPoint(p);
+ if owner.0.is_null() {
+ continue;
+ }
+ let owner_root = GetAncestor(owner, GA_ROOT);
+ if owner_root != target_root {
+ covered_samples += 1;
+ }
+ }
+ // Threshold of 2 of 5: a single corner being "covered" can be a layered
+ // overlay (e.g. the cua-driver agent cursor) that's not actually opaque
+ // over the content; we don't want to false-positive on that. Two or more
+ // sample points missing means real content is being covered.
+ covered_samples >= 2
+}
+
/// Fallback capture path: BitBlt the desktop DC over the rectangle covered
/// by `hwnd`'s on-screen bounds. Works for UWP / WinUI3 / DirectComposition
/// surfaces that PrintWindow can't reach, as long as the window is on-screen
@@ -128,7 +184,18 @@ unsafe fn screenshot_via_screen_region(hwnd: HWND) -> Result<(Vec<u8>, i32, i32)
/// Capture a window by HWND, returning raw PNG bytes.
pub fn screenshot_window_bytes(hwnd: u64) -> Result<Vec<u8>> {
- unsafe { screenshot_window_bytes_unsafe(hwnd) }
+ screenshot_window_bytes_with_occlusion(hwnd).map(|(b, _)| b)
+}
+
+/// Capture a window by HWND. Returns `(png_bytes, occluded_flag)` — the
+/// boolean is `true` when the capture had to fall through to the
+/// screen-region BitBlt path AND another window was visibly covering the
+/// target at sample time. In that case the bitmap reflects the COVERING
+/// window's pixels, not the target's. Callers that surface the image to a
+/// user / LLM should attach an explicit warning. See `target_is_obscured`
+/// for the sampling heuristic.
+pub fn screenshot_window_bytes_with_occlusion(hwnd: u64) -> Result<(Vec<u8>, bool)> {
+ unsafe { screenshot_window_bytes_with_occlusion_unsafe(hwnd) }
}
/// Capture a window by HWND, returning (base64_png, width, height).
@@ -143,24 +210,45 @@ pub fn screenshot_window(hwnd: u64) -> Result<(String, u32, u32)> {
Ok((BASE64.encode(&png_bytes), w, h))
}
-unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {
+unsafe fn screenshot_window_bytes_with_occlusion_unsafe(hwnd: u64) -> Result<(Vec<u8>, bool)> {
use windows::Win32::UI::WindowsAndMessaging::GetClientRect;
use windows::Win32::Foundation::RECT;
let hwnd_raw = hwnd;
let hwnd = HWND(hwnd as *mut _);
- // CUA-542 routing: for known XAML / WinUI3 / UWP targets, skip
- // PrintWindow entirely and go straight to the screen-region BitBlt
- // path. PrintWindow either returns all-black bitmaps for those
- // surfaces or — as observed for backgrounded Calculator — a tiny
- // clipped capture of the window's collapsed client rect. The
- // screen-region path reads from the live desktop DC, which has the
- // real composited image.
+ // CUA-542 routing: for known XAML / WinUI3 / UWP targets, the
+ // PrintWindow GDI path returns black (DirectComposition isn't in
+ // the GDI back-buffer). We try Windows.Graphics.Capture (WGC)
+ // first — that's the only API that returns the target's actual
+ // composited pixels even when it's occluded by another window
+ // (the Calculator-behind-terminal regression captured in the
+ // autonomous-test journal). On older Windows / GPU stalls / cloaked
+ // windows WGC may fail; fall back to the screen-region BitBlt
+ // path, then to PrintWindow as a last resort.
if crate::input::is_xaml_host_hwnd(hwnd_raw) {
+ match crate::wgc::screenshot_window_via_wgc(hwnd_raw) {
+ Ok((pixels, w, h)) => {
+ return Ok((
+ mcp_server::image_utils::encode_bgra_to_png(&pixels, w, h)?,
+ false, // WGC reads target's own pixels — never occluded by definition
+ ));
+ }
+ Err(e) => {
+ tracing::warn!(
+ target: "cua-driver",
+ "screenshot: WGC failed for XAML target hwnd 0x{hwnd_raw:x}: {e}; \
+ falling back to screen-region BitBlt (may be occluded)."
+ );
+ }
+ }
+ let occluded = target_is_obscured(hwnd);
match screenshot_via_screen_region(hwnd) {
Ok((pixels, w, h)) => {
- return mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32);
+ return Ok((
+ mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)?,
+ occluded,
+ ));
}
Err(e) => {
// Screen-region failed — fall through and try PrintWindow as a
@@ -218,12 +306,35 @@ unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {
if ok == 0 { bail!("GetDIBits returned 0"); }
// CUA-542: detect the all-black bitmap PrintWindow returns for
- // DirectComposition-backed UWP / WinUI3 surfaces and retry via
- // screen-region BitBlt. See module docs for the rationale.
+ // DirectComposition-backed UWP / WinUI3 surfaces. Recovery order:
+ // 1. WGC (occlusion-immune; works for UWP).
+ // 2. Screen-region BitBlt (works when target is on-screen and
+ // not covered).
+ // The WGC-first ordering covers backgrounded UWP targets the
+ // screen-region path mishandles (returns covering window's pixels).
if is_mostly_black_bgra(&pixels) {
+ match crate::wgc::screenshot_window_via_wgc(hwnd_raw) {
+ Ok((alt_pixels, w, h)) => {
+ return Ok((
+ mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, w, h)?,
+ false,
+ ));
+ }
+ Err(e) => {
+ tracing::warn!(
+ target: "cua-driver",
+ "screenshot: PrintWindow returned black AND WGC failed for hwnd 0x{hwnd_raw:x}: {e}; \
+ trying screen-region BitBlt next."
+ );
+ }
+ }
+ let occluded = target_is_obscured(hwnd);
match screenshot_via_screen_region(hwnd) {
Ok((alt_pixels, alt_w, alt_h)) => {
- return mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, alt_w as u32, alt_h as u32);
+ return Ok((
+ mcp_server::image_utils::encode_bgra_to_png(&alt_pixels, alt_w as u32, alt_h as u32)?,
+ occluded,
+ ));
}
Err(e) => {
// Screen-region path failed too — return the (black) PrintWindow
@@ -244,7 +355,10 @@ unsafe fn screenshot_window_bytes_unsafe(hwnd: u64) -> Result<Vec<u8>> {
// uncompressed-PNG path that produced ~5x larger output. The
// `image` crate's encoder is already a workspace dep so the
// smaller output is free).
- mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)
+ // PrintWindow itself reads from the target's own DC, so the bitmap
+ // we return here is the target's pixels even when occluded — no
+ // occluded warning needed on this path.
+ Ok((mcp_server::image_utils::encode_bgra_to_png(&pixels, w as u32, h as u32)?, false))
}
// NOTE: previously this module carried a hand-rolled
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs
new file mode 100644
index 0000000000..c1c0364e51
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs
@@ -0,0 +1,258 @@
+//! Shared dispatch-mode logic for cua-driver-rs Windows input tools.
+//!
+//! Each input tool accepts an optional `dispatch` field with three modes:
+//!
+//! - `background` (DEFAULT) — PostMessage / UIA path only. If
+//! `would_be_silently_dropped(target, event_kind)` returns true, the tool
+//! returns a structured `background_unavailable` error so the caller can
+//! `bring_to_front` then retry with `dispatch:"foreground"`. This is the
+//! default because cua-driver's value proposition is that input never
+//! steals foreground — making that the silent guarantee, not a hidden
+//! trade-off, is worth a breaking change to existing callers that
+//! previously got auto-fallback to SendInput.
+//!
+//! - `foreground` — SendInput with a brief `SetForegroundWindow(target)`
+//! swap, restoring the prior foreground after the events flush. Brief
+//! flash unless the target was already foreground. Required to drive
+//! Chromium-content or GTK-button widget targets reliably.
+//!
+//! - `auto` — current heuristics: each tool's existing PostMessage /
+//! SendInput routing is preserved bit-for-bit. Opt-in for callers that
+//! want the historical silent-fallback behavior. Not the default.
+//!
+//! The matrix of "which (window class, event kind) pairs are silently
+//! dropped by PostMessage" lives here as `would_be_silently_dropped`.
+//! Tools call into this helper instead of inlining the class checks.
+
+use serde_json::Value;
+
+/// Family of synthetic input events. Used together with the target HWND
+/// to decide whether PostMessage will be silently dropped.
+#[derive(Copy, Clone, Debug)]
+pub enum EventKind {
+ /// `WM_LBUTTONDOWN/UP`, `WM_RBUTTONDOWN/UP`, `WM_MBUTTONDOWN/UP`.
+ MouseClick,
+ /// `WM_MOUSEMOVE` (also covers drag intermediate steps).
+ MouseMove,
+ /// `WM_VSCROLL` / `WM_HSCROLL`.
+ MouseScroll,
+ /// `WM_KEYDOWN/UP` with no modifiers — plain key tap.
+ Keystroke,
+ /// `WM_KEYDOWN/UP` with modifiers — accelerator candidate (Ctrl+S etc).
+ KeyCombo,
+ /// `WM_CHAR` text input.
+ TextInput,
+}
+
+impl EventKind {
+ pub fn name(self) -> &'static str {
+ match self {
+ Self::MouseClick => "mouse_click",
+ Self::MouseMove => "mouse_move",
+ Self::MouseScroll => "mouse_scroll",
+ Self::Keystroke => "keystroke",
+ Self::KeyCombo => "key_combo",
+ Self::TextInput => "text_input",
+ }
+ }
+}
+
+/// Client-controlled dispatch policy.
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
+pub enum DispatchMode {
+ /// Current heuristics. Each tool keeps its existing routing.
+ Auto,
+ /// PostMessage / UIA only. Error if dispatch would be silently dropped.
+ Background,
+ /// SendInput with brief foreground swap.
+ Foreground,
+}
+
+impl DispatchMode {
+ /// Parse from a tool's `dispatch` JSON arg.
+ ///
+ /// Missing / unknown values resolve to `Background` — cua-driver's
+ /// default is now strict no-foreground. Callers that want the
+ /// historical silent-fallback heuristics must explicitly pass
+ /// `dispatch:"auto"`.
+ pub fn from_args(args: &Value) -> Self {
+ match args.get("dispatch").and_then(|v| v.as_str()) {
+ Some("auto") => Self::Auto,
+ Some("foreground") => Self::Foreground,
+ _ => Self::Background,
+ }
+ }
+}
+
+/// JSON-schema fragment for the `dispatch` field. Include this in every
+/// input tool's `input_schema.properties.dispatch`.
+pub fn dispatch_schema() -> Value {
+ serde_json::json!({
+ "type": "string",
+ "enum": ["background", "foreground", "auto"],
+ "default": "background",
+ "description":
+ "Dispatch mode. 'background' (default) refuses to swap \
+ foreground; returns a structured background_unavailable error \
+ if PostMessage would be silently dropped for this event kind \
+ on this target. 'foreground' explicitly accepts a brief \
+ SetForegroundWindow swap (SendInput path) — required to drive \
+ Chromium-content or GTK-button widget targets reliably. \
+ 'auto' uses cua-driver's internal heuristics (silent fallback \
+ to SendInput on known-problematic targets); opt-in for \
+ callers that prefer the historical behavior."
+ })
+}
+
+/// Returns true if PostMessage on `hwnd` for this `event_kind` is empirically
+/// known to be silently dropped by the target's input stack.
+///
+/// Combines the existing `is_chromium_target_window` detector with new GTK
+/// (`gdkWindowToplevel` / `gdkSurfaceToplevel`) detection.
+pub fn would_be_silently_dropped(hwnd: u64, kind: EventKind) -> bool {
+ use EventKind::*;
+ if crate::input::is_chromium_target_window(hwnd) {
+ // Chromium's input thread architecture requires SendInput-queue
+ // origin for mouse + key-combo events (#1623). Plain keystrokes and
+ // text input via WM_CHAR still work because they go through
+ // Chromium's IME path, which DOES consume Win32 messages.
+ return matches!(kind, MouseClick | MouseMove | MouseScroll | KeyCombo);
+ }
+ if is_gtk_target_window(hwnd) {
+ // Conservative flag for GTK: button widgets ignore PostMessage
+ // clicks, drawing-area widgets accept them. We cannot distinguish
+ // at the HWND level (single HWND for the whole GTK window) so we
+ // flag mouse clicks broadly. Canvas-style drag works in practice;
+ // caller can still opt to retry with dispatch:"background" on the
+ // drag path if the click error wasn't actually load-bearing.
+ return matches!(kind, MouseClick);
+ }
+ if is_vcl_target_window(hwnd) {
+ // VCL (LibreOffice / OpenOffice family) routes accelerator keys
+ // through TranslateAccelerator, which reads `GetKeyState`.
+ // PostMessage(WM_KEYDOWN) delivers the key but does NOT update
+ // GetKeyState, so single-key dialog accelerators (Y/N for
+ // confirmations, Esc / Enter / Tab) and modifier combos (Ctrl+S,
+ // Alt+F4) silently fail. Plain WM_CHAR text input through the
+ // document widgets still works (verified end-to-end against
+ // Writer's main editing area). Flag the keystroke-class events
+ // so dispatch:"background" surfaces a structured error instead
+ // of pretending to succeed.
+ return matches!(kind, Keystroke | KeyCombo);
+ }
+ false
+}
+
+/// Detect LibreOffice / OpenOffice (VCL framework) windows.
+///
+/// VCL on Windows registers window classes with a `SAL` prefix (StarOffice's
+/// "Service Abstraction Layer"): `SALFRAME` (top-level), `SALSUBFRAME`
+/// (dialogs / popups), `SALOBJECT` (embedded), `SALMENU` (menu host),
+/// `SALTMPSUBFRAME` (transient). The `SAL` prefix is unique enough to
+/// match by leading characters.
+pub fn is_vcl_target_window(hwnd: u64) -> bool {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
+ if hwnd == 0 {
+ return false;
+ }
+ let mut buf = [0u16; 64];
+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };
+ if n <= 0 {
+ return false;
+ }
+ let class = String::from_utf16_lossy(&buf[..n as usize]);
+ class.starts_with("SAL")
+}
+
+/// Detect GTK/GDK top-level windows.
+///
+/// GTK 3 on Windows uses class `gdkWindowToplevel`; GTK 4 uses
+/// `gdkSurfaceToplevel`. Match by prefix in case future GTK versions
+/// append a suffix the way Chromium does (`Chrome_WidgetWin_0` /
+/// `Chrome_WidgetWin_1`).
+pub fn is_gtk_target_window(hwnd: u64) -> bool {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
+ if hwnd == 0 {
+ return false;
+ }
+ let mut buf = [0u16; 64];
+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };
+ if n <= 0 {
+ return false;
+ }
+ let class = String::from_utf16_lossy(&buf[..n as usize]);
+ class.starts_with("gdkWindow") || class.starts_with("gdkSurface")
+}
+
+/// Read a window's class name into an owned String. Best-effort; returns
+/// `"<unknown>"` on any failure so it can drop straight into a diagnostic.
+pub fn read_class_name(hwnd: u64) -> String {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
+ if hwnd == 0 {
+ return "<unknown>".into();
+ }
+ let mut buf = [0u16; 256];
+ let n = unsafe { GetClassNameW(HWND(hwnd as *mut _), &mut buf) };
+ if n <= 0 {
+ return "<unknown>".into();
+ }
+ String::from_utf16_lossy(&buf[..n as usize])
+}
+
+/// Build the structured `background_unavailable` error returned when
+/// `dispatch:"background"` would silently drop.
+pub fn background_unavailable_error(
+ hwnd: u64,
+ kind: EventKind,
+) -> mcp_server::protocol::ToolResult {
+ let class = read_class_name(hwnd);
+ let text = format!(
+ "Background dispatch is not available for target window class \
+ '{class}' on this event kind ({}). Either call bring_to_front \
+ then retry with dispatch:\"foreground\", or accept the foreground \
+ swap directly by setting dispatch:\"foreground\".",
+ kind.name()
+ );
+ mcp_server::protocol::ToolResult::error(text)
+ .with_structured(serde_json::json!({
+ "code": "background_unavailable",
+ "target_class": class,
+ "event_kind": kind.name(),
+ "suggestion":
+ "Either call bring_to_front then retry with dispatch:\"foreground\", \
+ or accept the foreground swap by setting dispatch:\"foreground\" directly.",
+ }))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn dispatch_mode_parses_known_values() {
+ let j = |s: &str| serde_json::json!({"dispatch": s});
+ assert_eq!(DispatchMode::from_args(&j("auto")), DispatchMode::Auto);
+ assert_eq!(DispatchMode::from_args(&j("background")), DispatchMode::Background);
+ assert_eq!(DispatchMode::from_args(&j("foreground")), DispatchMode::Foreground);
+ }
+
+ #[test]
+ fn dispatch_mode_defaults_to_background() {
+ // Missing field, garbage value, null — all resolve to Background.
+ // This is the cua-driver no-foreground-by-default contract;
+ // breaking change to callers who used to rely on silent SendInput
+ // fallback against Chromium / GTK-button targets.
+ assert_eq!(DispatchMode::from_args(&serde_json::json!({})), DispatchMode::Background);
+ assert_eq!(
+ DispatchMode::from_args(&serde_json::json!({"dispatch": "garbage"})),
+ DispatchMode::Background
+ );
+ assert_eq!(
+ DispatchMode::from_args(&serde_json::json!({"dispatch": null})),
+ DispatchMode::Background
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
index 416dfd409e..ad7aee477a 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
@@ -176,35 +176,87 @@ pub fn post_char(hwnd: u64, ch: char) -> Result<()> {
Ok(())
}
+/// Post `\n` (or `\r`, or `\r\n`) as a real Enter keystroke pair —
+/// WM_KEYDOWN/WM_KEYUP(VK_RETURN) — to the focused child `h`.
+///
+/// Why a keystroke and not WM_CHAR(0x0A) or WM_CHAR(0x0D): standard Win32
+/// edit controls accept `WM_CHAR(0x0D)` (carriage return) and insert a
+/// newline; richer controls (LibreOffice's VCL edit, modern WPF/WinForms,
+/// Scintilla) ignore `0x0A` outright and only sometimes accept `0x0D`.
+/// A real `WM_KEYDOWN(VK_RETURN)` is the universally-honoured "Enter
+/// pressed" signal — it produces a paragraph break in word processors,
+/// activates the default button in dialogs, and submits forms in browsers.
+/// The previous `WM_CHAR(0x0A)`-as-newline path silently joined every
+/// line of multi-line `type_text` input into a single run (visible in the
+/// LibreOffice Writer ode-to-a-background-cursor screenshot).
+unsafe fn post_enter_keystroke(h: HWND) -> Result<()> {
+ let vk = windows::Win32::UI::Input::KeyboardAndMouse::VK_RETURN;
+ let scan = MapVirtualKeyW(vk.0 as u32, MAPVK_VK_TO_VSC);
+ let lp_down = 1u32 | (scan << 16);
+ let lp_up = lp_down | (1u32 << 30) | (1u32 << 31);
+ PostMessageW(h, WM_KEYDOWN, WPARAM(vk.0 as usize), LPARAM(lp_down as isize))?;
+ // Hold time between KEYDOWN and KEYUP — matches `post_key`'s pattern and
+ // gives the target's message loop time to TranslateMessage the KEYDOWN
+ // (which synthesizes WM_CHAR(0x0D)) and DispatchMessage the paragraph
+ // break before the KEYUP arrives. Without this gap, LibreOffice Writer
+ // intermittently produced a double paragraph break AND dropped the
+ // next character — visible in the "ABC\nDEF\nGHI" repro as "ABC / DEF /
+ // (gap) / HI".
+ sleep(Duration::from_millis(KEY_DELAY_MS));
+ PostMessageW(h, WM_KEYUP, WPARAM(vk.0 as usize), LPARAM(lp_up as isize))?;
+ Ok(())
+}
+
/// Post all characters in a string as WM_CHAR messages with inter-key delay.
+///
+/// Line breaks (`\n`, `\r`, or `\r\n`) are emitted as Enter keystrokes
+/// (see [`post_enter_keystroke`]) instead of literal `WM_CHAR(0x0A/0x0D)`,
+/// which most rich-text Win32 controls drop.
pub fn post_type_text(hwnd: u64, text: &str) -> Result<()> {
- if let Some(msg) = crate::input::post_message_blocked_by_uipi(hwnd) {
- anyhow::bail!(msg);
- }
- let h_parent = HWND(hwnd as *mut _);
- // Resolve focused-child once at entry — re-querying per-character would
- // race with text-insertion side effects on the focus.
- let h = focused_descendant(h_parent).unwrap_or(h_parent);
- for ch in text.chars() {
- let code = ch as u32 as usize;
- unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }
- sleep(Duration::from_millis(KEY_DELAY_MS));
- }
- Ok(())
+ post_type_text_with_delay(hwnd, text, 0)
}
/// Post all characters in `text` as WM_CHAR messages with a configurable
/// inter-character delay (on top of the baseline KEY_DELAY_MS gap).
+///
+/// Line breaks (`\n`, `\r`, or `\r\n`) are emitted as Enter keystrokes
+/// (see [`post_enter_keystroke`]) — see the LibreOffice screenshot in
+/// the PR description for the bug this fixes.
pub fn post_type_text_with_delay(hwnd: u64, text: &str, inter_char_ms: u64) -> Result<()> {
if let Some(msg) = crate::input::post_message_blocked_by_uipi(hwnd) {
anyhow::bail!(msg);
}
let h_parent = HWND(hwnd as *mut _);
+ // Resolve focused-child once at entry — re-querying per-character would
+ // race with text-insertion side effects on the focus.
let h = focused_descendant(h_parent).unwrap_or(h_parent);
+
+ let mut prev_was_cr = false;
for ch in text.chars() {
- let code = ch as u32 as usize;
- unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }
- sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms));
+ match ch {
+ // `\r\n` (Windows-style line ending) → single Enter. The `\r`
+ // does the Enter; the following `\n` is consumed silently.
+ '\n' if prev_was_cr => {
+ prev_was_cr = false;
+ }
+ '\n' | '\r' => {
+ unsafe { post_enter_keystroke(h)?; }
+ prev_was_cr = ch == '\r';
+ // Extra settle after Enter — paragraph creation in rich
+ // editors (VCL Writer, Scintilla, RichEdit) is heavier than
+ // a single-character insert, so the baseline KEY_DELAY_MS +
+ // inter_char_ms is not enough on slower hosts. The extra
+ // 20 ms covers the queue drain without noticeably slowing
+ // multi-line typing.
+ sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms + 20));
+ }
+ _ => {
+ prev_was_cr = false;
+ let code = ch as u32 as usize;
+ unsafe { PostMessageW(h, WM_CHAR, WPARAM(code), LPARAM(1))?; }
+ sleep(Duration::from_millis(KEY_DELAY_MS + inter_char_ms));
+ }
+ }
}
Ok(())
}
@@ -317,6 +369,31 @@ pub fn send_key_synthesized(hwnd: u64, key: &str, modifiers: &[&str]) -> Result<
// Brief settle so the foreground swap is processed before we send.
sleep(Duration::from_millis(8));
+ // Verify the swap actually happened. `SetForegroundWindow` returns a
+ // BOOL but Windows treats foreground-lock violations as a silent
+ // no-op in most cases (the call returns success-ish, the foreground
+ // doesn't change). The reliable check is observing the foreground
+ // after the settle. If we didn't get it, abort BEFORE SendInput —
+ // otherwise the events land on whatever app actually held foreground
+ // (typically the terminal hosting this process), which causes spooky
+ // side effects like Alt+Enter toggling the terminal into fullscreen.
+ let actual_fg = GetForegroundWindow();
+ if actual_fg != target {
+ // Don't restore — prev_fg is presumably still foreground anyway.
+ bail!(
+ "Foreground swap to target HWND {:?} was rejected by Windows \
+ (actual foreground is HWND {:?}). This daemon is not at \
+ UIAccess integrity, so SetForegroundWindow is subject to the \
+ foreground-lock and the swap silently fails. Without the \
+ swap, SendInput would land on the wrong window. Fix: install \
+ / spawn the cua-driver-uia worker (UIAccess-manifested PE) \
+ and route hotkey calls through it. Until then, the calling \
+ app must already be foreground for dispatch:\"foreground\" \
+ to be safe.",
+ target.0, actual_fg.0
+ );
+ }
+
let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);
if sent as usize != events.len() {
// SendInput returns the number of events successfully inserted.
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
index 114ac9bd5f..bf7b1c3f49 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mod.rs
@@ -11,6 +11,7 @@
pub mod mouse;
pub mod keyboard;
+pub mod dispatch;
pub use mouse::{is_chromium_target_window, post_click, post_click_screen, send_click_synthesized};
pub use keyboard::{
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
index 317d6a415b..d42f29430c 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
@@ -323,6 +323,25 @@ pub fn send_click_synthesized(
let _ = SetForegroundWindow(target);
sleep(Duration::from_millis(8));
+ // Verify the swap actually happened. Without UIAccess the call returns
+ // success but the foreground stays put — and SendInput then lands on
+ // whatever window IS foreground (typically the terminal hosting this
+ // process). Abort before injecting so we don't click random apps.
+ let actual_fg = GetForegroundWindow();
+ if actual_fg != target {
+ bail!(
+ "Foreground swap to target HWND {:?} was rejected by Windows \
+ (actual foreground is HWND {:?}). This daemon is not at \
+ UIAccess integrity, so SetForegroundWindow is subject to the \
+ foreground-lock and the swap silently fails. Without the \
+ swap, SendInput-based mouse events would land on the wrong \
+ window. Fix: install / spawn the cua-driver-uia worker \
+ (UIAccess-manifested PE) and route Chromium coord clicks \
+ through it.",
+ target.0, actual_fg.0
+ );
+ }
+
// Move the cursor first so the OS hover state matches before the click.
// `SetCursorPos` is the visible cursor move; the MOUSEEVENTF_MOVE input
// ensures Chromium's input filter sees a coordinated move event.
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/lib.rs b/libs/cua-driver/rust/crates/platform-windows/src/lib.rs
index 663b801b80..e56c19188e 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/lib.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/lib.rs
@@ -27,6 +27,9 @@ pub mod input;
#[cfg(target_os = "windows")]
pub mod capture;
+#[cfg(target_os = "windows")]
+pub mod wgc;
+
#[cfg(target_os = "windows")]
pub mod launch_uwp;
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
index d1729abab0..4650c5107b 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
@@ -22,7 +22,7 @@ use std::sync::{Mutex, OnceLock};
use std::time::Instant;
use cursor_overlay::{
- CursorConfig, MotionConfig, OverlayCommand, RenderStateCore,
+ CursorConfig, MotionConfig, OverlayCommand, RenderStateCore, ZOrderEnforcer,
};
// ── Global channel ────────────────────────────────────────────────────────
@@ -31,6 +31,14 @@ static CMD_TX: OnceLock<std::sync::mpsc::SyncSender<OverlayCommand>> = OnceLock:
static CMD_RX_CELL: Mutex<Option<std::sync::mpsc::Receiver<OverlayCommand>>> = Mutex::new(None);
static RENDER: Mutex<Option<RenderState>> = Mutex::new(None);
+// ── Arrival-signal channel ────────────────────────────────────────────────
+//
+// `animate_cursor_to` installs a oneshot sender here, the render thread's
+// `WM_TIMER` handler fires it the tick the planned path ends. Mirrors
+// macOS so click handlers can `.await` until the cursor visually lands
+// before dispatching the actual UIA / PostMessage action.
+static ARRIVAL_TX: Mutex<Option<tokio::sync::oneshot::Sender<()>>> = Mutex::new(None);
+
pub fn init(cfg: CursorConfig) {
let (tx, rx) = std::sync::mpsc::sync_channel(4096);
let _ = CMD_TX.set(tx);
@@ -84,6 +92,52 @@ pub fn is_at_initial_position() -> bool {
.unwrap_or(true)
}
+/// Animate the overlay cursor to `(x, y)` and suspend until the planned
+/// path completes (the spring-settle phase that follows is allowed to keep
+/// running — we only wait for the visible glide to land).
+///
+/// Mirrors `platform_macos::cursor::overlay::animate_cursor_to`. Returns
+/// immediately (no animation, no wait) when:
+/// - the overlay is disabled, or
+/// - the cursor is still at the off-screen sentinel `(-200, -200)` — in
+/// that case the caller should rely on `ClickPulse` to snap the cursor.
+pub async fn animate_cursor_to(x: f64, y: f64) {
+ let should_animate = {
+ let guard = RENDER.lock().unwrap();
+ match guard.as_ref() {
+ Some(rs) if rs.core.cfg.enabled && rs.core.visible && rs.core.pos.0 > -50.0 => true,
+ _ => false,
+ }
+ };
+ if !should_animate {
+ return;
+ }
+
+ // Install the oneshot sender BEFORE issuing MoveTo, so the render
+ // thread's arrival-fire can never lose a race against an immediate
+ // path-end (e.g. zero-length glide).
+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();
+ {
+ let mut guard = ARRIVAL_TX.lock().unwrap();
+ // A previous in-flight animation gets superseded by this one —
+ // unblock its waiter so it doesn't hang forever.
+ if let Some(old_tx) = guard.take() {
+ let _ = old_tx.send(());
+ }
+ *guard = Some(tx);
+ }
+
+ send_command(OverlayCommand::MoveTo {
+ x,
+ y,
+ // Arrive pointing upper-left (45°) — same convention as macOS /
+ // Swift reference (`endAngleDegrees: 45`).
+ end_heading_radians: std::f64::consts::FRAC_PI_4,
+ });
+
+ let _ = rx.await;
+}
+
/// Spin up the overlay on a dedicated thread (STA for Win32 message loop).
/// This is a non-blocking call — the overlay runs on its own thread.
pub fn run_on_thread() {
@@ -144,8 +198,11 @@ impl RenderState {
}
}
- fn tick(&mut self, dt: f64) {
- self.core.tick_motion(dt);
+ /// Advance the motion state by `dt`. Returns `true` the tick the
+ /// planned path completes, so the WM_TIMER handler can fire the
+ /// arrival oneshot that unblocks `animate_cursor_to`.
+ fn tick(&mut self, dt: f64) -> bool {
+ self.core.tick_motion(dt)
}
fn apply_command(&mut self, cmd: OverlayCommand) {
@@ -244,6 +301,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayCo
OVERLAY_HWND.store(hwnd.0 as isize, std::sync::atomic::Ordering::Relaxed);
*CMD_RX_WIN.lock().unwrap() = Some(rx);
LAST_ZTICK.store(0, std::sync::atomic::Ordering::Relaxed);
+ let _ = Z_ORDER.set(WinZOrderEnforcer { hwnd_isize: hwnd.0 as isize });
// Standard Win32 message loop.
let mut msg = MSG::default();
@@ -267,6 +325,7 @@ static OVERLAY_HWND: std::sync::atomic::AtomicIsize =
static CMD_RX_WIN: Mutex<Option<std::sync::mpsc::Receiver<OverlayCommand>>> = Mutex::new(None);
static LAST_ZTICK: std::sync::atomic::AtomicU64 =
std::sync::atomic::AtomicU64::new(0);
+static Z_ORDER: OnceLock<WinZOrderEnforcer> = OnceLock::new();
// ── Window procedure ──────────────────────────────────────────────────────
@@ -290,7 +349,7 @@ unsafe extern "system" fn wnd_proc(
// Drain commands and tick animation. Measure real dt from last
// tick — Windows timer resolution defaults to 15ms so the
// hardcoded 8ms was running the animation at half speed.
- let pixmap = {
+ let (pixmap, fire_arrival) = {
let mut guard = RENDER.lock().unwrap();
if let Some(rs) = guard.as_mut() {
// Drain the channel.
@@ -304,17 +363,17 @@ unsafe extern "system" fn wnd_proc(
let now = std::time::Instant::now();
let dt = now.duration_since(rs.last_tick).as_secs_f64().clamp(0.0, 0.05);
rs.last_tick = now;
- rs.tick(dt);
- Some(cursor_overlay::render_frame(
+ let arrived = rs.tick(dt);
+ (Some(cursor_overlay::render_frame(
&rs.core,
rs.virt_w.max(1) as u32,
rs.virt_h.max(1) as u32,
rs.virt_x as f64,
rs.virt_y as f64,
None, // focus-rect is macOS-only
- ))
+ )), arrived)
} else {
- None
+ (None, false)
}
};
@@ -322,11 +381,28 @@ unsafe extern "system" fn wnd_proc(
update_layered_window(hwnd, &pm);
}
- // Z-order maintenance every 80ms.
+ // Fire arrival oneshot the tick the path just ended — unblocks
+ // any `animate_cursor_to(...).await` so the click action only
+ // dispatches once the cursor has visually landed.
+ if fire_arrival {
+ if let Ok(mut guard) = ARRIVAL_TX.lock() {
+ if let Some(tx) = guard.take() {
+ let _ = tx.send(());
+ }
+ }
+ }
+
+ // Z-order maintenance every 80ms — delegate to the cross-platform
+ // ZOrderEnforcer so the contract for "z+1 of the application under
+ // test" is documented once in `cursor_overlay::z_order`.
let last = LAST_ZTICK.load(std::sync::atomic::Ordering::Relaxed);
if now_ms.wrapping_sub(last) >= 80 {
LAST_ZTICK.store(now_ms, std::sync::atomic::Ordering::Relaxed);
- reapply_z_order(hwnd);
+ let pinned_wid = RENDER.lock().ok()
+ .and_then(|g| g.as_ref().and_then(|rs| rs.core.pinned_wid));
+ if let Some(enforcer) = Z_ORDER.get() {
+ enforcer.reassert(pinned_wid);
+ }
}
LRESULT(0)
@@ -434,54 +510,81 @@ unsafe fn update_layered_window(
ReleaseDC(None, hdc_screen);
}
-#[cfg(target_os = "windows")]
-unsafe fn reapply_z_order(hwnd: windows::Win32::Foundation::HWND) {
- use windows::Win32::Foundation::HWND;
- use windows::Win32::UI::WindowsAndMessaging::*;
+// ── Z-order enforcer (Windows impl of cursor_overlay::ZOrderEnforcer) ────
- // Read pinned_wid from render state (RENDER lock released before this is called).
- let pinned_wid = {
- let guard = RENDER.lock().unwrap();
- guard.as_ref().and_then(|rs| rs.core.pinned_wid)
- };
-
- let pinned_target = pinned_wid.and_then(|wid| {
- let h = HWND(wid as *mut _);
- if IsWindow(h).as_bool() { Some(h) } else { None }
- });
-
- // The overlay must sit JUST above the pinned target window so the
- // user's foreground app (a different non-topmost window — say their
- // terminal) renders on top of the overlay. Two pitfalls:
- //
- // 1. HWND_TOPMOST was the previous fallback. Once Windows promotes
- // a window into the topmost band (sets WS_EX_TOPMOST), a later
- // SetWindowPos with a normal target_hwnd does NOT drop it back
- // out — the overlay stays above EVERYTHING non-topmost (incl.
- // the user's foreground). That was the symptom in #1688-style
- // reports: overlay sticks visible over a terminal even when
- // the pinned window (Calculator) is behind it.
- // 2. To drop out of the topmost band, we need an explicit
- // SetWindowPos(hwnd, HWND_NOTOPMOST, …) call. Then we can
- // issue a second SetWindowPos with the real target_hwnd so
- // the overlay lands ABOVE the pin but BELOW any window
- // currently above the pin (the user's foreground).
- //
- // Fallback when there's no live pin: HWND_TOP — top of non-topmost
- // band, NOT the topmost band. So a "no pin" overlay still respects
- // the user's foreground stack.
- let _ = SetWindowPos(
- hwnd,
- HWND_NOTOPMOST,
- 0, 0, 0, 0,
- SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOOWNERZORDER,
- );
+/// Win32 implementation of [`cursor_overlay::ZOrderEnforcer`].
+///
+/// Stores the overlay HWND as an `isize` (HWND is `*mut c_void` and not
+/// `Send`/`Sync`) and rehydrates it inside `reassert`. Driven by the
+/// `WM_TIMER` branch in `wnd_proc` on the overlay STA thread.
+struct WinZOrderEnforcer {
+ hwnd_isize: isize,
+}
- let insert_after = pinned_target.unwrap_or(HWND_TOP);
- let _ = SetWindowPos(
- hwnd,
- insert_after,
- 0, 0, 0, 0,
- SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_SHOWWINDOW | SWP_NOOWNERZORDER,
- );
+impl ZOrderEnforcer for WinZOrderEnforcer {
+ fn reassert(&self, target: Option<u64>) {
+ #[cfg(target_os = "windows")]
+ unsafe {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::*;
+
+ let hwnd = HWND(self.hwnd_isize as *mut _);
+
+ let pinned_target = target.and_then(|wid| {
+ let h = HWND(wid as *mut _);
+ if IsWindow(h).as_bool() { Some(h) } else { None }
+ });
+
+ // The overlay must sit JUST above the pinned target window so the
+ // user's foreground app (a different non-topmost window — say their
+ // terminal) renders on top of the overlay. Three Win32 pitfalls:
+ //
+ // 1. HWND_TOPMOST was the previous fallback. Once Windows promotes
+ // a window into the topmost band (sets WS_EX_TOPMOST), a later
+ // SetWindowPos with a normal target_hwnd does NOT drop it back
+ // out — the overlay stays above EVERYTHING non-topmost (incl.
+ // the user's foreground). That was the symptom in #1688-style
+ // reports.
+ // 2. To drop out of the topmost band, we need an explicit
+ // SetWindowPos(hwnd, HWND_NOTOPMOST, …) call before the real
+ // z-order placement.
+ // 3. `SetWindowPos(hwnd, target_hwnd, …)` does NOT mean "put hwnd
+ // above target_hwnd" — Win32 semantics are "insert hwnd
+ // *after* target_hwnd in z-order" (i.e. one slot BELOW
+ // target). To land overlay *above* target we have to insert
+ // it after target's previous sibling instead — the window
+ // currently just above target. `GetWindow(target, GW_HWNDPREV)`
+ // returns that (or null when target is already the topmost
+ // non-topmost window, in which case HWND_TOP raises overlay
+ // to the top and pushes target one slot down). This is the
+ // pitfall macOS / Linux dodge by virtue of their explicit
+ // `orderWindow:above:` / `StackMode::ABOVE` APIs.
+ //
+ // Fallback when there's no live pin: HWND_TOP — top of non-topmost
+ // band, NOT the topmost band. So a "no pin" overlay still respects
+ // the user's foreground stack.
+ let _ = SetWindowPos(
+ hwnd,
+ HWND_NOTOPMOST,
+ 0, 0, 0, 0,
+ SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOOWNERZORDER,
+ );
+
+ let insert_after = match pinned_target {
+ Some(target) => {
+ let prev = GetWindow(target, GW_HWNDPREV).unwrap_or(HWND(std::ptr::null_mut()));
+ if !prev.0.is_null() { prev } else { HWND_TOP }
+ }
+ None => HWND_TOP,
+ };
+ let _ = SetWindowPos(
+ hwnd,
+ insert_after,
+ 0, 0, 0, 0,
+ SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_SHOWWINDOW | SWP_NOOWNERZORDER,
+ );
+
+ let _ = target;
+ }
+ }
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 1515781681..92b80a6eb3 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -2,12 +2,37 @@
use async_trait::async_trait;
+/// Pin the agent-cursor overlay above `hwnd` in the z-order, normalising to
+/// the **root** ancestor so the pin lands on the window that actually appears
+/// in the global z-stack.
+///
+/// For UWP / packaged apps the inner `Windows.UI.Core.CoreWindow` is hosted
+/// inside `ApplicationFrameHost.exe` — UIA targets the inner window but only
+/// the host appears in the z-order. `GA_ROOT` normalises both inputs to the
+/// host so the overlay sits at z+1 of whatever is actually painted on screen.
+///
+/// No-op when the overlay is disabled; the command is just dropped by the
+/// render thread in that case.
+fn pin_overlay_above(hwnd: u64) {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT};
+ let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };
+ let wid = if !root.0.is_null() { root.0 as u64 } else { hwnd };
+ crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(wid));
+}
+
/// Animate the agent cursor to (sx, sy) in screen coordinates and wait for the
/// glide to finish before returning. No-op when the overlay is not enabled.
///
/// On the very first call the cursor is at the off-screen initial position
/// (-200, -200). Animating from there would cause a jarring off-screen fly-in,
/// so we snap to the target with a ClickPulse first and skip the glide wait.
+///
+/// For all subsequent calls this defers to
+/// [`crate::overlay::animate_cursor_to`], which sends `MoveTo` and awaits the
+/// render thread's arrival oneshot — that's how we keep the click action
+/// from firing before the cursor has visually landed (the old heuristic
+/// `tokio::sleep(80..600 ms)` was racing the spring-physics glide).
async fn overlay_glide_to(sx: f64, sy: f64) {
if !crate::overlay::is_enabled() { return; }
let pos = crate::overlay::current_position();
@@ -16,15 +41,7 @@ async fn overlay_glide_to(sx: f64, sy: f64) {
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });
return;
}
- // Compute travel time from distance and speed (matches Swift: peak=900, avg~467 pts/sec).
- let dist = ((sx - pos.0).powi(2) + (sy - pos.1).powi(2)).sqrt();
- let avg_speed = 467.0_f64; // (300 + 900 + 200) / 3 ≈ Swift avg
- let ms = (dist / avg_speed * 1000.0).clamp(80.0, 600.0) as u64;
- // Always arrive at 45° (upper-left) — matches Swift's endAngleDegrees: 45.0.
- crate::overlay::send_command(cursor_overlay::OverlayCommand::MoveTo {
- x: sx, y: sy, end_heading_radians: std::f64::consts::FRAC_PI_4,
- });
- tokio::time::sleep(std::time::Duration::from_millis(ms)).await;
+ crate::overlay::animate_cursor_to(sx, sy).await;
}
use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef, ToolRegistry}};
use serde_json::{json, Value};
@@ -571,7 +588,25 @@ impl Tool for GetWindowStateTool {
std::time::Duration::from_secs(4), blocking).await
{
Ok(join_result) => join_result.map_err(|e| anyhow::anyhow!("task panic: {e}")),
- Err(_elapsed) => Err(anyhow::anyhow!("get_window_state timed out after 4s (UIA provider unresponsive)")),
+ Err(_elapsed) => {
+ // Surface the target's window class + an actionable hint
+ // instead of just "UIA provider unresponsive". The class
+ // points the caller at the right workaround (e.g. SALFRAME
+ // → screenshot + pixel coords + dispatch:"foreground"; UWP
+ // class → retry with capture_mode:"vision" to skip the
+ // UIA walk and just get a screenshot).
+ let class = crate::input::dispatch::read_class_name(hwnd);
+ Err(anyhow::anyhow!(
+ "get_window_state timed out after 4s (UIA provider unresponsive on \
+ hwnd 0x{hwnd:x}, class '{class}'). Fallback options: \
+ (a) re-call this tool with `capture_mode:\"vision\"` to skip the \
+ UIA walk and just capture the screenshot — bypasses UIA entirely, \
+ then drive via pixel `click(x, y)` against the returned image; \
+ (b) if the target is a transient VCL / message-box dialog, send \
+ `press_key` with `dispatch:\"foreground\"` (SendInput) to fire the \
+ default accelerator (Esc / Enter / Y / N) without needing the tree."
+ ))
+ }
};
let result = result.and_then(|r| r);
@@ -991,7 +1026,8 @@ impl Tool for LaunchAppTool {
"additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process (or activation arguments for packaged apps)."},
"electron_debugging_port":{"type":"integer","description":"Accepted for cross-platform parity; currently no-op on Windows."},
"webkit_inspector_port":{"type":"integer","description":"Accepted for cross-platform parity; no-op on Windows."},
- "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."}
+ "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."},
+ "start_minimized":{"type":"boolean","description":"When true, launch the app's window minimized to the taskbar instead of restored-but-not-activated. Use this when the agent wants to drive the app entirely in the background — the user's previously-frontmost window (e.g. terminal) stays visually on top. Implementation uses SW_SHOWMINNOACTIVE for the ShellExecuteEx path and a follow-up ShowWindow(SW_MINIMIZE) on the AUMID path. UIA / background dispatch still work on a minimized window; only `screenshot` and `dispatch:\"foreground\"` need it restored."}
},"additionalProperties":false}),
read_only: false, destructive: false, idempotent: true, open_world: true,
})
@@ -1006,6 +1042,41 @@ impl Tool for LaunchAppTool {
let urls: Vec<String> = args.get("urls").and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect())
.unwrap_or_default();
+ let start_minimized = args.get("start_minimized")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ // `nShow` for the ShellExecuteEx-based launches. SW_SHOWNOACTIVATE
+ // keeps the new window from grabbing focus but still renders it on
+ // top of the previous frontmost window. SW_SHOWMINNOACTIVE both
+ // skips activation AND drops the window to the taskbar, which is
+ // what an agent doing background work wants — the user's terminal
+ // stays visually on top. UIA / PostMessage / set_value all keep
+ // working against a minimized window; only screenshot and
+ // dispatch:"foreground" pay a cost.
+ let n_show: i32 = if start_minimized {
+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWMINNOACTIVE.0
+ } else {
+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE.0
+ };
+ // Snapshot the set of currently-running pids BEFORE we launch. The
+ // detached start_minimized polling task uses this to detect "any
+ // process that came into existence as a result of our launch", so
+ // it can minimize their windows too — covers the launcher-stub
+ // case (LibreOffice swriter.exe → soffice.bin, GIMP gimp-3.exe →
+ // gimp-3.2.exe, etc.) where the launched pid exits and a child
+ // process with a different name owns the actual window. Captured
+ // here (before the launch) so the new soffice.bin pid won't be
+ // in the snapshot.
+ let pre_launch_pids: std::sync::Arc<std::collections::HashSet<u32>> =
+ if start_minimized {
+ let pids: std::collections::HashSet<u32> = crate::win32::list_processes()
+ .into_iter()
+ .map(|p| p.pid)
+ .collect();
+ std::sync::Arc::new(pids)
+ } else {
+ std::sync::Arc::new(std::collections::HashSet::new())
+ };
// Capture the foreground window BEFORE any launch path runs so the
// post-spawn polling restore (below) has a target HWND to flip back
@@ -1181,11 +1252,11 @@ impl Tool for LaunchAppTool {
let urls_clone = urls.clone();
let target_for_shell = target_file_opt.clone();
let extra_for_shell = extra_joined.clone();
+ let n_show_for_shell = n_show;
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<u32> {
use windows::Win32::UI::Shell::{
ShellExecuteExW, SHELLEXECUTEINFOW, SEE_MASK_NOCLOSEPROCESS,
};
- use windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE;
use windows::Win32::System::Threading::GetProcessId;
use windows::Win32::Foundation::CloseHandle;
use windows::core::PCWSTR;
@@ -1211,7 +1282,7 @@ impl Tool for LaunchAppTool {
} else {
PCWSTR(args_w.as_ptr())
},
- nShow: SW_SHOWNOACTIVATE.0,
+ nShow: n_show_for_shell,
..Default::default()
};
unsafe { ShellExecuteExW(&mut info)?; }
@@ -1230,7 +1301,7 @@ impl Tool for LaunchAppTool {
cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
lpVerb: PCWSTR(op_w.as_ptr()),
lpFile: PCWSTR(file.as_ptr()),
- nShow: SW_SHOWNOACTIVATE.0,
+ nShow: n_show_for_shell,
..Default::default()
};
unsafe { let _ = ShellExecuteExW(&mut url_info); }
@@ -1275,9 +1346,9 @@ impl Tool for LaunchAppTool {
// capture, mirroring Swift's NSWorkspace secondary-URL flow).
if aumid_for_uwp.is_some() && !urls.is_empty() {
let urls_clone = urls.clone();
+ let n_show_for_urls = n_show;
let _ = tokio::task::spawn_blocking(move || {
use windows::Win32::UI::Shell::{ShellExecuteExW, SHELLEXECUTEINFOW};
- use windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE;
use windows::core::PCWSTR;
fn to_wide(s: &str) -> Vec<u16> {
s.encode_utf16().chain(std::iter::once(0)).collect()
@@ -1289,7 +1360,7 @@ impl Tool for LaunchAppTool {
cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
lpVerb: PCWSTR(op_w.as_ptr()),
lpFile: PCWSTR(file.as_ptr()),
- nShow: SW_SHOWNOACTIVATE.0,
+ nShow: n_show_for_urls,
..Default::default()
};
unsafe { let _ = ShellExecuteExW(&mut url_info); }
@@ -1412,6 +1483,139 @@ impl Tool for LaunchAppTool {
}
let pid = resolved_pid;
+ // start_minimized post-pass: drop every resolved window to the
+ // taskbar. The desktop ShellExecuteEx path already minimizes via
+ // nShow=SW_SHOWMINNOACTIVE, but three cases still need this
+ // fallback:
+ // 1. AUMID / UWP activations come up un-minimized —
+ // ActivateApplication has no nShow.
+ // 2. Apps that ignore the nShow hint and decide their own geometry
+ // (notable offenders: LibreOffice, some Electron apps).
+ // 3. Launcher-stub flows (LO swriter.exe → soffice.bin, GIMP, etc.)
+ // where the launched pid exits before the real window appears,
+ // so `windows_json` is empty at this point.
+ //
+ // To cover (3) we spawn a background polling task that keeps
+ // looking for windows under the launched pid family for up to 5 s
+ // and minimizes anything that materializes. The task runs detached
+ // so the launch_app response isn't held up by it.
+ //
+ // SW_MINIMIZE itself activates "the next top-level window in z-order"
+ // which would shift the user's focus, but the foreground-restore
+ // polling task (spawned earlier in this method via
+ // `restore_foreground_polling_best_effort`) flips foreground back to
+ // the pre-launch window, so the net effect is "minimize and leave
+ // the user's window where it was".
+ if start_minimized {
+ // First, minimize anything already resolved (covers the common
+ // single-process path where windows_json was populated).
+ let immediate_hwnds: Vec<u64> = windows_json.iter()
+ .filter_map(|w| w["window_id"].as_u64())
+ .collect();
+ // Capture pid + basename for the post-launch polling task to
+ // pick up launcher-stub child processes.
+ let stub_basename = target_file_opt
+ .as_deref()
+ .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())
+ .unwrap_or("")
+ .to_owned();
+ // parent_pid is the launched pid family root — used below to
+ // constrain the post-launch minimize sweep to descendants /
+ // name-relatives of the launched app, so we don't accidentally
+ // minimize a user's unrelated app that started during the 5 s
+ // poll window.
+ let parent_pid = pid;
+ let _ = tokio::task::spawn_blocking(move || {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_MINIMIZE};
+ for h in immediate_hwnds {
+ unsafe { let _ = ShowWindow(HWND(h as *mut _), SW_MINIMIZE); }
+ }
+ }).await;
+ // Detached polling for launcher-stub late-window cases.
+ // Strategy: every 200 ms for 5 s, find pids that
+ // (a) weren't in the pre-launch snapshot, AND
+ // (b) are part of the launched app's family — either a
+ // descendant of the launched root pid, OR a process
+ // sharing the launched binary's name prefix (covers
+ // renamed-binary chains like swriter.exe → soffice.bin
+ // where the child's name doesn't match but it's a
+ // direct descendant).
+ // Minimize THEIR visible windows only. Without the family
+ // constraint, anything the user opens during the 5 s window
+ // (Chrome, terminal, IDE) would also get minimized — that's
+ // the regression CodeRabbit flagged.
+ //
+ // Loop ends early once we've minimized the first set of
+ // windows AND remained idle for one tick — the typical
+ // app has its main window up within ~2 s of launch.
+ let pre_pids = pre_launch_pids.clone();
+ let basename_for_poll = stub_basename.clone();
+ tokio::spawn(async move {
+ use std::collections::HashSet;
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_MINIMIZE};
+ let mut minimized: HashSet<u64> = HashSet::new();
+ let mut idle_ticks_after_any_hit: u8 = 0;
+ let mut hit_count_total: usize = 0;
+ for _ in 0..25 {
+ let pre_pids_clone = pre_pids.clone();
+ let basename_clone = basename_for_poll.clone();
+ // Build the family pid set fresh each tick — both
+ // descendant graph and name-relatives can grow as
+ // launcher-stub chains spawn deeper children.
+ let family_new_pids: Vec<u32> = tokio::task::spawn_blocking(move || {
+ let related: std::collections::HashSet<u32> =
+ crate::win32::related_processes(parent_pid, &basename_clone)
+ .into_iter()
+ .chain(std::iter::once(parent_pid))
+ .collect();
+ crate::win32::list_processes()
+ .into_iter()
+ .filter(|p| {
+ !pre_pids_clone.contains(&p.pid) && related.contains(&p.pid)
+ })
+ .map(|p| p.pid)
+ .collect()
+ })
+ .await
+ .unwrap_or_default();
+ let mut tick_hits: usize = 0;
+ for cpid in family_new_pids {
+ let wins = tokio::task::spawn_blocking(move || {
+ crate::win32::list_windows(Some(cpid))
+ })
+ .await
+ .unwrap_or_default();
+ for w in wins {
+ if minimized.insert(w.hwnd) {
+ tick_hits += 1;
+ hit_count_total += 1;
+ let hwnd_iso = w.hwnd as usize;
+ let _ = tokio::task::spawn_blocking(move || unsafe {
+ let _ = ShowWindow(
+ HWND(hwnd_iso as *mut _),
+ SW_MINIMIZE,
+ );
+ }).await;
+ }
+ }
+ }
+ if tick_hits == 0 {
+ idle_ticks_after_any_hit += 1;
+ if hit_count_total > 0 && idle_ticks_after_any_hit >= 3 {
+ // Stable: had hits, then 600 ms of nothing new.
+ // Done.
+ break;
+ }
+ } else {
+ idle_ticks_after_any_hit = 0;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;
+ }
+ });
+ }
+
// Match Swift text format 1:1.
let mut summary = format!("✅ Launched {display} (pid {pid}) in background.");
if !windows_json.is_empty() {
@@ -1499,7 +1703,8 @@ impl Tool for ClickTool {
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Windows-only convenience; Swift exposes right-click as a separate `right_click` tool."},
"count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},
- "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."}
+ "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false
}),
read_only: false, destructive: true, idempotent: false, open_world: true,
@@ -1508,6 +1713,7 @@ impl Tool for ClickTool {
async fn invoke(&self, args: Value) -> ToolResult {
use mcp_server::tool_args::ArgsExt;
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
let pid = match args.require_u32("pid") { Ok(v) => v, Err(e) => return e };
let hwnd_opt = args.opt_u64("window_id");
let elem_idx = args.opt_u64("element_index").map(|v| v as usize);
@@ -1515,6 +1721,7 @@ impl Tool for ClickTool {
let y = args.opt_f64("y");
let button = args.str_or("button", "left");
let count = args.u64_or("count", 1) as usize;
+ let dispatch = DispatchMode::from_args(&args);
// Resolve HWND: explicit, or auto from pid.
let hwnd = match hwnd_opt {
@@ -1534,13 +1741,37 @@ impl Tool for ClickTool {
Some(v) => v,
None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),
};
- // Step 2: animate cursor to screen coords (awaits glide_duration_ms).
+ // Step 2: pin overlay to target window, then animate to screen coords.
+ pin_overlay_above(hwnd);
overlay_glide_to(cx as f64, cy as f64).await;
// Step 3: click pulse + actual click.
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
x: cx as f64, y: cy as f64,
});
let btn = button.clone();
+
+ // dispatch:"foreground" — skip UIA Invoke and use SendInput at the
+ // cached element center. The caller explicitly chose foreground
+ // delivery; UIA Invoke would be background-safe (which they
+ // rejected) and might fail on elements that don't support
+ // InvokePattern anyway.
+ if dispatch == DispatchMode::Foreground {
+ let btn_fg = btn.clone();
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, cx, cy, count, &btn_fg)
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Performed SendInput click on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
// Try UIA Invoke first (it works for UWP / modern XAML / web
// content where PostMessage(WM_LBUTTONDOWN) hits the outer
// HWND but never reaches the inner XAML/composition element).
@@ -1594,6 +1825,15 @@ impl Tool for ClickTool {
}
}
// PostMessage fallback (legacy Win32 + non-Invokable elements).
+ // dispatch:"background" refuses the fallback on targets known
+ // to silently drop PostMessage clicks (Chromium content, GTK
+ // buttons). We surface a tagged error here so the outer match
+ // can convert to the structured background_unavailable result.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ anyhow::bail!("__CUA_BG_UNAVAILABLE_CLICK__");
+ }
crate::input::post_click_screen(hwnd, cx, cy, count, &btn)?;
let action_name = match btn.as_str() {
"right" => "ShowMenu",
@@ -1603,6 +1843,9 @@ impl Tool for ClickTool {
}).await;
match result {
Ok(Ok(msg)) => ToolResult::text(msg),
+ Ok(Err(e)) if e.to_string().contains("__CUA_BG_UNAVAILABLE_CLICK__") => {
+ background_unavailable_error(hwnd, EventKind::MouseClick)
+ }
Ok(Err(e)) => ToolResult::error(e.to_string()),
Err(e) => ToolResult::error(format!("Task error: {e}")),
}
@@ -1627,6 +1870,7 @@ impl Tool for ClickTool {
let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);
(pt.x as f64, pt.y as f64)
};
+ pin_overlay_above(hwnd);
overlay_glide_to(sx, sy).await;
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });
let btn = button.clone();
@@ -1660,6 +1904,30 @@ impl Tool for ClickTool {
// right-click (UIA has no by-coord `ShowContextMenu`) and
// multi-click (`Invoke()` is single-fire by definition)
// skip directly to PostMessage.
+ // dispatch:"foreground" short-circuit — caller explicitly chose
+ // foreground delivery. Skip the UIA hit-test (UIA Invoke is
+ // background-safe and would deliver the click without the
+ // foreground swap the caller asked for).
+ if dispatch == DispatchMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn)
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => {
+ let click_word = match count { 2 => "double-click", 3 => "triple-click", _ => "click" };
+ ToolResult::text(format!(
+ "✅ Sent {click_word} via SendInput to pid {pid} at ({sx},{sy}) (dispatch:foreground)."
+ ))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
let use_uia = (btn == "left" || btn == "middle") && count == 1;
if use_uia {
let invoked = tokio::task::spawn_blocking(move || {
@@ -1676,6 +1944,25 @@ impl Tool for ClickTool {
}
}
+ // UIA hit-test didn't land. Decide between PostMessage / SendInput
+ // based on dispatch mode.
+ //
+ // dispatch:"background" — refuse to swap foreground. If the target
+ // is known to silently drop PostMessage mouse events (Chromium
+ // DOM content, GTK button widgets), surface a structured
+ // background_unavailable error so the caller can bring_to_front
+ // then retry with dispatch:"foreground".
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+
+ // dispatch:"auto" — historical heuristic: Chromium targets get
+ // SendInput with foreground swap (#1623); everything else takes
+ // the PostMessage path. This branch is ONLY reached on Auto mode
+ // (Background returned above; Foreground was handled at the top).
+
// #1623: PostMessage(WM_LBUTTONDOWN) to Chromium frame HWNDs doesn't
// reach the DOM input pipeline — Chromium's input thread only accepts
// events with SendInput-queue origin. For Chromium targets, take the
@@ -1807,7 +2094,8 @@ impl Tool for TypeTextTool {
"text":{"type":"string","description":"Text to insert at the focused element's cursor."},
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used."},
"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},
- "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."}
+ "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false
}),
read_only: false, destructive: true, idempotent: false, open_world: true,
@@ -1816,11 +2104,13 @@ impl Tool for TypeTextTool {
async fn invoke(&self, args: Value) -> ToolResult {
use mcp_server::tool_args::ArgsExt;
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };
let pid = raw_pid as u32;
let text = match args.require_str("text") { Ok(v) => v, Err(e) => return e };
let hwnd_opt = args.opt_u64("window_id");
let elem_idx = args.opt_u64("element_index");
+ let dispatch = DispatchMode::from_args(&args);
if elem_idx.is_some() && hwnd_opt.is_none() {
return ToolResult::error(
"window_id is required when element_index is used — the element_index cache \
@@ -1840,6 +2130,36 @@ impl Tool for TypeTextTool {
};
let text_len = text.chars().count();
+ // dispatch:"background" — TextInput is currently never flagged as
+ // silently dropped (Chromium accepts WM_CHAR through its IME path),
+ // but call the helper so the policy stays centralised in dispatch.rs
+ // and future targets can be added without touching this site.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::TextInput)
+ {
+ return background_unavailable_error(hwnd, EventKind::TextInput);
+ }
+ // dispatch:"foreground" — no SendInput-based text-input helper yet.
+ // Caller can build a foreground-style "type" by hotkey/press_key per
+ // character; or use bring_to_front then dispatch:"auto" which keeps
+ // PostMessage WM_CHAR (works for the targets we know about).
+ if dispatch == DispatchMode::Foreground {
+ return ToolResult::error(
+ "dispatch:\"foreground\" is not yet implemented for type_text. \
+ Use bring_to_front and retry with dispatch:\"auto\" — \
+ PostMessage WM_CHAR works against an already-foreground \
+ window for legacy Win32 targets, and UIA ValuePattern.SetValue \
+ handles XAML/WinUI3 hosts when element_index is supplied. \
+ TODO: add a send_text_synthesized helper that drives per-char \
+ SendInput keystrokes.".to_string(),
+ );
+ }
+
+ // Pin the agent-cursor overlay above the target window so the synthetic
+ // cursor stays sandwiched at z+1 of the type target for the full
+ // duration of the keystrokes (both XAML/UIA and PostMessage paths).
+ pin_overlay_above(hwnd);
+
// CUA-543 routing: PostMessage WM_CHAR doesn't reach modern
// XAML / WinUI3 hosts. When the target is one of those AND the
// caller has supplied an element_index, route through UIA
@@ -1936,7 +2256,8 @@ impl Tool for PressKeyTool {
"key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},
"modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},
"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},
- "window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."}
+ "window_id":{"type":"integer","description":"HWND for the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false
}),
read_only: false, destructive: true, idempotent: false, open_world: true,
@@ -1945,11 +2266,13 @@ impl Tool for PressKeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use mcp_server::tool_args::ArgsExt;
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };
let pid = raw_pid as u32;
let key = match args.require_str("key") { Ok(v) => v, Err(e) => return e };
let mods: Vec<String> = args.str_array("modifiers");
let hwnd_opt = args.opt_u64("window_id");
+ let dispatch = DispatchMode::from_args(&args);
// Swift requires window_id when element_index is supplied — ports the
// same validation.
let elem_idx = args.opt_u64("element_index");
@@ -1970,6 +2293,30 @@ impl Tool for PressKeyTool {
}
};
let key_display = key.clone();
+ // Background mode: plain keystrokes (no modifiers) go through Chromium
+ // and GTK fine — would_be_silently_dropped returns false for the
+ // Keystroke variant by design. KeyCombo (modifiers) on Chromium IS
+ // dropped, so check that when modifiers are present.
+ let event_kind = if mods.is_empty() { EventKind::Keystroke } else { EventKind::KeyCombo };
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, event_kind)
+ {
+ return background_unavailable_error(hwnd, event_kind);
+ }
+ // Foreground: send_key_synthesized takes the SetForegroundWindow path.
+ if dispatch == DispatchMode::Foreground {
+ let send_result = tokio::task::spawn_blocking(move || {
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized(hwnd, &key, &m)
+ }).await;
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent {key_display} via SendInput on pid {raw_pid} (dispatch:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
crate::input::post_key(hwnd, &key, &m)
@@ -2034,7 +2381,8 @@ impl Tool for HotkeyTool {
"pid":{"type":"integer","description":"Target process ID."},
"keys":{"type":"array","items":{"type":"string"},"minItems":2,
"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},
- "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."}
+ "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false
}),
read_only: false, destructive: true, idempotent: false, open_world: true,
@@ -2043,9 +2391,11 @@ impl Tool for HotkeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use mcp_server::tool_args::ArgsExt;
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
let hwnd_opt = args.opt_u64("window_id");
let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e };
let pid = raw_pid as u32;
+ let dispatch = DispatchMode::from_args(&args);
// Parse keys array (Swift's only path). Legacy key+modifiers shape
// still accepted as a Rust-side convenience.
@@ -2175,8 +2525,24 @@ impl Tool for HotkeyTool {
// - Modifiers + non-Chromium Win32 → SendInput (the
// TranslateAccelerator path). The foreground swap stays.
let has_modifiers = !mods.is_empty();
+ // dispatch:"background" — refuse if PostMessage of the key combo
+ // would be silently dropped on this target (Chromium with modifiers).
+ let event_kind = if has_modifiers { EventKind::KeyCombo } else { EventKind::Keystroke };
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, event_kind)
+ {
+ return background_unavailable_error(hwnd, event_kind);
+ }
+ // dispatch:"foreground" — explicit SendInput swap (the path that
+ // unblocks TranslateAccelerator-style apps). Auto mode preserves the
+ // historical "Chromium → PostMessage, non-Chromium-with-modifiers →
+ // SendInput" heuristic.
let is_chromium = crate::input::is_chromium_target_window(hwnd);
- let use_send_input = has_modifiers && !is_chromium;
+ let use_send_input = match dispatch {
+ DispatchMode::Foreground => true,
+ DispatchMode::Background => false,
+ DispatchMode::Auto => has_modifiers && !is_chromium,
+ };
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
if use_send_input {
@@ -2310,7 +2676,8 @@ impl Tool for ScrollTool {
"amount":{"type":"integer","minimum":1,"maximum":50,
"description":"Number of scroll ticks. Default 3."},
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},
- "element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."}
+ "element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false
}),
read_only: false, destructive: false, idempotent: false, open_world: true,
@@ -2318,12 +2685,14 @@ impl Tool for ScrollTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
// Swift error wording 1:1.
let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(p) => p,
None => return ToolResult::error("Missing required integer field pid."),
};
let pid = raw_pid as u32;
+ let dispatch = DispatchMode::from_args(&args);
let direction = match args.get("direction").and_then(|v| v.as_str()) {
Some(d) => d.to_owned(),
None => return ToolResult::error("Missing required string field direction."),
@@ -2354,6 +2723,52 @@ impl Tool for ScrollTool {
}
};
+ // dispatch:"background" — WM_VSCROLL/HSCROLL is silently dropped by
+ // Chromium and may be by GTK. Don't use the generic
+ // `background_unavailable_error` helper here: its standard
+ // remediation suggests retrying with `dispatch:"foreground"`, but
+ // scroll's foreground path is unimplemented (see the explicit
+ // foreground rejection below), so the advertised recovery would
+ // dead-end at another error. Surface a scroll-specific structured
+ // error that points at `bring_to_front` + `dispatch:"auto"` (the
+ // PostMessage path against an already-foreground window works for
+ // most non-Chromium targets) as the actual viable recovery.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseScroll)
+ {
+ let class = crate::input::dispatch::read_class_name(hwnd);
+ return ToolResult::error(format!(
+ "Background scroll dispatch is not available for target window class \
+ '{class}' on this event kind (mouse_scroll). The Chromium / GTK input \
+ stack silently drops PostMessage WM_VSCROLL/HSCROLL events. \
+ dispatch:\"foreground\" is also unimplemented for scroll — there is no \
+ SendInput-based wheel-synthesis helper yet. Workaround: call \
+ bring_to_front(pid, window_id) to make the target frontmost, then \
+ retry with dispatch:\"auto\" — PostMessage scroll against an already-\
+ foreground window works for most non-Chromium targets."
+ ))
+ .with_structured(serde_json::json!({
+ "code": "scroll_unavailable",
+ "target_class": class,
+ "event_kind": "mouse_scroll",
+ "suggestion":
+ "bring_to_front + dispatch:\"auto\". dispatch:\"foreground\" is \
+ NOT a valid recovery for scroll — that path returns an explicit \
+ not-implemented error.",
+ }));
+ }
+ // dispatch:"foreground" — TODO: route through SendInput MOUSEEVENTF_WHEEL
+ // with brief SetForegroundWindow swap. No helper yet.
+ if dispatch == DispatchMode::Foreground {
+ return ToolResult::error(
+ "dispatch:\"foreground\" is not yet implemented for the scroll tool. \
+ Use bring_to_front to activate the target, then retry with \
+ dispatch:\"auto\" (PostMessage scroll works against an already-\
+ foreground window for most non-Chromium targets). TODO: add a \
+ send_wheel_synthesized helper.".to_string(),
+ );
+ }
+
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
use windows::Win32::Foundation::{HWND, LPARAM, WPARAM};
use windows::Win32::UI::WindowsAndMessaging::{
@@ -2459,24 +2874,31 @@ impl Tool for ScreenshotTool {
let is_jpeg = format == "jpeg";
let max_dim = self.state.config.read().unwrap().max_image_dimension;
- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(String, u32, u32)> {
- let raw = match hwnd_opt {
- Some(hwnd) => crate::capture::screenshot_window_bytes(hwnd)?,
- None => crate::capture::screenshot_display_bytes()?,
+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(String, u32, u32, bool)> {
+ // Use the occlusion-aware variant so we can flag captures that
+ // went through the screen-region BitBlt path while another
+ // window was visibly covering the target — those captures show
+ // the OBSCURING window's pixels, not the target's, and silent
+ // delivery would mislead the agent (regression captured in
+ // the autonomous-test journal: a backgrounded Notepad
+ // screenshot returned LibreOffice's pixels).
+ let (raw, occluded) = match hwnd_opt {
+ Some(hwnd) => crate::capture::screenshot_window_bytes_with_occlusion(hwnd)?,
+ None => (crate::capture::screenshot_display_bytes()?, false),
};
let png_bytes = crate::capture::resize_png_if_needed(&raw, max_dim)?;
let (w, h) = crate::capture::png_dimensions_pub(&png_bytes)?;
use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
if is_jpeg {
let jpeg = crate::capture::png_bytes_to_jpeg(&png_bytes, quality)?;
- Ok((B64.encode(&jpeg), w, h))
+ Ok((B64.encode(&jpeg), w, h, occluded))
} else {
- Ok((B64.encode(&png_bytes), w, h))
+ Ok((B64.encode(&png_bytes), w, h, occluded))
}
}).await;
match result {
- Ok(Ok((b64, w, h))) => {
+ Ok(Ok((b64, w, h, occluded))) => {
let label = if is_jpeg { "jpeg" } else { "png" };
// Match Swift `ScreenshotTool.swift` text format 1:1:
// "✅ Window screenshot — WxH png [window_id: ID]"
@@ -2493,7 +2915,33 @@ impl Tool for ScreenshotTool {
mcp_server::protocol::Content::image_png(b64)
};
tr.content.push(img_content);
- tr.structured_content = Some(json!({ "width": w, "height": h, "format": label }));
+ // Occlusion warning: when the screen-region BitBlt path was
+ // used AND another window was on top, the bitmap shows the
+ // covering window. Tell the caller so they don't reason from
+ // the wrong pixels. Always before the image in the content
+ // vec so it's visible regardless of how the consumer reads.
+ if occluded {
+ tr.content.insert(0, mcp_server::protocol::Content::text(
+ format!(
+ "⚠️ Screenshot may not show the target window — HWND \
+ {} appears to be obscured by another window. The \
+ returned bitmap reflects whatever pixels are on \
+ screen at the target's coordinates, not the \
+ target's own backing store. Call \
+ `bring_to_front(pid, window_id)` first if you need \
+ an accurate capture, or use `get_window_state` \
+ with `capture_mode:\"ax\"` for occlusion-immune \
+ UIA tree access.",
+ hwnd_opt.unwrap_or(0)
+ )
+ ));
+ }
+ tr.structured_content = Some(json!({
+ "width": w,
+ "height": h,
+ "format": label,
+ "occluded": occluded,
+ }));
tr
}
Ok(Err(e)) => ToolResult::error(e.to_string()),
@@ -2640,12 +3088,14 @@ impl Tool for DoubleClickTool {
"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys held during the gesture. Accepted for parity; currently no-op on Windows."},
- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."}
+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false}),
read_only: false, destructive: true, idempotent: false, open_world: true,
})
}
async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
// Swift error wording 1:1.
let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(p) => p,
@@ -2657,6 +3107,7 @@ impl Tool for DoubleClickTool {
let elem_idx = args.opt_u64("element_index").map(|v| v as usize);
let x = args.opt_f64("x");
let y = args.opt_f64("y");
+ let dispatch = DispatchMode::from_args(&args);
// Swift validates "both x and y or neither" and "no element_index without window_id".
let has_xy = x.is_some() && y.is_some();
let partial_xy = x.is_some() != y.is_some();
@@ -2693,8 +3144,32 @@ impl Tool for DoubleClickTool {
Some(v) => v,
None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),
};
+ pin_overlay_above(hwnd);
overlay_glide_to(cx as f64, cy as f64).await;
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 });
+ // dispatch:"background" — reject if PostMessage would be silently dropped.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+ // dispatch:"foreground" — route through SendInput at the cached coords.
+ if dispatch == DispatchMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, cx, cy, 2, "left")
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent double-click via SendInput on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<String> {
crate::input::post_click_screen(hwnd, cx, cy, 2, "left")?;
// Swift text format 1:1: `"✅ Posted double-click to [N] role \"title\" at screen-point (X, Y)."`.
@@ -2726,8 +3201,33 @@ impl Tool for DoubleClickTool {
let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);
(pt.x as f64, pt.y as f64)
};
+ pin_overlay_above(hwnd);
overlay_glide_to(sx, sy).await;
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });
+ // dispatch:"background" — reject if PostMessage would be silently dropped.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+ // dispatch:"foreground" — SendInput at screen coords with FG swap.
+ if dispatch == DispatchMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, 2, "left")
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent double-click via SendInput to pid {pid} at screen ({},{}) (dispatch:foreground).",
+ sx as i32, sy as i32
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
let (xi, yi) = (px as i32, py as i32);
let result = tokio::task::spawn_blocking(move || crate::input::post_click(hwnd, xi, yi, 2, "left")).await;
match result {
@@ -2784,12 +3284,14 @@ impl Tool for RightClickTool {
"x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
"y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys held during the right-click. Accepted for parity; currently no-op on Windows."},
- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."}
+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false}),
read_only: false, destructive: true, idempotent: false, open_world: true,
})
}
async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
// Swift error wording 1:1.
let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(p) => p,
@@ -2801,6 +3303,7 @@ impl Tool for RightClickTool {
let elem_idx = args.opt_u64("element_index").map(|v| v as usize);
let x = args.opt_f64("x");
let y = args.opt_f64("y");
+ let dispatch = DispatchMode::from_args(&args);
// Port Swift's full validation set.
let has_xy = x.is_some() && y.is_some();
let partial_xy = x.is_some() != y.is_some();
@@ -2837,8 +3340,30 @@ impl Tool for RightClickTool {
Some(v) => v,
None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")),
};
+ pin_overlay_above(hwnd);
overlay_glide_to(cx as f64, cy as f64).await;
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 });
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+ if dispatch == DispatchMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, cx, cy, 1, "right")
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent right-click via SendInput on [{idx}] at screen ({cx},{cy}) (dispatch:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<String> {
crate::input::post_click_screen(hwnd, cx, cy, 1, "right")?;
// Match Swift's element-path text 1:1
@@ -2871,8 +3396,31 @@ impl Tool for RightClickTool {
let _ = ClientToScreen(HWND(hwnd as *mut _), &mut pt);
(pt.x as f64, pt.y as f64)
};
+ pin_overlay_above(hwnd);
overlay_glide_to(sx, sy).await;
crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy });
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+ if dispatch == DispatchMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, 1, "right")
+ }).await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent right-click via SendInput to pid {pid} at screen ({},{}) (dispatch:foreground).",
+ sx as i32, sy as i32
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
let (xi, yi) = (px as i32, py as i32);
let result = tokio::task::spawn_blocking(move || crate::input::post_click(hwnd, xi, yi, 1, "right")).await;
match result {
@@ -2919,18 +3467,21 @@ impl Tool for DragTool {
"steps":{"type":"integer","minimum":1,"maximum":200,"description":"Number of intermediate WM_MOUSEMOVE events. Default: 20."},
"modifier":{"type":"array","items":{"type":"string"},"description":"Modifier keys: cmd/shift/option/ctrl (held via keyboard)."},
"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: left."},
- "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."}
+ "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."},
+ "dispatch": crate::input::dispatch::dispatch_schema()
},"additionalProperties":false}),
read_only: false, destructive: true, idempotent: false, open_world: true,
})
}
async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error};
// Swift error wording 1:1.
let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(p) => p,
None => return ToolResult::error("Missing required integer field pid."),
};
let pid = raw_pid as u32;
+ let dispatch = DispatchMode::from_args(&args);
use mcp_server::tool_args::ArgsExt;
// Accepts numeric JSON as either float or integer — coerce both to f64.
@@ -2990,6 +3541,45 @@ impl Tool for DragTool {
(p1.x, p1.y, p2.x, p2.y)
};
+ // dispatch:"background" — refuse if PostMessage drag would silently drop.
+ if dispatch == DispatchMode::Background
+ && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ return background_unavailable_error(hwnd, EventKind::MouseClick);
+ }
+ // dispatch:"foreground" — no SendInput-based drag helper yet. The
+ // PostMessage drag works for the GTK-canvas + plain-Win32 cases we
+ // care about today; Chromium DOM dragstart will need a real
+ // SendInput-MOUSE-MOVE sequence, tracked as TODO.
+ if dispatch == DispatchMode::Foreground {
+ return ToolResult::error(
+ "dispatch:\"foreground\" is not yet implemented for the drag tool. \
+ Use bring_to_front to activate the target first, then retry with \
+ dispatch:\"auto\" (PostMessage WM_MOUSEMOVE works against an \
+ already-foreground window in most cases). TODO: add a \
+ send_drag_synthesized helper analogous to send_click_synthesized."
+ .to_string(),
+ );
+ }
+
+ // Pin the agent-cursor overlay above the drag target so the synthetic
+ // cursor stays sandwiched at z+1 of the dragged window for the full
+ // path. Drag stays within a single HWND, so one pin at the start is
+ // sufficient — the 80 ms z-order tick keeps it asserted thereafter.
+ pin_overlay_above(hwnd);
+
+ // Animate the agent cursor to the drag-start, fire a press pulse,
+ // run the actual drag synthesis, then glide to the drag-end and
+ // fire a release pulse. Mirrors macOS `tools/drag.rs:191-242`.
+ // Cursor doesn't follow the interpolated PostMessage path during
+ // the drag itself (the timing coordination would be invasive);
+ // pre- and post-glides plus the press/release pulses are enough
+ // signal for a user watching the agent operate.
+ overlay_glide_to(sx_from as f64, sy_from as f64).await;
+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx_from as f64, y: sy_from as f64,
+ });
+
let button_c = button.clone();
let result = tokio::task::spawn_blocking(move || {
crate::input::mouse::post_drag(
@@ -3000,6 +3590,16 @@ impl Tool for DragTool {
)
}).await;
+ // Drag finished — glide the visual cursor to the drag-end and
+ // pulse the release. Skipped on error so the cursor doesn't lie
+ // about a successful endpoint.
+ if matches!(&result, Ok(Ok(()))) {
+ overlay_glide_to(sx_to as f64, sy_to as f64).await;
+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx_to as f64, y: sy_to as f64,
+ });
+ }
+
match result {
Ok(Ok(())) => {
// Match Swift's text format verbatim.
@@ -4017,6 +4617,135 @@ impl Tool for TypeTextCharsTool {
// keyed on pid. Marked `destructive: true` so MCP clients with permission
// gating prompt before invoking.
+// ── bring_to_front ────────────────────────────────────────────────────────
+
+pub struct BringToFrontTool;
+static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for BringToFrontTool {
+ fn def(&self) -> &ToolDef {
+ BTF_DEF.get_or_init(|| ToolDef {
+ name: "bring_to_front".into(),
+ description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \
+ the OS foreground. \n\n\
+ **This deliberately breaks the no-foreground contract.** Use only when an \
+ agent is about to drive a sequence of operations against a target whose input \
+ stack silently drops PostMessage (Chromium DOM content, GTK button widgets) \
+ and the agent wants to pay the foreground cost once instead of per call. \n\n\
+ Pairs with the `dispatch` field on input tools:\n\
+ 1. Try `click(..., dispatch:\"background\")`. If it returns \
+ `background_unavailable`, the target needs foreground delivery.\n\
+ 2. Call `bring_to_front(pid)` so the target is foreground.\n\
+ 3. Subsequent input calls with `dispatch:\"foreground\"` deliver via \
+ SendInput WITHOUT visible flashing -- the SetForegroundWindow swap \
+ inside SendInput is a no-op since target is already foreground.\n\n\
+ Implementation uses the `AttachThreadInput` trick to bypass Windows' \
+ foreground-lock when the daemon is not at UIAccess integrity. Returns \
+ structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \
+ Windows only; macOS / Linux return an error pointing at platform-native \
+ alternatives.".into(),
+ input_schema: json!({"type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"Optional HWND. Defaults to pid's first visible top-level window."}
+ },"additionalProperties":false}),
+ read_only: false,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use mcp_server::tool_args::ArgsExt;
+ let pid = match args.require_u32("pid") { Ok(v) => v, Err(e) => return e };
+ let hwnd_opt = args.opt_u64("window_id");
+
+ // Resolve hwnd: explicit, or auto from pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid))).await.unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => return ToolResult::error(format!("bring_to_front: no windows found for pid {pid}. Provide window_id.")),
+ }
+ }
+ };
+
+ // Run the foreground swap on a blocking thread. The AttachThreadInput
+ // trick mirrors `send_key_synthesized` (input/keyboard.rs:313-345)
+ // and is validated by `flash-repro/16-edge-launch-fg.ps1` for the
+ // Edge launch focus-steal recovery case.
+ let outcome = tokio::task::spawn_blocking(move || -> Result<(u64, u64), String> {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow,
+ SetForegroundWindow,
+ };
+ use windows::Win32::System::Threading::{AttachThreadInput, GetCurrentThreadId};
+
+ let target = HWND(hwnd as *mut _);
+ if !unsafe { IsWindow(target) }.as_bool() {
+ return Err(format!("hwnd 0x{hwnd:x} is not a valid window"));
+ }
+
+ let prev_fg = unsafe { GetForegroundWindow() };
+ let prev_fg_addr = prev_fg.0 as u64;
+
+ // Attach our thread to the current FG thread so we inherit its
+ // FG-lock token; SetForegroundWindow then succeeds even when
+ // the daemon is at Medium IL without UIAccess.
+ let my_tid = unsafe { GetCurrentThreadId() };
+ let mut fg_pid = 0u32;
+ let fg_tid = unsafe { GetWindowThreadProcessId(prev_fg, Some(&mut fg_pid)) };
+ let attached = fg_tid != 0 && fg_tid != my_tid;
+ if attached {
+ let _ = unsafe { AttachThreadInput(my_tid, fg_tid, true) };
+ }
+ let set_ok = unsafe { SetForegroundWindow(target) }.as_bool();
+ if attached {
+ let _ = unsafe { AttachThreadInput(my_tid, fg_tid, false) };
+ }
+
+ // SetForegroundWindow can silently return FALSE under UIPI /
+ // foreground-lock denial. We still report what we tried.
+ let _ = set_ok;
+ let now_fg = unsafe { GetForegroundWindow() };
+ Ok((prev_fg_addr, now_fg.0 as u64))
+ }).await;
+
+ match outcome {
+ Ok(Ok((prev, now))) => {
+ let landed_on_target = now == hwnd;
+ let msg = if landed_on_target {
+ format!("✅ bring_to_front: pid {pid} hwnd 0x{hwnd:x} is now foreground (was 0x{prev:x}).")
+ } else {
+ format!(
+ "bring_to_front returned but the OS did not honor the swap: target 0x{hwnd:x}, \
+ current foreground 0x{now:x}. Common cause: Windows foreground-lock denied \
+ SetForegroundWindow from a non-UIAccess process. Route through the \
+ cua-driver-uia worker if you have one running."
+ )
+ };
+ let r = if landed_on_target {
+ ToolResult::text(msg)
+ } else {
+ ToolResult::error(msg)
+ };
+ r.with_structured(json!({
+ "previous_fg_hwnd": format!("0x{prev:x}"),
+ "now_fg_hwnd": format!("0x{now:x}"),
+ "target_hwnd": format!("0x{hwnd:x}"),
+ "landed_on_target": landed_on_target,
+ }))
+ }
+ Ok(Err(e)) => ToolResult::error(format!("bring_to_front: {e}")),
+ Err(join) => ToolResult::error(format!("bring_to_front: blocking-task join error: {join}")),
+ }
+ }
+}
+
pub struct KillAppTool;
static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
@@ -4367,6 +5096,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry {
r.register(Box::new(GetWindowStateTool { state: state.clone() }));
r.register(Box::new(LaunchAppTool));
r.register(Box::new(KillAppTool));
+ r.register(Box::new(BringToFrontTool));
r.register(Box::new(DebugWindowInfoTool));
r.register(Box::new(ClickTool { state: state.clone() }));
r.register(Box::new(DoubleClickTool { state: state.clone() }));
@@ -4377,11 +5107,24 @@ pub fn build_registry(compat: bool) -> ToolRegistry {
r.register(Box::new(HotkeyTool));
r.register(Box::new(SetValueTool { state: state.clone() }));
r.register(Box::new(ScrollTool));
- if compat {
- r.register(Box::new(ScreenshotCompatTool { state: state.clone() }));
- } else {
- r.register(Box::new(ScreenshotTool { state: state.clone() }));
- }
+ // `screenshot` / `ScreenshotCompatTool` removed from the tool surface
+ // — `get_window_state` with `capture_mode:"vision"` is the single
+ // canonical path for getting a window screenshot. Reasons:
+ // 1. One entry point means callers always have a window_id context,
+ // which `screenshot` made ambiguous (whole-display fallback
+ // always picked the wrong thing for headless agent workflows).
+ // 2. `get_window_state` already returns a JPEG in vision/som mode
+ // — there's no information `screenshot` exposed that's not in
+ // `get_window_state`.
+ // 3. Claude Code's vision pipeline can be retargeted at
+ // `get_window_state` via its tool-anchor config — no need for
+ // a dedicated "screenshot" name.
+ // The `ScreenshotTool` and `ScreenshotCompatTool` structs are kept in
+ // this file for now as private impl helpers — the `capture::*`
+ // functions they wrap are still the actual screenshot machinery.
+ // Delete the structs in a follow-up after confirming no callers
+ // depend on them via `Tool` trait reflection.
+ let _ = compat; // formerly drove the ScreenshotCompatTool branch
r.register(Box::new(GetScreenSizeTool));
r.register(Box::new(GetCursorPositionTool));
r.register(Box::new(MoveCursorTool { state: state.clone() }));
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs
index 0c958d87bd..120d6999d3 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs
@@ -28,7 +28,7 @@
//! Returns an actionable error if neither path works.
use async_trait::async_trait;
-use mcp_server::page::PageBackend;
+use mcp_server::page::{ClickElementResult, PageBackend};
use windows::core::Interface;
use windows::Win32::Foundation::HWND;
@@ -128,6 +128,146 @@ impl PageBackend for WindowsPageBackend {
let result = mcp_server::cdp::evaluate(port, javascript, true).await?;
Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))
}
+
+ async fn click_element(
+ &self,
+ pid: i32,
+ window_id: u32,
+ selector: &str,
+ ) -> anyhow::Result<ClickElementResult> {
+ // Two-step:
+ // (1) JS to derive element center + viewport screen position.
+ // (2) Animate cursor overlay to those screen coords + ClickPulse.
+ // (3) JS to fire element.click().
+ //
+ // Steps 1 and 3 reuse self.execute_javascript so they take the same
+ // bookmark / CDP routing the agent already knows. Step 2 uses the
+ // existing overlay primitives (pin + animate_cursor_to + ClickPulse).
+
+ let selector_js = escape_js_string_literal(selector);
+
+ // Step 1 — query coords.
+ let probe_js = format!(
+ "(function(){{\
+ var el = document.querySelector('{selector_js}');\
+ if (!el) throw new Error('element_not_found:{selector_js}');\
+ var r = el.getBoundingClientRect();\
+ return JSON.stringify({{\
+ vx: r.left + r.width/2,\
+ vy: r.top + r.height/2,\
+ sx: window.screenX + (window.outerWidth - window.innerWidth)/2,\
+ sy: window.screenY + (window.outerHeight - window.innerHeight),\
+ dpr: window.devicePixelRatio || 1\
+ }});\
+ }})()"
+ );
+ let probe_raw = self.execute_javascript(pid, window_id, &probe_js).await?;
+ let probe_json = strip_dispatch_prefix(&probe_raw);
+
+ // The wrapper paths return the result JSON-stringified. The bookmark
+ // wrapper wraps the user expression's return in JSON.stringify(),
+ // which means the inner string itself is JSON-encoded — we need to
+ // parse twice to get the actual coord object.
+ let parsed: serde_json::Value = serde_json::from_str(&probe_json)
+ .or_else(|_| {
+ // Bookmark wrapper double-encoded the string — decode the
+ // outer JSON-string, then parse the inner JSON.
+ let inner: String = serde_json::from_str(&probe_json)?;
+ serde_json::from_str::<serde_json::Value>(&inner)
+ })
+ .map_err(|e| anyhow::anyhow!(
+ "click_element: could not parse coord JSON from probe (raw: {probe_raw:?}): {e}"
+ ))?;
+
+ // Required-field validation. The previous version defaulted any
+ // missing field to 0.0, which silently animated the visible cursor
+ // to (0,0) AND still fired element.click() — the click would still
+ // hit the DOM element correctly but the visual cursor would lie.
+ // For a fast-fail loud-error contract, require all four screen-
+ // coord inputs from the JS probe and reject anything missing/NaN.
+ // `dpr` retains its 1.0 default because devicePixelRatio is
+ // genuinely optional on the JS side (older webviews / unusual
+ // contexts may not expose it).
+ let require = |key: &str| -> Result<f64, anyhow::Error> {
+ parsed.get(key).and_then(|v| v.as_f64()).filter(|f| f.is_finite())
+ .ok_or_else(|| anyhow::anyhow!(
+ "click_element: probe JSON missing/invalid required field '{key}' \
+ (raw: {probe_raw:?}). All four of vx/vy/sx/sy must be finite numbers."
+ ))
+ };
+ let vx = require("vx")?;
+ let vy = require("vy")?;
+ let sx = require("sx")?;
+ let sy = require("sy")?;
+ let dpr = parsed.get("dpr").and_then(|v| v.as_f64())
+ .filter(|f| f.is_finite() && *f > 0.0)
+ .unwrap_or(1.0);
+
+ let screen_x = sx + vx * dpr;
+ let screen_y = sy + vy * dpr;
+
+ // Step 2 — drive the cursor overlay. Pin above the browser's root
+ // HWND so the overlay sits at z+1 of the page, glide, click-pulse.
+ // Inlined GA_ROOT lookup mirrors the helper in tools/impl_.rs but
+ // keeps page.rs from depending on a private symbol there.
+ let hwnd = window_id as u64;
+ {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT};
+ let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };
+ let pin_wid = if !root.0.is_null() { root.0 as u64 } else { hwnd };
+ crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(pin_wid));
+ }
+ crate::overlay::animate_cursor_to(screen_x, screen_y).await;
+ crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
+ x: screen_x,
+ y: screen_y,
+ });
+
+ // Step 3 — fire the real click.
+ let click_js = format!(
+ "(function(){{\
+ var el = document.querySelector('{selector_js}');\
+ if (!el) throw new Error('element_not_found_on_click:{selector_js}');\
+ el.click();\
+ return 'clicked:{selector_js}';\
+ }})()"
+ );
+ let _ = self.execute_javascript(pid, window_id, &click_js).await?;
+
+ Ok(ClickElementResult {
+ screen_x,
+ screen_y,
+ viewport_x: vx,
+ viewport_y: vy,
+ message: format!(
+ "✅ click_element('{selector}') at screen ({screen_x:.0},{screen_y:.0}) on pid {pid}."
+ ),
+ })
+ }
+}
+
+/// Escape `s` for safe interpolation inside a single-quoted JS string
+/// literal. Backslash and apostrophe are the only characters that can
+/// break out of `'...'` after the opening quote; newlines also confuse
+/// the bookmark wrapper which strips them, so we replace with space.
+fn escape_js_string_literal(s: &str) -> String {
+ s.replace('\\', "\\\\")
+ .replace('\'', "\\'")
+ .replace('\r', " ")
+ .replace('\n', " ")
+}
+
+/// `execute_javascript` returns the raw JS result prefixed with the
+/// dispatch route (e.g. `"uia.bookmark_exec: ..."`, `"cdp.runtime.evaluate.user_gesture: ..."`).
+/// Strip that prefix so coord parsing sees the bare value.
+fn strip_dispatch_prefix(s: &str) -> String {
+ for prefix in &["uia.bookmark_exec: ", "cdp.runtime.evaluate.user_gesture: "] {
+ if let Some(rest) = s.strip_prefix(prefix) {
+ return rest.to_owned();
+ }
+ }
+ s.to_owned()
}
// ── implementation ────────────────────────────────────────────────────────
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
index 1a58886f7c..39cb9fb9e9 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs
@@ -115,10 +115,74 @@ unsafe fn walk_tree_unsafe(hwnd: u64, query: Option<&str>) -> UiaTreeResult {
}
let hwnd_win = windows::Win32::Foundation::HWND(hwnd as *mut _);
- let root_elem = match automation.ElementFromHandleBuildCache(hwnd_win, &cache_req) {
+
+ // SAL (LibreOffice / OpenOffice) fast-path: VCL's UIA provider hangs
+ // on `BuildUpdatedCache(TreeScope.Subtree)` for transient dialogs
+ // (SALSUBFRAME class — Confirmation, Warning, modal Yes/No). Both
+ // the atomic `ElementFromHandleBuildCache` and the split
+ // `ElementFromHandle + BuildUpdatedCache(Subtree)` paths block
+ // indefinitely under the daemon's MTA thread pool (the same code
+ // runs fine via CLI in-process, suggesting a COM apartment /
+ // concurrent-access interaction with SAL's provider). The 4 s
+ // outer timeout fires and the caller gets the structured
+ // diagnostic, but that's a wasted 4 s every call.
+ //
+ // Short-circuit by class detection: if the target's class starts
+ // with "SAL", skip the bulk-cache walk entirely and return the
+ // same synthetic stub the post-walk SAL-skip would emit, with the
+ // three actionable fallback options. Saves the 4 s wait and the
+ // ambiguity of "did the walk run? did it find nothing? did it
+ // hang?" The caller sees the stub immediately and routes to the
+ // appropriate workaround.
+ // Class lookup once — used both for the fast-path skip decision
+ // and the diagnostic stub. SALSUBFRAME is the empirically-confirmed
+ // hang class (modal Confirmation / Warning dialogs). SALFRAME
+ // (the document window, the Recovery dialog) walks fine, so don't
+ // include it in the skip list — pixel + element_index against
+ // Recovery's Discard All button worked end-to-end through the
+ // primary path. SALMENU / SALTMPSUBFRAME are transient menu hosts
+ // that the user typically doesn't get_window_state on; default to
+ // skipping out of caution. If a real use-case emerges, narrow.
+ let sal_class: Option<String> = {
+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
+ let mut buf = [0u16; 64];
+ let n = GetClassNameW(hwnd_win, &mut buf);
+ if n > 0 {
+ let s = String::from_utf16_lossy(&buf[..n as usize]);
+ if s == "SALSUBFRAME" || s == "SALMENU" || s == "SALTMPSUBFRAME" {
+ Some(s)
+ } else { None }
+ } else { None }
+ };
+ if let Some(class) = sal_class {
+ let stub = format!(
+ "- Window <{class} — SAL/VCL target, UIA walk skipped>\n\
+ (SAL's UIA provider hangs on TreeScope.Subtree BuildUpdatedCache \
+ when the daemon is running in MTA; the cua-driver fast-path \
+ returns immediately rather than wait 4 s. Use one of: \
+ (a) `screenshot(pid, window_id)` + pixel `click(x, y)`; \
+ (b) `press_key` with `dispatch:\"foreground\"` (Esc / Enter / Y / N); \
+ (c) `get_window_state` with `capture_mode:\"vision\"`.)\n"
+ );
+ return UiaTreeResult { tree_markdown: stub, nodes: Vec::new() };
+ }
+
+ // Two-call sequence (ElementFromHandle + BuildUpdatedCache) instead of
+ // the atomic ElementFromHandleBuildCache. Empirically, the single-call
+ // batched variant hangs against SAL even in CLI in-process tests;
+ // the two-call path works in-process. Kept for non-SAL targets where
+ // the difference matters for performance-sensitive providers.
+ let uncached = match automation.ElementFromHandle(hwnd_win) {
Ok(e) => e,
Err(e) => return UiaTreeResult {
- tree_markdown: format!("ElementFromHandleBuildCache failed: {e}"),
+ tree_markdown: format!("ElementFromHandle failed: {e}"),
+ nodes: Vec::new(),
+ },
+ };
+ let root_elem = match uncached.BuildUpdatedCache(&cache_req) {
+ Ok(e) => e,
+ Err(e) => return UiaTreeResult {
+ tree_markdown: format!("BuildUpdatedCache failed: {e}"),
nodes: Vec::new(),
},
};
@@ -147,34 +211,86 @@ unsafe fn walk_tree_unsafe(hwnd: u64, query: Option<&str>) -> UiaTreeResult {
// erasing it AND leaving the consumed `MAX_TOTAL_ELEMENTS` budget intact
// for the fallback (which would then truncate large trees prematurely).
if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {
- if let Some(target_pid) = pid_from_hwnd(hwnd_win) {
- let mut fallback_nodes: Vec<UiaNode> = Vec::new();
- let mut fallback_lines: Vec<(usize, String)> = Vec::new();
- let mut fallback_counter = 0usize;
- let mut fallback_total = 0usize;
-
+ // Skip the desktop-root walk-by-pid fallback for VCL / SAL
+ // targets (LibreOffice, OpenOffice). The fallback does its own
+ // `BuildUpdatedCache(TreeScope.Subtree)` per matched top-level
+ // window — which is exactly the bulk-cache RPC shape that SAL
+ // hangs on. The primary two-call path (ElementFromHandle +
+ // BuildUpdatedCache on the dialog's own HWND) ALREADY dodged the
+ // hang on that one specific call, but the fallback re-introduces
+ // it. Returning the empty tree here lets the outer
+ // get_window_state timeout fire its structured diagnostic
+ // promptly instead of stalling 4 s on the fallback's hang.
+ //
+ // The diagnostic tells callers exactly how to drive the SAL
+ // dialog without the tree: pixel click via screenshot,
+ // capture_mode:"vision", or press_key with dispatch:"foreground"
+ // for accelerator-style dismissal. That's enough for the
+ // common modal-dismissal case (Yes/No/Esc on a Confirmation),
+ // which is what SAL dialogs almost always need.
+ let is_sal = {
+ use windows::Win32::UI::WindowsAndMessaging::GetClassNameW;
+ let mut buf = [0u16; 64];
+ let n = GetClassNameW(hwnd_win, &mut buf);
+ n > 0 && {
+ let class = String::from_utf16_lossy(&buf[..n as usize]);
+ class.starts_with("SAL")
+ }
+ };
+ if !is_sal {
+ if let Some(target_pid) = pid_from_hwnd(hwnd_win) {
+ let mut fallback_nodes: Vec<UiaNode> = Vec::new();
+ let mut fallback_lines: Vec<(usize, String)> = Vec::new();
+ let mut fallback_counter = 0usize;
+ let mut fallback_total = 0usize;
+
+ tracing::debug!(
+ target: "uia",
+ "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \
+ falling back to GetRootElement + filter ProcessId={target_pid}"
+ );
+ walk_root_by_pid(
+ &automation,
+ &cache_req,
+ target_pid,
+ &mut fallback_nodes,
+ &mut fallback_lines,
+ &mut fallback_counter,
+ &mut fallback_total,
+ );
+
+ if fallback_nodes.iter().any(|n| n.element_index.is_some()) {
+ nodes = fallback_nodes;
+ lines = fallback_lines;
+ // counter/total aren't read after this point — they're
+ // only used by walk_cached's &mut params for element
+ // indexing inside that call.
+ }
+ }
+ } else {
tracing::debug!(
target: "uia",
- "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \
- falling back to GetRootElement + filter ProcessId={target_pid}"
+ "SAL target hwnd 0x{hwnd:x} returned empty primary tree; \
+ skipping walk_root_by_pid fallback (known to re-hang on SAL Subtree fetch). \
+ Caller should use press_key/screenshot fallbacks per the get_window_state diagnostic."
);
- walk_root_by_pid(
- &automation,
- &cache_req,
- target_pid,
- &mut fallback_nodes,
- &mut fallback_lines,
- &mut fallback_counter,
- &mut fallback_total,
+ // Return a tree_markdown that mirrors the get_window_state
+ // timeout diagnostic so callers get the same actionable
+ // fallback options even though the walk itself didn't hit
+ // the 4 s outer timeout (because we skipped the hang-prone
+ // fallback). Without this the caller sees an empty tree
+ // and no error, which is less actionable.
+ let stub = format!(
+ "- Window <SAL class — empty primary UIA tree>\n\
+ (SAL providers don't expose modal-dialog children via \
+ ElementFromHandle, and the desktop-root fallback walk that \
+ would normally find them is known to hang on SAL Subtree \
+ BuildUpdatedCache. Use one of: \
+ (a) `screenshot(pid, window_id)` + pixel `click(x, y)`; \
+ (b) `press_key` with `dispatch:\"foreground\"` (Esc / Enter / Y / N); \
+ (c) `get_window_state` with `capture_mode:\"vision\"`.)\n"
);
-
- if fallback_nodes.iter().any(|n| n.element_index.is_some()) {
- nodes = fallback_nodes;
- lines = fallback_lines;
- // counter/total aren't read after this point — they're
- // only used by walk_cached's &mut params for element
- // indexing inside that call.
- }
+ return UiaTreeResult { tree_markdown: stub, nodes: Vec::new() };
}
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/wgc.rs b/libs/cua-driver/rust/crates/platform-windows/src/wgc.rs
new file mode 100644
index 0000000000..c8e8f38f06
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/wgc.rs
@@ -0,0 +1,226 @@
+//! Windows.Graphics.Capture (WGC) backend for window screenshots.
+//!
+//! Used when PrintWindow returns black (UWP / DirectComposition) AND/OR
+//! when the target window is occluded by another window. WGC captures
+//! the window's own composited frames out of DWM regardless of z-order,
+//! which is the only correct way to screenshot UWP apps like Calculator
+//! when they're behind the user's terminal.
+//!
+//! Constraints:
+//! - Minimum OS: Windows 10 version 1903 (~April 2019). ~100% of users
+//! today; we surface a structured error on older builds.
+//! - Cannot capture **minimized** windows. They have no rendered content.
+//! For minimized targets, the caller should use UIA via
+//! `get_window_state` `capture_mode:"ax"`. We surface this as an
+//! error with the recommended fallback.
+//! - First call per process pays ~50ms for D3D11 device creation +
+//! COM activation factory lookup. Subsequent calls don't cache the
+//! device — could be optimized later, but a single-shot screenshot
+//! tool isn't the place to fight that.
+
+use anyhow::{bail, Context, Result};
+use std::time::Duration;
+
+use windows::{
+ Graphics::{
+ Capture::{Direct3D11CaptureFramePool, GraphicsCaptureItem},
+ DirectX::{Direct3D11::IDirect3DDevice, DirectXPixelFormat},
+ },
+ Win32::{
+ Foundation::HWND,
+ Graphics::{
+ Direct3D::{D3D_DRIVER_TYPE_HARDWARE, D3D_FEATURE_LEVEL_11_0},
+ Direct3D11::{
+ D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D,
+ D3D11_CPU_ACCESS_READ, D3D11_CREATE_DEVICE_BGRA_SUPPORT,
+ D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_READ, D3D11_SDK_VERSION,
+ D3D11_TEXTURE2D_DESC, D3D11_USAGE_STAGING,
+ },
+ Dxgi::IDXGIDevice,
+ },
+ System::WinRT::{
+ Direct3D11::{CreateDirect3D11DeviceFromDXGIDevice, IDirect3DDxgiInterfaceAccess},
+ Graphics::Capture::IGraphicsCaptureItemInterop,
+ },
+ UI::WindowsAndMessaging::IsIconic,
+ },
+ core::Interface,
+};
+
+/// Capture a window via WGC, returning BGRA pixels + (width, height).
+/// Caller is responsible for encoding to PNG (or whatever format) via
+/// `mcp_server::image_utils::encode_bgra_to_png`.
+pub fn screenshot_window_via_wgc(hwnd: u64) -> Result<(Vec<u8>, u32, u32)> {
+ unsafe { wgc_capture_impl(HWND(hwnd as *mut _)) }
+}
+
+unsafe fn wgc_capture_impl(hwnd: HWND) -> Result<(Vec<u8>, u32, u32)> {
+ if IsIconic(hwnd).as_bool() {
+ bail!(
+ "WGC cannot capture a minimized window (no rendered content). \
+ Restore the window first, or use `get_window_state` with \
+ `capture_mode:\"ax\"` for UIA tree access — that works on \
+ minimized windows."
+ );
+ }
+
+ // 1. D3D11 device — feature level 11.0 + BGRA support (required by WGC).
+ let mut d3d_device: Option<ID3D11Device> = None;
+ let mut d3d_context: Option<ID3D11DeviceContext> = None;
+ D3D11CreateDevice(
+ None,
+ D3D_DRIVER_TYPE_HARDWARE,
+ windows::Win32::Foundation::HMODULE(std::ptr::null_mut()),
+ D3D11_CREATE_DEVICE_BGRA_SUPPORT,
+ Some(&[D3D_FEATURE_LEVEL_11_0]),
+ D3D11_SDK_VERSION,
+ Some(&mut d3d_device),
+ None,
+ Some(&mut d3d_context),
+ )
+ .context("D3D11CreateDevice failed (hardware feature-level 11.0 + BGRA)")?;
+ let d3d_device = d3d_device.context("D3D11CreateDevice returned a None device")?;
+ let d3d_context = d3d_context.context("D3D11CreateDevice returned a None context")?;
+
+ // 2. DXGI device interface for the D3D11 device, then the WinRT
+ // IDirect3DDevice wrapper that GraphicsCaptureFramePool needs.
+ let dxgi_device: IDXGIDevice = d3d_device.cast().context("ID3D11Device → IDXGIDevice cast")?;
+ let inspectable = CreateDirect3D11DeviceFromDXGIDevice(&dxgi_device)
+ .context("CreateDirect3D11DeviceFromDXGIDevice failed")?;
+ let direct3d_device: IDirect3DDevice = inspectable
+ .cast()
+ .context("CreateDirect3D11DeviceFromDXGIDevice → IDirect3DDevice cast")?;
+
+ // 3. GraphicsCaptureItem for our target HWND.
+ let interop_factory: IGraphicsCaptureItemInterop =
+ windows::core::factory::<GraphicsCaptureItem, IGraphicsCaptureItemInterop>()
+ .context("IGraphicsCaptureItemInterop activation factory")?;
+ let item: GraphicsCaptureItem = interop_factory
+ .CreateForWindow(hwnd)
+ .context(
+ "IGraphicsCaptureItemInterop::CreateForWindow failed — \
+ requires Win10 1903+ and the target window must exist",
+ )?;
+ let item_size = item.Size().context("GraphicsCaptureItem::Size")?;
+ if item_size.Width <= 0 || item_size.Height <= 0 {
+ bail!(
+ "WGC item size is {}x{} — target may be hidden/cloaked. \
+ Try restoring the window or use UIA.",
+ item_size.Width,
+ item_size.Height
+ );
+ }
+
+ // 4. Frame pool + capture session.
+ // `CreateFreeThreaded` (not `Create`) — `Create` requires the
+ // calling thread to have a CoreDispatcher/event loop running
+ // for FrameArrived to fire, which we don't have inside a
+ // daemon's spawn_blocking thread. `CreateFreeThreaded`
+ // dispatches FrameArrived on a system thread-pool thread, which
+ // is exactly what our mpsc channel handler expects. Without
+ // this, FrameArrived silently never fires and our 1500 ms wait
+ // always times out.
+ //
+ // Frame count 2 — WGC docs recommend ≥2 to avoid GPU stalls
+ // while the consumer reads the previous frame. We only consume
+ // one but the GPU side is happier with breathing room.
+ let pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
+ &direct3d_device,
+ DirectXPixelFormat::B8G8R8A8UIntNormalized,
+ 2,
+ item_size,
+ )
+ .context("Direct3D11CaptureFramePool::CreateFreeThreaded")?;
+ let session = pool
+ .CreateCaptureSession(&item)
+ .context("Direct3D11CaptureFramePool::CreateCaptureSession")?;
+
+ // 5. Hide the WGC capture indicator (yellow border) if the build
+ // supports it. Available since Win11 (IsBorderRequired property).
+ // Best-effort — older Win10 won't have the setter; ignore failure.
+ let _ = session.SetIsBorderRequired(false);
+ let _ = session.SetIsCursorCaptureEnabled(false);
+
+ // 6. Start capture, then poll TryGetNextFrame until a frame is
+ // available or we time out. We tried FrameArrived events first
+ // (both `Create` and `CreateFreeThreaded` pool variants) but
+ // the event never fired in our daemon's thread environment —
+ // likely an STA/COM-apartment mismatch we couldn't isolate.
+ // Polling is what windows-capture and OBS-style consumers do
+ // for single-shot capture anyway: WGC drops frames in the pool
+ // as DWM composes them, and TryGetNextFrame returns Ok(None) /
+ // NULL until one's available. 50 ms polling interval keeps the
+ // "frame finally arrived" detection latency low without burning
+ // CPU on a tight spin.
+ session.StartCapture().context("GraphicsCaptureSession::StartCapture")?;
+
+ let deadline = std::time::Instant::now() + Duration::from_millis(1500);
+ let mut frame_opt = None;
+ while std::time::Instant::now() < deadline {
+ match pool.TryGetNextFrame() {
+ Ok(f) => { frame_opt = Some(f); break; }
+ Err(_) => {
+ // TryGetNextFrame returns an error when no frame is
+ // available yet. Wait briefly, retry.
+ std::thread::sleep(Duration::from_millis(50));
+ }
+ }
+ }
+ let frame = frame_opt.ok_or_else(|| anyhow::anyhow!(
+ "WGC TryGetNextFrame returned no frame within 1500 ms — \
+ DWM may not be compositing this window (cloaked / not actually \
+ rendered). Fall through to screen-region BitBlt."
+ ))?;
+
+ // 8. Pull the underlying ID3D11Texture2D out of the WinRT frame
+ // surface via the IDirect3DDxgiInterfaceAccess shim.
+ let surface = frame.Surface().context("frame.Surface")?;
+ let access: IDirect3DDxgiInterfaceAccess = surface
+ .cast()
+ .context("IDirect3DSurface → IDirect3DDxgiInterfaceAccess cast")?;
+ let frame_texture: ID3D11Texture2D = access
+ .GetInterface()
+ .context("IDirect3DDxgiInterfaceAccess::GetInterface(ID3D11Texture2D)")?;
+
+ // 9. Create a staging texture we can map for CPU read, copy frame in.
+ let mut desc = D3D11_TEXTURE2D_DESC::default();
+ frame_texture.GetDesc(&mut desc);
+ desc.Usage = D3D11_USAGE_STAGING;
+ desc.BindFlags = 0;
+ desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ.0 as u32;
+ desc.MiscFlags = 0;
+ let mut staging: Option<ID3D11Texture2D> = None;
+ d3d_device
+ .CreateTexture2D(&desc, None, Some(&mut staging))
+ .context("CreateTexture2D(staging)")?;
+ let staging = staging.context("CreateTexture2D returned a None texture")?;
+ d3d_context.CopyResource(&staging, &frame_texture);
+
+ // 10. Map the staging texture, copy BGRA out row-by-row (RowPitch may
+ // be padded — can't just copy `Width * 4 * Height` straight).
+ let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
+ d3d_context
+ .Map(&staging, 0, D3D11_MAP_READ, 0, Some(&mut mapped))
+ .context("ID3D11DeviceContext::Map")?;
+ let width = desc.Width as usize;
+ let height = desc.Height as usize;
+ let stride = mapped.RowPitch as usize;
+ let mut bgra = vec![0u8; width * height * 4];
+ let src_base = mapped.pData as *const u8;
+ for row in 0..height {
+ std::ptr::copy_nonoverlapping(
+ src_base.add(row * stride),
+ bgra.as_mut_ptr().add(row * width * 4),
+ width * 4,
+ );
+ }
+ d3d_context.Unmap(&staging, 0);
+
+ // 11. Cleanup: close the session and pool explicitly. RAII would
+ // handle this but being explicit makes the lifetimes easier to
+ // audit. (No FrameArrived to unhook — we polled instead.)
+ let _ = session.Close();
+ let _ = pool.Close();
+
+ Ok((bgra, desc.Width, desc.Height))
+}
diff --git a/libs/cua-driver/scripts/_install-common.psm1 b/libs/cua-driver/scripts/_install-common.psm1
index c5ee0b5dbd..d781ff5d08 100644
--- a/libs/cua-driver/scripts/_install-common.psm1
+++ b/libs/cua-driver/scripts/_install-common.psm1
@@ -52,20 +52,95 @@ function Stop-CuaDriverDaemons {
return @(Get-Process -Name "cua-driver","cua-driver-uia" -ErrorAction SilentlyContinue)
}
+# Probe whether `\\.\pipe\cua-driver` is currently accepting connections.
+# Distinguishes a *healthy* High-IL daemon (process alive AND pipe
+# responsive - install can proceed by deferring to the user to restart
+# from elevated PS) from a *stale* daemon (process alive but pipe gone
+# - daemon process is hung, no new daemon can bind, MCP is broken until
+# someone kills the zombie).
+#
+# Returns $true iff a real serve daemon is listening. Uses a short
+# 200 ms timeout so install latency stays bounded.
+function Test-CuaDriverPipeAlive {
+ [CmdletBinding()]
+ param()
+ try {
+ $fs = [System.IO.File]::Open(
+ '\\.\pipe\cua-driver',
+ [System.IO.FileMode]::Open,
+ [System.IO.FileAccess]::ReadWrite,
+ [System.IO.FileShare]::ReadWrite
+ )
+ $fs.Close()
+ return $true
+ } catch {
+ return $false
+ }
+}
+
+# Stop-CuaDriverDaemons + stale-daemon detection in one go. Returns a
+# PSCustomObject with both the survivor list AND a `Stale` boolean
+# flagging "processes are alive but the named pipe is dead". Used by
+# install/install-local to swap the user-facing message from "you need
+# to kill these from elevated PS" to the stronger "your daemon is
+# WEDGED - kill it now or reboot, MCP is currently broken".
+function Stop-CuaDriverDaemonsWithHealth {
+ [CmdletBinding()]
+ param()
+ $survivors = Stop-CuaDriverDaemons
+ $pipeAlive = Test-CuaDriverPipeAlive
+ # @() forces array context so .Count works even when PowerShell
+ # collapsed a 1-element array to a single PSObject (strict-mode
+ # otherwise errors on .Count missing on the bare object).
+ $count = @($survivors).Count
+ $stale = ($count -gt 0 -and -not $pipeAlive)
+ return [pscustomobject]@{
+ Survivors = $survivors
+ PipeAlive = $pipeAlive
+ Stale = $stale
+ }
+}
+
# Prints a clear hint when Stop-CuaDriverDaemons leaves something
# running (almost always a High-IL daemon spawned by the
# RunLevel=Highest autostart task - Medium-IL kill returns Access
# Denied for those).
function Show-CuaDriverDaemonSurvivors {
[CmdletBinding()]
- param([Parameter(Mandatory = $true)][array]$Survivors)
+ # AllowNull + non-mandatory: PowerShell collapses `return @()` from
+ # Stop-CuaDriverDaemons to $null at the call site, which would fail
+ # a `Mandatory = $true [array]` bind. The body below already treats
+ # null and empty array as the no-survivors case, so accept both.
+ #
+ # `-Stale` toggles the user-facing wording from "the old binary is
+ # still running" (mild - install completed, OLD binary in memory
+ # until the daemon restarts) to "MCP is currently broken because
+ # the daemon process is alive but its named pipe is dead" (loud -
+ # nothing will work until the zombie process is killed).
+ param(
+ [Parameter()][AllowNull()][array]$Survivors,
+ [switch]$Stale
+ )
if (-not $Survivors -or $Survivors.Count -eq 0) { return }
$pids = ($Survivors | ForEach-Object { $_.Id }) -join ', '
- Write-Host "Note: $($Survivors.Count) cua-driver process(es) still running after best-effort kill (pid: $pids)." -ForegroundColor Yellow
- Write-Host " They are likely High-IL (spawned by RunLevel=Highest autostart task)." -ForegroundColor Yellow
- Write-Host " From an elevated PowerShell:" -ForegroundColor Yellow
- Write-Host " taskkill /IM cua-driver.exe /F" -ForegroundColor Yellow
- Write-Host " Or just reboot. Until they exit, the OLD binary keeps running." -ForegroundColor Yellow
+ if ($Stale) {
+ Write-Host "" -ForegroundColor Red
+ Write-Host "ERROR: cua-driver daemon is STALE - process alive (pid: $pids) but \\.\pipe\cua-driver" -ForegroundColor Red
+ Write-Host " is not accepting connections. The daemon is wedged; MCP / CLI calls will fail" -ForegroundColor Red
+ Write-Host " with 'cannot find the file specified' until this process is killed." -ForegroundColor Red
+ Write-Host "" -ForegroundColor Red
+ Write-Host " From an ELEVATED PowerShell (right-click PowerShell, 'Run as Administrator'):" -ForegroundColor Yellow
+ Write-Host " Stop-Process -Id $pids -Force" -ForegroundColor Yellow
+ Write-Host " schtasks /Run /TN 'cua-driver-serve'" -ForegroundColor Yellow
+ Write-Host "" -ForegroundColor Yellow
+ Write-Host " Reboot also clears it. After the kill+restart, MCP recovers automatically." -ForegroundColor Yellow
+ } else {
+ Write-Host "Note: $($Survivors.Count) cua-driver process(es) still running after best-effort kill (pid: $pids)." -ForegroundColor Yellow
+ Write-Host " They are likely High-IL (spawned by RunLevel=Highest autostart task)." -ForegroundColor Yellow
+ Write-Host " From an elevated PowerShell:" -ForegroundColor Yellow
+ Write-Host " taskkill /IM cua-driver.exe /F" -ForegroundColor Yellow
+ Write-Host " Or just reboot. Until they exit, the OLD binary keeps running." -ForegroundColor Yellow
+ }
}
# Load this module from disk if `$LocalDir/_install-common.psm1` exists
@@ -107,7 +182,139 @@ function Import-CuaDriverInstallModule {
}
}
+# Trigger UAC and spawn a brief elevated PowerShell that kills the stale
+# (High-IL) cua-driver pids and restarts the scheduled task. Returns
+# $true iff the elevated helper actually killed everything AND the
+# named pipe is alive again afterward. Returns $false on UAC cancel,
+# elevation failure, or post-recovery health-check still failing.
+#
+# Why a helper rather than self-elevating the whole install: the
+# install only needs admin to break the wedge. Binary copy, PATH
+# munging, scheduled-task re-registration (which has its own UAC
+# prompt anyway) all run fine at Medium IL. Asking for admin just
+# for those steps would needlessly broaden the prompt's blast
+# radius.
+#
+# The helper is a single short -Command string so we don't need a
+# temp file. UAC users see "Windows PowerShell" as the requesting
+# app, with the command in the elevation prompt's "Show details"
+# panel.
+function Invoke-CuaDriverStaleDaemonKill {
+ [CmdletBinding()]
+ param(
+ [Parameter(Mandatory = $true)][int[]]$Pids
+ )
+ if (-not $Pids -or $Pids.Count -eq 0) { return $true }
+ $pidList = ($Pids -join ',')
+ # Build the elevated payload:
+ # 1. Stop-Process each pid, force, swallow errors (process already gone is OK).
+ # 2. Re-run the scheduled task so a fresh daemon binds the pipe.
+ # 3. Brief wait + verify the pipe is reachable.
+ # The exit code from the helper signals success (0) / pipe still
+ # dead post-restart (3) so the install-side caller can tell the
+ # user whether to expect MCP to work.
+ $payload = @"
+`$ErrorActionPreference = 'Continue'
+foreach (`$p in @($pidList)) {
+ try { Stop-Process -Id `$p -Force -ErrorAction Stop } catch {}
+}
+& schtasks.exe /Run /TN 'cua-driver-serve' | Out-Null
+Start-Sleep -Milliseconds 1500
+try {
+ `$fs = [System.IO.File]::Open('\\.\pipe\cua-driver','Open','ReadWrite','ReadWrite')
+ `$fs.Close()
+ exit 0
+} catch {
+ exit 3
+}
+"@
+ try {
+ $proc = Start-Process `
+ -FilePath powershell.exe `
+ -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', $payload) `
+ -Verb RunAs `
+ -PassThru `
+ -Wait `
+ -WindowStyle Hidden `
+ -ErrorAction Stop
+ return ($proc.ExitCode -eq 0)
+ } catch {
+ # UAC cancel surfaces as System.ComponentModel.Win32Exception ("The
+ # operation was canceled by the user"). Other failures (no
+ # interactive desktop, etc.) take the same path. Return false so
+ # the caller falls back to the printed instructions.
+ return $false
+ }
+}
+
+# Convenience wrapper: detect stale state, if stale prompt the user, on
+# yes invoke the elevated kill, re-probe, report. Either way return the
+# post-recovery state so the install script can decide whether to
+# proceed quietly or print the manual-recovery instructions.
+#
+# Designed to be a drop-in replacement for the
+# `Stop-CuaDriverDaemonsWithHealth + Show-CuaDriverDaemonSurvivors`
+# pair at the call site.
+function Repair-CuaDriverStaleDaemon {
+ [CmdletBinding()]
+ param(
+ # Default-true so install scripts auto-prompt. Pass `-AutoConfirm`
+ # to skip the y/n prompt (CI / automated runs) — UAC still prompts
+ # for elevation itself.
+ [switch]$AutoConfirm
+ )
+ $result = Stop-CuaDriverDaemonsWithHealth
+ if (-not $result.Stale) {
+ Show-CuaDriverDaemonSurvivors -Survivors $result.Survivors
+ return $result
+ }
+ # Stale: tell user what we see, offer to elevate.
+ $survivorPids = @($result.Survivors | ForEach-Object { $_.Id })
+ Write-Host ""
+ Write-Host "Detected STALE cua-driver daemon (pid: $($survivorPids -join ', '))." -ForegroundColor Yellow
+ Write-Host " Process is alive but \\.\pipe\cua-driver is not accepting connections." -ForegroundColor Yellow
+ Write-Host " Fixing this requires admin to terminate the High-IL daemon process." -ForegroundColor Yellow
+ Write-Host ""
+
+ $proceed = $AutoConfirm
+ if (-not $proceed) {
+ # Single-key prompt; default Y on Enter. Y/yes/[Enter] -> elevate,
+ # anything else -> skip elevation and fall through to manual-recovery
+ # instructions.
+ $ans = Read-Host " Trigger UAC prompt to kill the stale daemon now? [Y/n]"
+ $proceed = ($ans -eq '' -or $ans -match '^[Yy]')
+ }
+
+ if (-not $proceed) {
+ Show-CuaDriverDaemonSurvivors -Survivors $result.Survivors -Stale
+ return $result
+ }
+
+ Write-Host " Triggering UAC prompt (accept to kill the stale daemon)..." -ForegroundColor Cyan
+ $ok = Invoke-CuaDriverStaleDaemonKill -Pids $survivorPids
+ if ($ok) {
+ Write-Host " Stale daemon killed; new cua-driver-serve started; pipe is healthy." -ForegroundColor Green
+ return [pscustomobject]@{
+ Survivors = @()
+ PipeAlive = $true
+ Stale = $false
+ }
+ } else {
+ Write-Host " Elevated kill did not complete (UAC cancelled or pipe still dead)." -ForegroundColor Red
+ # Re-probe; if the user accepted UAC but the daemon restart
+ # failed, the survivors list may have shrunk to just the
+ # un-killable processes — re-show with current state.
+ $rep = Stop-CuaDriverDaemonsWithHealth
+ Show-CuaDriverDaemonSurvivors -Survivors $rep.Survivors -Stale:$rep.Stale
+ return $rep
+ }
+}
+
Export-ModuleMember -Function `
Stop-CuaDriverDaemons, `
+ Stop-CuaDriverDaemonsWithHealth, `
+ Test-CuaDriverPipeAlive, `
Show-CuaDriverDaemonSurvivors, `
+ Invoke-CuaDriverStaleDaemonKill, `
+ Repair-CuaDriverStaleDaemon, `
Import-CuaDriverInstallModule
diff --git a/libs/cua-driver/scripts/install-local.ps1 b/libs/cua-driver/scripts/install-local.ps1
index 6e4d73499d..2e45fde4a6 100644
--- a/libs/cua-driver/scripts/install-local.ps1
+++ b/libs/cua-driver/scripts/install-local.ps1
@@ -34,9 +34,19 @@
[CmdletBinding()]
param(
- [switch]$AutoStart,
+ # Default-on: most users want the daemon to come back at every
+ # logon. Opt out with `-AutoStart:$false` (or the dedicated
+ # `-NoAutoStart`) when running install-local.ps1 from CI / a
+ # container build / a sandbox where you specifically don't want
+ # a scheduled task registered.
+ [switch]$AutoStart = $true,
+ [switch]$NoAutoStart,
[switch]$NoPathUpdate
)
+# `-NoAutoStart` is the explicit opt-out; takes precedence over the
+# default-true `-AutoStart` for callers who'd rather read negative
+# than `-AutoStart:$false`.
+if ($NoAutoStart) { $AutoStart = $false }
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
@@ -198,8 +208,16 @@ if (Test-Path -LiteralPath $DestBinary) {
ForEach-Object { try { Remove-Item -LiteralPath $_.FullName -Force -ErrorAction SilentlyContinue } catch {} }
Write-Step "killing previous cua-driver processes (best-effort; High-IL needs admin)"
- $survivors = Stop-CuaDriverDaemons
- Show-CuaDriverDaemonSurvivors -Survivors $survivors
+ # Repair- variant does Stop-CuaDriverDaemonsWithHealth + stale
+ # detection + UAC self-elevation when wedged: if survivors are
+ # present AND the pipe is dead, prompts the user (y/n), and on
+ # yes triggers a UAC prompt to spawn a brief elevated
+ # PowerShell that kills the High-IL pids and re-runs the
+ # scheduled task. On UAC accept + healthy pipe afterward, the
+ # install proceeds at Medium IL like nothing happened. On UAC
+ # cancel / failure, falls back to the same printed manual
+ # recovery instructions the previous flow used.
+ $null = Repair-CuaDriverStaleDaemon
}
Write-Step "staging into $VersionedDir"
@@ -335,13 +353,23 @@ if (Test-Path -LiteralPath $HintsTxt) {
}
# Windows-specific autostart hint (kept inline; per-shell natural location).
-if (-not $AutoStart) {
+if ($AutoStart) {
+ # Default branch: autostart was enabled (either by default or explicitly).
+ # Surface the management subcommands so the user knows how to inspect /
+ # disable later without digging through Task Scheduler.
+ Write-Host ""
+ Write-Host "Auto-start: 'cua-driver-serve' is registered at RunLevel=Highest." -ForegroundColor Cyan
+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan
+ Write-Host ""
+} else {
+ # Opt-out branch (-NoAutoStart or -AutoStart:`$false`).
Write-Host ""
- Write-Host "Auto-start at logon (Windows-native equivalent of macOS LaunchAgent):" -ForegroundColor Cyan
+ Write-Host "Auto-start at logon (NOT enabled - re-run without -NoAutoStart to register, or:):" -ForegroundColor Cyan
Write-Host " cua-driver autostart enable (register Scheduled Task at RunLevel=Highest)" -ForegroundColor Cyan
Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan
Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan
Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan
- Write-Host " Or re-run install-local.ps1 with -AutoStart for the same result." -ForegroundColor Cyan
Write-Host ""
}
diff --git a/libs/cua-driver/scripts/install.ps1 b/libs/cua-driver/scripts/install.ps1
index 176c8fa820..60945bfd95 100644
--- a/libs/cua-driver/scripts/install.ps1
+++ b/libs/cua-driver/scripts/install.ps1
@@ -78,9 +78,19 @@
[CmdletBinding()]
param(
[string]$Release = "latest",
- [switch]$AutoStart,
+ # Default-on: cua-driver-serve is what makes the agent flow work
+ # across logon / reboot. Without the scheduled task the user has
+ # to remember to run `cua-driver autostart kick` every time, and
+ # MCP-style flows go silently in-process. Opt out with
+ # `-AutoStart:$false` or `-NoAutoStart` for CI / sandbox installs
+ # that specifically don't want a scheduled task registered.
+ [switch]$AutoStart = $true,
+ [switch]$NoAutoStart,
[switch]$NoPathUpdate
)
+# `-NoAutoStart` is the explicit opt-out and takes precedence over
+# the default-true `-AutoStart`.
+if ($NoAutoStart) { $AutoStart = $false }
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
@@ -1205,8 +1215,12 @@ else {
# Medium-IL kill and get reported via Show-CuaDriverDaemonSurvivors.
Write-Host ""
Write-Host "Stopping any previous cua-driver processes (best-effort; High-IL needs admin)..." -ForegroundColor Cyan
-$survivors = Stop-CuaDriverDaemons
-Show-CuaDriverDaemonSurvivors -Survivors $survivors
+# Repair- handles the wedged-daemon case: detect stale (process alive
+# but pipe dead), prompt the user, and on consent self-elevate via
+# UAC to kill the High-IL pids + restart the scheduled task. On UAC
+# cancel / failure it falls back to printing the manual recovery
+# instructions, same as the previous behavior.
+$null = Repair-CuaDriverStaleDaemon
if ($AutoStart) {
Write-Host ""
@@ -1268,12 +1282,18 @@ catch {
# Windows-specific autostart hint (kept inline; OS-natural location).
Write-Host ""
-Write-Host "Auto-start at logon (Windows-native equivalent of macOS LaunchAgent):" -ForegroundColor Cyan
-Write-Host " cua-driver autostart enable (Scheduled Task at RunLevel=Highest)" -ForegroundColor Cyan
-Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan
-Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan
-Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan
-Write-Host " Or re-run this installer with -AutoStart for the same result." -ForegroundColor Cyan
+if ($AutoStart) {
+ Write-Host "Auto-start: 'cua-driver-serve' is registered at RunLevel=Highest." -ForegroundColor Cyan
+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan
+} else {
+ Write-Host "Auto-start at logon (NOT enabled - re-run without -NoAutoStart to register, or:):" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart enable (Scheduled Task at RunLevel=Highest)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart kick (start now without re-logging)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart status (inspect)" -ForegroundColor Cyan
+ Write-Host " cua-driver autostart disable (remove)" -ForegroundColor Cyan
+}
Write-Host ""
Write-Host "WARNING -- BETA: cua-driver-rs is a cross-platform Rust port of the Swift" -ForegroundColor Yellow
Write-Host " cua-driver. Windows and Linux support is feature-complete; macOS" -ForegroundColor Yellow