Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

refactor(cua-driver): split Windows and Linux tools/impl_.rs into per-tool modules (#5957)

ruicong38dad235d3aa
diff --git a/.github/workflows/smoke-cua-driver-live.yml b/.github/workflows/smoke-cua-driver-live.yml
index e18c2fd57b..7132866bf9 100644
--- a/.github/workflows/smoke-cua-driver-live.yml
+++ b/.github/workflows/smoke-cua-driver-live.yml
@@ -20,9 +20,9 @@ on:
- "libs/cua-driver/rust/crates/cua-driver-core/src/snapshot_store.rs"
- "libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs"
- "libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs"
- - "libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs"
+ - "libs/cua-driver/rust/crates/platform-windows/src/tools/*.rs"
- "libs/cua-driver/rust/crates/platform-windows/src/win32/combo.rs"
- - "libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs"
+ - "libs/cua-driver/rust/crates/platform-linux/src/tools/*.rs"
- "libs/cua-driver/rust/crates/platform-linux/src/atspi/**"
permissions:
diff --git a/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md b/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
index c322988537..c1c9ef131e 100644
--- a/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
+++ b/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
@@ -13,12 +13,12 @@ the items listed.
| Project | Copyright | Used in | What |
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/impl_.rs`); `crates/cursor-overlay/src/motion.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default and the `with_overrides` method shape |
-| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/impl_.rs`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
+| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/`); `crates/cursor-overlay/src/motion.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default and the `with_overrides` method shape |
+| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
| [yabai](https://github.com/koekeishiya/yabai) | Copyright (c) 2019 Åsmund Vikane | `crates/platform-macos/src/input/skylight.rs` (`activate_without_raise`) | The focus-without-raise SkyLight event-record sequence |
| [QuickJS-ng](https://github.com/quickjs-ng/quickjs) | Copyright (c) 2017-2026 Fabrice Bellard; (c) 2017-2024 Charlie Gordon; (c) 2023-2026 Ben Noordhuis; (c) 2023-2026 Saúl Ibarra Corretgé | `crates/cua-driver-core/src/script_tool/` (compiled in through rquickjs-sys) | The JavaScript engine behind the experimental `run_script` tool |
| [rquickjs](https://github.com/DelSkayn/rquickjs) | Copyright (c) 2020 Mees Delzenne; (c) 2025 Rquickjs Contributors | `crates/cua-driver-core/src/script_tool/` | Rust bindings to QuickJS-ng |
-| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name`); `crates/platform-linux/src/wayland/shell_helper.rs` and `shell_helper/transport.rs`; `crates/platform-linux/src/wayland/libei.rs` (input acknowledgement); `crates/platform-linux/src/wayland/mod.rs` and `src/tools/impl_.rs` (GNOME window capture); `../wayland-helper/winrects@cua/` (`CaptureWindow`, `CaptureWindowPreview`) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through, the persistent session-bus transport for the GNOME helper, waiting for EIS to acknowledge libei input, and per-window GNOME capture with compositor-side preview downscaling (helper API 9 and 10) |
+| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name`); `crates/platform-linux/src/wayland/shell_helper.rs` and `shell_helper/transport.rs`; `crates/platform-linux/src/wayland/libei.rs` (input acknowledgement); `crates/platform-linux/src/wayland/mod.rs` and `src/tools/` (GNOME window capture); `../wayland-helper/winrects@cua/` (`CaptureWindow`, `CaptureWindowPreview`) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through, the persistent session-bus transport for the GNOME helper, waiting for EIS to acknowledge libei input, and per-window GNOME capture with compositor-side preview downscaling (helper API 9 and 10) |
## MIT License
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_libreoffice_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_libreoffice_test.rs
index a194b9dcd3..6638d9a178 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_libreoffice_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_libreoffice_test.rs
@@ -150,7 +150,7 @@ fn setup() -> LoSession {
/// - (1) MSAA walker not finding Font Color (check msaa.rs walker
/// budget / depth, or LO renamed the button).
/// - (2) cua-driver click tool's MSAA dispatch broke (check
-/// `tools/impl_.rs` BUTTONDROPDOWN branch).
+/// `tools/click.rs` BUTTONDROPDOWN branch).
/// - (3) Right-edge offset wrong for current LO version's toolbar
/// scale (check `rect.right - 4` heuristic in
/// `tools/impl_.rs`).
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_winui3_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_winui3_test.rs
index 70302c8269..91f91d271a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_winui3_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_winui3_test.rs
@@ -564,7 +564,7 @@ fn harness_winui3_radio_select() {
/// Documents cua-driver gap: WinUI3 Slider implements
/// `RangeValuePattern`, not `ValuePattern`. cua-driver's `set_value` tool
-/// queries `ValuePatternId` specifically (impl_.rs:2640), so it silently
+/// queries `ValuePatternId` specifically (tools/set_value.rs), so it silently
/// fails on RangeValuePattern-only elements. Real fix: cua-driver should
/// try RangeValuePattern.SetValue (coercing the string to a double) when
/// ValuePattern isn't found.
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/bring_to_front.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/bring_to_front.rs
new file mode 100644
index 0000000000..32e6d37181
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/bring_to_front.rs
@@ -0,0 +1,127 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::Value;
+
+use super::impl_::{foreground_hyprland_action, foreground_hyprland_refusal};
+
+// ── bring_to_front (Linux) ───────────────────────────────────────────────────
+
+pub struct BringToFrontTool;
+
+static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for BringToFrontTool {
+ fn def(&self) -> &ToolDef {
+ BTF_DEF.get_or_init(|| ToolDef {
+ name: "bring_to_front".into(),
+ description:
+ "Persistently activate a window so subsequent input lands on it. \
+ This deliberately breaks the no-foreground contract and is not part \
+ of the normal input ladder. For an ordinary `background_unavailable` \
+ response, retry only the refused action with \
+ `delivery_mode:\"foreground\"`; the input tool performs its own \
+ activate, act, and restore sequence. Use `bring_to_front` only for a \
+ focus-proxy surface that must remain foreground across multiple calls, \
+ such as a remote desktop session, or when repeated action-scoped \
+ activation prevents the remote surface from accepting input. \
+ X11: EWMH _NET_ACTIVE_WINDOW activation (the `wmctrl -a` equivalent, \
+ with proper timestamp handling to beat focus-stealing prevention). \
+ Wayland: activates through a target-addressable compositor adapter \
+ (wlroots foreign-toplevel or the GNOME Shell helper) and refuses \
+ when the compositor offers no safe adapter. Matches the macOS / \
+ Windows bring_to_front rung."
+ .into(),
+ input_schema: serde_json::json!({
+ "type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"Process ID of the app to activate."},
+ "window_id":{"type":"integer","description":"X11 window id (xid) to activate. If omitted, the first window of `pid` is used."}
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let pid = args.u64_or("pid", 0) as u32;
+ if crate::wayland::is_wayland() {
+ let window_id = match args.opt_u64("window_id") {
+ Some(window_id) => window_id,
+ None => match crate::wayland::list_windows_dispatch(Some(pid)).first() {
+ Some(window) => window.xid,
+ None => {
+ return ToolResult::error(format!(
+ "bring_to_front: no window_id given and no Wayland windows found for pid {pid}."
+ ))
+ }
+ },
+ };
+ if crate::wayland::wayland_input_enabled() && crate::wayland::hyprland::is_session() {
+ if args.opt_u64("window_id").is_none() {
+ return foreground_hyprland_refusal("an exact window_id is required");
+ }
+ return foreground_hyprland_action(
+ &args,
+ pid,
+ window_id,
+ crate::wayland::hyprland_input::Action::Activate,
+ )
+ .await;
+ }
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::activate_window_for_input_target(window_id, Some(pid))
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => {
+ ToolResult::text(format!("Brought Wayland window {window_id} to front."))
+ .with_structured(serde_json::json!({
+ "window_id": window_id,
+ "platform": "linux",
+ "session": "wayland",
+ }))
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ // X11: resolve the target xid (window_id, else first window for pid).
+ let xid = match args.opt_u64("window_id") {
+ Some(x) => x,
+ None => {
+ let windows =
+ tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.xid,
+ None => {
+ return ToolResult::error(format!(
+ "bring_to_front: no window_id given and no windows found for pid {pid}."
+ ))
+ }
+ }
+ }
+ };
+ let r =
+ tokio::task::spawn_blocking(move || crate::input::x11_activate_window_persistent(xid))
+ .await;
+ match r {
+ Ok(Ok(prior)) => ToolResult::text(format!(
+ "✅ Brought window {xid} to front (X11 _NET_ACTIVE_WINDOW)."
+ ))
+ .with_structured(serde_json::json!({
+ "window_id": xid,
+ "prior_active": prior,
+ "platform": "linux",
+ })),
+ Ok(Err(e)) => ToolResult::error(format!("bring_to_front failed: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/check_permissions.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/check_permissions.rs
new file mode 100644
index 0000000000..a7509ea49f
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/check_permissions.rs
@@ -0,0 +1,78 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── check_permissions ─────────────────────────────────────────────────────────
+
+pub struct CheckPermissionsTool;
+static PERMS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for CheckPermissionsTool {
+ fn def(&self) -> &ToolDef {
+ PERMS_DEF.get_or_init(|| ToolDef {
+ name: "check_permissions".into(),
+ description: "Check required permissions for cua-driver-rs on Linux.".into(),
+ input_schema: json!({"type":"object","properties":{"prompt":{"type":"boolean","description":"Ignored: Linux has no permission prompts."}},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ // Check X11 connectivity (required for window enumeration and input injection).
+ let x11_ok = tokio::task::spawn_blocking(|| {
+ x11rb::rust_connection::RustConnection::connect(None).is_ok()
+ })
+ .await
+ .unwrap_or(false);
+
+ // Check AT-SPI: not merely "is there a session bus?" but "does
+ // org.a11y.Bus actually answer on it?" — the previous env-var-or-
+ // /run/user heuristic false-passed exactly the headless/container case
+ // (/run/user exists, but no a11y bus → empty trees). Probe for real.
+ let dbus_address = std::env::var("DBUS_SESSION_BUS_ADDRESS").ok();
+ let atspi_ok = tokio::task::spawn_blocking(crate::health_report::probe_a11y_bus)
+ .await
+ .unwrap_or(false);
+
+ let wayland_display = std::env::var("WAYLAND_DISPLAY").ok();
+ let atspi_status = if atspi_ok {
+ match &dbus_address {
+ Some(a) => format!("✅ org.a11y.Bus reachable (DBUS_SESSION_BUS_ADDRESS={a})"),
+ None => "✅ org.a11y.Bus reachable".to_string(),
+ }
+ } else if dbus_address.is_none() {
+ "❌ no session bus (DBUS_SESSION_BUS_ADDRESS unset and none auto-discovered) — \
+ AT-SPI trees will be empty; start the daemon inside the desktop session"
+ .to_string()
+ } else {
+ "❌ session bus present but org.a11y.Bus has no owner — enable accessibility \
+ (gsettings set org.gnome.desktop.interface toolkit-accessibility true) / \
+ start at-spi-bus-launcher"
+ .to_string()
+ };
+ let status_text = format!(
+ "X11 display: {}\nWayland: {}\nAT-SPI (D-Bus): {}\nXSendEvent injection: {}",
+ if x11_ok { "✅ connected" } else { "❌ DISPLAY not set or X11 unavailable" },
+ match &wayland_display {
+ Some(s) if crate::wayland::wayland_enabled() =>
+ format!("✅ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend ENABLED"),
+ Some(s) => format!(
+ "⚠️ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend OFF; \
+ set {}=1 to enable it",
+ crate::wayland::ENABLE_WAYLAND_ENV
+ ),
+ None => "❌ not a Wayland session".to_string(),
+ },
+ atspi_status,
+ if x11_ok { "✅ available" } else { "❌ requires X11" }
+ );
+ ToolResult::text(status_text)
+ .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "wayland_enabled": crate::wayland::wayland_enabled(), "atspi": atspi_ok, "dbus_session_bus_address": dbus_address, "xsend_event": x11_ok }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/click.rs
new file mode 100644
index 0000000000..a4bb776cbc
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/click.rs
@@ -0,0 +1,1857 @@
+use async_trait::async_trait;
+use cua_driver_contract::ClickButton;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::{parse_legacy_click_input, ArgsExt as _},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ apply_foreground_report, attach_focus_guard, capture_admission_error,
+ checked_element_local_coords, coordinate_frame_schema, desktop_frame_requested,
+ desktop_frame_unmapped, desktop_input_space, desktop_to_window_local, element_is_menu_role,
+ element_needs_real_click, element_screen_center, foreground_budget, foreground_hyprland_action,
+ foreground_hyprland_refusal, foreground_structured, hyprland_foreground, input_error_result,
+ isolated_hyprland_action, isolated_hyprland_background, isolated_hyprland_refusal,
+ isolated_hyprland_result, isolated_hyprland_task_error, named_session_cursor_key,
+ non_ax_escalation, parse_mouse_button, prove_token_element, push_evidence, require_point_args,
+ resolve_cursor_key, resolve_element_local_coords, reveal_pointer_action_for,
+ reveal_pointer_action_target_for, screenshot_scale, spawn_blocking_bounded,
+ spawn_isolated_hyprland, synthetic_pointer_is_dropped, unavailable_chromium_background,
+ unavailable_wayland_focused_input_background, window_local_to_screen,
+ window_scope_unproven_refusal, with_pixel_points, x11_pixel_click_no_focus_steal, AimedPoint,
+ CoordinateContext, PointerRoute, ToolState,
+};
+
+#[cfg(test)]
+use super::impl_::{
+ chromium_background_must_refuse, maps_indicate_gtk, stale_observed_element, token_element_for,
+};
+
+/// A snapshot frame `(x, y, w, h)` as the overlay target rect, only when it
+/// is non-empty and actually contains the glide point (a redirected press or
+/// a frame in another space gets no rect rather than a wrong one).
+fn frame_target_rect(bounds: Option<(i32, i32, u32, u32)>, sx: f64, sy: f64) -> Option<[f64; 4]> {
+ let (x, y, w, h) = bounds?;
+ let rect = [f64::from(x), f64::from(y), f64::from(w), f64::from(h)];
+ (w > 0
+ && h > 0
+ && sx >= rect[0]
+ && sy >= rect[1]
+ && sx <= rect[0] + rect[2]
+ && sy <= rect[1] + rect[3])
+ .then_some(rect)
+}
+
+/// The cursor glides to an element before its click. A failed lookup skips
+/// only the glide, except a window-membership failure, which refuses.
+fn element_click_placement(
+ placement: anyhow::Result<(u64, f64, f64)>,
+ xid: Option<u64>,
+) -> Result<Option<(u64, f64, f64)>, ToolResult> {
+ match placement {
+ Ok(point) => Ok(Some(point)),
+ Err(error) if crate::atspi::is_window_membership_unproven(&error) => Err(
+ window_scope_unproven_refusal("click", xid.unwrap_or(0), &error.to_string()),
+ ),
+ Err(_) => Ok(None),
+ }
+}
+
+#[cfg(test)]
+mod token_window_scope_tests {
+ use super::{
+ element_click_placement, prove_token_element, stale_observed_element, token_element_for,
+ };
+ use crate::atspi::native::{window_element_membership, ObjectRef};
+ use crate::atspi::snapshot::{cached_element, forget_window, update_snapshot, CachedElement};
+ use crate::atspi::{AtspiIdentity, AtspiNode, Snapshots, TokenElement, WindowProof};
+ use cua_driver_core::element_token::{token_for, ResolvedElement};
+ use cua_driver_core::protocol::ToolResult;
+
+ // One process owns windows A and B; each has a text field at the same index.
+ const PID: u32 = 515_151;
+ const WINDOW_A: u64 = 1;
+ const WINDOW_B: u64 = 2;
+ const INDEX: usize = 3;
+
+ fn identity(path: &str) -> AtspiIdentity {
+ AtspiIdentity {
+ bus_name: ":1.42".into(),
+ path: path.into(),
+ frame_bus_name: ":1.42".into(),
+ frame_path: "/frame".into(),
+ }
+ }
+
+ fn object_ref(path: &str) -> Option<ObjectRef> {
+ Some(ObjectRef {
+ bus: ":1.42".into(),
+ path: path.into(),
+ })
+ }
+
+ fn observed(path: &str) -> CachedElement {
+ CachedElement {
+ key: 0,
+ identity: identity(path),
+ object_ref: object_ref(path),
+ role: "text".into(),
+ in_web_content: false,
+ bounds: None,
+ }
+ }
+
+ fn text_field(path: &str) -> AtspiNode {
+ AtspiNode {
+ element_index: Some(INDEX),
+ role: "text".into(),
+ name: None,
+ value: None,
+ checked: None,
+ enabled: Some(true),
+ selected: None,
+ description: None,
+ actions: Vec::new(),
+ element_key: 0,
+ identity: Some(identity(path)),
+ depth: 0,
+ parent_element_index: None,
+ in_web_content: false,
+ object_ref: object_ref(path),
+ }
+ }
+
+ fn refusal_code(refusal: &ToolResult) -> &str {
+ assert_eq!(refusal.is_error, Some(true));
+ let structured = refusal.structured_content.as_ref().expect("structured");
+ assert_eq!(structured["status"], "refused");
+ structured["refusal"]["code"]
+ .as_str()
+ .expect("refusal code")
+ }
+
+ #[test]
+ fn an_unproven_window_element_refuses_before_dispatch() {
+ // The membership walk no longer finds the element in window 2's scope.
+ let proven = |membership: anyhow::Result<()>| membership.map(|()| WindowProof::Proven);
+ let unproven = proven(window_element_membership(Some(0), None, || false));
+ let refusal =
+ token_element_for("click", 2, 3, observed("/a"), unproven).expect_err("refused");
+ assert_eq!(refusal_code(&refusal), "window_scope_unproven");
+ let gone = proven(window_element_membership(None, Some(0), || true));
+ let refusal =
+ token_element_for("type_text", 2, 3, observed("/a"), gone).expect_err("refused");
+ assert_eq!(refusal_code(&refusal), "window_scope_unproven");
+ assert!(token_element_for("click", 2, 3, observed("/a"), Ok(WindowProof::Proven)).is_ok());
+ }
+
+ #[test]
+ fn a_membership_failure_in_the_click_glide_refuses_instead_of_acting() {
+ let unproven = window_element_membership(Some(0), None, || false).map(|()| (2, 0.0, 0.0));
+ let refusal = element_click_placement(unproven, Some(2)).expect_err("refused");
+ assert_eq!(refusal_code(&refusal), "window_scope_unproven");
+ // Any other lookup failure only skips the cursor glide.
+ let other = element_click_placement(
+ Err(anyhow::anyhow!("element exposes no Component interface")),
+ Some(2),
+ );
+ assert!(matches!(other, Ok(None)));
+ assert!(matches!(
+ element_click_placement(Ok((2, 10.0, 20.0)), Some(2)),
+ Ok(Some((2, x, y))) if x == 10.0 && y == 20.0
+ ));
+ }
+
+ #[tokio::test]
+ async fn a_target_without_an_element_token_needs_no_membership_walk() {
+ let none = ResolvedElement::<CachedElement>::None;
+ assert!(matches!(
+ prove_token_element(1, &none, "click").await,
+ Ok(None)
+ ));
+ }
+
+ #[test]
+ fn a_proven_token_dispatches_to_its_own_element_while_another_window_is_retained() {
+ // Both windows' snapshots are retained: in the runtime store and in the
+ // (pid, window) side index.
+ let snapshots = Snapshots::new();
+ let tokens =
+ [(WINDOW_A, "/window_a/text"), (WINDOW_B, "/window_b/text")].map(|(window, path)| {
+ let payload = update_snapshot(PID, window, &[text_field(path)], &[]);
+ (snapshots.publish(PID as i32, window, payload), window, path)
+ });
+ let mut pid_only = Vec::new();
+ for (snapshot, window, path) in tokens {
+ let args = serde_json::json!({ "element_token": token_for(snapshot, INDEX) });
+ let Ok(ResolvedElement::Element {
+ window_id,
+ element_index,
+ element,
+ }) = snapshots.resolve(PID as i32, &args)
+ else {
+ panic!("the token for window {window} resolves to its element");
+ };
+ assert_eq!(window_id, window);
+ // The fresh walk scoped to the window finds the observed identity.
+ let proof =
+ window_element_membership(Some(0), Some(0), || true).map(|()| WindowProof::Proven);
+ let token = token_element_for("type_text", window_id, element_index, element, proof)
+ .expect("membership proven");
+ // Text, focus, value and scroll dispatch address this object.
+ assert_eq!(token.object().expect("addressable").path, path);
+ assert!(!token.may_reresolve, "a proven token never re-resolves");
+ pid_only.push(cached_element(PID, None, INDEX).and_then(|element| element.object_ref));
+ }
+ // The PID-only lookup the earlier dispatch used gives both tokens the same
+ // window's element, so it acted on the wrong window for one of them.
+ assert_eq!(pid_only[0], pid_only[1]);
+ forget_window(PID, WINDOW_A);
+ forget_window(PID, WINDOW_B);
+ }
+
+ #[test]
+ fn only_an_unproven_session_may_re_resolve_a_token_index() {
+ let proven = TokenElement::new(INDEX, observed("/window_b/text"), WindowProof::Proven);
+ assert!(!proven.may_reresolve);
+ let unproven =
+ TokenElement::new(INDEX, observed("/window_b/text"), WindowProof::NotRequired);
+ assert!(unproven.may_reresolve);
+ }
+
+ #[test]
+ fn a_gone_observed_object_refuses_as_a_stale_token() {
+ let mut gone = observed("/window_b/text");
+ gone.object_ref = None;
+ let token = TokenElement::new(INDEX, gone, WindowProof::Proven);
+ let error = token.object().expect_err("no D-Bus address");
+ let refusal = stale_observed_element("type_text", INDEX, &error).expect("stale");
+ assert_eq!(
+ refusal.structured_content.expect("structured")["code"],
+ "stale_element_token"
+ );
+ // A write the toolkit rejected is not a stale token.
+ let rejected = anyhow::anyhow!("EditableText write was rejected");
+ assert!(stale_observed_element("type_text", INDEX, &rejected).is_none());
+ }
+}
+
+/// `scope:"desktop"` alongside a pid/window target: the caller read `x,y` off
+/// `get_desktop_state` (screen pixels) but still names the window to act on.
+/// Upper bound for one AT-SPI element operation (bounds lookup / action).
+/// The AT-SPI layer bounds single bus calls, but a full tree walk over a large
+/// GTK/VCL window chains many of them; observed stalls ran to minutes.
+const ELEMENT_AX_BUDGET: std::time::Duration = std::time::Duration::from_secs(30);
+
+/// After a pointer press: name the grid cell that holds the widget focus now
+/// (LibreOffice reports "D2"), so a one-row miss shows in the click result
+/// instead of after six typed values. Read from the focus-event log only,
+/// so it costs nothing when the press did not land on a grid.
+async fn attach_focused_cell(mut result: ToolResult, pid: u32) -> ToolResult {
+ if result.is_error == Some(true) {
+ return result;
+ }
+ let focused = tokio::task::spawn_blocking(move || crate::atspi::focused_control(pid))
+ .await
+ .ok()
+ .flatten();
+ let Some((role, name)) = focused.filter(|(role, name)| {
+ let role = role.to_ascii_lowercase();
+ (role.contains("cell") || role == "table cell") && !name.trim().is_empty()
+ }) else {
+ return result;
+ };
+ result
+ .content
+ .push(cua_driver_core::protocol::Content::text(format!(
+ "focus: cell {name}"
+ )));
+ let mut structured = result
+ .structured_content
+ .take()
+ .unwrap_or_else(|| json!({}));
+ structured["focused_cell"] = json!(name);
+ push_evidence(
+ &mut structured,
+ json!({
+ "kind": "native_api_result",
+ "detail": format!("the accessible holding the focus after the press is {role} \"{name}\""),
+ }),
+ );
+ result.structured_content = Some(structured);
+ result
+}
+
+fn coordinate_click_context(
+ native_refusal: Option<ToolResult>,
+ resolve_context: impl FnOnce() -> Result<CoordinateContext, ToolResult>,
+) -> Result<CoordinateContext, ToolResult> {
+ if let Some(refusal) = native_refusal {
+ return Err(refusal);
+ }
+ resolve_context()
+}
+
+fn element_click_prefers_ax(
+ hyprland_foreground: bool,
+ button: u8,
+ count: usize,
+ has_modifiers: bool,
+) -> bool {
+ !has_modifiers && (!hyprland_foreground || (button == 1 && count == 1))
+}
+
+fn foreground_ax_click_unknown(pid: u32, idx: usize) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
+ };
+ let record = ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::LinuxAtSpiAction,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Unknown)
+ .build()
+ .expect("uncertain semantic activation has no delivered count or replay");
+ let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
+ ToolResult::error(format!(
+ "click: AT-SPI activation outcome is unknown for element [{idx}] (pid {pid}); refresh state before another action."
+ ))
+ .with_structured(public)
+ .with_action_record(record)
+}
+
+#[cfg(test)]
+#[test]
+fn foreground_ax_click_error_does_not_claim_delivery_or_request_replay() {
+ let result = foreground_ax_click_unknown(123, 4);
+ assert_eq!(result.is_error, Some(true));
+ let wire = serde_json::to_value(&result).unwrap();
+ assert_eq!(wire["isError"], true);
+ let public = &wire["structuredContent"];
+ assert_eq!(
+ *public,
+ serde_json::to_value(result.action_record.unwrap().public_result().unwrap()).unwrap()
+ );
+ assert_eq!(public["effect"], "unverifiable");
+ assert_eq!(public["route"], "accessibility");
+ assert_eq!(public["delivery"]["mode"], "unknown");
+ assert!(public.get("escalation").is_none());
+ assert!(public["delivery"].get("delivered_count").is_none());
+}
+
+#[cfg(test)]
+#[test]
+fn hyprland_foreground_semantic_click_preserves_pointer_gestures() {
+ assert!(element_click_prefers_ax(true, 1, 1, false));
+ for (button, count) in [(2, 1), (3, 1), (1, 0), (1, 2), (1, 3), (2, 2), (3, 3)] {
+ assert!(!element_click_prefers_ax(true, button, count, false));
+ }
+ // Preserve the existing AT-SPI eligibility for all other delivery routes.
+ for button in [1, 2, 3] {
+ for count in [0, 1, 2, 3] {
+ assert!(element_click_prefers_ax(false, button, count, false));
+ assert!(!element_click_prefers_ax(false, button, count, true));
+ assert!(!element_click_prefers_ax(true, button, count, true));
+ }
+ }
+}
+
+#[cfg(test)]
+#[test]
+fn coordinate_click_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
+ use std::sync::atomic::{AtomicBool, Ordering};
+
+ let resolved = AtomicBool::new(false);
+ let native_refusal = crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ let result = coordinate_click_context(Some(native_refusal), || {
+ resolved.store(true, Ordering::SeqCst);
+ Ok(CoordinateContext::Screenshot(7.35))
+ })
+ .expect_err("unsupported Wayland background delivery must refuse");
+
+ assert_eq!(
+ result.structured_content.as_ref().unwrap()["code"],
+ "background_unavailable"
+ );
+ assert!(
+ !resolved.load(Ordering::SeqCst),
+ "snapshot context must not be consulted before the native refusal"
+ );
+}
+
+fn x11_pixel_click_no_focus_steal_modifiers(
+ xid: u64,
+ lx: i32,
+ ly: i32,
+ button: u8,
+ count: usize,
+ modifiers: &[&str],
+) -> anyhow::Result<()> {
+ crate::input::send_click_with_modifiers(xid, lx, ly, count, button, modifiers)
+}
+
+// ── click ─────────────────────────────────────────────────────────────────────
+
+pub struct ClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+/// Presses in one click train: 1 to 3, as the contract states. A niri click
+/// train is one gated action, so a larger count would mean seconds of
+/// presses no recheck covers.
+fn click_train(requested: u64) -> usize {
+ requested.clamp(1, 3) as usize
+}
+
+#[cfg(test)]
+#[test]
+fn click_trains_stop_at_the_advertised_triple_click() {
+ // A niri click train is rechecked once, before its first press, so a
+ // `count: 20` must not become seconds of ungated presses.
+ let tool = ClickTool {
+ state: ToolState::new(),
+ };
+ let count = &tool.def().input_schema["properties"]["count"];
+ assert_eq!(
+ (count["minimum"].as_u64(), count["maximum"].as_u64()),
+ (Some(1), Some(3))
+ );
+ assert_eq!(click_train(20), 3);
+ assert_eq!(click_train(2), 2);
+}
+
+/// Where a foreground element click may put the real pointer.
+enum ForegroundElementPlacement {
+ /// The element centre lies inside the target window (or a popup of the
+ /// same process): XTest-click it there.
+ Point { xid: u64, sx: f64, sy: f64 },
+ /// The element has no usable on-screen bounds (its menu is closed, its
+ /// page is unmapped, its centre is outside every window of the process):
+ /// only the AT-SPI action can reach it.
+ NoPoint { reason: String },
+}
+
+impl ClickTool {
+ /// X11 foreground element click. Behave like a user when the element is
+ /// really on screen: activate the window and XTest-click the element's
+ /// centre. A degenerate centre — the walk answering (0,0) for a menu item
+ /// whose menu is closed used to glide the real pointer into the GNOME hot
+ /// corner — is never clicked; the observed object's AT-SPI action is fired
+ /// instead (`path: "ax_fg"`, a label the core's action record maps to
+ /// the AT-SPI transport), with the foreground activation kept, and when
+ /// that is not possible either the call is refused
+ /// (`element_bounds_unavailable`).
+ #[allow(clippy::too_many_arguments)]
+ async fn foreground_element_click(
+ &self,
+ cursor_id: &str,
+ pid: u32,
+ idx: usize,
+ observed: &crate::atspi::snapshot::CachedElement,
+ xid_hint: Option<u64>,
+ placement: Option<(u64, f64, f64)>,
+ button: u8,
+ count: usize,
+ modifiers: &[String],
+ ) -> ToolResult {
+ let observed_for_state = observed.clone();
+ let decided = tokio::task::spawn_blocking(move || {
+ decide_foreground_element_placement(pid, &observed_for_state, xid_hint, placement)
+ })
+ .await;
+ let (xid, decision) = match decided {
+ Ok(v) => v,
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ };
+ match decision {
+ ForegroundElementPlacement::Point { xid, sx, sy } => {
+ // Ask the toolkit who actually owns this screen point before
+ // pressing it: a snapshot frame is where the toolkit *said*
+ // the element is, and a hidden sibling view (a QFileDialog's
+ // off-page QListView mirroring the visible QTreeView) can
+ // report a frame nothing is drawn at. See atspi::native::hit_test.
+ // Web content is exempt: Chromium's GetAccessibleAtPoint
+ // answers with an unrelated `panel` for web elements, so it
+ // cannot vouch for (or against) the point.
+ let owned_ref = observed
+ .object_ref
+ .clone()
+ .filter(|_| !observed.in_web_content);
+ let (sx, sy, redirect_note) = if let Some(oref) = owned_ref {
+ let (pid_c, oref_c, sx_i, sy_i) =
+ (pid, oref, sx.round() as i32, sy.round() as i32);
+ let ownership = tokio::task::spawn_blocking(move || {
+ crate::atspi::native::hit_test::point_ownership(
+ pid_c, xid, &oref_c, sx_i, sy_i,
+ )
+ })
+ .await
+ .unwrap_or(crate::atspi::native::hit_test::PointOwnership::Unknown);
+ match ownership {
+ crate::atspi::native::hit_test::PointOwnership::Redirect {
+ target,
+ screen_point,
+ ..
+ } => (
+ screen_point.0 as f64,
+ screen_point.1 as f64,
+ Some(format!("redirected the press to {}", target.describe())),
+ ),
+ crate::atspi::native::hit_test::PointOwnership::Mismatch { owner } => {
+ return ToolResult::error(format!(
+ "click: element [{idx}] (pid {pid}) has a snapshot frame at \
+ ({sx_i}, {sy_i}), but that screen point is owned by {} rather \
+ than the observed element; a real pointer click there would hit \
+ the wrong control. Re-snapshot with get_window_state, or click by \
+ pixel from a screenshot.",
+ owner.describe()
+ ))
+ .with_structured(json!({
+ "code": "element_bounds_unavailable",
+ "effect": "none",
+ "element_index": idx,
+ "reason": "point_owned_by_another_element",
+ "owner": owner.describe(),
+ }));
+ }
+ _ => (sx, sy, None),
+ }
+ } else {
+ (sx, sy, None)
+ };
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ // A redirected press targets another control: no rect.
+ let target_rect = redirect_note
+ .is_none()
+ .then(|| frame_target_rect(observed.bounds, sx, sy))
+ .flatten();
+ reveal_pointer_action_target_for(&self.state, cursor_id, sx, sy, true, target_rect)
+ .await;
+ let modifier_owned = modifiers.to_vec();
+ let result = spawn_blocking_bounded(
+ "foreground element click",
+ foreground_budget(0),
+ move || {
+ let modifier_refs: Vec<&str> =
+ modifier_owned.iter().map(String::as_str).collect();
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx.round() as i32,
+ sy.round() as i32,
+ button,
+ count,
+ &modifier_refs,
+ )
+ },
+ )
+ },
+ )
+ .await;
+ let window_point = crate::x11::window_info(xid)
+ .map(|w| (sx.round() as i32 - w.x, sy.round() as i32 - w.y))
+ .unwrap_or((sx.round() as i32, sy.round() as i32));
+ match result {
+ Ok(Ok(((), report))) => {
+ let confirmed = report.window_change_confirms();
+ let result = ToolResult::text(format!(
+ "Clicked element [{idx}] (pid {pid}) with a real pointer click \
+ (delivery_mode=foreground, focus_after={}){}{}",
+ report.focus_after.as_str(),
+ if confirmed {
+ format!("; {}.", report.window_change.as_deref().unwrap_or(""))
+ } else {
+ "; not verified — confirm with a screenshot.".to_owned()
+ },
+ redirect_note
+ .as_deref()
+ .map(|n| format!(" ({n})."))
+ .unwrap_or_default(),
+ ))
+ .with_structured(foreground_structured(
+ "x11_xtest_fg",
+ report,
+ serde_json::Map::new(),
+ ));
+ with_pixel_points(
+ result,
+ Some((sx.round() as i32, sy.round() as i32)),
+ window_point,
+ )
+ }
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ ForegroundElementPlacement::NoPoint { reason } => {
+ let Some(xid) = xid else {
+ return ToolResult::error(format!(
+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
+ ({reason}) and pid {pid} has no window to activate; no input was sent."
+ ))
+ .with_structured(json!({
+ "code": "element_bounds_unavailable",
+ "effect": "none",
+ "element_index": idx,
+ "reason": reason,
+ }));
+ };
+ if observed.object_ref.is_none()
+ || button != 1
+ || count != 1
+ || !modifiers.is_empty()
+ {
+ return ToolResult::error(format!(
+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
+ ({reason}); a real pointer click cannot reach it and this click \
+ shape (button {button} × {count}{}) has no AT-SPI action form. No \
+ input was sent. Open the containing menu first, re-snapshot with \
+ get_window_state, or click by pixel from a screenshot.",
+ if modifiers.is_empty() { String::new() } else { format!(", modifiers {modifiers:?}") }
+ ))
+ .with_structured(json!({
+ "code": "element_bounds_unavailable",
+ "effect": "none",
+ "element_index": idx,
+ "reason": reason,
+ "suggestion": "open the containing menu first, or re-snapshot with get_window_state and click by x/y",
+ }));
+ }
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ let observed_for_ax = observed.clone();
+ let result = spawn_blocking_bounded(
+ "foreground element action",
+ foreground_budget(0),
+ move || {
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || crate::atspi::perform_action_observed(&observed_for_ax),
+ )
+ },
+ )
+ .await;
+ match result {
+ Ok(Ok(((action, suspected_noop, unacknowledged), report))) => {
+ let mut extra = serde_json::Map::new();
+ extra.insert("ax_action".into(), json!(action));
+ extra.insert("bounds_reason".into(), json!(reason));
+ if suspected_noop {
+ extra.insert("effect".into(), json!("suspected_noop"));
+ extra.insert("escalation".into(), non_ax_escalation());
+ } else if unacknowledged {
+ // No doAction reply arrived in time: never let a
+ // coincidental window-change observation promote
+ // this to `confirmed` — the AT-SPI layer itself
+ // never confirmed dispatch, let alone effect.
+ extra.insert("effect".into(), json!("unverifiable"));
+ extra.insert("verified".into(), json!(false));
+ extra.insert(
+ "note".into(),
+ json!(
+ "no acknowledgement received for the doAction call within \
+ the AT-SPI timeout; its effect is unknown"
+ ),
+ );
+ }
+ ToolResult::text(format!(
+ "Clicked element [{idx}] (pid {pid}) through its AT-SPI action \
+ \"{action}\" with the window activated (delivery_mode=foreground, \
+ path=ax_fg, focus_after={}): the element had no usable \
+ on-screen bounds ({reason}), so no pointer was moved.{}",
+ report.focus_after.as_str(),
+ match &report.window_change {
+ Some(change) => format!(" {change}."),
+ None => " Not verified — confirm with a screenshot.".to_owned(),
+ }
+ ))
+ .with_structured(foreground_structured("ax_fg", report, extra))
+ }
+ Ok(Err(error)) => {
+ if error.is::<crate::atspi::native::CachedElementGone>() {
+ return ToolResult::error(format!(
+ "click: stale_element_token: observed AT-SPI object [{idx}] is no \
+ longer present ({error}); re-snapshot with get_window_state"
+ ))
+ .with_structured(json!({
+ "code": "stale_element_token",
+ "effect": "none",
+ }));
+ }
+ if crate::input::foreground::error_code(&error).is_some() {
+ return input_error_result(error);
+ }
+ ToolResult::error(format!(
+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
+ ({reason}) and its AT-SPI action failed ({error:#}); no input was \
+ sent. Open the containing menu first, re-snapshot with \
+ get_window_state, or click by pixel from a screenshot."
+ ))
+ .with_structured(json!({
+ "code": "element_bounds_unavailable",
+ "effect": "none",
+ "element_index": idx,
+ "reason": reason,
+ "ax_error": format!("{error:#}"),
+ }))
+ }
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ }
+ }
+}
+
+/// Decide whether a foreground element click has a real point to press.
+/// Returns the window to activate (when the process has one) and the
+/// placement decision. Runs blocking X11 / AT-SPI queries.
+fn decide_foreground_element_placement(
+ pid: u32,
+ observed: &crate::atspi::snapshot::CachedElement,
+ xid_hint: Option<u64>,
+ placement: Option<(u64, f64, f64)>,
+) -> (Option<u64>, ForegroundElementPlacement) {
+ let xid = xid_hint
+ .or_else(|| placement.map(|(xid, ..)| xid).filter(|xid| *xid != 0))
+ .or_else(|| {
+ crate::x11::list_windows(Some(pid))
+ .into_iter()
+ .next()
+ .map(|w| w.xid)
+ });
+ // The object's own state decides whether it is on screen at all: a menu
+ // item in a closed menu still answers a (stale or zero) extent.
+ let showing = match crate::atspi::element_showing_observed(observed) {
+ Ok(showing) => Some(showing),
+ Err(error) => {
+ tracing::debug!(
+ "foreground element [{}] showing-state unavailable: {error:#}",
+ observed.key
+ );
+ None
+ }
+ };
+ if showing == Some(false) {
+ return (
+ xid,
+ ForegroundElementPlacement::NoPoint {
+ reason: format!(
+ "the {} is not showing (its menu or page is closed)",
+ observed.role
+ ),
+ },
+ );
+ }
+ let Some((placed_xid, sx, sy)) = placement.filter(|(xid, ..)| *xid != 0) else {
+ return (
+ xid,
+ ForegroundElementPlacement::NoPoint {
+ reason: "no usable on-screen bounds were resolved for it".to_owned(),
+ },
+ );
+ };
+ let (px, py) = (sx.round() as i32, sy.round() as i32);
+ let inside = |w: &crate::x11::WindowInfo| {
+ px >= w.x && py >= w.y && px < w.x + w.width as i32 && py < w.y + w.height as i32
+ };
+ if crate::x11::window_info(placed_xid).is_some_and(|w| inside(&w)) {
+ return (
+ xid,
+ ForegroundElementPlacement::Point {
+ xid: placed_xid,
+ sx,
+ sy,
+ },
+ );
+ }
+ // The centre is outside the named window: a popup menu of the same
+ // process (an open menu's item) is the one legitimate case.
+ if let Some(popup) = crate::input::popup_under_screen_point(px, py) {
+ if popup.pid.is_none_or(|p| p == pid) {
+ return (
+ xid,
+ ForegroundElementPlacement::Point {
+ xid: placed_xid,
+ sx,
+ sy,
+ },
+ );
+ }
+ }
+ (
+ xid,
+ ForegroundElementPlacement::NoPoint {
+ reason: format!(
+ "its centre ({px}, {py}) lies outside window {placed_xid} and every popup of \
+ pid {pid}"
+ ),
+ },
+ )
+}
+
+#[async_trait]
+impl Tool for ClickTool {
+ fn def(&self) -> &ToolDef {
+ CLICK_DEF.get_or_init(|| ToolDef {
+ name: "click".into(),
+ description: "Click against a target pid. **Prefer `element_token` over pixel \
+ coordinates** — element_token works on backgrounded / hidden windows, surfaces \
+ a stable handle, and tells you what you're clicking via the cached AT-SPI \
+ element's role + label. Reach for `x, y` only when the target is a canvas / \
+ custom-drawn surface that doesn't appear in the AT-SPI tree.\n\n\
+ Provide either (window_id + x/y) or (pid + element_token). x/y are \
+ WINDOW-LOCAL pixels of window_id (the get_window_state screenshot frame); \
+ a point outside the window is refused (point_outside_window) and every \
+ pixel result reports the resolved window_point / screen_point. Background \
+ delivery first resolves the accessible under the point and fires its action \
+ (the result says `hit: <role> \"<name>\" action=...`; a file / list item is \
+ selected through its container: `selected: ...`), else presses the real \
+ virtual pointer there. A click that opens a popup menu names it (`popup: \
+ {window_id, bounds}`): pass that window_id to get_window_state to index its \
+ items. The token is stale once the next get_window_state of the same \
+ window replaces its snapshot — \
+ re-snapshot every turn before clicking.\n\n\
+ After a zoom call, pass from_zoom=true to auto-translate zoom-image coords \
+ back to full-window space.\n\n\
+ `count`: 1 (default), 2 for a double-click, 3 for a triple-click (selects a \
+ line / paragraph in editors) — one press train with real double-click cadence, \
+ in background and foreground alike; there is no separate triple_click tool.\n\n\
+ button: \"left\" (default), \"right\", or \"middle\". Defaults to left so the \
+ field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release \
+ with the matching button code. Native Wayland: only left-button is supported \
+ via the virtual-pointer protocol — right/middle return an error rather than \
+ silently degrading to left. `modifier` holds ctrl/shift/alt/super for the \
+ click on X11. Native Wayland refuses modified pointer clicks until its input \
+ protocol can carry keyboard modifier state.".into(),
+ input_schema: json!({
+ // `pid` is conditionally required (validated in code: needed for
+ // window/element clicks, omitted for windowless scope="desktop"),
+ // so it is NOT pinned in `required` — matches the click→[] canon
+ // in cua_driver_core::tool_schema.
+ "type":"object","required":[],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "capture_id":{"type":"string","description":"Optional one-shot binding to the exact PNG returned by get_window_state or get_desktop_state. When present, x/y are interpreted in that capture and admitted before native dispatch."},
+ // Shape matches the shared button_schema() canon (string +
+ // [left,right,middle]); kept inline to carry the Linux/Wayland
+ // back-compat prose the click button-schema test asserts on.
+ "button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},
+ "count":{"type":"integer","minimum":1,"maximum":3,"description":"Number of clicks in one press train: 1 (default), 2 = double-click, 3 = triple-click (line/paragraph selection). There is no separate triple_click tool — pass count: 3 here."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Coordinate frame (default \"window\"). Pass \"desktop\" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let modifiers: Vec<String> = args.str_array("modifier");
+
+ // ── Window-less screen-absolute branch (desktop target) ───────────────
+ // x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,
+ // desktop-scope click (the Linux peer of the Windows WindowFromPoint /
+ // macOS global-HID path). The core registry has already enforced the
+ // session policy; this branch validates the explicit action form.
+ let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
+ let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
+ let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
+ && args.get("y").map(|v| v.is_number()).unwrap_or(false);
+ if args.get("capture_id").is_some() && !has_xy {
+ return ToolResult::error("click.capture_id requires both x and y coordinates.")
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ if has_xy && !has_pid && !has_window_id {
+ if args.get("scope").and_then(Value::as_str) != Some("desktop") {
+ return ToolResult::error(
+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to read true screen pixels first."
+ .to_string(),
+ )
+ .with_structured(json!({
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
+ }));
+ }
+ let input = match parse_legacy_click_input(&args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let button = parse_mouse_button(input.button.unwrap_or(ClickButton::Left).as_str());
+ let n = input.count.unwrap_or(1);
+ if n == 0 {
+ return ToolResult::error("click.count must be at least 1.")
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ let n = click_train(u64::from(n));
+ let space = match desktop_input_space().await {
+ Ok(space) => space,
+ Err(error) => {
+ return ToolResult::error(format!("desktop-scope click failed: {error}"))
+ }
+ };
+ let (action_x, action_y) = if let Some(capture_id) = args.opt_str("capture_id") {
+ match crate::capture_action_frame::admit_desktop_click(
+ &self.state.capture_service,
+ &args,
+ &capture_id,
+ input.x,
+ input.y,
+ ) {
+ Ok(point) => point,
+ Err(error) => return capture_admission_error(error),
+ }
+ } else {
+ (input.x, input.y)
+ };
+ let sx = action_x.round() as i32;
+ let sy = action_y.round() as i32;
+ // Glide the agent-cursor overlay to the click point first (the macOS
+ // / Windows desktop paths already do this). Without it the overlay
+ // sits idle elsewhere while only the real pointer warps, so a viewer
+ // sees the cursor "click somewhere else."
+ // The overlay draws in layout coordinates, like the input below.
+ let (overlay_x, overlay_y) = space.to_layout(sx, sy);
+ reveal_pointer_action_for(
+ &self.state,
+ &cursor_id,
+ f64::from(overlay_x),
+ f64::from(overlay_y),
+ true,
+ )
+ .await;
+ let r = spawn_blocking_bounded("desktop click", foreground_budget(0), move || {
+ if crate::wayland::wayland_input_enabled() {
+ if !modifiers.is_empty() {
+ anyhow::bail!(
+ "modified desktop clicks are unavailable on native Wayland: \
+ the virtual-pointer route cannot carry keyboard modifier state"
+ );
+ }
+ crate::wayland::click_desktop(&space, sx, sy, n as u32, button)
+ } else {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx,
+ sy,
+ button,
+ n,
+ &modifier_refs,
+ )
+ }
+ })
+ .await;
+ return match r {
+ // Screen-absolute click — never driver-verifiable (no
+ // read-back); the caller confirms via screenshot.
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent screen-absolute click at ({sx},{sy}) (desktop scope)."
+ ))
+ .with_structured(
+ json!({ "path": if crate::wayland::wayland_input_enabled() { "wayland_desktop" } else { "xtest_desktop" }, "verified": false, "effect": "unverifiable" }),
+ ),
+ Ok(Err(e)) if crate::input::foreground::error_code(&e).is_some() => {
+ input_error_result(e)
+ }
+ Ok(Err(e)) => ToolResult::error(format!("desktop-scope click failed: {e}")),
+ Err(e) => ToolResult::error(format!("task error: {e}")),
+ };
+ }
+
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ let count = click_train(args.u64_or("count", 1));
+ let isolated_background = isolated_hyprland_background(delivery);
+ // Surface 5: reject unknown buttons so a typo can't silently fall through
+ // to a left-click. Empty string keeps back-compat with old clients.
+ let button_str_raw = args.str_or("button", "left").to_lowercase();
+ if !matches!(button_str_raw.as_str(), "" | "left" | "right" | "middle") {
+ return ToolResult::error(format!(
+ "click: unknown button \"{button_str_raw}\" — expected one of left, right, middle."
+ ));
+ }
+ let button_str = if button_str_raw.is_empty() {
+ "left"
+ } else {
+ button_str_raw.as_str()
+ };
+ let button = parse_mouse_button(button_str);
+
+ // Surface 6: element_token / element_index precedence resolution.
+ // We resolve before the legacy `opt_u64("element_index")` branch
+ // so a token-only call (no integer arg) still takes the element path.
+ let capture_id_arg = args.opt_str("capture_id");
+ let window_id_arg = args.opt_u64("window_id");
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "click").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let elem_idx_resolved: Option<usize> = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ let window_id_resolved: Option<u64> = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
+ };
+
+ if capture_id_arg.is_some() && elem_idx_resolved.is_some() {
+ return ToolResult::error(
+ "click.capture_id is only valid with x/y coordinates, not an element target.",
+ )
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+
+ if let Some(token) = token {
+ // The observed element: its proven AT-SPI identity and the frame
+ // the snapshot recorded. Every route below acts on that identity
+ // (or its cached frame); the live ordinal `idx` is only an
+ // address within the snapshot, never used to retarget.
+ let idx = token.index;
+ let observed = token.observed.clone();
+ let xid_hint = window_id_resolved;
+ // Resolve the element's screen center + its window FIRST, so the
+ // agent cursor glides to the target *before* the click fires —
+ // matching the coordinate path below and the macOS/Windows backends.
+ // Previously perform_action ran inside this spawn_blocking, so the
+ // app updated before the cursor visibly arrived.
+ // Bounded: resolving bounds re-walks the AT-SPI tree, which on a
+ // large GTK/VCL tree can stall for minutes; a missing placement
+ // degrades to the AX action (background) or a refusal (foreground).
+ let placed = token.clone();
+ let placement = tokio::time::timeout(
+ ELEMENT_AX_BUDGET,
+ tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
+ let (cx, cy) = element_screen_center(pid, &placed, xid_hint)?;
+ let xid = xid_hint
+ .or_else(|| {
+ crate::x11::list_windows(Some(pid))
+ .into_iter()
+ .next()
+ .map(|w| w.xid)
+ })
+ .unwrap_or(0);
+ Ok((xid, cx, cy))
+ }),
+ )
+ .await;
+ let placement = match placement {
+ Ok(Ok(placement)) => placement,
+ Ok(Err(error)) => Err(anyhow::anyhow!("task error: {error}")),
+ Err(_elapsed) => Err(anyhow::anyhow!("element placement timed out")),
+ };
+ let placement = match element_click_placement(placement, xid_hint) {
+ Ok(placement) => placement,
+ Err(refusal) => return refusal,
+ };
+ // X11 foreground element click: behave like a user — activate the
+ // window and XTest-click the element's screen centre. No AT-SPI
+ // action, no second tree walk. Element bounds come from the
+ // (bounded) placement above.
+ if delivery.is_foreground()
+ && !crate::wayland::wayland_input_enabled()
+ && !hyprland_foreground(delivery)
+ {
+ return self
+ .foreground_element_click(
+ &cursor_id, pid, idx, &observed, xid_hint, placement, button, count,
+ &modifiers,
+ )
+ .await;
+ }
+ if let Some((xid, sx, sy)) = placement {
+ if xid != 0 {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ }
+ reveal_pointer_action_target_for(
+ &self.state,
+ &cursor_id,
+ sx,
+ sy,
+ true,
+ frame_target_rect(observed.bounds, sx, sy),
+ )
+ .await;
+ }
+
+ // Chromium can execute a genuine AT-SPI action without focus. Try
+ // that route before applying its background synthetic-input gate.
+ // Plain Hyprland foreground clicks also retain semantic activation
+ // for offscreen controls; other pointer gestures need raw input.
+ let foreground_hyprland = hyprland_foreground(delivery);
+ if foreground_hyprland && window_id_resolved.is_none() {
+ return foreground_hyprland_refusal(
+ "an exact window_id or window-bound element token is required",
+ );
+ }
+ // An AT-SPI action can open a menu or press a button, but it cannot
+ // give an entry / spin button / slider the widget focus a following
+ // type_text needs (GIMP spin scales, VS Code settings inputs). With
+ // the MPX real pointer available, click those like a user would.
+ let needs_real_click = element_needs_real_click(&observed.role);
+ // A menu / menu item's `doAction` is declined or silently ignored
+ // by LibreOffice VCL (and opens GTK menus without the pointer
+ // grab a following item click expects): with a real pointer, press
+ // it like a user would and let the popup be listed. That holds
+ // for an entry that is on screen (SHOWING: a menubar entry, an
+ // item of an open GTK3 / VCL popup). A menu item that is not
+ // showing has no point to press: gail (GTK2, GIMP) never marks
+ // menu items SHOWING, open or closed, and an item of a closed
+ // menu is still reached through its AT-SPI `click` action (the
+ // same rule the foreground route applies), so those keep the
+ // action route.
+ let real_pointer = !delivery.is_foreground()
+ && !crate::wayland::wayland_input_enabled()
+ && crate::input::real_pointer_input_available();
+ let menu_entry_on_screen = real_pointer && element_is_menu_role(&observed.role) && {
+ // A bounded D-Bus round-trip: off the runtime thread.
+ let observed_for_state = observed.clone();
+ let showing = tokio::task::spawn_blocking(move || {
+ crate::atspi::element_showing_observed(&observed_for_state)
+ })
+ .await;
+ !matches!(showing, Ok(Ok(false)))
+ };
+ let real_click_role = real_pointer && (needs_real_click || menu_entry_on_screen);
+ // Without a real pointer, an editable / table cell cannot take a
+ // background click that gives it focus: refuse instead of firing
+ // an AT-SPI action whose effect is not a click.
+ if !delivery.is_foreground()
+ && !crate::wayland::is_wayland()
+ && needs_real_click
+ && !crate::input::real_pointer_input_available()
+ {
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ return crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ }
+ if !real_click_role
+ && element_click_prefers_ax(
+ foreground_hyprland,
+ button,
+ count,
+ !modifiers.is_empty(),
+ )
+ {
+ let observed_for_ax = observed.clone();
+ let guard_pid = (!delivery.is_foreground()).then_some(pid);
+ let ax_result = match tokio::time::timeout(
+ ELEMENT_AX_BUDGET,
+ tokio::task::spawn_blocking(move || {
+ // Background: an AT-SPI action can open a menu (GTK/VCL
+ // grab the keyboard and activate the toplevel) or map a
+ // dialog mutter then focuses. Snapshot, act, restore.
+ match guard_pid {
+ Some(pid) => crate::input::focus_guard::guarded(Some(pid), || {
+ crate::atspi::perform_action_observed(&observed_for_ax)
+ }),
+ None => crate::atspi::perform_action_observed(&observed_for_ax)
+ .map(|value| (value, None)),
+ }
+ }),
+ )
+ .await
+ {
+ Ok(joined) => joined,
+ Err(_elapsed) => {
+ return ToolResult::error(format!(
+ "click: the AT-SPI action for element [{idx}] did not complete within {ELEMENT_AX_BUDGET:?}; its effect is unknown. Take a screenshot, then retry by pixel (x/y) or with delivery_mode:\"foreground\"."
+ ))
+ .with_structured(json!({
+ "code": "ax_timeout",
+ "effect": "unknown",
+ "escalation": non_ax_escalation(),
+ }));
+ }
+ };
+ if let Ok(Ok(((_action, suspected_noop, unacknowledged), guard))) = ax_result {
+ let mut structured = json!({
+ "path": "ax",
+ "verified": false,
+ "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },
+ });
+ if suspected_noop {
+ structured["escalation"] = non_ax_escalation();
+ } else if unacknowledged {
+ structured["note"] = json!(
+ "no acknowledgement received for the doAction call within the \
+ AT-SPI timeout; its effect is unknown"
+ );
+ }
+ return attach_focus_guard(
+ ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))
+ .with_structured(structured),
+ guard.as_ref(),
+ );
+ }
+ // The observed object is gone: the snapshot is stale. Refuse
+ // rather than retarget whatever now lives at that index.
+ if let Ok(Err(error)) = &ax_result {
+ if error.is::<crate::atspi::native::CachedElementGone>() {
+ return ToolResult::error(format!(
+ "AT-SPI element click failed: stale_element_token: observed \
+ AT-SPI object [{idx}] is no longer present ({error}); \
+ re-snapshot with get_window_state"
+ ))
+ .with_structured(json!({
+ "code": "stale_element_token",
+ "effect": "none",
+ }));
+ }
+ }
+ // AT-SPI errors can arrive after dispatch. Do not introduce a
+ // primary-seat replay when its semantic outcome is uncertain.
+ if foreground_hyprland {
+ return foreground_ax_click_unknown(pid, idx);
+ }
+ }
+ if foreground_hyprland {
+ if !modifiers.is_empty() {
+ return foreground_hyprland_refusal("modified clicks are unsupported");
+ }
+ let Some(xid) = window_id_resolved else {
+ return foreground_hyprland_refusal(
+ "an exact window_id or window-bound element token is required",
+ );
+ };
+ let point = tokio::task::spawn_blocking(move || {
+ resolve_element_local_coords(pid, &token, Some(xid))
+ })
+ .await;
+ let (x, y) = match point {
+ Ok(Ok((_, x, y))) => (x, y),
+ _ => {
+ return foreground_hyprland_refusal(
+ "could not resolve exact element coordinates",
+ )
+ }
+ };
+ return foreground_hyprland_action(
+ &args,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Click {
+ x,
+ y,
+ button: u32::from(button),
+ count,
+ },
+ )
+ .await;
+ }
+ if isolated_background {
+ if !modifiers.is_empty() {
+ return isolated_hyprland_refusal(
+ "modified element clicks are unavailable on native Wayland: \
+ isolated input does not carry pointer modifier state",
+ );
+ }
+ let Some(exact_xid) = window_id_resolved else {
+ return isolated_hyprland_refusal(
+ "an exact window_id or window-bound element token is required",
+ );
+ };
+ let owner = named_session_cursor_key(&args);
+ let (_cancellation, dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ let (_, x, y) = resolve_element_local_coords(pid, &token, Some(exact_xid))?;
+ crate::wayland::hyprland_input::execute(
+ owner,
+ pid,
+ exact_xid,
+ crate::wayland::hyprland_input::Action::Click {
+ x,
+ y,
+ button: u32::from(button),
+ count,
+ },
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(refusal) => return refusal,
+ };
+ return match dispatch.await {
+ Ok(result) => isolated_hyprland_result(result),
+ Err(error) => isolated_hyprland_task_error(error, false),
+ };
+ }
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+
+ // The AX route was unavailable. Fall back to a real MPX pointer
+ // click at the element (no focus steal), then to a target-addressed
+ // X11 event for toolkits that accept it.
+ let cursor_id_for_fallback = cursor_id.clone();
+ let result = spawn_blocking_bounded(
+ "element click",
+ foreground_budget(0),
+ move || -> anyhow::Result<PointerRoute> {
+ // Refuse before resolving coordinates: the refusal does not
+ // depend on where the element is, and a window without
+ // compositor geometry must not turn it into a lookup error.
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {
+ anyhow::bail!(
+ "modified element clicks are unavailable on native Wayland: \
+ the pointer route cannot carry keyboard modifier state"
+ );
+ }
+ let (xid2, lx, ly) = checked_element_local_coords(pid, &token, xid_hint)?;
+ // An explicit X11 foreground request needs the real XTest path
+ // even for a plain click. Some native widgets (notably GTK
+ // selectable rows) expose bounds but no AT-SPI Action and
+ // ignore a targeted XSendEvent; limiting XTest to modified
+ // clicks made those rows addressable but not selectable.
+ if delivery.is_foreground() && !crate::wayland::wayland_input_enabled() {
+ let (_, sx, sy) = placement
+ .ok_or_else(|| anyhow::anyhow!("element screen bounds unavailable"))?;
+ crate::interaction::with_x11_foreground(
+ xid2,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx.round() as i32,
+ sy.round() as i32,
+ button,
+ count,
+ &modifier_refs,
+ )
+ },
+ )
+ .map(|(value, _report)| value)
+ .map(|()| PointerRoute::Foreground)
+ } else if modifier_refs.is_empty() && !crate::wayland::wayland_input_enabled() {
+ x11_pixel_click_no_focus_steal(
+ &cursor_id_for_fallback,
+ xid2,
+ lx as i32,
+ ly as i32,
+ button,
+ count,
+ )
+ } else {
+ crate::input::send_click_with_modifiers(
+ xid2,
+ lx as i32,
+ ly as i32,
+ count,
+ button,
+ &modifier_refs,
+ )
+ .map(|()| PointerRoute::Synthetic)
+ }
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ return match result {
+ // The MPX route reports its cheap post-checks (screen region
+ // around the element, focus untouched); the other routes stay
+ // unverifiable and the caller confirms via screenshot.
+ Ok(Ok(route)) => {
+ attach_focused_cell(
+ ToolResult::text(format!(
+ "Clicked element [{idx}] (pid {pid}) with a real pointer click {}",
+ route.text_suffix(mode_label)
+ ))
+ .with_structured(route.structured(mode_label)),
+ pid,
+ )
+ .await
+ }
+ Ok(Err(e)) => input_error_result(e.context("AT-SPI element click failed")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ if !isolated_background {
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ }
+
+ // Coordinate-based path.
+ if let Some(refusal) = require_point_args("click", &args) {
+ return refusal;
+ }
+ let xid = match args.opt_u64("window_id") {
+ Some(v) => v,
+ None => return ToolResult::error("Provide either element_token or window_id + x/y."),
+ };
+ let desktop_frame = desktop_frame_requested(&args);
+ let from_zoom = args.bool_or("from_zoom", false);
+ let mut x = args.f64_or("x", 0.0);
+ let mut y = args.f64_or("y", 0.0);
+ if capture_id_arg.is_some() && from_zoom {
+ return ToolResult::error(
+ "click.capture_id cannot be combined with from_zoom; use coordinates from the published capture.",
+ )
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ if capture_id_arg.is_some() && desktop_frame {
+ return ToolResult::error(
+ "click.capture_id cannot be combined with desktop-frame coordinates; use coordinates from the published capture.",
+ )
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ let native_refusal = (!isolated_background)
+ .then(|| unavailable_wayland_focused_input_background(delivery, true))
+ .flatten();
+ if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated once below.
+ if let Some(refusal) = native_refusal {
+ return refusal;
+ }
+ } else if let Some(capture_id) = capture_id_arg {
+ if let Some(refusal) = native_refusal {
+ return refusal;
+ }
+ match crate::capture_action_frame::admit_window_click(
+ &self.state.capture_service,
+ &args,
+ &capture_id,
+ pid,
+ xid,
+ x,
+ y,
+ ) {
+ Ok(point) => (x, y) = point,
+ Err(error) => return capture_admission_error(error),
+ }
+ } else {
+ let context = match coordinate_click_context(native_refusal, || {
+ if from_zoom {
+ self.state
+ .zoom_context(&args, pid, Some(xid))
+ .map(CoordinateContext::Zoom)
+ } else {
+ screenshot_scale(&self.state, &args, pid, Some(xid))
+ .map(CoordinateContext::Screenshot)
+ }
+ }) {
+ Ok(context) => context,
+ Err(refusal) => return refusal,
+ };
+ match context {
+ CoordinateContext::Zoom(context) => {
+ (x, y) = context.zoom_to_window(x, y);
+ }
+ CoordinateContext::Screenshot(scale) => {
+ x *= scale;
+ y *= scale;
+ }
+ }
+ }
+ if desktop_frame {
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
+ Ok(Ok((lx, ly))) => {
+ x = lx;
+ y = ly;
+ }
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ // Resolve the screen point the cursor glides to. Tool coordinates are
+ // always window-local screenshot pixels; native Wayland translates
+ // through compositor/AT-SPI geometry while X11 uses XTranslateCoordinates.
+ let wayland_output_point = if crate::wayland::wayland_input_enabled() {
+ match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
+ Ok(point) => Some(point),
+ Err(error) => return input_error_result(error),
+ }
+ } else {
+ None
+ };
+ let glide_target = if let Some(AimedPoint {
+ point: (sx, sy), ..
+ }) = wayland_output_point
+ {
+ Some((sx as f64, sy as f64))
+ } else {
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
+ .await
+ .ok()
+ .and_then(|r| r.ok())
+ };
+ if let Some((sx, sy)) = glide_target {
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
+ }
+
+ let (xi, yi) = (x as i32, y as i32);
+ let pixel_screen_point =
+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32));
+ let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
+ let (output_x, output_y) = aimed.point;
+ if hyprland_foreground(delivery) {
+ if !modifiers.is_empty() {
+ return foreground_hyprland_refusal("modified clicks are unsupported");
+ }
+ if window_id_resolved.is_none() {
+ return foreground_hyprland_refusal("an exact window_id is required");
+ }
+ return foreground_hyprland_action(
+ &args,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Click {
+ x,
+ y,
+ button: u32::from(button),
+ count,
+ },
+ )
+ .await;
+ }
+ if isolated_background {
+ if !modifiers.is_empty() {
+ return isolated_hyprland_refusal(
+ "modified clicks are unsupported by isolated input",
+ );
+ }
+ if button == 1 && count == 1 {
+ let semantic = tokio::task::spawn_blocking(move || {
+ crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
+ })
+ .await;
+ if matches!(semantic, Ok(Ok(Some(_)))) {
+ return ToolResult::text("Dispatched AT-SPI click.").with_structured(json!({
+ "path": "wayland_atspi", "verified": false, "effect": "unverifiable"
+ }));
+ }
+ }
+ return isolated_hyprland_action(
+ &args,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Click {
+ x,
+ y,
+ button: u32::from(button),
+ count,
+ },
+ )
+ .await;
+ }
+ let cursor_id_for_task = cursor_id.clone();
+ let modifiers_for_task = modifiers.clone();
+ // delivery_mode: background (default) = no-focus-steal injection;
+ // foreground = activate the target window (EWMH) first, then inject,
+ // then restore prior active. Mirrors macOS/Windows.
+ let fg_budget = foreground_budget(0);
+ let result = spawn_blocking_bounded("foreground click", fg_budget, move || -> anyhow::Result<(&'static str, Option<crate::input::ForegroundReport>, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {
+ if crate::wayland::wayland_input_enabled() {
+ if !modifiers_for_task.is_empty() {
+ anyhow::bail!(
+ "modified coordinate clicks are unavailable on native Wayland: \
+ the pointer route cannot carry keyboard modifier state"
+ );
+ }
+ // Vision/pixel click on native Wayland. Mutter drops synthetic
+ // virtual-pointer events (the `wayland::click` warp doesn't land),
+ // so for a plain left single click resolve the screen pixel to the
+ // covering accessible element and fire its action by
+ // `element_index` — the coordinate-free path already verified
+ // working. (x,y) are screen coords here, matching the frames in
+ // `get_window_state`. Miss → fall through to the injection paths.
+ if !delivery.is_foreground() && button == 1 && count == 1 {
+ if let Ok(Some(_)) =
+ crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
+ {
+ return Ok(("wayland_atspi", None, None, None));
+ }
+ }
+ if crate::wayland::is_inject_mode() {
+ crate::wayland::inject_click(pid, xid, x, y, count as u32, button)?;
+ return Ok(("wayland_cua_compositor", None, None, None));
+ }
+ if !delivery.is_foreground() {
+ return Ok(("background_unavailable", None, None, None));
+ }
+ // Native Wayland: focus+raise the target toplevel
+ // (foreign-toplevel `activate`), then drive `count` virtual-pointer
+ // button events. Wayland injection routes to the compositor focus.
+ crate::interaction::with_target_foreground(pid, xid, crate::interaction::RestoreMode::RestorePrevious, || {
+ crate::wayland::click_focused(aimed, count as u32, button)
+ })?;
+ return Ok(("wayland_activate", None, None, None));
+ }
+ // X11 injection. Tiered no-focus-steal delivery (background):
+ // 1. Plain left single-click → AT-SPI doAction at that point.
+ // 2. Right / middle / double click → real MPX uinput pointer + XI2
+ // shield grab (real Xorg only; skipped on Xvfb/Wayland).
+ // 3. Fallback → synthetic XSendEvent.
+ // Foreground skips the AT-SPI shortcut and does a real activated pixel
+ // click (the agent's escalation when background didn't land).
+ let inject = |fg: bool| -> anyhow::Result<(&'static str, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {
+ if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {
+ // The accessible action under the point may open a menu or
+ // a dialog that takes the focus: guard and restore. With a
+ // real pointer available, an entry / spin button / cell
+ // under the point is left to the MPX click below (its
+ // `doAction` would not focus it for a following type_text).
+ let real_click_for_focus_roles =
+ crate::input::real_pointer_input_available();
+ let (hit, guard) = crate::input::focus_guard::guarded(Some(pid), || {
+ Ok(crate::atspi::perform_action_at_point_in(
+ pid,
+ xid,
+ xi,
+ yi,
+ real_click_for_focus_roles,
+ )
+ .ok()
+ .flatten())
+ })?;
+ if let Some(hit) = hit {
+ return Ok(("x11_atspi", Some(PointerRoute::Atspi(hit)), guard));
+ }
+ }
+ if !fg
+ && !crate::input::real_pointer_input_available()
+ && synthetic_pointer_is_dropped(pid)
+ {
+ // No accessible actuator covers the point and the only
+ // route left is a synthetic XSendEvent that this toolkit
+ // (GTK3/4 XInput2, LibreOffice VCL) discards. Say so
+ // instead of reporting a click that changed nothing.
+ return Ok(("background_unavailable_pointer", None, None));
+ }
+ if fg {
+ // Foreground: the window is already activated. Deliver a REAL
+ // XTest warp+button click. Synthetic XSendEvent button events
+ // are dropped by GTK/Qt/Chromium/Firefox, and the MPX uinput
+ // path needs /dev/uinput, which headless X servers
+ // (Xvfb/Xtigervnc) lack — so neither focuses the clicked
+ // widget. XTest is accepted as real input and gives the
+ // widget keyboard focus, so a following type lands.
+ if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64) {
+ let modifier_refs: Vec<&str> =
+ modifiers_for_task.iter().map(String::as_str).collect();
+ crate::input::send_click_xtest_desktop_with_modifiers(
+ sx.round() as i32,
+ sy.round() as i32,
+ button,
+ count,
+ &modifier_refs,
+ )?;
+ return Ok(("x11_xtest_fg", None, None));
+ }
+ }
+ if modifiers_for_task.is_empty() {
+ // The MPX press train runs under its own (settled) focus
+ // guard inside the helper; its report rides on the route.
+ let route = x11_pixel_click_no_focus_steal(
+ &cursor_id_for_task,
+ xid,
+ xi,
+ yi,
+ button,
+ count,
+ )?;
+ return Ok((route.path().unwrap_or("x11_pixel"), Some(route), None));
+ } else {
+ let modifier_refs: Vec<&str> =
+ modifiers_for_task.iter().map(String::as_str).collect();
+ x11_pixel_click_no_focus_steal_modifiers(
+ xid,
+ xi,
+ yi,
+ button,
+ count,
+ &modifier_refs,
+ )?;
+ }
+ Ok((if fg { "x11_pixel_fg" } else { "x11_pixel" }, None, None))
+ };
+ if delivery.is_foreground() {
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || inject(true),
+ )
+ .map(|((path, route, guard), report)| (path, Some(report), route, guard))
+ } else {
+ inject(false).map(|(path, route, guard)| (path, None, route, guard))
+ }
+ })
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ let outcome = match result {
+ Ok(Ok(("background_unavailable", _, _, _))) => {
+ crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ )
+ }
+ Ok(Ok(("background_unavailable_pointer", _, _, _))) => {
+ let mut refusal = crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ let hint = "No accessible control covers this point and the toolkit drops \
+ synthetic pointer events, so a background pixel click here would \
+ change nothing. Use get_window_state and click by element_token \
+ (AT-SPI action), or retry with delivery_mode='foreground'.";
+ refusal
+ .content
+ .push(cua_driver_core::protocol::Content::text(hint));
+ if let Some(structured) = refusal.structured_content.as_mut() {
+ structured["hint"] = json!(hint);
+ structured["path"] = json!("background_unavailable_pointer");
+ }
+ refusal
+ }
+ // A pixel/coordinate click is never driver-verifiable (no read-back) —
+ // verified:false, effect:"unverifiable"; the caller confirms via
+ // screenshot. path reports the rung taken; a foreground click also
+ // reports the activation/focus transaction it confirmed first.
+ Ok(Ok((path, report, route, guard))) => {
+ let mut structured = match &route {
+ Some(route) => route.structured(mode_label),
+ None => json!({
+ "path": path,
+ "verified": false,
+ "effect": "unverifiable",
+ "delivery_mode": mode_label,
+ }),
+ };
+ if let Some(report) = report {
+ apply_foreground_report(&mut structured, &report);
+ }
+ let suffix = match &route {
+ Some(route) => route.text_suffix(mode_label),
+ None => format!(
+ "(delivery_mode={mode_label}, path={path}); not verified — confirm \
+ with a screenshot."
+ ),
+ };
+ attach_focus_guard(
+ ToolResult::text(format!("Clicked at ({x:.1}, {y:.1}) × {count} {suffix}"))
+ .with_structured(structured),
+ guard.as_ref(),
+ )
+ }
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ let outcome = attach_focused_cell(outcome, pid).await;
+ with_pixel_points(outcome, pixel_screen_point, (xi, yi))
+ }
+}
+
+#[cfg(test)]
+mod click_button_schema_tests {
+ use super::{chromium_background_must_refuse, maps_indicate_gtk, ClickTool};
+ use cua_driver_core::tool::Tool;
+
+ /// Surface 5: schema must advertise the three canonical button values and
+ /// describe the back-compat default. Linux already routed button=middle/right
+ /// pre-Surface-5; this freezes the schema shape so the contract can't drift.
+ #[test]
+ fn schema_advertises_button_enum_and_description() {
+ let tool = ClickTool {
+ state: super::ToolState::new(),
+ };
+ let d = tool.def();
+ let props = d.input_schema.get("properties").expect("properties");
+ let button = props.get("button").expect("button field present");
+ assert_eq!(button.get("type").and_then(|v| v.as_str()), Some("string"));
+ let enum_vals: Vec<&str> = button
+ .get("enum")
+ .and_then(|v| v.as_array())
+ .expect("button.enum present")
+ .iter()
+ .filter_map(|v| v.as_str())
+ .collect();
+ for need in ["left", "right", "middle"] {
+ assert!(enum_vals.contains(&need), "missing {need} in button.enum");
+ }
+ let desc = button
+ .get("description")
+ .and_then(|v| v.as_str())
+ .expect("button.description present");
+ let lc = desc.to_ascii_lowercase();
+ assert!(lc.contains("left"), "description should mention default");
+ assert!(
+ lc.contains("wayland"),
+ "description should call out wayland fallback"
+ );
+ }
+
+ #[test]
+ fn chromium_background_requires_focus_free_inject_mode() {
+ assert!(chromium_background_must_refuse(false, false, true));
+ assert!(!chromium_background_must_refuse(false, true, true));
+ assert!(!chromium_background_must_refuse(true, false, true));
+ assert!(!chromium_background_must_refuse(false, false, false));
+ }
+
+ #[test]
+ fn gtk_process_maps_are_detected_without_matching_unrelated_libraries() {
+ assert!(maps_indicate_gtk(
+ "7f00-7f01 r-xp /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.32"
+ ));
+ assert!(maps_indicate_gtk(
+ "7f00-7f01 r-xp /nix/store/hash-gtk4/lib/libgtk-4.so.1"
+ ));
+ assert!(!maps_indicate_gtk(
+ "7f00-7f01 r-xp /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so"
+ ));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/cursor_tools.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/cursor_tools.rs
new file mode 100644
index 0000000000..40a92a56ec
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/cursor_tools.rs
@@ -0,0 +1,231 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{resolve_cursor_key, ToolState};
+
+// ── set_agent_cursor_enabled ──────────────────────────────────────────────────
+
+pub struct SetAgentCursorEnabledV2Tool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_ENABLED_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetAgentCursorEnabledV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_ENABLED_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_enabled"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let enabled = match args.get("enabled").and_then(Value::as_bool) {
+ Some(value) => value,
+ None => return ToolResult::error("Missing required boolean field `enabled`."),
+ };
+ let session = resolve_cursor_key(&args);
+ self.state.cursor_registry.set_enabled(&session, enabled);
+ crate::overlay::send_command_for(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetEnabled(enabled),
+ );
+ ToolResult::text(format!(
+ "Agent cursor for session '{session}' {}.",
+ if enabled { "enabled" } else { "disabled" }
+ ))
+ .with_structured(json!({"session":session,"enabled":enabled}))
+ }
+}
+
+pub struct SetAgentCursorMotionV2Tool;
+
+static CURSOR_MOTION_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn cursor_number(value: Option<&Value>) -> Option<f64> {
+ value.and_then(|value| {
+ value
+ .as_f64()
+ .or_else(|| value.as_i64().map(|integer| integer as f64))
+ })
+}
+
+#[async_trait]
+impl Tool for SetAgentCursorMotionV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_MOTION_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_motion"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let session = resolve_cursor_key(&args);
+ let current = crate::overlay::current_motion_for(&session);
+ let motion = current.with_overrides(
+ cursor_number(args.get("start_handle")),
+ cursor_number(args.get("end_handle")),
+ cursor_number(args.get("arc_size")),
+ cursor_number(args.get("arc_flow")),
+ cursor_number(args.get("spring")),
+ cursor_number(args.get("glide_duration_ms")),
+ cursor_number(args.get("dwell_after_click_ms")),
+ cursor_number(args.get("idle_hide_ms")),
+ None,
+ cursor_number(args.get("turn_radius")),
+ );
+ let motion = match motion.with_style_args(&args) {
+ Ok(motion) => motion,
+ Err(message) => return ToolResult::error(message),
+ };
+ crate::overlay::send_command_for(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetMotion(motion.clone()),
+ );
+ ToolResult::text(format!(
+ "Agent cursor motion updated for session '{session}'."
+ ))
+ .with_structured(json!({"session":session,"motion":motion.output_json()}))
+ }
+}
+
+pub struct SetAgentCursorThemeTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_THEME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn cursor_reduced_motion(value: Option<&str>) -> cursor_overlay::ReducedMotion {
+ match value {
+ Some("on") => cursor_overlay::ReducedMotion::On,
+ Some("off") => cursor_overlay::ReducedMotion::Off,
+ _ => cursor_overlay::ReducedMotion::Auto,
+ }
+}
+
+#[async_trait]
+impl Tool for SetAgentCursorThemeTool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_THEME_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_theme"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let Some(theme_id) = args.get("theme_id").and_then(Value::as_str) else {
+ return ToolResult::error("Missing required string field `theme_id`.");
+ };
+ let session = resolve_cursor_key(&args);
+ let resolved_theme = match cursor_overlay::resolve_theme_selection(theme_id) {
+ Ok(theme) => theme,
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Cursor theme '{theme_id}' cannot be selected: {error}"
+ ));
+ }
+ };
+ let (version, profile) = resolved_theme
+ .as_deref()
+ .map(|theme| (theme.version.as_str(), theme.profile.as_str()))
+ .unwrap_or((
+ cursor_overlay::DEFAULT_THEME_VERSION,
+ cursor_overlay::THEME_PROFILE,
+ ));
+ let reduced_motion =
+ cursor_reduced_motion(args.get("reduced_motion").and_then(Value::as_str));
+ self.state
+ .cursor_registry
+ .update_config(&session, |config| {
+ config.theme_id = theme_id.to_owned();
+ config.reduced_motion = reduced_motion;
+ });
+ crate::overlay::send_command_for(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetTheme {
+ theme_id: theme_id.to_owned(),
+ reduced_motion,
+ },
+ );
+ ToolResult::text(format!(
+ "Agent cursor theme for session '{session}' set to '{theme_id}'."
+ ))
+ .with_structured(json!({"session":session,"theme":{
+ "id":theme_id,
+ "version":version,
+ "profile":profile,
+ "reduced_motion":reduced_motion,
+ "fallback":null
+ }}))
+ }
+}
+
+pub struct GetAgentCursorStateV2Tool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_STATE_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetAgentCursorStateV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_STATE_V2_DEF.get_or_init(|| canonical_cursor_def("get_agent_cursor_state"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let session = resolve_cursor_key(&args);
+ let enabled = crate::overlay::is_enabled_for(&session);
+ let motion = crate::overlay::current_motion_for(&session);
+ let (theme_id, version, profile, fallback, visual) =
+ crate::overlay::current_theme_state_for(&session).unwrap_or_else(|| {
+ (
+ cursor_overlay::DEFAULT_THEME_ID.into(),
+ cursor_overlay::DEFAULT_THEME_VERSION.into(),
+ cursor_overlay::THEME_PROFILE.into(),
+ None,
+ cursor_overlay::CursorVisualState::default(),
+ )
+ });
+ let modifiers: Vec<&str> = [
+ visual.delivery.map(|value| value.as_str()),
+ visual.target.map(|value| value.as_str()),
+ ]
+ .into_iter()
+ .flatten()
+ .collect();
+ // The last point this session's cursor was placed at, `null` until
+ // it first moves (same registry source as macOS).
+ let position = self
+ .state
+ .cursor_registry
+ .get(&session)
+ .and_then(|cursor| cursor.x.zip(cursor.y))
+ .map(|(x, y)| json!({"x": x, "y": y}));
+ ToolResult::text(format!("Agent cursor state for session '{session}'.")).with_structured(
+ json!({
+ "session":session,
+ "enabled":enabled,
+ "position":position,
+ "theme":{
+ "id":theme_id,
+ "version":version,
+ "profile":profile,
+ "reduced_motion":visual.reduced_motion,
+ "fallback":fallback
+ },
+ "visual_state":{
+ "requested_action":visual.requested_action,
+ "resolved_action":visual.resolved_action,
+ "modifiers":modifiers,
+ "phase":visual.phase(),
+ "frame":visual.frame(),
+ "preempted_count":visual.preempted_count
+ },
+ "motion":motion.output_json()
+ }),
+ )
+ }
+}
+
+fn canonical_cursor_def(name: &str) -> ToolDef {
+ let contract = cua_driver_contract::tool_contract(name)
+ .unwrap_or_else(|| panic!("missing canonical cursor contract for {name}"));
+ ToolDef::from_contract(&contract)
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/double_click.rs
new file mode 100644
index 0000000000..471c2e7a60
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/double_click.rs
@@ -0,0 +1,342 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::ArgsExt as _,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::{
+ click::ClickTool,
+ impl_::{
+ checked_element_local_coords, coordinate_frame_schema, desktop_frame_requested,
+ desktop_frame_unmapped, desktop_to_window_local, element_screen_center_and_rect,
+ foreground_budget, hyprland_foreground, input_error_result, prove_token_element,
+ require_point_args, resolve_cursor_key, reveal_pointer_action_for,
+ reveal_pointer_action_target_for, screenshot_scale, spawn_blocking_bounded,
+ unavailable_chromium_background, unavailable_gtk_pointer_background,
+ unavailable_wayland_focused_input_background, unavailable_webkit_background,
+ window_local_to_screen, with_pixel_points, x11_pixel_click_no_focus_steal, AimedPoint,
+ PointerRoute, ToolState,
+ },
+};
+
+// `ScreenshotTool` and `ScreenshotCompatTool` removed in PR #1692 — see the
+// matching note in platform-windows/src/tools/impl_.rs. `get_window_state` is
+// the canonical screenshot path (it always returns a screenshot now); the
+// underlying capture machinery (XGetImage / `import` shell-out / etc.)
+// stays reachable through GetWindowStateTool.
+
+// ── double_click ──────────────────────────────────────────────────────────────
+
+pub struct DoubleClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static DCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for DoubleClickTool {
+ fn def(&self) -> &ToolDef {
+ DCLICK_DEF.get_or_init(|| ToolDef {
+ name: "double_click".into(),
+ description: "Double-click at (x,y) or an element_token (AT-SPI bounds). Background \
+ delivery on X11 sends real button events from the session's virtual master pointer \
+ (path mpx_pointer); a point covered by another application's window is refused \
+ (target_occluded), one covered by this app's own dialog is retargeted to it \
+ (retargeted_to). No focus steal. Provide either (window_id + x/y) or (pid + element_token). \
+ After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),
+ input_schema: json!({"type":"object","required":[],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
+ return refusal;
+ }
+ if hyprland_foreground(delivery) {
+ // Share exact-target validation and the admitted native click lifecycle.
+ let mut args = args;
+ args["button"] = json!("left");
+ args["count"] = json!(2);
+ return ClickTool {
+ state: self.state.clone(),
+ }
+ .invoke(args)
+ .await;
+ }
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "double_click").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let window_id_resolved: Option<u64> = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
+ };
+ if let Some(token) = token {
+ let idx = token.index;
+ let xid_hint = window_id_resolved;
+ let target = token.clone();
+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
+ checked_element_local_coords(pid, &target, xid_hint)
+ })
+ .await;
+ return match result {
+ Ok(Ok((xid, lx, ly))) => {
+ if let Ok(Ok(((sx, sy), rect))) = tokio::task::spawn_blocking(move || {
+ element_screen_center_and_rect(pid, &token, Some(xid))
+ })
+ .await
+ {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ reveal_pointer_action_target_for(
+ &self.state,
+ &cursor_id,
+ sx,
+ sy,
+ true,
+ Some(rect),
+ )
+ .await;
+ }
+ let lxi = lx as i32;
+ let lyi = ly as i32;
+ let wayland_point = match crate::wayland::wayland_input_enabled()
+ .then(|| crate::wayland::window_local_to_output(xid, lxi, lyi))
+ .transpose()
+ {
+ Ok(point) => point,
+ Err(error) => return input_error_result(error),
+ };
+ let cursor_id_for_task = cursor_id.clone();
+ let click_result = spawn_blocking_bounded(
+ "double_click",
+ foreground_budget(0),
+ move || -> anyhow::Result<PointerRoute> {
+ if crate::wayland::is_inject_mode() {
+ return crate::wayland::inject_click(pid, xid, lx, ly, 2, 1)
+ .map(|()| PointerRoute::Wayland);
+ }
+ if crate::wayland::wayland_input_enabled() {
+ let aimed = wayland_point.unwrap_or_else(|| (lxi, lyi).into());
+ return crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::pointer_gesture_restore(),
+ || crate::wayland::click(xid, aimed, 2, 1),
+ )
+ .map(|()| PointerRoute::Wayland);
+ }
+ if delivery.is_foreground() {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ let (sx, sy) =
+ window_local_to_screen(xid, lxi as f64, lyi as f64)?;
+ crate::input::send_click_xtest_desktop(
+ sx.round() as i32,
+ sy.round() as i32,
+ 1,
+ 2,
+ )
+ },
+ )
+ .map(|(value, _report)| value)
+ .map(|()| PointerRoute::Foreground);
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 1, 2)
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ match click_result {
+ Ok(Ok(route)) => ToolResult::text(format!(
+ "Double-clicked element [{idx}] {}",
+ route.text_suffix(mode_label)
+ ))
+ .with_structured(route.structured(mode_label)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ if let Some(refusal) = require_point_args("double_click", &args) {
+ return refusal;
+ }
+ let xid = match window_id_resolved {
+ Some(v) => v,
+ None => return ToolResult::error("Provide either element_token or window_id + x/y."),
+ };
+ let desktop_frame = desktop_frame_requested(&args);
+ let from_zoom = args.bool_or("from_zoom", false);
+ let mut x = args.f64_or("x", 0.0);
+ let mut y = args.f64_or("y", 0.0);
+ if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated once below.
+ } else if from_zoom {
+ match self.state.zoom_context(&args, pid, Some(xid)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(x, y);
+ x = wx;
+ y = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ x *= ratio;
+ y *= ratio;
+ }
+ if desktop_frame {
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
+ Ok(Ok((lx, ly))) => {
+ x = lx;
+ y = ly;
+ }
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ let wayland_output_point = if crate::wayland::wayland_input_enabled() {
+ match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
+ Ok(point) => Some(point),
+ Err(error) => return input_error_result(error),
+ }
+ } else {
+ None
+ };
+ let glide_target = if let Some(AimedPoint {
+ point: (sx, sy), ..
+ }) = wayland_output_point
+ {
+ Some((sx as f64, sy as f64))
+ } else {
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
+ .await
+ .ok()
+ .and_then(|r| r.ok())
+ };
+ if let Some((sx, sy)) = glide_target {
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
+ }
+ let (xi, yi) = (x as i32, y as i32);
+ let cursor_id_for_task = cursor_id.clone();
+ let result = spawn_blocking_bounded(
+ "double_click",
+ foreground_budget(0),
+ move || -> anyhow::Result<PointerRoute> {
+ if crate::wayland::is_inject_mode() {
+ return crate::wayland::inject_click(pid, xid, x, y, 2, 1)
+ .map(|()| PointerRoute::Wayland);
+ }
+ if crate::wayland::wayland_input_enabled() {
+ let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
+ return crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::pointer_gesture_restore(),
+ || crate::wayland::click(xid, aimed, 2, 1),
+ )
+ .map(|()| PointerRoute::Wayland);
+ }
+ if delivery.is_foreground() {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ // Real XTest double-click at the screen point — synthetic
+ // XSendEvent button events are dropped by GTK/Qt and the MPX
+ // uinput path needs /dev/uinput (absent on Xvfb/Xtigervnc), so
+ // neither lands. Mirrors the single-click foreground path.
+ if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64)
+ {
+ crate::input::send_click_xtest_desktop(
+ sx.round() as i32,
+ sy.round() as i32,
+ 1,
+ 2,
+ )?;
+ return Ok(());
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)
+ .map(|_| ())
+ },
+ )
+ .map(|(value, _report)| value)
+ .map(|()| PointerRoute::Foreground);
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ let outcome = match result {
+ Ok(Ok(route)) => ToolResult::text(format!(
+ "Double-clicked at ({x:.1}, {y:.1}) {}",
+ route.text_suffix(mode_label)
+ ))
+ .with_structured(route.structured(mode_label)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ with_pixel_points(
+ outcome,
+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),
+ (xi, yi),
+ )
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/drag.rs
new file mode 100644
index 0000000000..488beacbef
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/drag.rs
@@ -0,0 +1,950 @@
+use async_trait::async_trait;
+use cua_driver_contract::{ClickButton, DragInput};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::{parse_typed_projection, ArgsExt as _},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ coordinate_frame_schema, desktop_frame_requested, desktop_frame_unmapped, desktop_input_space,
+ desktop_to_window_local, drag_steps, foreground_budget, foreground_hyprland_refusal,
+ hyprland_foreground, hyprland_input_result, input_error_result, invalid_pointer_arguments,
+ isolated_hyprland_background, isolated_hyprland_refusal, isolated_hyprland_result,
+ isolated_hyprland_task_error, named_session_cursor_key, overlay_glide_to_for,
+ overlay_snap_to_for, parse_mouse_button, press_x11_drag_start, resolve_cursor_key,
+ screenshot_scale, spawn_blocking_bounded, spawn_isolated_hyprland,
+ synthetic_pointer_is_dropped, unavailable_chromium_background,
+ unavailable_gtk_pointer_background, unavailable_wayland_focused_input_background,
+ unavailable_webkit_background, window_local_to_screen, CoordinateContext, PointerRoute,
+ ToolState,
+};
+
+fn coordinate_drag_context(
+ native_refusal: Option<ToolResult>,
+ resolve_context: impl FnOnce() -> Result<CoordinateContext, ToolResult>,
+) -> Result<CoordinateContext, ToolResult> {
+ if let Some(refusal) = native_refusal {
+ return Err(refusal);
+ }
+ resolve_context()
+}
+
+#[cfg(test)]
+#[test]
+fn coordinate_drag_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
+ use std::sync::atomic::{AtomicBool, Ordering};
+
+ let resolved = AtomicBool::new(false);
+ let native_refusal = crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ let result = coordinate_drag_context(Some(native_refusal), || {
+ resolved.store(true, Ordering::SeqCst);
+ Ok(CoordinateContext::Screenshot(7.35))
+ })
+ .expect_err("unsupported Wayland background delivery must refuse drag");
+
+ assert_eq!(
+ result.structured_content.as_ref().unwrap()["code"],
+ "background_unavailable"
+ );
+ assert!(
+ !resolved.load(Ordering::SeqCst),
+ "drag snapshot context must not be consulted before the native refusal"
+ );
+}
+
+async fn track_overlay_drag_for(
+ cursor_id: String,
+ from: (f64, f64),
+ to: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+) {
+ track_overlay_drag_with(
+ |command| crate::overlay::send_command_for(cursor_id.clone(), command),
+ from,
+ to,
+ duration_ms,
+ steps,
+ )
+ .await;
+}
+
+async fn track_overlay_drag_with(
+ send: impl Fn(cursor_overlay::OverlayCommand),
+ from: (f64, f64),
+ to: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+) {
+ send(cursor_overlay::OverlayCommand::SetEnabled(true));
+ let _pressed = cursor_overlay::PressedVisualGuard::new(&send);
+ let steps = steps.max(1);
+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
+ for index in 0..=steps {
+ let t = index as f64 / steps as f64;
+ let x = from.0 + (to.0 - from.0) * t;
+ let y = from.1 + (to.1 - from.1) * t;
+ send(cursor_overlay::track_pointer_command(x, y));
+ if index < steps && !step_delay.is_zero() {
+ tokio::time::sleep(step_delay).await;
+ }
+ }
+}
+
+#[cfg(test)]
+#[tokio::test]
+async fn cancelled_overlay_drag_releases_visual_press_without_native_commands() {
+ use std::{cell::RefCell, future::Future, task::Context};
+ let commands = RefCell::new(Vec::new());
+ let mut drag = Box::pin(track_overlay_drag_with(
+ |command| commands.borrow_mut().push(command),
+ (10.0, 20.0),
+ (30.0, 40.0),
+ 2000,
+ 20,
+ ));
+ let mut context = Context::from_waker(std::task::Waker::noop());
+ assert!(drag.as_mut().poll(&mut context).is_pending());
+ assert!(matches!(
+ commands.borrow()[1],
+ cursor_overlay::OverlayCommand::SetPressed(true)
+ ));
+ drop(drag);
+ let commands = commands.borrow();
+ assert_eq!(commands.len(), 4);
+ assert!(matches!(
+ commands.last(),
+ Some(cursor_overlay::OverlayCommand::SetPressed(false))
+ ));
+}
+
+#[cfg(test)]
+#[tokio::test]
+async fn completed_overlay_drag_balances_visual_press_once() {
+ use std::cell::RefCell;
+ let pressed = RefCell::new(Vec::new());
+ track_overlay_drag_with(
+ |command| {
+ if let cursor_overlay::OverlayCommand::SetPressed(value) = command {
+ pressed.borrow_mut().push(value);
+ }
+ },
+ (10.0, 20.0),
+ (30.0, 40.0),
+ 0,
+ 2,
+ )
+ .await;
+ assert_eq!(*pressed.borrow(), vec![true, false]);
+}
+
+// ── drag ─────────────────────────────────────────────────────────────────────
+
+/// Typed refusal for a modified drag on native Wayland. The Wayland pointer
+/// routes (virtual pointer, libei, and the nested cua-compositor socket) do not
+/// hold keyboard modifier state across a pointer gesture, so the drag is
+/// refused before any input is dispatched rather than degraded to an
+/// unmodified drag. The stable `code` lets callers branch without parsing text.
+fn wayland_modified_drag_refusal() -> ToolResult {
+ const DETAIL: &str = "the pointer route cannot carry keyboard modifier state";
+ ToolResult::error(format!(
+ "modified drags are unavailable on native Wayland: {DETAIL}"
+ ))
+ .with_structured(json!({
+ "code": "modified_pointer_unavailable",
+ "effect": "refused",
+ "verified": false,
+ "detail": DETAIL,
+ }))
+}
+
+pub struct DragTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static DRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for DragTool {
+ fn has_independent_input_lane(&self, args: &Value) -> bool {
+ // This opt-in branch below either uses its own leased synthetic seat
+ // or refuses. It never falls back to primary-seat input. All ordinary
+ // Linux routes retain the common process-wide input coordinator.
+ // Socket availability can change between admission and invocation.
+ // Native Hyprland window drags remain fail-closed on their own lane.
+ args.opt_str("scope").as_deref() != Some("desktop")
+ && !crate::input::delivery::DeliveryMode::from_args(args).is_foreground()
+ && crate::wayland::wayland_input_enabled()
+ && crate::wayland::hyprland::is_session()
+ }
+
+ fn def(&self) -> &ToolDef {
+ DRAG_DEF.get_or_init(|| ToolDef {
+ name: "drag".into(),
+ description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in \
+ window-local screenshot pixels. Background delivery on X11 is one real held \
+ gesture on the session's virtual master pointer (path mpx_pointer: press, \
+ interpolated motion, release) — it lands on GTK/VCL/Qt/Chromium; a point \
+ covered by another application's window is refused (target_occluded), one \
+ covered by this app's own dialog is retargeted to it. Headless servers fall \
+ back to XSendEvent. duration_ms (default 500), steps (default 20). No focus steal.".into(),
+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\"."},
+ "window_id":{"type":"integer","description":"Target window XID. Required."},
+ "from_x":{"type":"number","description":"Drag start X: window-local pixels of the target window's own get_window_state screenshot; pass scope:\"desktop\" for get_desktop_state pixels."},
+ "from_y":{"type":"number","description":"Drag start Y (same frame as from_x)."},
+ "to_x":{"type":"number","description":"Drag end X (same frame as from_x)."},
+ "to_y":{"type":"number","description":"Drag end Y (same frame as from_x)."},
+ "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},
+ "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "button": cua_driver_core::tool_schema::button_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid/window_id for get_desktop_state screen coordinates. Default \"window\"."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let modifiers: Vec<String> = args.str_array("modifier");
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<DragInput>("drag", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let (from_x, from_y) = (input.from_x, input.from_y);
+ let (to_x, to_y) = (input.to_x, input.to_y);
+ let button = parse_mouse_button(input.button.unwrap_or(ClickButton::Left).as_str());
+ let duration_ms = input.duration_ms.unwrap_or(500).min(10_000);
+ let steps = input.steps.unwrap_or(20).clamp(1, 200) as usize;
+ let modifiers = input.modifier.unwrap_or_default();
+ let wayland = crate::wayland::wayland_input_enabled();
+ if wayland && !modifiers.is_empty() {
+ return wayland_modified_drag_refusal();
+ }
+ let path = if wayland { "wayland_desktop" } else { "xtest" };
+ let space = match desktop_input_space().await {
+ Ok(space) => space,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ // The overlay draws in layout coordinates, like the input below.
+ let overlay_point = |x: f64, y: f64| {
+ let (lx, ly) = space.to_layout(x.round() as i32, y.round() as i32);
+ (f64::from(lx), f64::from(ly))
+ };
+ let (overlay_from, overlay_to) =
+ (overlay_point(from_x, from_y), overlay_point(to_x, to_y));
+ let result = spawn_blocking_bounded(
+ "desktop drag",
+ foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
+ move || {
+ if wayland {
+ crate::wayland::drag_desktop(
+ &space,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ steps as u32,
+ duration_ms,
+ button,
+ )
+ } else {
+ let modifier_refs: Vec<&str> =
+ modifiers.iter().map(String::as_str).collect();
+ crate::input::send_drag_xtest_desktop_with_modifiers(
+ from_x.round() as i32,
+ from_y.round() as i32,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ button,
+ duration_ms,
+ steps,
+ &modifier_refs,
+ )
+ }
+ },
+ );
+ let visual_drag = track_overlay_drag_for(
+ cursor_id.clone(),
+ overlay_from,
+ overlay_to,
+ duration_ms,
+ steps,
+ );
+ let (result, ()) = tokio::join!(result, visual_drag);
+ if matches!(&result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, overlay_to.0, overlay_to.1);
+ }
+ return match result {
+ Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
+ input_error_result(error)
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let xid = match args.opt_u64("window_id") {
+ Some(v) => v,
+ None => return ToolResult::error("window_id is required on Linux."),
+ };
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ let isolated_background = isolated_hyprland_background(delivery);
+ let native_refusal = if !isolated_background {
+ if crate::wayland::wayland_input_enabled() && !modifiers.is_empty() {
+ return wayland_modified_drag_refusal();
+ }
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
+ return refusal;
+ }
+ unavailable_wayland_focused_input_background(delivery, true)
+ } else {
+ None
+ };
+
+ let from_zoom = args.bool_or("from_zoom", false);
+ let desktop_frame = desktop_frame_requested(&args);
+ let context = match coordinate_drag_context(native_refusal, || {
+ if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated below.
+ Ok(CoordinateContext::Screenshot(1.0))
+ } else if from_zoom {
+ self.state
+ .zoom_context(&args, pid, Some(xid))
+ .map(CoordinateContext::Zoom)
+ } else {
+ screenshot_scale(&self.state, &args, pid, Some(xid))
+ .map(CoordinateContext::Screenshot)
+ }
+ }) {
+ Ok(context) => context,
+ Err(refusal) => return refusal,
+ };
+
+ let coerce = |key: &str| -> Option<f64> {
+ args.opt_f64(key)
+ .or_else(|| args.opt_i64(key).map(|i| i as f64))
+ };
+ let mut from_x = match coerce("from_x") {
+ Some(v) => v,
+ None => {
+ return invalid_pointer_arguments(
+ "drag",
+ "from_x is missing (from_x/from_y/to_x/to_y are all required)",
+ )
+ }
+ };
+ let mut from_y = match coerce("from_y") {
+ Some(v) => v,
+ None => {
+ return invalid_pointer_arguments(
+ "drag",
+ "from_y is missing (from_x/from_y/to_x/to_y are all required)",
+ )
+ }
+ };
+ let mut to_x = match coerce("to_x") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing: to_x"),
+ };
+ let mut to_y = match coerce("to_y") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing: to_y"),
+ };
+
+ let duration_ms = args.u64_or("duration_ms", 500);
+ let steps = drag_steps(&args);
+ let button_str = args.str_or("button", "left");
+ let button = parse_mouse_button(button_str.as_str());
+ match context {
+ CoordinateContext::Zoom(context) => {
+ (from_x, from_y) = context.zoom_to_window(from_x, from_y);
+ (to_x, to_y) = context.zoom_to_window(to_x, to_y);
+ }
+ CoordinateContext::Screenshot(scale) => {
+ from_x *= scale;
+ from_y *= scale;
+ to_x *= scale;
+ to_y *= scale;
+ }
+ }
+ if desktop_frame {
+ let mapped = tokio::task::spawn_blocking(move || {
+ Ok::<_, anyhow::Error>((
+ desktop_to_window_local(xid, from_x, from_y)?,
+ desktop_to_window_local(xid, to_x, to_y)?,
+ ))
+ })
+ .await;
+ match mapped {
+ Ok(Ok(((fx, fy), (tx, ty)))) => {
+ from_x = fx;
+ from_y = fy;
+ to_x = tx;
+ to_y = ty;
+ }
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+
+ if hyprland_foreground(delivery) {
+ if button_str != "left" || args.get("modifier").is_some_and(|value| !value.is_null()) {
+ return foreground_hyprland_refusal(
+ "foreground drag supports only an unmodified left button",
+ );
+ }
+ if !crate::wayland::hyprland_input::enabled() {
+ return foreground_hyprland_refusal(
+ "production Hyprland input plugin is unavailable",
+ );
+ }
+ let owner = named_session_cursor_key(&args);
+ let points = crate::wayland::window_local_to_output(
+ xid,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ )
+ .and_then(|from| {
+ crate::wayland::window_local_to_output(
+ xid,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ )
+ .map(|to| (from.point, to.point))
+ });
+ let (from, to) = match points {
+ Ok(points) => points,
+ Err(error) => return input_error_result(error),
+ };
+ let (started, acknowledged) = tokio::sync::oneshot::channel();
+ let (_cancellation, mut dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ crate::wayland::hyprland_input::execute_foreground_with_started(
+ owner,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Drag {
+ from: (from_x, from_y),
+ to: (to_x, to_y),
+ duration_ms,
+ },
+ Some(started),
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(_) => {
+ return foreground_hyprland_refusal(
+ "authenticated admitted lifecycle required",
+ )
+ }
+ };
+ let acknowledged = acknowledged.await.is_ok();
+ let result = if acknowledged {
+ overlay_snap_to_for(&cursor_id, from.0 as f64, from.1 as f64, None);
+ tokio::select! {
+ result = &mut dispatch => result,
+ () = track_overlay_drag_for(cursor_id.clone(), (from.0 as f64, from.1 as f64),
+ (to.0 as f64, to.1 as f64), duration_ms, steps) => dispatch.await,
+ }
+ } else {
+ dispatch.await
+ };
+ return hyprland_input_result(
+ match result {
+ Ok(result) => result,
+ Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
+ error.into(),
+ u32::from(acknowledged),
+ )),
+ },
+ true,
+ );
+ }
+
+ if isolated_background {
+ if button_str != "left" || args.get("modifier").is_some_and(|value| !value.is_null()) {
+ return isolated_hyprland_refusal(
+ "isolated drag supports only an unmodified left button",
+ );
+ }
+ let owner = named_session_cursor_key(&args);
+ let points = crate::wayland::window_local_to_output(
+ xid,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ )
+ .and_then(|from| {
+ crate::wayland::window_local_to_output(
+ xid,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ )
+ .map(|to| (from.point, to.point))
+ });
+ let (from, to) = match points {
+ Ok(points) => points,
+ Err(error) => return input_error_result(error),
+ };
+ let (started, acknowledged) = tokio::sync::oneshot::channel();
+ let (_cancellation, mut dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ crate::wayland::hyprland_input::execute_with_started(
+ owner,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Drag {
+ from: (from_x, from_y),
+ to: (to_x, to_y),
+ duration_ms,
+ },
+ Some(started),
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(refusal) => return refusal,
+ };
+ // Do not animate a drag that was refused. The compositor starts
+ // the overlay clock only after accepting and pressing the button.
+ let acknowledged = acknowledged.await.is_ok();
+ if acknowledged {
+ overlay_snap_to_for(&cursor_id, from.0 as f64, from.1 as f64, None);
+ tokio::select! {
+ result = &mut dispatch => {
+ return match result { Ok(result) => isolated_hyprland_result(result), Err(error) => isolated_hyprland_task_error(error, acknowledged) };
+ }
+ () = track_overlay_drag_for(cursor_id.clone(), (from.0 as f64, from.1 as f64),
+ (to.0 as f64, to.1 as f64), duration_ms, steps) => {}
+ }
+ }
+ return match dispatch.await {
+ Ok(result) => isolated_hyprland_result(result),
+ Err(error) => isolated_hyprland_task_error(error, acknowledged),
+ };
+ }
+
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ let wayland_points = match crate::wayland::wayland_input_enabled()
+ .then(|| {
+ crate::wayland::window_local_to_output(
+ xid,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ )
+ .and_then(|from| {
+ crate::wayland::window_local_to_output(
+ xid,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ )
+ .map(|to| (from, to))
+ })
+ })
+ .transpose()
+ {
+ Ok(points) => points,
+ Err(error) => return input_error_result(error),
+ };
+ let screen_from = if let Some((from, _)) = wayland_points {
+ Some((from.point.0 as f64, from.point.1 as f64))
+ } else {
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, from_x, from_y))
+ .await
+ .ok()
+ .and_then(|result| result.ok())
+ };
+ if let Some((sx_from, sy_from)) = screen_from {
+ overlay_glide_to_for(&cursor_id, sx_from, sy_from).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx_from, sy_from);
+ overlay_snap_to_for(&cursor_id, sx_from, sy_from, None);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx_from,
+ y: sy_from,
+ },
+ );
+ }
+ // Native Wayland: emit press + interpolated motion + release as one
+ // virtual-pointer (wlroots) or libei (GNOME/KDE) sequence, output-relative
+ // coords. Returns early so we don't fall into the X11 XSendEvent loop below.
+ if crate::wayland::wayland_input_enabled() {
+ let steps_u32 = steps as u32;
+ let inject = crate::wayland::is_inject_mode();
+ let (from_aimed, to_aimed) = wayland_points.unwrap_or_else(|| {
+ (
+ (from_x.round() as i32, from_y.round() as i32).into(),
+ (to_x.round() as i32, to_y.round() as i32).into(),
+ )
+ });
+ let drag_result = spawn_blocking_bounded(
+ "drag",
+ foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
+ move || {
+ if inject {
+ return crate::wayland::inject_drag(
+ pid,
+ xid,
+ (from_x, from_y),
+ (to_x, to_y),
+ steps,
+ button as u32,
+ );
+ }
+ crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::pointer_gesture_restore(),
+ || {
+ crate::wayland::drag(
+ xid,
+ from_aimed,
+ to_aimed,
+ steps_u32,
+ duration_ms,
+ button,
+ )
+ },
+ )
+ },
+ );
+ let ((from_output_x, from_output_y), (to_output_x, to_output_y)) =
+ (from_aimed.point, to_aimed.point);
+ let visual_drag = track_overlay_drag_for(
+ cursor_id.clone(),
+ (f64::from(from_output_x), f64::from(from_output_y)),
+ (f64::from(to_output_x), f64::from(to_output_y)),
+ duration_ms,
+ steps,
+ );
+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
+ if matches!(&drag_result, Ok(Ok(()))) {
+ self.state.cursor_registry.update_position(
+ &cursor_id,
+ f64::from(to_output_x),
+ f64::from(to_output_y),
+ );
+ }
+ return match drag_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Posted drag ({button_str}) to pid {pid} \
+ from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \
+ in {duration_ms}ms / {steps} steps."
+ )),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ if delivery.is_foreground() {
+ let screen_points = tokio::task::spawn_blocking(move || {
+ Ok::<_, anyhow::Error>((
+ window_local_to_screen(xid, from_x, from_y)?,
+ window_local_to_screen(xid, to_x, to_y)?,
+ ))
+ })
+ .await;
+ let ((screen_from_x, screen_from_y), (screen_to_x, screen_to_y)) = match screen_points {
+ Ok(Ok(points)) => points,
+ Ok(Err(e)) => return ToolResult::error(e.to_string()),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ };
+ let drag_result = spawn_blocking_bounded(
+ "foreground drag",
+ foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
+ move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ crate::input::send_drag_xtest_desktop_with_modifiers(
+ screen_from_x.round() as i32,
+ screen_from_y.round() as i32,
+ screen_to_x.round() as i32,
+ screen_to_y.round() as i32,
+ button,
+ duration_ms,
+ steps,
+ &modifier_refs,
+ )
+ },
+ )
+ .map(|(value, _report)| value)
+ },
+ );
+ let visual_drag = track_overlay_drag_for(
+ cursor_id.clone(),
+ (screen_from_x, screen_from_y),
+ (screen_to_x, screen_to_y),
+ duration_ms,
+ steps,
+ );
+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
+ if matches!(&drag_result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, screen_to_x, screen_to_y);
+ }
+ return match drag_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Dragged ({button_str}) to pid {pid} from ({from_x:.0}, {from_y:.0}) \
+ to ({to_x:.0}, {to_y:.0}) in {duration_ms}ms / {steps} steps \
+ (delivery_mode=foreground)."
+ ))
+ .with_structured(json!({
+ "path": "x11_xtest_fg",
+ "verified": false,
+ "delivery_mode": "foreground"
+ })),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Background X11 drag. With a real Xorg server and /dev/uinput the
+ // gesture is a held press + interpolated motion + release on the
+ // session's MPX virtual master pointer (GTK/VCL/Qt/Chromium drop the
+ // synthetic XSendEvent gesture below); the target's screen points are
+ // resolved once and the agent cursor tracks the glide.
+ if crate::input::real_pointer_input_available() {
+ let screen_points = tokio::task::spawn_blocking(move || {
+ Ok::<_, anyhow::Error>((
+ window_local_to_screen(xid, from_x, from_y)?,
+ window_local_to_screen(xid, to_x, to_y)?,
+ ))
+ })
+ .await;
+ let ((sfx, sfy), (stx, sty)) = match screen_points {
+ Ok(Ok(points)) => points,
+ Ok(Err(e)) => return ToolResult::error(format!("{e:#}")),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ };
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let cursor_id_for_task = cursor_id.clone();
+ let drag_result = spawn_blocking_bounded(
+ "drag",
+ foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
+ move || {
+ let (outcome, guard) =
+ crate::input::focus_guard::guarded_settled(Some(pid), || {
+ Ok(crate::input::send_virtual_pointer_drag(
+ &cursor_id_for_task,
+ &crate::input::VirtualPointerDrag {
+ target_window: xid,
+ button,
+ path: vec![
+ (sfx.round() as i32, sfy.round() as i32),
+ (stx.round() as i32, sty.round() as i32),
+ ],
+ duration_ms,
+ steps,
+ },
+ ))
+ })?;
+ outcome.map(|mut effect| {
+ effect.focus_guard = guard;
+ effect
+ })
+ },
+ )
+ .await;
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ if matches!(&drag_result, Ok(Ok(_))) {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::track_pointer_command(stx, sty),
+ );
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, stx, sty);
+ }
+ return match drag_result {
+ Ok(Ok(effect)) => {
+ let route = PointerRoute::Mpx(effect);
+ ToolResult::text(format!(
+ "Dragged ({button_str}) pid {pid} from ({from_x:.0}, {from_y:.0}) to \
+ ({to_x:.0}, {to_y:.0}) in {duration_ms}ms / {steps} steps with a real \
+ held pointer gesture {}",
+ route.text_suffix("background")
+ ))
+ .with_structured(route.structured("background"))
+ }
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Legacy target-addressed XSendEvent gesture (headless servers / no
+ // uinput): only core-protocol toolkits accept it.
+ if synthetic_pointer_is_dropped(pid) {
+ return crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let press_result = press_x11_drag_start(
+ foreground_budget(0),
+ move || {
+ crate::input::send_button_down(
+ xid,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ button,
+ )
+ },
+ move || {
+ crate::input::send_button_up(
+ xid,
+ from_x.round() as i32,
+ from_y.round() as i32,
+ button,
+ )
+ },
+ )
+ .await;
+ let mut result: anyhow::Result<()> = match press_result {
+ Ok(Ok(())) => Ok(()),
+ Ok(Err(e)) => Err(e),
+ Err(e) => Err(anyhow::anyhow!("Task error: {e}")),
+ };
+
+ if result.is_ok() {
+ let step_delay_ms = if steps > 1 {
+ duration_ms / steps as u64
+ } else {
+ duration_ms
+ };
+ for i in 1..=steps {
+ let t = i as f64 / steps.max(1) as f64;
+ let ix = from_x + (to_x - from_x) * t;
+ let iy = from_y + (to_y - from_y) * t;
+ let motion_result =
+ spawn_blocking_bounded("drag", foreground_budget(0), move || {
+ crate::input::send_motion(
+ xid,
+ ix.round() as i32,
+ iy.round() as i32,
+ Some(button),
+ )
+ })
+ .await;
+ match motion_result {
+ Ok(Ok(())) => {
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy))
+ .await
+ {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::track_pointer_command(sx, sy),
+ );
+ }
+ if step_delay_ms > 0 {
+ tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms))
+ .await;
+ }
+ }
+ Ok(Err(e)) => {
+ result = Err(e);
+ break;
+ }
+ Err(e) => {
+ result = Err(anyhow::anyhow!("Task error: {e}"));
+ break;
+ }
+ }
+ }
+ }
+
+ // XSendEvent does not read the call's action, so the release still
+ // goes out if the call ends first.
+ let release_result = spawn_blocking_bounded("drag", foreground_budget(0), move || {
+ crate::input::send_button_up(xid, to_x.round() as i32, to_y.round() as i32, button)
+ })
+ .await;
+ if result.is_ok() {
+ result = match release_result {
+ Ok(Ok(())) => Ok(()),
+ Ok(Err(e)) => Err(e),
+ Err(e) => Err(anyhow::anyhow!("Task error: {e}")),
+ };
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+
+ if result.is_ok() {
+ if let Ok(Ok((sx_to, sy_to))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y)).await
+ {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::track_pointer_command(sx_to, sy_to),
+ );
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx_to, sy_to);
+ }
+ }
+
+ let route = PointerRoute::Synthetic;
+ match result {
+ Ok(()) => ToolResult::text(format!(
+ "Posted drag ({button_str}) to pid {pid} \
+ from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \
+ in {duration_ms}ms / {steps} steps {}",
+ route.text_suffix("background")
+ ))
+ .with_structured(route.structured("background")),
+ Err(e) => input_error_result(e),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_accessibility_tree.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_accessibility_tree.rs
new file mode 100644
index 0000000000..f612f8a096
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_accessibility_tree.rs
@@ -0,0 +1,91 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+use super::impl_::{visible_live_desktop_windows, window_record_json};
+
+// ── get_accessibility_tree ────────────────────────────────────────────────────
+
+pub struct GetAccessibilityTreeTool;
+
+static GAX_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetAccessibilityTreeTool {
+ fn def(&self) -> &ToolDef {
+ GAX_DEF.get_or_init(|| ToolDef {
+ name: "get_accessibility_tree".into(),
+ description: "Return a lightweight snapshot of the desktop: running processes and \
+ on-screen visible windows with their bounds and owner pid.\n\n\
+ For the full AT-SPI subtree of a single window (with interactive element indices \
+ you can click by), use get_window_state instead — this is a fast discovery read."
+ .into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ let (procs, windows) = tokio::task::spawn_blocking(|| {
+ (
+ crate::proc_fs::list_processes(),
+ visible_live_desktop_windows(crate::wayland::list_windows_dispatch(None)),
+ )
+ })
+ .await
+ .unwrap_or_default();
+
+ if crate::wayland::is_wayland() {
+ crate::wayland::remember_observed_window_origins(&windows);
+ }
+
+ let mut lines = vec![format!(
+ "{} running process(es), {} visible window(s)",
+ procs.len(),
+ windows.len()
+ )];
+ for p in &procs {
+ let cmd = if p.cmdline.is_empty() {
+ p.name.clone()
+ } else {
+ p.cmdline.clone()
+ };
+ lines.push(format!("- {} (pid {})", cmd, p.pid));
+ }
+ if !windows.is_empty() {
+ lines.push(String::new());
+ lines.push("Windows:".to_owned());
+ for w in &windows {
+ let title = if w.title.is_empty() {
+ "(no title)".to_owned()
+ } else {
+ format!("\"{}\"", w.title)
+ };
+ lines.push(format!(
+ "- pid={} window_id={} {} {}x{}+{}+{}",
+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
+ w.xid,
+ title,
+ w.width,
+ w.height,
+ w.x,
+ w.y
+ ));
+ }
+ lines.push(
+ "→ Call get_window_state(pid, window_id) to inspect a window's UI.".to_owned(),
+ );
+ }
+
+ let structured = json!({
+ "processes": procs.iter().map(|p| json!({"pid":p.pid,"name":p.name})).collect::<Vec<_>>(),
+ "windows": windows.iter().map(window_record_json).collect::<Vec<_>>()
+ });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_config.rs
new file mode 100644
index 0000000000..d4951b3867
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_config.rs
@@ -0,0 +1,45 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+// ── get_config ────────────────────────────────────────────────────────────────
+
+pub struct GetConfigTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static GCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetConfigTool {
+ fn def(&self) -> &ToolDef {
+ GCFG_DEF.get_or_init(|| ToolDef {
+ name: "get_config".into(),
+ description: "Return current cua-driver-rs configuration.".into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ let cfg = self.state.config.read().unwrap();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
+ ToolResult::text("cua-driver-rs configuration").with_structured(json!({
+ "version": env!("CARGO_PKG_VERSION"),
+ "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
+ "platform": "linux",
+ "capture_mode": cfg.capture_mode,
+ "max_image_dimension": cfg.max_image_dimension,
+ "cursor": cursor_overlay::cursor_options::saved_config_json(),
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry
+ }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_cursor_position.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_cursor_position.rs
new file mode 100644
index 0000000000..60066fbe6c
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_cursor_position.rs
@@ -0,0 +1,93 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetCursorPositionInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+// ── get_cursor_position ───────────────────────────────────────────────────────
+
+pub struct GetCursorPositionTool;
+static GCP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetCursorPositionTool {
+ fn def(&self) -> &ToolDef {
+ GCP_DEF.get_or_init(|| ToolDef {
+ name: "get_cursor_position".into(),
+ description:
+ "Return the current mouse cursor position in screen points (origin top-left)."
+ .into(),
+ input_schema: json!({"type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema()
+ },"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Err(result) =
+ parse_typed_input::<GetCursorPositionInput>("get_cursor_position", args)
+ {
+ return result;
+ }
+ // Hyprland reports the real pointer over IPC; return it in the
+ // desktop-frame coordinates that get_desktop_state and desktop actions use.
+ if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
+ let position = tokio::task::spawn_blocking(|| -> anyhow::Result<(i32, i32)> {
+ let (x, y) = crate::wayland::hyprland::cursor_position()?;
+ Ok(crate::wayland::hyprland::desktop_frame()?
+ .from_layout(x.round() as i32, y.round() as i32))
+ })
+ .await;
+ // The synthetic registry below holds layout coordinates, not the
+ // desktop frame, so a failed compositor query is an error here.
+ return match position {
+ Ok(Ok((x, y))) => {
+ ToolResult::text(format!("✅ Cursor at ({x}, {y}) in the desktop frame"))
+ .with_structured(json!({ "x": x, "y": y, "source": "compositor" }))
+ }
+ Ok(Err(e)) => ToolResult::error(format!(
+ "Could not read the Hyprland pointer position: {e:#}"
+ )),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // Native Wayland: there's no protocol for clients to query the real
+ // global cursor position. Fall back to the synthetic registry that
+ // records every `motion_absolute` this process emits.
+ if crate::wayland::is_wayland() {
+ return match crate::wayland::last_synth_cursor_pos() {
+ Some((x, y)) => ToolResult::text(
+ format!("✅ Cursor at ({x}, {y}) (synthetic — last move_cursor in this process)")
+ ).with_structured(json!({
+ "x": x, "y": y, "source": "synthetic"
+ })),
+ None => ToolResult::text(
+ "Cursor position unknown on Wayland — no move_cursor has been issued in this process yet.".to_string()
+ ).with_structured(json!({ "source": "synthetic", "available": false })),
+ };
+ }
+ let result = tokio::task::spawn_blocking(|| {
+ use x11rb::connection::Connection;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
+ let (conn, screen_num) = RustConnection::connect(None)?;
+ let root = conn.setup().roots[screen_num].root;
+ let reply = conn.query_pointer(root)?.reply()?;
+ Ok::<(i32, i32), anyhow::Error>((reply.root_x as i32, reply.root_y as i32))
+ })
+ .await;
+ match result {
+ // Text format matches Swift `GetCursorPositionTool` 1:1.
+ Ok(Ok((x, y))) => ToolResult::text(format!("✅ Cursor at ({x}, {y})"))
+ .with_structured(json!({ "x": x, "y": y, "source": "x11" })),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_desktop_state.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_desktop_state.rs
new file mode 100644
index 0000000000..016f545318
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_desktop_state.rs
@@ -0,0 +1,346 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetDesktopStateInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ normalize_desktop_capture_for_action_frame, window_record_json, x11_screen_size, ToolState,
+};
+
+#[cfg(test)]
+use super::impl_::maps_indicate_synthetic_pointer_dropped;
+
+// ── get_desktop_state ─────────────────────────────────────────────────────────
+
+pub struct GetDesktopStateTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static GDS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetDesktopStateTool {
+ fn def(&self) -> &ToolDef {
+ GDS_DEF.get_or_init(|| ToolDef {
+ name: "get_desktop_state".into(),
+ description: "Capture the full display in the desktop action coordinate frame. \
+ Use the returned PNG directly as the coordinate source for actions whose target is \
+ {kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "session":{"type":"string","description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session."},
+ "screenshot_out_file":{"type":"string","description":"Write PNG here instead of base64."},
+ "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()
+ },"additionalProperties":false}),
+ read_only: true, destructive: false, idempotent: false, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let capture_args = args.clone();
+ let input = match parse_typed_input::<GetDesktopStateInput>("get_desktop_state", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let out_file = input.screenshot_out_file;
+ let max_image_dimension = input.max_image_dimension;
+ let capture_service = self.state.capture_service.clone();
+
+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
+ // Capture the full display at native size first. When the
+ // compositor consumes logical input coordinates, normalize the
+ // image below so screenshot pixels still land exactly.
+ //
+ // The agent reads this image, so the Driver's own cursor and
+ // session pill are hidden around the grab (or the limitation is
+ // reported) instead of being baked over the controls it reads.
+ // Hyprland: read the desktop frame once. The screenshot, its action
+ // size and the window geometry below all use it.
+ let hyprland_plan =
+ if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
+ Some(crate::wayland::hyprland::desktop_capture_plan()?)
+ } else {
+ None
+ };
+ // niri: the single output's frame, read before the capture and
+ // compared after it.
+ let niri_frame = (crate::wayland::is_wayland() && crate::wayland::niri::is_session())
+ .then(|| crate::wayland::niri::desktop_snapshot().map(|snapshot| snapshot.frame))
+ .transpose()?;
+ let (native_png, overlay_capture) =
+ cursor_overlay::capture_exclusion::capture_excluding_overlays(
+ &crate::overlay_capture::OverlayExcluder,
+ |hidden| {
+ // A multi-monitor layout is composed from the frame
+ // read above, so the image is exactly that frame.
+ let png = match &hyprland_plan {
+ Some(plan) if plan.composite => {
+ crate::wayland::hyprland::compose_desktop_capture(&plan.frame)?
+ }
+ _ => crate::capture::screenshot_display_bytes()?,
+ };
+ Ok::<_, anyhow::Error>(crate::overlay_capture::verify_hidden_capture(
+ png, hidden,
+ ))
+ },
+ )?;
+ let (native_w, native_h) = crate::capture::png_dimensions_pub(&native_png)?;
+ // True screen size. On a pure-Wayland session (native backend
+ // opted in, no X11 DISPLAY) the capture above came from the
+ // wlroots `zwlr_screencopy` cascade, whose full-display buffer is
+ // the whole output at native (physical) pixels — so the PNG
+ // dimensions ARE the true screen size. Querying the X11 root
+ // window here would fail with "$DISPLAY variable not set" and
+ // abort the tool even though the screenshot already succeeded.
+ // Only fall back to the X11 root-window geometry off Wayland, so
+ // the X11 / XWayland path is unchanged. See #2017 / Sway testing.
+ let hyprland_frame = hyprland_plan.map(|plan| plan.frame);
+ let (screen_w, screen_h) = if let Some(frame) = &hyprland_frame {
+ (frame.width, frame.height)
+ } else if crate::wayland::is_wayland() {
+ crate::capture_action_frame::desktop_action_dimensions((native_w, native_h))?
+ } else {
+ x11_screen_size()?
+ };
+ let (png, shot_w, shot_h, scale_factor) =
+ normalize_desktop_capture_for_action_frame(native_png, screen_w, screen_h)?;
+ // Opt-in long-edge cap: the full-size capture is the desktop
+ // action frame; a capped image is mapped back at dispatch
+ // (cua_driver_core::desktop_capture_scale) and by its capture_id.
+ let (png, shot_w, shot_h) = match max_image_dimension.filter(|cap| *cap > 0) {
+ Some(cap) if shot_w.max(shot_h) > cap => {
+ let png = crate::capture::resize_png_if_needed(&png, cap)?;
+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;
+ (png, w, h)
+ }
+ _ => (png, shot_w, shot_h),
+ };
+ // Optional: write PNG to disk instead of returning base64.
+ let written = if let Some(path) = out_file.as_deref() {
+ std::fs::write(path, &png)?;
+ Some(path.to_string())
+ } else {
+ None
+ };
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ let b64 = if written.is_some() {
+ None
+ } else {
+ Some(B64.encode(&png))
+ };
+ // Which pid owns which window is what the agent needs before its
+ // first click; without it the first action guesses `pid: 1`.
+ let mut windows = crate::wayland::list_windows_dispatch(None);
+ windows
+ .retain(|w| w.is_on_screen && w.pid.is_some_and(crate::proc_fs::is_process_live));
+ if let Some(frame) = &hyprland_frame {
+ // Window geometry is in layout coordinates; screenshot pixel
+ // (0, 0) is the frame origin. Refuse rather than pair this
+ // capture with a layout that changed meanwhile.
+ anyhow::ensure!(
+ crate::wayland::hyprland::desktop_frame()? == *frame,
+ "the Hyprland monitor layout changed during get_desktop_state; call it again"
+ );
+ frame.rebase_windows(&mut windows);
+ } else if let Some(frame) = &niri_frame {
+ // niri window geometry is in layout coordinates; screenshot
+ // pixel (0, 0) is the single output's logical origin.
+ anyhow::ensure!(
+ crate::wayland::niri::desktop_snapshot()?.frame == *frame,
+ "the niri output layout changed during get_desktop_state; call it again"
+ );
+ frame.rebase_windows(&mut windows);
+ }
+ let capture_id = crate::capture_action_frame::publish_desktop(
+ &capture_service,
+ &capture_args,
+ &png,
+ (shot_w, shot_h),
+ (screen_w, screen_h),
+ )?;
+ Ok((
+ b64,
+ shot_w,
+ shot_h,
+ screen_w,
+ screen_h,
+ scale_factor,
+ written,
+ windows,
+ capture_id,
+ overlay_capture,
+ ))
+ })
+ .await;
+
+ match result {
+ Ok(Ok((
+ b64_opt,
+ shot_w,
+ shot_h,
+ screen_w,
+ screen_h,
+ scale_factor,
+ written,
+ windows,
+ capture_id,
+ overlay_capture,
+ ))) => {
+ let frame_scale = if shot_w > 0 {
+ f64::from(screen_w) / f64::from(shot_w)
+ } else {
+ 1.0
+ };
+ let mut content = Vec::new();
+ let mut structured = json!({
+ "platform": "linux",
+ "display": "primary",
+ "screenshot_width": shot_w,
+ "screenshot_height": shot_h,
+ "screen_width": screen_w,
+ "screen_height": screen_h,
+ "scale_factor": scale_factor,
+ "frame_scale": frame_scale,
+ "screenshot_mime_type": "image/png",
+ "windows": windows.iter().map(window_record_json).collect::<Vec<_>>(),
+ "capture_id": capture_id,
+ "agent_overlay_capture": overlay_capture,
+ });
+ if (frame_scale - 1.0).abs() > 0.001 {
+ // Capped: the uncapped capture is the action frame.
+ structured["screenshot_original_width"] = json!(screen_w);
+ structured["screenshot_original_height"] = json!(screen_h);
+ }
+ if let Some(b64) = b64_opt {
+ content.push(cua_driver_core::protocol::Content::image_png(b64));
+ }
+ let window_lines = desktop_window_lines(&windows, frame_scale);
+ let mut frame_note = if (frame_scale - 1.0).abs() > 0.001 {
+ format!(
+ "; x/y for scope:\"desktop\" actions are pixels of THIS screenshot \
+ (mapped ×{frame_scale:.2} back to the screen automatically)"
+ )
+ } else {
+ String::new()
+ };
+ cursor_overlay::capture_exclusion::append_summary_note(
+ &mut frame_note,
+ &overlay_capture,
+ );
+ if let Some(path) = written {
+ structured["screenshot_file_path"] = json!(path);
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "✅ Desktop screenshot {shot_w}x{shot_h} written to {path} (screen {screen_w}x{screen_h}{frame_note}){window_lines}"
+ )));
+ } else {
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "✅ Desktop screenshot {shot_w}x{shot_h} (screen {screen_w}x{screen_h}{frame_note}){window_lines}"
+ )));
+ }
+ ToolResult {
+ content,
+ is_error: None,
+ structured_content: Some(structured),
+ action_record: None,
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("Capture error: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+/// Text block naming every visible window with the `pid` / `window_id` pair
+/// that every other tool takes, so the model's first action targets a real
+/// process instead of guessing.
+fn desktop_window_lines(windows: &[crate::x11::WindowInfo], frame_scale: f64) -> String {
+ let px = |v: i32| (f64::from(v) / frame_scale).round() as i32;
+ let sz = |v: u32| (f64::from(v) / frame_scale).round() as u32;
+ if windows.is_empty() {
+ return "\nVisible windows: none (use list_windows / launch_app).".to_owned();
+ }
+ let mut out =
+ String::from("\nVisible windows (use these pid + window_id values in every tool call):");
+ for w in windows {
+ let title = if w.title.is_empty() {
+ "(no title)".to_owned()
+ } else {
+ format!("\"{}\"", w.title)
+ };
+ out.push_str(&format!(
+ "\n- pid={} window_id={} {} {}x{} at ({},{}){}",
+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
+ w.xid,
+ title,
+ sz(w.width),
+ sz(w.height),
+ px(w.x),
+ px(w.y),
+ if w.app_name.is_empty() {
+ String::new()
+ } else {
+ format!(" app={}", w.app_name)
+ }
+ ));
+ }
+ out.push_str("\n→ get_window_state(pid, window_id) lists clickable elements; click/type_text take the same pid (+ window_id for x,y).");
+ out
+}
+
+#[cfg(test)]
+mod background_budget_tests {
+ use super::*;
+
+ #[test]
+ fn synthetic_pointer_drop_detection_covers_gtk_vcl_and_qt() {
+ assert!(maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libgtk-3.so.0\n"
+ ));
+ assert!(maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libgtk-4.so.1\n"
+ ));
+ assert!(maps_indicate_synthetic_pointer_dropped(
+ "7f /opt/libreoffice/program/libmergedlo.so\n"
+ ));
+ assert!(maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libvcllo.so\n"
+ ));
+ assert!(maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libQt5Gui.so.5\n"
+ ));
+ assert!(!maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libgtk-x11-2.0.so.0\n"
+ ));
+ assert!(!maps_indicate_synthetic_pointer_dropped(
+ "7f /usr/lib/libX11.so.6\n"
+ ));
+ }
+
+ #[test]
+ fn desktop_window_lines_expose_pid_and_window_id() {
+ let windows = vec![crate::x11::WindowInfo {
+ xid: 0x2e00003,
+ pid: Some(4321),
+ app_name: "libreoffice".into(),
+ title: "Untitled 1 - LibreOffice Calc".into(),
+ is_on_screen: true,
+ z_index: None,
+ x: 0,
+ y: 27,
+ width: 1920,
+ height: 1053,
+ }];
+ let text = desktop_window_lines(&windows, 1.0);
+ assert!(text.contains("pid=4321 window_id=48234499"));
+ assert!(text.contains("LibreOffice Calc"));
+ assert!(text.contains("app=libreoffice"));
+ assert!(text.contains("get_window_state(pid, window_id)"));
+ assert!(desktop_window_lines(&[], 1.0).contains("none"));
+ // A downsized desktop screenshot reports window rects in ITS pixels.
+ let scaled = desktop_window_lines(&windows, 1.5);
+ assert!(scaled.contains("1280x702 at (0,18)"), "{scaled}");
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_screen_size.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_screen_size.rs
new file mode 100644
index 0000000000..8fb3d5d40b
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_screen_size.rs
@@ -0,0 +1,107 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetScreenSizeInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+use super::impl_::x11_screen_size;
+
+// ── get_screen_size ───────────────────────────────────────────────────────────
+
+pub struct GetScreenSizeTool;
+static GSS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetScreenSizeTool {
+ fn def(&self) -> &ToolDef {
+ GSS_DEF.get_or_init(|| ToolDef {
+ name: "get_screen_size".into(),
+ description: "Return the logical size of the main display in points plus its backing \
+ scale factor. Agents click in points; Retina displays have scale_factor 2.0. \
+ Requires no TCC permissions."
+ .into(),
+ input_schema: json!({"type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema()
+ },"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Err(result) = parse_typed_input::<GetScreenSizeInput>("get_screen_size", args) {
+ return result;
+ }
+ let result = tokio::task::spawn_blocking(|| {
+ if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
+ // Shared manifest admission needs content-free display
+ // metadata even when this native desktop has no X11 DISPLAY.
+ // The adapter attests the IPC/Wayland compositor peer and
+ // refuses layouts its logical frame cannot represent: no
+ // powered output, or a rotated powered output.
+ // Scaled outputs report logical pixels plus the output scale.
+ // The size and the monitor list come from one snapshot.
+ let (frame, monitors) = crate::wayland::hyprland::screen_report()?;
+ return Ok((frame.width, frame.height, frame.scale, Some(monitors)));
+ }
+ if crate::wayland::is_wayland() && crate::wayland::niri::is_session() {
+ // niri reports logical output geometry and its fractional
+ // scale over attested IPC, for a single-output layout.
+ let (frame, monitors) = crate::wayland::niri::screen_report()?;
+ return Ok((frame.width, frame.height, frame.scale, Some(monitors)));
+ }
+ // X11 reports pixel dimensions; scale factor on X11 is not
+ // well-defined per-monitor, so report 1.0 (matches DPI-unaware
+ // assumption). Other Wayland compositors retain their existing
+ // limitation; do not infer native metadata support there.
+ let (w, h) = x11_screen_size()?;
+ Ok::<(u32, u32, f64, Option<Value>), anyhow::Error>((w, h, 1.0, None))
+ })
+ .await;
+ match result {
+ Ok(Ok((w, h, scale, Some(monitors)))) => {
+ // Tell the agent which monitors are actually on: the desktop
+ // frame covers powered monitors only and changes as they turn
+ // on or off, so re-read it after any display change.
+ let describe = |m: &Value| {
+ let name = m["name"].as_str().unwrap_or("?");
+ match (
+ m["powered"].as_bool(),
+ m["frame_x"].as_i64(),
+ m["frame_y"].as_i64(),
+ ) {
+ (Some(true), Some(x), Some(y)) => format!(
+ "{name} {}x{} on, at ({x},{y}) in the desktop frame",
+ m["width"], m["height"]
+ ),
+ _ => format!("{name} off (standby), not in the desktop frame"),
+ }
+ };
+ let lines: Vec<String> = monitors
+ .as_array()
+ .into_iter()
+ .flatten()
+ .map(describe)
+ .collect();
+ ToolResult::text(format!(
+ "✅ Desktop frame: {w}x{h} points @ {scale}x, spanning every monitor that is on.\n{}",
+ lines.join("\n")
+ ))
+ .with_structured(json!({
+ "width": w, "height": h, "scale_factor": scale, "monitors": monitors,
+ }))
+ }
+ // Matches Swift text format 1:1.
+ Ok(Ok((w, h, scale, None))) => {
+ ToolResult::text(format!("✅ Main display: {w}x{h} points @ {scale}x"))
+ .with_structured(json!({ "width": w, "height": h, "scale_factor": scale }))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_window_state.rs
new file mode 100644
index 0000000000..d4ff3057d7
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/get_window_state.rs
@@ -0,0 +1,1188 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ framed_elements_first, non_ax_escalation, popup_menu_elements, rects_intersect, ToolState,
+ WindowOverlays,
+};
+
+// ── get_window_state ─────────────────────────────────────────────────────────
+
+/// The largest window screenshot delivered as-is. The Anthropic API downsizes
+/// any image above ~1.15 megapixels before the model sees it, so a 1568x861
+/// PNG reached the model as ~1447x795 while the driver still mapped its x/y
+/// as 1568x861: every pixel click landed a uniform 0.94x short. Capping here
+/// keeps "pixels of THIS screenshot" true for the image the model reads.
+const WINDOW_SCREENSHOT_MAX_PIXELS: u64 = 1_150_000;
+
+/// The long edge a `w`x`h` image must shrink to so that it holds at most
+/// `max_pixels` pixels; `None` when it already fits.
+fn megapixel_long_edge_cap(w: u32, h: u32, max_pixels: u64) -> Option<u32> {
+ let pixels = u64::from(w) * u64::from(h);
+ if w == 0 || h == 0 || pixels <= max_pixels {
+ return None;
+ }
+ let long = w.max(h) as f64;
+ let short = w.min(h) as f64;
+ let mut edge = (long * (max_pixels as f64 / pixels as f64).sqrt()).floor() as u32;
+ // The resizer rounds both sides; step down until the rounded image fits.
+ while edge > 1 {
+ let scale = edge as f64 / long;
+ let fits = ((long * scale).round() as u64) * ((short * scale).round() as u64) <= max_pixels;
+ if fits {
+ break;
+ }
+ edge -= 1;
+ }
+ Some(edge)
+}
+
+#[cfg(test)]
+mod megapixel_cap_tests {
+ use super::{megapixel_long_edge_cap, WINDOW_SCREENSHOT_MAX_PIXELS};
+
+ #[test]
+ fn a_window_shot_above_the_api_threshold_is_capped_below_it() {
+ let edge =
+ megapixel_long_edge_cap(1568, 861, WINDOW_SCREENSHOT_MAX_PIXELS).expect("capped");
+ let scale = edge as f64 / 1568.0;
+ let (w, h) = (
+ (1568.0 * scale).round() as u64,
+ (861.0 * scale).round() as u64,
+ );
+ assert!(w * h <= WINDOW_SCREENSHOT_MAX_PIXELS, "{w}x{h}");
+ assert!((1440..1568).contains(&edge), "{edge}");
+ }
+
+ #[test]
+ fn a_shot_within_the_threshold_is_left_alone() {
+ assert_eq!(
+ megapixel_long_edge_cap(1280, 800, WINDOW_SCREENSHOT_MAX_PIXELS),
+ None
+ );
+ assert_eq!(
+ megapixel_long_edge_cap(0, 800, WINDOW_SCREENSHOT_MAX_PIXELS),
+ None
+ );
+ }
+}
+
+/// Build a single structured element entry for `get_window_state`.
+/// Returns `None` when the node has no `element_index` (non-actionable rows).
+fn build_element_entry(
+ n: &crate::atspi::AtspiNode,
+ snapshot_id: Option<u32>,
+ bounds: Option<(i32, i32, u32, u32)>,
+) -> Option<serde_json::Value> {
+ let idx = n.element_index?;
+ // `label` mirrors what a human reading the markdown row would call this
+ // element: its name, else its description (Qt keeps a button's tooltip
+ // there: "Play"/"Pause"). Never its value: four spin buttons all labelled
+ // "0.0" cannot be told apart — such a control is `unlabelled` instead,
+ // with its place among its siblings in `description`.
+ let unlabelled = n.name.is_none()
+ && n.description
+ .as_deref()
+ .is_some_and(|d| d.starts_with(crate::atspi::native::UNLABELLED_NOTE_PREFIX));
+ let label = n
+ .name
+ .clone()
+ .or_else(|| n.description.clone().filter(|_| !unlabelled));
+ let mut entry = json!({
+ "element_index": idx,
+ "role": n.role,
+ "depth": n.depth,
+ });
+ if let Some(snapshot_id) = snapshot_id {
+ entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx));
+ }
+ if n.in_web_content {
+ entry["in_web_content"] = json!(true);
+ }
+ if let Some(label) = label {
+ entry["label"] = json!(label);
+ }
+ if unlabelled {
+ entry["unlabelled"] = json!(true);
+ }
+ // Surface the element's value separately from `label` (which collapses
+ // name→value→description): a field with both a name AND typed text would
+ // otherwise hide the text from a caller reading the structured side,
+ // leaving it only in tree_markdown. See the macOS get_window_state builder
+ // for the rationale.
+ if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) {
+ entry["value"] = json!(value);
+ }
+ if let Some(enabled) = n.enabled {
+ entry["enabled"] = json!(enabled);
+ }
+ if let Some(selected) = n.selected {
+ entry["selected"] = json!(selected);
+ }
+ let actions: Vec<String> = n
+ .actions
+ .iter()
+ .filter(|a| !a.trim().is_empty())
+ .cloned()
+ .collect();
+ if !actions.is_empty() {
+ entry["actions"] = json!(actions);
+ }
+ if let Some(parent) = n.parent_element_index {
+ entry["parent_index"] = json!(parent);
+ }
+ if let Some((x, y, w, h)) = bounds {
+ entry["frame"] = json!({ "x": x, "y": y, "w": w, "h": h });
+ }
+ if let Some(description) = n.description.clone().filter(|d| !d.is_empty()) {
+ entry["description"] = json!(description);
+ }
+ Some(entry)
+}
+
+fn window_overlays(pid: u32, xid: u64) -> WindowOverlays {
+ let mut out = WindowOverlays {
+ window_rect: crate::x11::window_info(xid).map(|w| (w.x, w.y, w.width, w.height)),
+ ..Default::default()
+ };
+ let over = |rect: (i32, i32, u32, u32)| {
+ out.window_rect
+ .is_some_and(|target| rects_intersect(rect, target))
+ };
+ let mut covers = false;
+ for window in crate::x11::list_windows(Some(pid)) {
+ if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {
+ continue;
+ }
+ let Some(owner) = crate::x11::transient_for(window.xid) else {
+ continue;
+ };
+ covers |= over((window.x, window.y, window.width, window.height));
+ out.dialogs.push(json!({
+ "window_id": window.xid,
+ "title": window.title,
+ "transient_for": owner,
+ "modal": crate::x11::window_is_modal(window.xid),
+ "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },
+ }));
+ }
+ // A dialog/plugin window can legitimately run as a DIFFERENT process than
+ // the application it belongs to (GIMP's separate-process export dialogs,
+ // LibreOffice's Document Recovery dialog under a distinct soffice.bin).
+ // Exact-pid matching above would make such a window invisible even though
+ // it is unambiguously this application's own popup. WM_TRANSIENT_FOR
+ // resolving to one of pid's own windows is the correlation signal (see
+ // `list_cross_pid_transient_windows`); such entries are tagged with
+ // `owning_pid` so callers can tell it apart from a same-process dialog.
+ for window in crate::x11::list_cross_pid_transient_windows(pid) {
+ if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {
+ continue;
+ }
+ let Some(owner) = crate::x11::transient_for(window.xid) else {
+ continue;
+ };
+ covers |= over((window.x, window.y, window.width, window.height));
+ out.dialogs.push(json!({
+ "window_id": window.xid,
+ "title": window.title,
+ "transient_for": owner,
+ "modal": crate::x11::window_is_modal(window.xid),
+ "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },
+ "owning_pid": window.pid,
+ }));
+ }
+ for popup in crate::input::mapped_popup_windows() {
+ if popup.window == xid || popup.pid.is_some_and(|owner| owner != pid) {
+ continue;
+ }
+ // Desktop-wide override-redirect windows that are nobody's menu:
+ // mutter's guard window and the driver's own cursor overlay.
+ if popup.pid.is_none()
+ && (popup.title.contains("guard window") || popup.title.starts_with("Cua."))
+ {
+ continue;
+ }
+ covers |= over((popup.x, popup.y, popup.width, popup.height));
+ out.popups.push(popup.to_json());
+ }
+ out.covers_window = covers;
+ out
+}
+
+pub struct GetWindowStateTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+const COLD_START_WALK_TIMEOUT_MS: u64 = 2_000;
+
+/// The AT-SPI walker's own node budget when the caller gives none.
+const LINUX_DEFAULT_MAX_ELEMENTS: usize = 5_000;
+
+fn linux_snapshot_timeout_ms(timeout: Option<&Value>, has_prior_snapshot: bool) -> u64 {
+ cua_driver_core::tool_schema::resolve_timeout_ms_with_first_snapshot_grace(
+ timeout,
+ has_prior_snapshot,
+ COLD_START_WALK_TIMEOUT_MS,
+ )
+}
+
+static GWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetWindowStateTool {
+ fn def(&self) -> &ToolDef {
+ GWS_DEF.get_or_init(|| ToolDef {
+ name: "get_window_state".into(),
+ description: "Walk a running app's AT-SPI tree and return it ONCE, as \
+ compact Markdown by default (`tree_format:\"markdown\"`). Every actionable \
+ element is tagged `[N]` (an element_index); its element_token is \
+ `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and \
+ returned in structuredContent. Pass `tree_format:\"elements\"` for the \
+ structured `elements` array instead (explicit `element_token`, `role`, \
+ `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when \
+ AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for \
+ both (about twice the size). Set `query` to project the tree to matching rows \
+ plus their ancestor chain while preserving original indices; \
+ `total_element_count` reports the complete snapshot.\n\n\
+ Size: a read is bounded by default (`max_elements` 250 actionable nodes). \
+ When the walk stops at that budget the response says `Tree truncated at \
+ max_elements=…` and `truncated: true`; pass a larger `max_elements`, or \
+ `query` / `max_depth`, to reach the rest. `_note` is omitted unless \
+ `verbose:true`. `full_output:true` restores the previous full response: both \
+ representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\n\
+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \
+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
+ back to a full read and `since_status` says why. The focused element is not \
+ reported on Linux yet.\n\n\
+ Always returns BOTH the element tree AND a screenshot — ground on \
+ both and cross-check (the tree lies on some surfaces). Choose the \
+ modality at ACTION time: an element ax action \
+ (element_token → accessibility rung) or an element px \
+ action (x,y → pixel rung off this screenshot). capture_mode is \
+ deprecated and ignored. On Wayland, where output capture cannot prove \
+ the requested surface's identity, the truthful tree is returned without \
+ a screenshot and `screenshot_error.code` is \
+ `surface_identity_unproven`.\n\n\
+ The mirror image: pass `include_accessibility_tree:false` to SKIP \
+ the AT-SPI walk entirely and return just the screenshot plus \
+ window metadata (window_bounds, app_name, window_title) — the \
+ capture-only path for a live window preview / picture-in-picture. \
+ Setting BOTH `include_accessibility_tree:false` and \
+ `include_screenshot:false` is an error. Optional `max_dimension` \
+ caps the returned screenshot's long edge in pixels for a cheap \
+ thumbnail.\n\n\
+ Optional `max_elements` / `max_depth` bound the AT-SPI walk \
+ (Electron / large web apps produce 10k+ element trees). Markdown and \
+ structured elements are truncated identically. Default: `max_elements` 250.\n\n\
+ TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI \
+ walk. When omitted, a window's first snapshot gets a 2000 ms \
+ cold-start budget so Chromium/Electron can finish publishing its \
+ initial tree. Explicit values are always honored. Large apps \
+ (LibreOffice, GIMP, file managers) can exceed the budget; \
+ the call then returns the PARTIAL tree with `truncated: true`, \
+ `truncation_reason`, `nodes_visited`/`nodes_pending` and \
+ `elements_complete: false`. Every element listed is real and \
+ clickable; elements after the cut are simply missing. Retry with a \
+ larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` \
+ when the element you need is absent.\n\n\
+ SCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels \
+ (long edge <= max_image_dimension, 1568 by default), because larger images \
+ are downsized before a model reads them and its pixel coordinates would then \
+ be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools \
+ are pixels of the delivered screenshot (`frame` stays in screen coordinates, \
+ as on every platform); `frame_scale` < 1 reports the downsizing. An \
+ explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\n\
+ POPUP MENUS: a context menu / popover / combo list is an \
+ override-redirect window that list_windows never shows. A click or \
+ right_click that opened one names it in its result (`popup: \
+ {window_id, bounds, title}`); pass that window_id here to walk the \
+ popup's own AT-SPI toplevel so its menu items get element indices \
+ (then click them by element_token). Omitting window_id while a popup \
+ of this pid is open walks that popup.".into(),
+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Process ID that owns the window."},
+ "window_id":{"type":"integer","description":"Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window."},
+ "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
+ "include_accessibility_tree":{"type":"boolean",
+ "description":"Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
+ "include_screenshot":{"type":"boolean",
+ "description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
+ "screenshot_out_file":{"type":"string",
+ "description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},
+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
+ "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},
+ "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
+ "max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted."},
+ "max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior."}
+ },"additionalProperties":false})),
+ read_only: true, destructive: false, idempotent: false, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {
+ Ok(display_only) => display_only,
+ Err(refusal) => return refusal,
+ };
+ use cua_driver_core::tool_args::ArgsExt;
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ // `window_id` omitted: the open popup menu of this pid when one is
+ // mapped (a context menu the caller wants to read), else the pid's
+ // focused / dialog / active / largest window.
+ let xid = match args.opt_u64("window_id") {
+ Some(v) => v,
+ None => {
+ let chosen = tokio::task::spawn_blocking(move || {
+ if let Some(popup) = crate::input::mapped_popup_windows()
+ .into_iter()
+ .rev()
+ .find(|p| p.pid == Some(pid))
+ {
+ return Some(popup.window);
+ }
+ let windows = crate::x11::list_windows(Some(pid));
+ let candidates: Vec<u64> = windows.iter().map(|w| w.xid).collect();
+ crate::x11::pick_pid_window(
+ &windows,
+ crate::x11::focused_window_among(&candidates),
+ crate::x11::transient_for,
+ crate::x11::active_window(),
+ )
+ })
+ .await
+ .ok()
+ .flatten();
+ match chosen {
+ Some(v) => v,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ // Retain the legacy additive cap, but let the canonical per-call field
+ // replace the configured ceiling entirely (including 0 = native size).
+ let max_dimension = args
+ .get("max_dimension")
+ .and_then(|v| v.as_u64())
+ .map(|v| v.max(1) as u32);
+ let max_image_dimension = match args.get("max_image_dimension") {
+ None => None,
+ Some(value) => match value.as_u64().and_then(|value| u32::try_from(value).ok()) {
+ Some(value) => Some(value),
+ None => {
+ return ToolResult::error(
+ "get_window_state.max_image_dimension must be an integer from 0 through 4294967295.",
+ )
+ .with_structured(json!({ "code": "invalid_arguments" }))
+ }
+ },
+ };
+ let explicit_max_image_dimension = max_image_dimension.is_some();
+ let max_dim = {
+ let cfg = self.state.config.read().unwrap();
+ cua_driver_core::image_utils::ImageDimensionLimits {
+ configured: cfg.max_image_dimension,
+ legacy_max_dimension: max_dimension,
+ max_image_dimension,
+ }
+ .resolve()
+ };
+ // `capture_mode` is DEPRECATED and ignored — get_window_state always
+ // returns BOTH the AT-SPI tree and a screenshot now, so the agent grounds
+ // on both and cross-checks (the tree lies often enough that a grounding
+ // screenshot should always be present). The modality is chosen at action
+ // time: an element ax action (element_token) or element px action (x,y).
+ // We don't even read the arg; it stays in the schema only so old callers
+ // don't trip additionalProperties:false.
+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
+ Ok(view) => view,
+ Err(refusal) => return refusal,
+ };
+ let query = args.opt_str("query");
+ let session_id = args.opt_str("_session_id");
+ // include_screenshot (default true) — the perf opt-out. The tree+screenshot
+ // pair is the default; `include_screenshot:false` skips the grab and returns
+ // tree only (the cheap re-index path before an element ax action). A
+ // screenshot_out_file still forces a capture (to disk), regardless.
+ let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
+ // `include_accessibility_tree` (default true) mirrors include_screenshot:
+ // set false to SKIP the AT-SPI walk and return just the screenshot +
+ // window metadata (the capture-only / preview path).
+ let want_tree = args
+ .get("include_accessibility_tree")
+ .and_then(|v| v.as_bool())
+ != Some(false);
+ // screenshot_out_file: when set, write the PNG to disk and surface the
+ // path instead of embedding base64 in the response. `~` expands.
+ let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {
+ if let Some(rest) = s.strip_prefix("~/") {
+ let home = std::env::var("HOME").unwrap_or_default();
+ format!("{home}/{rest}")
+ } else {
+ s
+ }
+ });
+ // Optional caps — when omitted, the AT-SPI walker uses its built-in
+ // defaults (#22865).
+ let max_elements = Some(
+ view.max_elements(
+ &args,
+ LINUX_DEFAULT_MAX_ELEMENTS,
+ display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true),
+ ),
+ );
+ let max_depth = args
+ .get("max_depth")
+ .and_then(|v| v.as_u64())
+ .map(|v| v.max(1) as usize);
+ let timeout_ms = linux_snapshot_timeout_ms(
+ args.get("timeout_ms"),
+ self.state
+ .snapshots
+ .contains_semantic_window(pid as i32, xid),
+ );
+ let walk_timeout = std::time::Duration::from_millis(timeout_ms);
+
+ let process_is_live = crate::proc_fs::is_process_live(pid);
+ // Enumerate the pid's windows ONCE and reuse the result for both the
+ // window-ownership check (Wayland) and the additive window metadata
+ // below, instead of paying for the compositor/X11 enumeration twice.
+ // `window_meta` also names the surface + its on-screen rectangle on the
+ // capture-only path, where no AT-SPI tree identifies it.
+ let popup_meta = (!crate::wayland::is_wayland())
+ .then(|| crate::input::popup_window_info(xid))
+ .flatten();
+ let window_meta = crate::wayland::list_windows_dispatch(Some(pid))
+ .into_iter()
+ .find(|w| w.xid == xid)
+ .or_else(|| {
+ popup_meta.as_ref().map(|popup| crate::x11::WindowInfo {
+ xid: popup.window,
+ pid: popup.pid,
+ app_name: String::new(),
+ title: popup.title.clone(),
+ is_on_screen: true,
+ z_index: None,
+ x: popup.x,
+ y: popup.y,
+ width: popup.width,
+ height: popup.height,
+ })
+ });
+ let window_matches = if crate::wayland::is_wayland() {
+ window_meta.as_ref().is_some_and(|w| w.pid == Some(pid))
+ || crate::wayland::window_was_listed_for_pid(pid, xid)
+ } else {
+ crate::x11::window_belongs_to_pid(xid, pid)
+ || crate::input::popup_window_info(xid)
+ .is_some_and(|popup| popup.pid.is_none_or(|owner| owner == pid))
+ // A cross-process dialog/plugin window transient-for one of
+ // pid's own windows (GIMP export dialogs, LibreOffice's
+ // Document Recovery) legitimately belongs to a different pid
+ // than the caller's target; accept it here too so the
+ // get_window_state(window_id=...) follow-up named in the
+ // dialogs[] overlay actually works.
+ || crate::x11::list_cross_pid_transient_windows(pid)
+ .iter()
+ .any(|w| w.xid == xid)
+ };
+ if !process_is_live || !window_matches {
+ return ToolResult::error(format!(
+ "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows."
+ ));
+ }
+
+ // Always walk the AT-SPI tree; capture the screenshot by default. The
+ // tree+screenshot pair is the default so the agent grounds on both and
+ // cross-checks the (sometimes-lying) tree against the frame. An explicit
+ // `include_screenshot:false` skips the grab; an unproven Wayland surface
+ // returns the tree with a typed screenshot error instead of unrelated pixels.
+ let should_capture = include_screenshot != Some(false) || screenshot_out_file.is_some();
+ if !want_tree && !should_capture {
+ return ToolResult::error(
+ "Nothing to return: both include_accessibility_tree:false and \
+ include_screenshot:false. Set at least one to true, or pass \
+ screenshot_out_file to force a capture.",
+ );
+ }
+ // `display_only` is the public form of the internal observation-only
+ // read: pixels for a preview, with no snapshot or capture change.
+ let observation_only = display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true);
+ let state = self.state.clone();
+ let state_for_capture = state.clone();
+ let query_for_walk = query.clone();
+
+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
+ // Skip the AT-SPI walk on the capture-only path
+ // (include_accessibility_tree:false).
+ let tree_result = if want_tree {
+ Some(crate::atspi::walk_tree_bounded_within(
+ pid,
+ xid,
+ query_for_walk.as_deref(),
+ max_elements,
+ max_depth,
+ walk_timeout,
+ ))
+ } else {
+ None
+ };
+ let binds_window = tree_result.as_ref().is_some_and(|tree| {
+ crate::atspi::snapshot_binds_window(pid, xid, tree.window_scoped, tree.trusted)
+ });
+ // Bounds and element indices come from the same captured AT-SPI
+ // traversal. Joining two live walks by ordinal mis-associated
+ // Chromium controls when its lazy subtree changed between walks.
+ let bounds = tree_result
+ .as_ref()
+ .map(|tree| tree.bounds.clone())
+ .unwrap_or_default();
+ // Capture and DELIVER the screenshot alongside the tree by default — the
+ // grounding frame the agent cross-checks the tree against. With
+ // screenshot_out_file set, write to disk and surface the path instead
+ // of embedding base64; otherwise embed base64. Skipped only when
+ // include_screenshot:false and no disk path was requested.
+ // Keep the exact delivered PNG for capture publication. The action
+ // binding must identify the bytes returned to the caller, including
+ // any max-dimension resize, rather than the backend's raw frame.
+ let mut screenshot_error = None;
+ // Same-pid popups (menus, combo lists) and transient dialogs mapped
+ // over this window: listed for the caller, and when one overlaps
+ // the window the screenshot is taken from the screen, since the
+ // window's own drawable never shows them.
+ let overlays = if crate::wayland::is_wayland() {
+ WindowOverlays::default()
+ } else {
+ window_overlays(pid, xid)
+ };
+ let screenshot = if should_capture {
+ // A screenshot-only read with a size cap (a live preview) asks
+ // a GNOME helper with API 10 to downscale in the compositor,
+ // and keeps the window's full size for frame_scale.
+ let preview = if !want_tree && max_dim > 0 && crate::wayland::is_wayland() {
+ crate::wayland::shell_helper::screenshot_window_preview(xid, pid, max_dim)
+ } else {
+ None
+ };
+ let captured = if let Some((png, width, height)) = preview {
+ Ok((png, Some((width, height))))
+ } else if overlays.covers_window {
+ overlays
+ .window_rect
+ .ok_or_else(|| anyhow::anyhow!("window geometry unavailable"))
+ .and_then(|(x, y, w, h)| {
+ crate::capture::screenshot_root_region_png(x, y, w, h)
+ })
+ .or_else(|_| crate::wayland::screenshot_dispatch_with_pid(xid, pid))
+ .map(|png| (png, None))
+ } else {
+ crate::wayland::screenshot_dispatch_with_pid(xid, pid).map(|png| (png, None))
+ };
+ match captured {
+ Ok((raw, original_size)) => {
+ let (orig_w, orig_h) = match original_size {
+ Some(size) => size,
+ None => crate::capture::png_dimensions_pub(&raw)?,
+ };
+ let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;
+ // An explicit per-call max_image_dimension (0 = native)
+ // is authoritative; the model-safe cap applies otherwise.
+ let cap = if explicit_max_image_dimension {
+ None
+ } else {
+ megapixel_long_edge_cap(w, h, WINDOW_SCREENSHOT_MAX_PIXELS)
+ };
+ let png = match cap {
+ Some(edge) => crate::capture::resize_png_if_needed(&png, edge)?,
+ None => png,
+ };
+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;
+ let original_w = if w < orig_w { Some(orig_w) } else { None };
+ let (b64, file_path) = if let Some(ref path) = screenshot_out_file {
+ std::fs::write(path, &png)?;
+ (None, Some(path.clone()))
+ } else {
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ (Some(B64.encode(&png)), None)
+ };
+ let capture_id = if observation_only {
+ None
+ } else {
+ Some(crate::capture_action_frame::publish_window(
+ &state_for_capture.capture_service,
+ &args,
+ &png,
+ pid,
+ xid,
+ (w, h),
+ (orig_w, orig_h),
+ )?)
+ };
+ Some((b64, file_path, w, h, original_w, capture_id))
+ }
+ Err(error) if crate::wayland::is_surface_identity_unproven(&error) => {
+ screenshot_error = Some(error.to_string());
+ None
+ }
+ // The window passed the ownership check above and went
+ // away during the AT-SPI walk (a document closed by the
+ // action being observed): a stale target, not a capture
+ // failure.
+ Err(_) if !crate::wayland::is_wayland() && !crate::x11::window_exists(xid) => {
+ return Err(anyhow::anyhow!(
+ "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows."
+ ));
+ }
+ Err(error) => {
+ return Err(anyhow::anyhow!(
+ "window screenshot failed for window {xid}: {error}"
+ ));
+ }
+ }
+ } else {
+ None
+ };
+ Ok((
+ tree_result,
+ screenshot,
+ bounds,
+ screenshot_error,
+ overlays,
+ binds_window,
+ ))
+ })
+ .await;
+
+ match result {
+ Ok(Ok((tree_opt, shot_opt, bounds, screenshot_error, overlays, binds_window))) => {
+ let mut content = Vec::new();
+ let mut structured = json!({ "window_id": xid, "pid": pid });
+ let screenshot_scale = shot_opt.as_ref().map(|(_, _, w, _, original_w, _)| {
+ original_w.map_or(1.0, |ow| ow as f64 / *w as f64)
+ });
+ let mut published_snapshot = false;
+ let mut invalidated = Vec::new();
+
+ if let Some(tr) = tree_opt {
+ let source_trusted = tr.trusted;
+ let source_degraded_reason = tr.degraded_reason.clone();
+ let count = tr
+ .nodes
+ .iter()
+ .filter(|n| n.element_index.is_some())
+ .count();
+ let mut header = format!(
+ "window_id={xid} pid={pid} elements={count} walk_ms={}\n",
+ tr.elapsed_ms
+ );
+ if tr.truncated {
+ header.push_str(&cua_driver_core::walk_budget::truncation_note(
+ tr.truncation_reason.as_deref(),
+ timeout_ms,
+ tr.nodes_visited,
+ tr.nodes_pending,
+ ));
+ header.push('\n');
+ } else if !tr.bounds_complete {
+ header.push_str(
+ "⚠️ bounds phase ran out of time: some elements have no frame \
+ (element_token clicks still work; pixel targeting may not). \
+ Retry with a larger timeout_ms if you need frames.\n",
+ );
+ }
+ header.push('\n');
+ content.push(cua_driver_core::protocol::Content::text(
+ header + &tr.tree_markdown,
+ ));
+ structured["element_count"] = json!(count);
+ // `elements_complete` is a real claim now: a native walk that
+ // finished without hitting the deadline / node budget saw
+ // every node the application publishes. Truncated or
+ // fallback trees keep negative existence unknown.
+ structured["elements_complete"] = json!(tr.trusted && !tr.truncated);
+ structured["truncated"] = json!(tr.truncated);
+ if let Some(reason) = &tr.truncation_reason {
+ structured["truncation_reason"] = json!(reason);
+ }
+ structured["nodes_visited"] = json!(tr.nodes_visited);
+ structured["nodes_pending"] = json!(tr.nodes_pending);
+ structured["bounds_complete"] = json!(tr.bounds_complete);
+ structured["walk_elapsed_ms"] = json!(tr.elapsed_ms as u64);
+ structured["timeout_ms"] = json!(timeout_ms);
+ structured["tree_markdown"] = json!(tr.tree_markdown);
+
+ // Elements bind to this window only when the walk proved
+ // the window's scope (`snapshot_binds_window`).
+ let target_scoped = binds_window;
+ if !observation_only && !target_scoped {
+ invalidated.extend(state.snapshots.remove(pid as i32, xid));
+ crate::atspi::snapshot::forget_window(pid, xid);
+ }
+ let (snapshot_id, replaced) = (!observation_only && target_scoped)
+ .then(|| {
+ state.snapshots.publish_for_session(
+ pid as i32,
+ xid,
+ crate::atspi::snapshot::update_snapshot(
+ pid, xid, &tr.nodes, &tr.bounds,
+ ),
+ session_id.as_deref(),
+ screenshot_scale,
+ )
+ })
+ .flatten()
+ .unzip();
+ published_snapshot = snapshot_id.is_some();
+ invalidated.extend(replaced.into_iter().flatten());
+
+ // Structured `elements` array: one entry per actionable node.
+ // Shape: `{element_index, element_token, role, label,
+ // depth, actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is
+ // included whenever AT-SPI Component.GetExtents(Screen)
+ // reported usable bounds; omitted otherwise (some
+ // toolkits leave bounds unset on hidden / virtual
+ // elements).
+ use std::collections::HashMap;
+ // The walk produces screen extents (what the element
+ // cache and hit-tests compare against). The published
+ // `frame` is the element's screen rectangle, as on macOS and
+ // Windows; `screenshot_frame` (added below) is the same
+ // rectangle in pixels of the screenshot in this response,
+ // the space of the pointer tools' window-local x/y. The
+ // window-local origin is the X11 window's root-relative
+ // origin, the one `window_bounds` and
+ // `coordinate_frame:"desktop"` translation use. A popup
+ // (override-redirect) window is not in the WM's client
+ // list; when its origin cannot be read, the popup's own
+ // screen rectangle is the origin.
+ let local_origin = (!crate::wayland::is_wayland())
+ .then(|| crate::atspi::native::x11_window_origin(xid))
+ .flatten()
+ .or_else(|| popup_meta.as_ref().map(|popup| (popup.x, popup.y)));
+ let bounds_by_idx: HashMap<usize, (i32, i32, u32, u32)> = bounds
+ .into_iter()
+ .map(|(i, x, y, w, h)| (i, (x, y, w, h)))
+ .collect();
+ let elements: Vec<serde_json::Value> = tr
+ .nodes
+ .iter()
+ .filter_map(|n| {
+ build_element_entry(
+ n,
+ snapshot_id,
+ bounds_by_idx.get(&n.element_index?).copied(),
+ )
+ })
+ .collect();
+ let elements = cua_driver_core::element_query::project_elements_for_query(
+ elements,
+ query.as_deref(),
+ &tr.tree_markdown,
+ );
+ let elements = framed_elements_first(elements);
+ // Screenshot pixels: the capture is the window's own X11
+ // drawable (screen pixels), downsized by delivered/native.
+ let elements = match (local_origin, shot_opt.as_ref()) {
+ (Some((ox, oy)), Some((_, _, w, _, orig_w, _))) => {
+ let scale = orig_w.map_or(1.0, |ow| *w as f64 / ow as f64);
+ cua_driver_core::element_frame::with_screenshot_frames(
+ elements,
+ (f64::from(ox), f64::from(oy)),
+ scale,
+ )
+ }
+ _ => elements,
+ };
+ // A popup that has no AT-SPI frame of its own (LibreOffice
+ // VCL menus live under the menubar's `menu` node): return
+ // the open menu's items, the ones drawn inside the popup,
+ // instead of the whole application.
+ let elements = match (&popup_meta, tr.window_scoped) {
+ (Some(popup), false) => popup_menu_elements(
+ elements,
+ (popup.x, popup.y),
+ popup.width,
+ popup.height,
+ ),
+ _ => elements,
+ };
+ structured["total_element_count"] = json!(count);
+ structured["returned_element_count"] = json!(elements.len());
+ structured["elements"] = json!(elements);
+ // Surface 6: snapshot id mirror for debug correlation.
+ if let Some(snapshot_id) = snapshot_id {
+ structured["snapshot_id"] = json!(
+ cua_driver_core::element_token::format_snapshot_id(snapshot_id)
+ );
+ }
+ structured["_note"] = json!(
+ "Prefer `elements` — `tree_markdown` will continue to work \
+ but new fields will only be added to the structured side. \
+ Use `max_elements` / `max_depth` to bound the \
+ AT-SPI walk on apps with very large trees."
+ );
+ // Best-effort-background ladder parity with macOS/Windows: an
+ // AT-SPI walk that ran but found zero actionable elements is
+ // NOT a clean "this window has no controls" — far more often
+ // the bridge wasn't ready (toolkit-accessibility off, or the
+ // daemon isn't on the desktop session bus so the registry is
+ // empty), or it's a non-AX surface (canvas/WebGL). Mark it
+ // degraded so callers don't read `elements: []` as authoritative.
+ if !target_scoped {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!("accessibility_window_identity_unproven: tree is application-scoped; exact-window element tokens and bounds are unavailable");
+ } else if !source_trusted {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!(source_degraded_reason
+ .unwrap_or_else(|| {
+ "x11_property_fallback_partial: AT-SPI was unavailable and \
+ Cua Driver only recovered window metadata. Treat it as \
+ discovery evidence; it cannot prove checked state."
+ .to_owned()
+ }));
+ } else if count == 0 && tr.truncated {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] =
+ json!(cua_driver_core::walk_budget::truncation_note(
+ tr.truncation_reason.as_deref(),
+ timeout_ms,
+ tr.nodes_visited,
+ tr.nodes_pending,
+ ));
+ } else if count == 0 {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!(
+ "atspi_tree_empty: the AT-SPI walk returned no actionable \
+ elements. Common causes: the a11y bridge is off (enable \
+ `gsettings set org.gnome.desktop.interface \
+ toolkit-accessibility true`), the daemon is not on the \
+ desktop session bus (DBUS_SESSION_BUS_ADDRESS unreachable — \
+ run `cua-driver doctor`), or the window is a non-AX surface \
+ (canvas/WebGL/custom-drawn). Do not treat element data as \
+ authoritative — verify via the screenshot, and re-snapshot \
+ after enabling a11y or if the app just launched."
+ );
+ // Point the agent at the next rung explicitly: an empty
+ // tree means element_index has nothing to bind to. The
+ // recommendation is session-dependent (X11 → px,
+ // Wayland → foreground) — see non_ax_escalation.
+ structured["escalation"] = non_ax_escalation();
+ }
+ }
+
+ // A screenshot-only read adds an image but no element rows: it
+ // refreshes the current snapshot's pixel frame in place and
+ // keeps its tokens valid. Only a window with no element
+ // snapshot yet gets a new capture-only one.
+ let mut refreshed = None;
+ if let (false, false, Some(scale)) =
+ (observation_only, published_snapshot, screenshot_scale)
+ {
+ refreshed = state.snapshots.refresh_screenshot_for_session(
+ pid as i32,
+ xid,
+ session_id.as_deref(),
+ scale,
+ None,
+ );
+ if refreshed.is_none() {
+ if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(
+ pid as i32,
+ xid,
+ crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
+ session_id.as_deref(),
+ screenshot_scale,
+ ) {
+ invalidated.extend(replaced);
+ }
+ }
+ }
+ if let Some(id) = refreshed {
+ let sid = cua_driver_core::element_token::format_snapshot_id(id);
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
+ ({sid}:N) stay valid."
+ )));
+ structured["snapshot_id"] = json!(sid);
+ structured["screenshot_refreshed_snapshot"] = json!(true);
+ }
+ if !invalidated.is_empty() {
+ let ids: Vec<String> = invalidated
+ .into_iter()
+ .map(cua_driver_core::element_token::format_snapshot_id)
+ .collect();
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Invalidated snapshots {}: their element_tokens are stale.",
+ ids.join(", ")
+ )));
+ structured["invalidated_snapshot_ids"] = json!(ids);
+ }
+
+ if let Some((b64_opt, file_path, w, h, orig_w, capture_id)) = shot_opt {
+ // ax mode + screenshot_out_file writes the PNG to disk and
+ // returns b64=None — never embed the image bytes in that case.
+ // Keep a text content part when the image went to disk so the
+ // response is never empty on the capture-only path (which has
+ // no tree markdown either).
+ if let Some(b64) = b64_opt {
+ content.push(cua_driver_core::protocol::Content::image_png(b64));
+ } else if let Some(fp) = &file_path {
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "window_id={xid} pid={pid} size={w}x{h} screenshot written to {fp}"
+ )));
+ }
+ structured["screenshot_width"] = json!(w);
+ structured["screenshot_height"] = json!(h);
+ structured["screenshot_frame_valid"] = json!(true);
+ if let Some(ow) = orig_w {
+ if ow > 0 {
+ structured["frame_scale"] = json!(w as f64 / ow as f64);
+ structured["screenshot_original_width"] = json!(ow);
+ }
+ }
+ // Surface 7: mirror the MCP image part's `mimeType` onto
+ // the structured payload so consumers don't have to sniff
+ // magic bytes off the base64 to know the format.
+ structured["screenshot_mime_type"] = json!("image/png");
+ if let Some(capture_id) = capture_id {
+ structured["capture_id"] = json!(capture_id);
+ }
+ if let Some(fp) = file_path {
+ structured["screenshot_file_path"] = json!(fp);
+ }
+ }
+ if let Some(reason) = &screenshot_error {
+ structured["screenshot_frame_valid"] = json!(false);
+ structured["screenshot_error"] =
+ surface_identity_unproven_error(xid, reason.clone());
+ }
+ // Window identity metadata (additive): app + title + on-screen
+ // rectangle for the requested window_id, useful on the
+ // capture-only path where no AT-SPI tree names the surface.
+ if popup_meta.is_some() {
+ structured["popup"] = json!(true);
+ }
+ if let Some(meta) = &window_meta {
+ if !meta.app_name.is_empty() {
+ structured["app_name"] = json!(meta.app_name);
+ }
+ if !meta.title.is_empty() {
+ structured["window_title"] = json!(meta.title);
+ }
+ structured["window_bounds"] = json!({
+ "x": meta.x, "y": meta.y, "width": meta.width, "height": meta.height
+ });
+ }
+ // Transient dialogs and popups of this pid that are open over
+ // the window, each with the call that targets it.
+ if !overlays.dialogs.is_empty() {
+ structured["dialogs"] = json!(overlays.dialogs);
+ }
+ if !overlays.popups.is_empty() {
+ structured["popups"] = json!(overlays.popups);
+ }
+ if overlays.covers_window {
+ structured["screenshot_composited"] = json!(true);
+ }
+ if let Some(note) = overlays.follow_up(pid) {
+ structured["follow_up"] = json!(note);
+ content.push(cua_driver_core::protocol::Content::text(note));
+ }
+ structured["coordinate_frame"] = json!("window");
+ structured["frame_note"] = json!(
+ "x/y for click / double_click / right_click / drag / scroll on this \
+ window are pixels of THIS screenshot (window-local, 0..screenshot_width \
+ x 0..screenshot_height), as are the element frames; the screenshot is \
+ kept at or below 1.15 megapixels so the image you read is the image these \
+ pixels index (frame_scale < 1 says the window was downsized to it). \
+ window_bounds is where it sits on the screen. Pass scope:\"desktop\" only \
+ for get_desktop_state pixels."
+ );
+ if display_only {
+ cua_driver_core::window_state_view::mark_display_only(&mut structured);
+ }
+
+ // The capture-only path (include_accessibility_tree:false) leaves
+ // `content` empty when the screenshot was also unavailable — most
+ // often on Wayland, where per-window capture cannot prove surface
+ // identity. Return a structured error rather than a "successful"
+ // response with no content parts.
+ if content.is_empty() {
+ let reason_note = match &screenshot_error {
+ Some(reason) => format!(" and no screenshot could be captured ({reason})"),
+ None => " and no screenshot was returned".to_string(),
+ };
+ return ToolResult::error(format!(
+ "No content produced for window_id {xid}: the accessibility tree was \
+ skipped (include_accessibility_tree:false){reason_note}."
+ ))
+ .with_structured(structured);
+ }
+
+ if !observation_only {
+ cua_driver_core::window_state_view::apply(
+ &view,
+ &cua_driver_core::window_state_view::ViewContext {
+ pid: i64::from(pid),
+ window_id: xid,
+ key: cua_driver_core::window_state_view::ViewKey {
+ query: query.clone(),
+ max_elements: max_elements.unwrap_or(LINUX_DEFAULT_MAX_ELEMENTS),
+ max_depth,
+ },
+ focus_probe: None,
+ },
+ &mut content,
+ &mut structured,
+ );
+ }
+
+ ToolResult {
+ content,
+ is_error: None,
+ structured_content: Some(structured),
+ action_record: None,
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("Capture error: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+/// One-line, model-facing explanation of a partial tree and what to do about
+/// it. Shared by the text header and the structured `degraded_reason`.
+fn surface_identity_unproven_error(xid: u64, reason: String) -> Value {
+ json!({
+ "code": "surface_identity_unproven",
+ "window_id": xid,
+ "reason": reason,
+ "suggestion": "capture the full output explicitly, or retry on a compositor backend that supports identified per-window capture"
+ })
+}
+
+#[cfg(test)]
+mod get_window_state_actions_tests {
+ use super::*;
+ use crate::atspi::AtspiNode;
+
+ fn node(actions: Vec<String>) -> AtspiNode {
+ AtspiNode {
+ element_index: Some(1),
+ role: "button".to_owned(),
+ name: Some("ok".to_owned()),
+ value: None,
+ checked: None,
+ enabled: Some(true),
+ selected: None,
+ description: None,
+ actions,
+ element_key: 1,
+ identity: None,
+ depth: 0,
+ parent_element_index: None,
+ in_web_content: false,
+ object_ref: None,
+ }
+ }
+
+ #[test]
+ fn element_entry_omits_actions_when_empty() {
+ let n = node(Vec::new());
+ let entry = build_element_entry(&n, None, None).unwrap();
+ assert!(entry.get("actions").is_none());
+ }
+
+ #[test]
+ fn element_entry_filters_blank_action_names() {
+ let n = node(vec![
+ "Press".to_owned(),
+ "".to_owned(),
+ " ".to_owned(),
+ "Open".to_owned(),
+ ]);
+ let entry = build_element_entry(&n, None, None).unwrap();
+ assert_eq!(entry["actions"], json!(["Press", "Open"]));
+ }
+}
+
+#[cfg(test)]
+mod display_only_tests {
+ use super::{GetWindowStateTool, ToolState};
+ use cua_driver_core::tool::Tool;
+
+ // T3 Code feature-detects the preview mode from this schema property.
+ #[test]
+ fn display_only_is_advertised() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(),
+ };
+ assert_eq!(
+ tool.def().input_schema["properties"]["display_only"]["type"],
+ "boolean"
+ );
+ }
+
+ #[tokio::test]
+ async fn display_only_refuses_accessibility_snapshot_requests() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(),
+ };
+ for tree in [None, Some(true)] {
+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
+ if let Some(tree) = tree {
+ args["include_accessibility_tree"] = serde_json::json!(tree);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ serde_json::to_value(result).unwrap(),
+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
+ "display_only requires include_accessibility_tree:false"
+ ))
+ .unwrap()
+ );
+ }
+ }
+}
+#[cfg(test)]
+mod window_capture_dimension_tests {
+ use super::{GetWindowStateTool, ToolState};
+ use cua_driver_core::tool::Tool;
+
+ #[test]
+ fn schema_exposes_zero_as_native_resolution() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(),
+ };
+ assert_eq!(
+ tool.def().input_schema["properties"]["max_image_dimension"]["minimum"],
+ 0
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/hotkey.rs
new file mode 100644
index 0000000000..2feff91cd9
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/hotkey.rs
@@ -0,0 +1,464 @@
+use async_trait::async_trait;
+use cua_driver_contract::HotkeyInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ background_key_route, focus_by_pixel, focus_hyprland_foreground, focus_nested_inject_target,
+ foreground_budget, foreground_hyprland_action, foreground_hyprland_key,
+ foreground_hyprland_refusal, hyprland_foreground, input_error_result,
+ invalid_pointer_arguments, is_chromium_embedder, is_modifier, isolated_hyprland_action,
+ isolated_hyprland_background, isolated_hyprland_refusal, key_route_result,
+ position_named_session_keyboard_cursor, prove_token_element, resolve_cursor_key,
+ resolve_element_local_coords, spawn_blocking_bounded, stale_observed_element,
+ unavailable_chromium_background, unavailable_gtk_keyboard_background,
+ unavailable_wayland_focused_input_background, unavailable_webkit_keyboard_background,
+ wm_chord_background, KeyRoute, ToolState,
+};
+
+// ── hotkey ────────────────────────────────────────────────────────────────────
+
+pub struct HotkeyTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static HOTKEY_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for HotkeyTool {
+ fn def(&self) -> &ToolDef {
+ HOTKEY_DEF.get_or_init(|| ToolDef {
+ name: "hotkey".into(),
+ description: "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. \
+ Delivered to the target window without focus steal (real key events from a virtual \
+ master keyboard on X11); target does NOT need to be frontmost. Window-manager chords \
+ cannot reach the WM that way: alt+F4 on a window target closes that window through \
+ the WM (_NET_CLOSE_WINDOW, effect confirmed when it unmaps); super+*, alt+tab, \
+ ctrl+alt+* and other WM bindings are refused with code wm_chord_unavailable — use \
+ bring_to_front / list_windows / kill_app or delivery_mode:\"foreground\" instead.".into(),
+ input_schema: json!({
+ "type":"object","required":["keys"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to send the chord to the focused application."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
+ "keys":{"type":"array","items":{"type":"string"},"minItems":2,
+ "description":"Modifier(s) + one non-modifier key, e.g. [\"ctrl\",\"c\"]."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
+ "y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to send the chord to the focused application. Default \"window\"."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {
+ let input = match parse_typed_projection::<HotkeyInput>("hotkey", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let keys = input.keys;
+ if keys.len() < 2 {
+ return ToolResult::error("hotkey.keys must contain at least two keys.")
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ let modifiers: Vec<String> = keys
+ .iter()
+ .filter(|key| is_modifier(key))
+ .cloned()
+ .collect();
+ let Some(key) = keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {
+ return ToolResult::error("keys must include at least one non-modifier key.");
+ };
+ let display = keys.join("+");
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result =
+ spawn_blocking_bounded("desktop hotkey", foreground_budget(1), move || {
+ if wayland {
+ crate::wayland::hotkey_focused(&keys)
+ } else {
+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_key_xtest(&key, &refs)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let mut pid = args.u64_or("pid", 0) as u32;
+ let window_id_arg = args.opt_u64("window_id");
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(resolved) => resolved,
+ Err(error) => return error,
+ };
+ let token = match prove_token_element(pid, &resolved, "hotkey").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let resolved_element_index = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ let xid_opt = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
+ };
+
+ // Resolve XID: use window_id if given, else first window for pid.
+ let xid = match xid_opt {
+ Some(x) => x,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || {
+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
+ })
+ .await
+ .unwrap_or_default();
+ // pid omitted: the keys go to the active window, like a
+ // physical keyboard would, and the action adopts its pid.
+ let chosen = if pid == 0 {
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active && w.pid.is_some())
+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
+ } else {
+ // The pid's active window, else its largest on-screen
+ // toplevel: LibreOffice and GIMP own hidden/utility
+ // toplevels that a plain `first()` could pick, and an
+ // unmapped window cannot take the virtual keyboard focus.
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active)
+ .or_else(|| {
+ windows
+ .iter()
+ .filter(|w| w.is_on_screen)
+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
+ })
+ .or_else(|| windows.first())
+ };
+ match chosen {
+ Some(w) => {
+ if pid == 0 {
+ pid = w.pid.unwrap_or(0);
+ }
+ w.xid
+ }
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // Parse keys array (preferred) or fall back to legacy key+modifiers.
+ let (key, mods) = if let Some(arr) = args.get("keys").and_then(|v| v.as_array()) {
+ let keys: Vec<String> = arr
+ .iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect();
+ let modifiers: Vec<String> = keys.iter().filter(|k| is_modifier(k)).cloned().collect();
+ let non_mods: Vec<String> = keys.iter().filter(|k| !is_modifier(k)).cloned().collect();
+ if non_mods.is_empty() {
+ return ToolResult::error("keys must include at least one non-modifier key.");
+ }
+ (non_mods.last().unwrap().clone(), modifiers)
+ } else if let Some(k) = args.opt_str("key") {
+ let mods: Vec<String> = args.str_array("modifiers");
+ (k, mods)
+ } else {
+ return ToolResult::error(
+ "Provide 'keys' array (e.g. [\"ctrl\",\"c\"]) or 'key'+'modifiers' parameters.",
+ );
+ };
+
+ let key_display = format!("{}+{}", mods.join("+"), key);
+ let key_for_wayland = key.clone();
+ let mods_for_wayland = mods.clone();
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+
+ if isolated_hyprland_background(delivery) {
+ if xid_opt.is_none() {
+ return isolated_hyprland_refusal(
+ "an exact window_id is required for isolated hotkeys",
+ );
+ }
+ if resolved_element_index.is_some()
+ || args.get("x").is_some()
+ || args.get("y").is_some()
+ {
+ return isolated_hyprland_refusal("isolated hotkeys address the exact top-level; first click the child explicitly");
+ }
+ if let Some(keys) = args.get("keys").and_then(Value::as_array) {
+ if keys
+ .iter()
+ .filter(|value| value.as_str().is_some_and(|key| !is_modifier(key)))
+ .count()
+ != 1
+ || keys.iter().any(|value| !value.is_string())
+ {
+ return isolated_hyprland_refusal(
+ "isolated hotkeys require exactly one non-modifier key",
+ );
+ }
+ }
+ position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, false)
+ .await;
+ return isolated_hyprland_action(
+ &args,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Key {
+ key,
+ modifiers: mods,
+ },
+ )
+ .await;
+ }
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
+ return refusal;
+ }
+ if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {
+ return result;
+ }
+
+ let px = args.get("x").and_then(|value| value.as_f64());
+ let py = args.get("y").and_then(|value| value.as_f64());
+ if px.is_some() != py.is_some() {
+ return invalid_pointer_arguments(
+ "hotkey",
+ "x and y must be passed together (a pixel focus point), or neither",
+ );
+ }
+ if px.is_some() && resolved_element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to hotkey, not both.",
+ );
+ }
+
+ if hyprland_foreground(delivery) {
+ if xid_opt.is_none() {
+ return foreground_hyprland_refusal("an exact window_id is required");
+ }
+ if let Some(keys) = args.get("keys").and_then(Value::as_array) {
+ if keys.iter().any(|key| !key.is_string())
+ || keys
+ .iter()
+ .filter(|key| key.as_str().is_some_and(|key| !is_modifier(key)))
+ .count()
+ != 1
+ {
+ return foreground_hyprland_refusal(
+ "hotkeys require exactly one non-modifier key",
+ );
+ }
+ }
+ let action = match foreground_hyprland_key(key, mods) {
+ Ok(action) => action,
+ Err(error) => return error,
+ };
+ if let Err(error) =
+ focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
+ .await
+ {
+ return error;
+ }
+ if px.is_none() {
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ None,
+ false,
+ )
+ .await;
+ }
+ return foreground_hyprland_action(&args, pid, xid, action).await;
+ }
+
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ px.zip(py),
+ false,
+ )
+ .await;
+
+ if crate::wayland::is_inject_mode() {
+ if let Err(error) =
+ focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
+ {
+ return error;
+ }
+ let mut chord = mods.clone();
+ chord.push(key.clone());
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_hotkey(pid, xid, &chord)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Pressed hotkey '{key_display}' (focus-free via cua-compositor)."
+ )),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ // Chromium renderer shortcuts need DOM focus established by a real
+ // pointer event. Component.GrabFocus alone can report success while
+ // the renderer still drops the virtual-keyboard chord.
+ if delivery.is_foreground()
+ && crate::wayland::wayland_input_enabled()
+ && is_chromium_embedder(pid)
+ {
+ if let Some(element) = token.clone() {
+ let coordinates = tokio::task::spawn_blocking(move || {
+ resolve_element_local_coords(pid, &element, Some(xid))
+ })
+ .await;
+ let (_, x, y) = match coordinates {
+ Ok(Ok(value)) => value,
+ Ok(Err(error)) => return ToolResult::error(error.to_string()),
+ Err(error) => return ToolResult::error(format!("Task error: {error}")),
+ };
+ if let Err(error) =
+ focus_by_pixel(&self.state, pid, Some(xid), (x, y), true, &args, false).await
+ {
+ return error;
+ }
+ }
+ }
+
+ // Foreground: GrabFocus the addressed element up front. Background
+ // does it inside `background_key_route`, under the focus guard.
+ if let Some(element) = token.clone().filter(|_| delivery.is_foreground()) {
+ let element_index = element.index;
+ let focused =
+ tokio::task::spawn_blocking(move || crate::atspi::focus_token(pid, &element)).await;
+ match focused {
+ Ok(Ok(true)) => {}
+ Ok(Ok(false)) => {
+ return ToolResult::error(format!(
+ "AT-SPI Component.GrabFocus returned false for element {element_index}"
+ ))
+ }
+ Ok(Err(error)) => {
+ return stale_observed_element("hotkey", element_index, &error)
+ .unwrap_or_else(|| ToolResult::error(error.to_string()))
+ }
+ Err(error) => return ToolResult::error(format!("Task error: {error}")),
+ }
+ }
+
+ // ── px form: pixel-click to focus, then the combo acts on the focused field ──
+ // (e.g. Ctrl+V into a Chromium input). Reuses click's translation +
+ // delivery_mode; after it, deliver the combo via the plain background path
+ // (the focus-click already fronted when fg).
+ let px_target = {
+ if let (Some(cx), Some(cy)) = (px, py) {
+ let from_zoom = args.bool_or("from_zoom", false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ Some(xid),
+ (cx, cy),
+ delivery.is_foreground(),
+ &args,
+ from_zoom,
+ )
+ .await
+ {
+ return e;
+ }
+ Some((cx.round() as i32, cy.round() as i32))
+ } else {
+ None
+ }
+ };
+ let deliver_fg = delivery.is_foreground();
+ let cursor_id = resolve_cursor_key(&args);
+
+ let result = spawn_blocking_bounded(
+ "hotkey",
+ foreground_budget(1),
+ move || -> anyhow::Result<KeyRoute> {
+ if crate::wayland::wayland_input_enabled() {
+ // Native Wayland: route the modifier combo through wtype's
+ // -M/-k/-m sequence — the closest equivalent to the X11
+ // state-mask path. window_id is irrelevant once focused.
+ let mut combo: Vec<String> = mods_for_wayland.clone();
+ combo.push(key_for_wayland.clone());
+ return crate::wayland::hotkey(xid, &combo).map(|()| KeyRoute::Synthetic);
+ }
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ // foreground: activate the target first, then inject the accelerator
+ // as REAL key events via XTest. Synthetic XSendEvent keys
+ // (`send_key`) are dropped by GTK/Qt/Chromium/Firefox, so the
+ // foreground rung must use XTest, which reaches the focused window.
+ if deliver_fg {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::keyboard(),
+ || crate::input::send_key_xtest(&key, &m),
+ )
+ .map(|((), report)| KeyRoute::Foreground(report));
+ }
+ background_key_route(&cursor_id, pid, xid, px_target, &key, &m, token.as_ref())
+ },
+ )
+ .await;
+ let mode_label = if deliver_fg {
+ "foreground"
+ } else {
+ "background"
+ };
+ match result {
+ Ok(Ok(route)) => key_route_result(
+ &format!("Pressed {key_display} on pid {pid}"),
+ route,
+ mode_label,
+ ),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 02b72b5e29..165c593687 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1,14 +1,9 @@
//! Real Linux tool implementations (compiled only on Linux).
-use async_trait::async_trait;
-use cua_driver_contract::{
- ClickButton, DragInput, GetCursorPositionInput, GetDesktopStateInput, GetScreenSizeInput,
- HotkeyInput, InvokeMenuInput, MoveCursorInput, PressKeyInput, ScrollInput, TypeTextInput,
-};
use cua_driver_core::{
protocol::ToolResult,
- tool::{Tool, ToolDef, ToolRegistry},
- tool_args::{parse_legacy_click_input, parse_typed_input, parse_typed_projection, ArgsExt},
+ tool::{Tool, ToolRegistry},
+ tool_args::ArgsExt,
window_target::{PidOnlyWindowTargetGuard, WindowTargetCandidate, WindowTargetCandidates},
};
use serde_json::{json, Value};
@@ -17,9 +12,48 @@ use std::path::PathBuf;
use std::sync::{Arc, RwLock};
use crate::atspi::Snapshots;
-use crate::wayland::AimedPoint;
+pub(crate) use crate::input::action::spawn_blocking_bounded;
+pub(crate) use crate::wayland::AimedPoint;
use cursor_overlay::CursorRegistry;
+use super::{
+ bring_to_front::BringToFrontTool,
+ check_permissions::CheckPermissionsTool,
+ click::ClickTool,
+ cursor_tools::{
+ GetAgentCursorStateV2Tool, SetAgentCursorEnabledV2Tool, SetAgentCursorMotionV2Tool,
+ SetAgentCursorThemeTool,
+ },
+ double_click::DoubleClickTool,
+ drag::DragTool,
+ get_accessibility_tree::GetAccessibilityTreeTool,
+ get_config::GetConfigTool,
+ get_cursor_position::GetCursorPositionTool,
+ get_desktop_state::GetDesktopStateTool,
+ get_screen_size::GetScreenSizeTool,
+ get_window_state::GetWindowStateTool,
+ hotkey::HotkeyTool,
+ invoke_menu::InvokeMenuTool,
+ kill_app::KillAppTool,
+ launch_app::LaunchAppTool,
+ list_apps::ListAppsTool,
+ list_windows::ListWindowsTool,
+ mouse_button_down::MouseButtonDownTool,
+ mouse_button_up::MouseButtonUpTool,
+ mouse_drag::MouseDragTool,
+ move_cursor::MoveCursorTool,
+ parallel_mouse_drag::ParallelMouseDragTool,
+ press_key::PressKeyTool,
+ right_click::RightClickTool,
+ scroll::ScrollTool,
+ set_config::SetConfigTool,
+ set_value::SetValueTool,
+ set_window_frame::SetWindowFrameTool,
+ type_text::TypeTextTool,
+ type_text_chars::TypeTextCharsTool,
+ zoom::ZoomTool,
+};
+
fn window_target_candidates_for_pid(
windows: impl IntoIterator<Item = crate::x11::WindowInfo>,
pid: u32,
@@ -122,7 +156,103 @@ fn pid_fallback_window_resolver() -> cua_driver_core::window_target::PidFallback
}
#[cfg(test)]
-mod pid_window_resolver_tests;
+mod pid_window_resolver_tests {
+ use super::*;
+
+ fn win(xid: u64, z: Option<usize>, on_screen: bool, w: u32, h: u32) -> crate::x11::WindowInfo {
+ crate::x11::WindowInfo {
+ xid,
+ pid: Some(7),
+ app_name: "gimp".into(),
+ title: format!("w{xid}"),
+ is_on_screen: on_screen,
+ z_index: z,
+ x: 0,
+ y: 0,
+ width: w,
+ height: h,
+ }
+ }
+
+ #[test]
+ fn point_resolver_prefers_the_topmost_window_by_stacking_position() {
+ // Main window listed last (as `_NET_CLIENT_LIST_STACKING` bottom-up
+ // enumeration does when a dialog sits above it); the dialog wins.
+ let dialog = crate::x11::WindowInfo {
+ z_index: Some(5),
+ ..win(2, None, true, 400, 300)
+ };
+ let main = crate::x11::WindowInfo {
+ z_index: Some(3),
+ ..win(1, None, true, 1920, 1080)
+ };
+ assert_eq!(
+ topmost_window_at(&[main.clone(), dialog.clone()], 7, 100.0, 100.0),
+ Some(2)
+ );
+ assert_eq!(topmost_window_at(&[dialog, main], 7, 100.0, 100.0), Some(2));
+ }
+
+ #[test]
+ fn point_resolver_ignores_windows_without_a_stacking_position_when_one_has_it() {
+ let ranked = win(2, Some(0), true, 400, 300);
+ let unranked = win(1, None, true, 1920, 1080);
+ assert_eq!(
+ topmost_window_at(&[ranked, unranked], 7, 10.0, 10.0),
+ Some(2)
+ );
+ }
+
+ #[test]
+ fn pid_only_resolver_orders_focus_then_dialog_then_active_then_largest() {
+ let main = win(1, Some(1), true, 1920, 1080);
+ let dialog = win(2, Some(2), true, 400, 300);
+ let dock = win(3, Some(0), true, 200, 900);
+ let hidden = win(4, None, false, 3000, 3000);
+ let windows = vec![main, dialog, dock, hidden];
+ let transient = |xid: u64| (xid == 2).then_some(1u64);
+ // 1. core focus inside the pid wins outright (even over a dialog).
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, Some(3), transient, Some(1)),
+ Some(3)
+ );
+ // 2. no focus in the pid: its topmost transient dialog.
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, None, transient, Some(1)),
+ Some(2)
+ );
+ // 3. no dialog: the WM's active window when it is the pid's.
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, None, |_| None, Some(3)),
+ Some(3)
+ );
+ // 4. otherwise the largest mapped toplevel (never the unmapped one).
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, None, |_| None, None),
+ Some(1)
+ );
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, None, |_| None, Some(99)),
+ Some(1)
+ );
+ // A focused window that is not the pid's is ignored.
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, Some(42), |_| None, None),
+ Some(1)
+ );
+ }
+
+ #[test]
+ fn pid_only_resolver_skips_unmapped_dialogs() {
+ let main = win(1, Some(1), true, 800, 600);
+ let unmapped_dialog = win(2, Some(2), false, 400, 300);
+ let windows = vec![main, unmapped_dialog];
+ assert_eq!(
+ crate::x11::pick_pid_window(&windows, None, |xid| (xid == 2).then_some(1u64), None),
+ Some(1)
+ );
+ }
+}
type PidWindowGuardParts = (
WindowTargetCandidates,
@@ -246,7 +376,7 @@ impl ToolState {
})
}
- fn zoom_context(
+ pub(crate) fn zoom_context(
&self,
args: &Value,
pid: u32,
@@ -260,7 +390,7 @@ impl ToolState {
}
}
-fn screenshot_scale(
+pub(crate) fn screenshot_scale(
state: &ToolState,
args: &Value,
pid: u32,
@@ -279,263 +409,10 @@ fn screenshot_scale(
})
}
-// ── list_apps ────────────────────────────────────────────────────────────────
-
-pub struct ListAppsTool;
-static LIST_APPS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ListAppsTool {
- fn def(&self) -> &ToolDef {
- LIST_APPS_DEF.get_or_init(|| ToolDef {
- name: "list_apps".into(),
- description:
- "List Linux apps — both currently running and installed-but-not-running — \
- with per-app state flags:\n\n\
- - running: is a process for this app live? (pid is 0 when false)\n\
- - active: reserved (Linux X11/Wayland focus model differs from frontmost-app); \
- always false.\n\
- - kind: `\"desktop\"` for XDG `.desktop` launcher entries.\n\
- - launch_path: the launcher command from `Exec=` (field codes stripped). \
- Pass to `launch_app(launch_path=...)`.\n\
- - bundle_id: the XDG \"desktop file id\" — the `.desktop` file's path \
- relative to its XDG `applications/` root with the `.desktop` suffix \
- stripped and path separators replaced with `-` \
- (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).\n\
- - last_used: RFC3339 mtime of the `.desktop` file, when readable.\n\
- - windows: the app's current top-level windows (same records as list_windows); \
- empty for a process without one.\n\n\
- Running apps come from `/proc`. Installed apps come from XDG Desktop Entry \
- files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's \
- applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are \
- filtered. A `.desktop` file whose launcher matches a running process \
- (by basename), or whose WM_CLASS a top-level window carries, is merged \
- into a single entry with `running: true`.\n\n\
- Use this for \"is X installed?\" as well as \"is X running?\". For per-window \
- state — visibility, geometry, titles — call list_windows instead."
- .into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
-
- async fn invoke(&self, _args: Value) -> ToolResult {
- let apps = tokio::task::spawn_blocking(|| -> Vec<serde_json::Value> {
- let procs = crate::proc_fs::list_processes();
- let installed = crate::installed_apps::list_installed_apps();
- let windows = crate::wayland::list_windows_dispatch(None);
-
- // An app whose launcher is not its process (`libreoffice --calc`
- // runs `soffice.bin`, `google-chrome` runs `chrome`, `gnome-terminal`
- // hands off to `gnome-terminal-server`) never matches by
- // basename. Its top-level windows still carry its WM_CLASS, so
- // owning one is what makes it running.
- let by_window: std::collections::HashMap<usize, u32> = installed
- .iter()
- .enumerate()
- .filter_map(|(i, app)| {
- windows
- .iter()
- .find(|w| w.pid.is_some() && app.owns_window_class(&w.app_name))
- .and_then(|w| w.pid)
- .map(|pid| (i, pid))
- })
- .collect();
-
- // Match running processes to installed apps by executable
- // basename (Exec=firefox %u → "firefox"; cmdline /usr/bin/firefox
- // → "firefox"). Many distros register multiple .desktop entries
- // sharing the same basename (e.g. several `firefox` profiles, a
- // `code` and a `code-insiders` both shelling `code`), so the
- // bucket is `Vec<usize>` — we pick a winner per running process
- // via `disambiguate_installed_match` instead of overwriting.
- let mut by_exe: std::collections::HashMap<String, Vec<usize>> =
- std::collections::HashMap::new();
- for (i, app) in installed.iter().enumerate() {
- let basename = exec_basename(&app.launch_path);
- if !basename.is_empty() {
- by_exe.entry(basename).or_default().push(i);
- }
- }
-
- let mut consumed: std::collections::HashSet<usize> = std::collections::HashSet::new();
- let mut out = Vec::new();
- for p in &procs {
- let key_source = if !p.cmdline.is_empty() {
- &p.cmdline
- } else {
- &p.name
- };
- let basename = exec_basename(key_source);
- if basename.is_empty() {
- continue;
- }
- let candidates = by_exe.get(&basename).map(|v| v.as_slice()).unwrap_or(&[]);
- let merged = disambiguate_installed_match(candidates, &installed, key_source)
- .or_else(|| {
- by_window
- .iter()
- .filter(|(_, pid)| **pid == p.pid)
- .map(|(idx, _)| *idx)
- .min()
- });
- if let Some(idx) = merged {
- consumed.insert(idx);
- }
- let (name, bundle_id, launch_path, kind, last_used) = match merged {
- Some(idx) => {
- let a = &installed[idx];
- (
- a.name.clone(),
- Some(a.bundle_id.clone()),
- Some(a.launch_path.clone()),
- Some("desktop".to_owned()),
- a.last_used.clone(),
- )
- }
- None => (
- if !p.name.is_empty() {
- p.name.clone()
- } else {
- basename.clone()
- },
- None,
- None,
- None,
- None,
- ),
- };
- let owned: Vec<Value> = windows
- .iter()
- .filter(|w| w.pid == Some(p.pid))
- .filter(|w| {
- merged.is_none_or(|idx| installed[idx].owns_window_class(&w.app_name))
- })
- .map(window_record_json)
- .collect();
- out.push(json!({
- "pid": p.pid,
- "bundle_id": bundle_id,
- "name": name,
- "running": true,
- "active": false,
- "kind": kind,
- "launch_path": launch_path,
- "last_used": last_used,
- "windows": owned,
- }));
- }
- for (i, app) in installed.iter().enumerate() {
- if consumed.contains(&i) {
- continue;
- }
- // One process can own windows of several entries (a single
- // `soffice.bin` showing Calc and Writer): each entry with a
- // window is running under its own id.
- let window_pid = by_window.get(&i).copied();
- let owned: Vec<Value> = windows
- .iter()
- .filter(|w| w.pid.is_some() && w.pid == window_pid)
- .filter(|w| app.owns_window_class(&w.app_name))
- .map(window_record_json)
- .collect();
- out.push(json!({
- "pid": window_pid.unwrap_or(0),
- "bundle_id": app.bundle_id.clone(),
- "name": app.name.clone(),
- "running": window_pid.is_some(),
- "active": false,
- "kind": "desktop",
- "launch_path": app.launch_path.clone(),
- "last_used": app.last_used.clone(),
- "windows": owned,
- }));
- }
- out
- })
- .await
- .unwrap_or_default();
-
- let running_count = apps
- .iter()
- .filter(|a| a["running"].as_bool().unwrap_or(false))
- .count();
- let total = apps.len();
- let installed_only = total - running_count;
- let mut lines = vec![format!(
- "✅ Found {total} app(s): {running_count} running, {installed_only} installed-not-running."
- )];
- for app in apps
- .iter()
- .filter(|a| a["running"].as_bool().unwrap_or(false))
- {
- let name = app["name"].as_str().unwrap_or("?");
- let pid = app["pid"].as_u64().unwrap_or(0);
- lines.push(format!("- {name} (pid {pid})"));
- }
- // Unified `apps` array + legacy `processes` alias for older callers.
- let structured = json!({
- "apps": apps,
- "processes": apps.iter().filter(|a| a["running"].as_bool().unwrap_or(false))
- .map(|a| json!({
- "pid": a["pid"], "name": a["name"]
- })).collect::<Vec<_>>(),
- });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
-}
-
-/// Pick which of several `.desktop`-derived installed apps best matches a
-/// running process whose basename collided. Precedence:
-/// 1. exact full-launch-path equality with the process's cmdline token
-/// (so `/usr/bin/firefox` beats `/snap/firefox/current/firefox`)
-/// 2. most recently modified launcher (`last_used` desc, RFC3339 string
-/// ordering is lexicographic-safe for our format)
-/// 3. first candidate (deterministic by source order)
-fn disambiguate_installed_match(
- candidates: &[usize],
- installed: &[crate::installed_apps::InstalledApp],
- proc_key_source: &str,
-) -> Option<usize> {
- if candidates.is_empty() {
- return None;
- }
- if candidates.len() == 1 {
- return Some(candidates[0]);
- }
-
- // Look for an exact path match against the cmdline's first token.
- let proc_path = proc_key_source.split_whitespace().next().unwrap_or("");
- if !proc_path.is_empty() {
- if let Some(&idx) = candidates
- .iter()
- .find(|&&i| installed[i].launch_path == proc_path)
- {
- return Some(idx);
- }
- }
-
- // Fall back to the candidate with the most recent `last_used`.
- candidates
- .iter()
- .copied()
- .max_by(|&a, &b| {
- installed[a]
- .last_used
- .as_deref()
- .unwrap_or("")
- .cmp(installed[b].last_used.as_deref().unwrap_or(""))
- })
- .or_else(|| candidates.first().copied())
-}
-
/// Return the lowercase basename of an executable token, stripping any
/// leading shell-wrapper words and quoting. `env FOO=1 /usr/bin/firefox %U`
/// → `firefox`. `firefox %u` → `firefox`.
-fn exec_basename(s: &str) -> String {
+pub(crate) fn exec_basename(s: &str) -> String {
// Take the first whitespace-separated token that looks like a binary,
// skipping `env`-style prefixes and `K=V` assignments.
for tok in s.split_whitespace() {
@@ -556,77 +433,6 @@ fn exec_basename(s: &str) -> String {
String::new()
}
-// ── list_windows ─────────────────────────────────────────────────────────────
-
-pub struct ListWindowsTool;
-static LIST_WINDOWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ListWindowsTool {
- fn def(&self) -> &ToolDef {
- LIST_WINDOWS_DEF.get_or_init(|| ToolDef {
- name: "list_windows".into(),
- description: "List top-level windows. Each record includes z_index (integer or null; \
- higher values are closer to the front; null means stacking order is unavailable \
- and callers must not infer one). To select a frontmost candidate, take the maximum \
- integer z_index; if every value is null, use an explicit fallback instead of \
- relying on array order. Filter with `app` or `pid`: an unfiltered call lists \
- every window on the host.".into(),
- input_schema: json!({"type":"object","properties":{
- "pid":{"type":"integer","description":"Only list windows owned by this process ID."},
- "app":{"type":"string","description":cua_driver_core::tool_args::LIST_WINDOWS_APP_FILTER_DESCRIPTION},
- "on_screen_only":{"type":"boolean","description":"When true, filter to visible windows only. Default false."}
- },"additionalProperties":false}),
- read_only: true, destructive: false, idempotent: true, open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let filter_pid = args.opt_u64("pid").map(|v| v as u32);
- let on_screen_only = args.bool_or("on_screen_only", false);
- let mut windows =
- tokio::task::spawn_blocking(move || crate::wayland::list_windows_dispatch(filter_pid))
- .await
- .unwrap_or_default();
- // Exited applications can remain visible in AT-SPI/X11 as zombies;
- // never return those stale targets to callers.
- windows.retain(|window| window.pid.is_none_or(crate::proc_fs::is_process_live));
- if on_screen_only {
- windows.retain(|window| window.is_on_screen);
- }
- if let Some(filter) = args.opt_str("app") {
- windows.retain(|window| {
- cua_driver_core::tool_args::app_filter_matches(&filter, &window.app_name, None)
- });
- }
- if crate::wayland::is_wayland() {
- crate::wayland::remember_observed_window_origins(&windows);
- }
- let mut lines = vec![format!("Found {} windows:", windows.len())];
- for w in &windows {
- lines.push(format!(
- " window_id={} pid={} \"{}\" {}x{}+{}+{}{}",
- w.xid,
- w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
- w.title,
- w.width,
- w.height,
- w.x,
- w.y,
- if w.app_name.is_empty() {
- String::new()
- } else {
- format!(" app={}", w.app_name)
- }
- ));
- }
- let structured =
- json!({ "windows": windows.iter().map(window_record_json).collect::<Vec<_>>() });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
-}
-
/// Build the structured `list_windows` record for one Linux window.
///
/// Emits the canonical cross-platform shape — geometry nested under a
@@ -636,7 +442,7 @@ impl Tool for ListWindowsTool {
/// existing Linux callers don't break. Fully additive; no field removed,
/// no schema_version bump.
///
-fn window_record_json(w: &crate::x11::WindowInfo) -> Value {
+pub(crate) fn window_record_json(w: &crate::x11::WindowInfo) -> Value {
json!({
"window_id": w.xid,
"pid": w.pid,
@@ -652,7 +458,7 @@ fn window_record_json(w: &crate::x11::WindowInfo) -> Value {
})
}
-fn visible_live_desktop_windows(
+pub(crate) fn visible_live_desktop_windows(
mut windows: Vec<crate::x11::WindowInfo>,
) -> Vec<crate::x11::WindowInfo> {
windows.retain(|window| {
@@ -662,113 +468,109 @@ fn visible_live_desktop_windows(
}
#[cfg(test)]
-mod list_windows_tests;
-
-// ── get_window_state ─────────────────────────────────────────────────────────
-
-/// The largest window screenshot delivered as-is. The Anthropic API downsizes
-/// any image above ~1.15 megapixels before the model sees it, so a 1568x861
-/// PNG reached the model as ~1447x795 while the driver still mapped its x/y
-/// as 1568x861: every pixel click landed a uniform 0.94x short. Capping here
-/// keeps "pixels of THIS screenshot" true for the image the model reads.
-const WINDOW_SCREENSHOT_MAX_PIXELS: u64 = 1_150_000;
-
-/// The long edge a `w`x`h` image must shrink to so that it holds at most
-/// `max_pixels` pixels; `None` when it already fits.
-fn megapixel_long_edge_cap(w: u32, h: u32, max_pixels: u64) -> Option<u32> {
- let pixels = u64::from(w) * u64::from(h);
- if w == 0 || h == 0 || pixels <= max_pixels {
- return None;
- }
- let long = w.max(h) as f64;
- let short = w.min(h) as f64;
- let mut edge = (long * (max_pixels as f64 / pixels as f64).sqrt()).floor() as u32;
- // The resizer rounds both sides; step down until the rounded image fits.
- while edge > 1 {
- let scale = edge as f64 / long;
- let fits = ((long * scale).round() as u64) * ((short * scale).round() as u64) <= max_pixels;
- if fits {
- break;
- }
- edge -= 1;
- }
- Some(edge)
-}
+mod list_windows_tests {
+ use super::*;
-#[cfg(test)]
-mod megapixel_cap_tests;
-
-/// Build a single structured element entry for `get_window_state`.
-/// Returns `None` when the node has no `element_index` (non-actionable rows).
-fn build_element_entry(
- n: &crate::atspi::AtspiNode,
- snapshot_id: Option<u32>,
- bounds: Option<(i32, i32, u32, u32)>,
-) -> Option<serde_json::Value> {
- let idx = n.element_index?;
- // `label` mirrors what a human reading the markdown row would call this
- // element: its name, else its description (Qt keeps a button's tooltip
- // there: "Play"/"Pause"). Never its value: four spin buttons all labelled
- // "0.0" cannot be told apart — such a control is `unlabelled` instead,
- // with its place among its siblings in `description`.
- let unlabelled = n.name.is_none()
- && n.description
- .as_deref()
- .is_some_and(|d| d.starts_with(crate::atspi::native::UNLABELLED_NOTE_PREFIX));
- let label = n
- .name
- .clone()
- .or_else(|| n.description.clone().filter(|_| !unlabelled));
- let mut entry = json!({
- "element_index": idx,
- "role": n.role,
- "depth": n.depth,
- });
- if let Some(snapshot_id) = snapshot_id {
- entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx));
- }
- if n.in_web_content {
- entry["in_web_content"] = json!(true);
- }
- if let Some(label) = label {
- entry["label"] = json!(label);
- }
- if unlabelled {
- entry["unlabelled"] = json!(true);
- }
- // Surface the element's value separately from `label` (which collapses
- // name→value→description): a field with both a name AND typed text would
- // otherwise hide the text from a caller reading the structured side,
- // leaving it only in tree_markdown. See the macOS get_window_state builder
- // for the rationale.
- if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) {
- entry["value"] = json!(value);
- }
- if let Some(enabled) = n.enabled {
- entry["enabled"] = json!(enabled);
- }
- if let Some(selected) = n.selected {
- entry["selected"] = json!(selected);
- }
- let actions: Vec<String> = n
- .actions
- .iter()
- .filter(|a| !a.trim().is_empty())
- .cloned()
- .collect();
- if !actions.is_empty() {
- entry["actions"] = json!(actions);
+ #[test]
+ fn record_has_bounds_and_flat_legacy_fields() {
+ let w = crate::x11::WindowInfo {
+ xid: 42,
+ pid: Some(1234),
+ app_name: "example-app".to_owned(),
+ title: "Example".to_owned(),
+ is_on_screen: true,
+ z_index: Some(3),
+ x: 10,
+ y: 20,
+ width: 300,
+ height: 400,
+ };
+ let rec = window_record_json(&w);
+
+ // Canonical cross-platform shape: nested `bounds` object.
+ let bounds = rec
+ .get("bounds")
+ .expect("record must carry a `bounds` object");
+ assert_eq!(bounds["x"], json!(10));
+ assert_eq!(bounds["y"], json!(20));
+ assert_eq!(bounds["width"], json!(300));
+ assert_eq!(bounds["height"], json!(400));
+
+ // Legacy alias: flat fields must still be present.
+ assert_eq!(rec["x"], json!(10));
+ assert_eq!(rec["y"], json!(20));
+ assert_eq!(rec["width"], json!(300));
+ assert_eq!(rec["height"], json!(400));
+
+ // Cross-platform companions.
+ assert_eq!(rec["app_name"], json!("example-app"));
+ assert_eq!(rec["is_on_screen"], json!(true));
+ assert_eq!(rec["z_index"], json!(3));
+ assert_eq!(rec["window_id"], json!(42));
+ assert_eq!(rec["title"], json!("Example"));
}
- if let Some(parent) = n.parent_element_index {
- entry["parent_index"] = json!(parent);
+
+ #[test]
+ fn unavailable_wayland_order_serializes_as_null() {
+ let w = crate::x11::WindowInfo {
+ xid: 43,
+ pid: Some(1234),
+ app_name: "native-wayland-app".to_owned(),
+ title: "Example".to_owned(),
+ is_on_screen: true,
+ z_index: None,
+ x: 0,
+ y: 0,
+ width: 300,
+ height: 400,
+ };
+
+ assert_eq!(window_record_json(&w)["z_index"], Value::Null);
}
- if let Some((x, y, w, h)) = bounds {
- entry["frame"] = json!({ "x": x, "y": y, "w": w, "h": h });
+
+ #[test]
+ fn desktop_discovery_retains_only_visible_windows_with_live_or_unknown_owners() {
+ let window = |xid, pid, is_on_screen| crate::x11::WindowInfo {
+ xid,
+ pid,
+ app_name: "native-wayland-app".to_owned(),
+ title: "Example".to_owned(),
+ is_on_screen,
+ z_index: None,
+ x: 10,
+ y: 20,
+ width: 300,
+ height: 400,
+ };
+ let windows = visible_live_desktop_windows(vec![
+ window(1, Some(std::process::id()), true),
+ window(2, Some(std::process::id()), false),
+ window(3, Some(u32::MAX), true),
+ window(4, None, true),
+ window(5, None, false),
+ ]);
+
+ assert_eq!(
+ windows.iter().map(|w| w.xid).collect::<Vec<_>>(),
+ vec![1, 4]
+ );
+ let record = window_record_json(&windows[0]);
+ assert_eq!(
+ record["bounds"],
+ json!({"x": 10, "y": 20, "width": 300, "height": 400})
+ );
+ assert_eq!(record["app_name"], json!("native-wayland-app"));
+ assert_eq!(record["is_on_screen"], json!(true));
+ assert_eq!(record["z_index"], Value::Null);
}
- if let Some(description) = n.description.clone().filter(|d| !d.is_empty()) {
- entry["description"] = json!(description);
+
+ #[test]
+ fn chromium_launch_detection_uses_executable_basename() {
+ assert!(chromium_family_program("/usr/bin/google-chrome-stable"));
+ assert!(chromium_family_program("CuaTestHarness.Electron"));
+ assert!(chromium_family_program("chromium-browser"));
+ assert!(!chromium_family_program("/usr/bin/gnome-text-editor"));
}
- Some(entry)
}
/// Roles a popup's rows carry: menu entries, and the list / tree / table
@@ -792,7 +594,7 @@ fn popup_item_role(role: &str) -> bool {
/// Falls back to the input when nothing matches, so a popup that is not a
/// menu (a GTK popover without its own frame) still returns what the walk
/// found.
-fn popup_menu_elements(
+pub(crate) fn popup_menu_elements(
elements: Vec<serde_json::Value>,
(origin_x, origin_y): (i32, i32),
width: u32,
@@ -902,7 +704,7 @@ fn popup_menu_elements(
/// Elements that have a frame (visible, hit-testable on the screenshot)
/// come first, in walk order; frameless ones follow. A client that caps the
/// result then still shows the document, toolbar and sidebar controls.
-fn framed_elements_first(elements: Vec<serde_json::Value>) -> Vec<serde_json::Value> {
+pub(crate) fn framed_elements_first(elements: Vec<serde_json::Value>) -> Vec<serde_json::Value> {
let (framed, frameless): (Vec<_>, Vec<_>) = elements
.into_iter()
.partition(|entry| entry.get("frame").is_some());
@@ -914,16 +716,16 @@ fn framed_elements_first(elements: Vec<serde_json::Value>) -> Vec<serde_json::Va
/// overlaps the target's rectangle (then its own drawable is not what the
/// user sees).
#[derive(Default)]
-struct WindowOverlays {
- dialogs: Vec<serde_json::Value>,
- popups: Vec<serde_json::Value>,
- covers_window: bool,
- window_rect: Option<(i32, i32, u32, u32)>,
+pub(crate) struct WindowOverlays {
+ pub(crate) dialogs: Vec<serde_json::Value>,
+ pub(crate) popups: Vec<serde_json::Value>,
+ pub(crate) covers_window: bool,
+ pub(crate) window_rect: Option<(i32, i32, u32, u32)>,
}
impl WindowOverlays {
/// One sentence per overlay naming the call that acts on it.
- fn follow_up(&self, pid: u32) -> Option<String> {
+ pub(crate) fn follow_up(&self, pid: u32) -> Option<String> {
let mut parts = Vec::new();
for dialog in &self.dialogs {
let id = dialog["window_id"].as_u64().unwrap_or(0);
@@ -962,2588 +764,1093 @@ impl WindowOverlays {
}
}
-fn rects_intersect(a: (i32, i32, u32, u32), b: (i32, i32, u32, u32)) -> bool {
+pub(crate) fn rects_intersect(a: (i32, i32, u32, u32), b: (i32, i32, u32, u32)) -> bool {
a.0 < b.0 + b.2 as i32
&& a.0 + a.2 as i32 > b.0
&& a.1 < b.1 + b.3 as i32
&& a.1 + a.3 as i32 > b.1
}
-fn window_overlays(pid: u32, xid: u64) -> WindowOverlays {
- let mut out = WindowOverlays {
- window_rect: crate::x11::window_info(xid).map(|w| (w.x, w.y, w.width, w.height)),
- ..Default::default()
- };
- let over = |rect: (i32, i32, u32, u32)| {
- out.window_rect
- .is_some_and(|target| rects_intersect(rect, target))
+pub(crate) fn chromium_family_program(program: &str) -> bool {
+ let basename = std::path::Path::new(program)
+ .file_name()
+ .and_then(|name| name.to_str())
+ .unwrap_or(program)
+ .to_ascii_lowercase();
+ ["chrome", "chromium", "electron", "brave", "edge"]
+ .iter()
+ .any(|needle| basename.contains(needle))
+}
+
+// ── shared helpers ────────────────────────────────────────────────────────────
+
+/// Resolve a token element's center in window-local coordinates.
+///
+/// Returns `(xid, window_local_x, window_local_y)`.
+/// Looks up the element's bounds (see [`crate::atspi::token_bounds`]), finds
+/// the owning window (via `xid_hint` or the first window for `pid`), then
+/// converts screen-absolute → window-local coords via compositor metadata on
+/// Wayland or XTranslateCoordinates on X11.
+pub(crate) fn resolve_element_local_coords(
+ pid: u32,
+ token: &crate::atspi::TokenElement,
+ xid_hint: Option<u64>,
+) -> anyhow::Result<(u64, f64, f64)> {
+ let (bx, by, bw, bh) = crate::atspi::token_bounds(pid, xid_hint, token)?;
+ let screen_cx = bx as f64 + bw as f64 / 2.0;
+ let screen_cy = by as f64 + bh as f64 / 2.0;
+
+ let xid = if let Some(x) = xid_hint {
+ x
+ } else if crate::wayland::wayland_input_enabled() {
+ crate::wayland::list_windows_dispatch(Some(pid))
+ .into_iter()
+ .next()
+ .map(|window| window.xid)
+ .ok_or_else(|| anyhow::anyhow!("No Wayland windows for pid {pid}"))?
+ } else {
+ crate::x11::list_windows(Some(pid))
+ .into_iter()
+ .next()
+ .map(|w| w.xid)
+ .ok_or_else(|| anyhow::anyhow!("No windows for pid {pid}"))?
};
- let mut covers = false;
- for window in crate::x11::list_windows(Some(pid)) {
- if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {
- continue;
- }
- let Some(owner) = crate::x11::transient_for(window.xid) else {
- continue;
- };
- covers |= over((window.x, window.y, window.width, window.height));
- out.dialogs.push(json!({
- "window_id": window.xid,
- "title": window.title,
- "transient_for": owner,
- "modal": crate::x11::window_is_modal(window.xid),
- "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },
- }));
- }
- // A dialog/plugin window can legitimately run as a DIFFERENT process than
- // the application it belongs to (GIMP's separate-process export dialogs,
- // LibreOffice's Document Recovery dialog under a distinct soffice.bin).
- // Exact-pid matching above would make such a window invisible even though
- // it is unambiguously this application's own popup. WM_TRANSIENT_FOR
- // resolving to one of pid's own windows is the correlation signal (see
- // `list_cross_pid_transient_windows`); such entries are tagged with
- // `owning_pid` so callers can tell it apart from a same-process dialog.
- for window in crate::x11::list_cross_pid_transient_windows(pid) {
- if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {
- continue;
- }
- let Some(owner) = crate::x11::transient_for(window.xid) else {
- continue;
- };
- covers |= over((window.x, window.y, window.width, window.height));
- out.dialogs.push(json!({
- "window_id": window.xid,
- "title": window.title,
- "transient_for": owner,
- "modal": crate::x11::window_is_modal(window.xid),
- "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },
- "owning_pid": window.pid,
- }));
- }
- for popup in crate::input::mapped_popup_windows() {
- if popup.window == xid || popup.pid.is_some_and(|owner| owner != pid) {
- continue;
- }
- // Desktop-wide override-redirect windows that are nobody's menu:
- // mutter's guard window and the driver's own cursor overlay.
- if popup.pid.is_none()
- && (popup.title.contains("guard window") || popup.title.starts_with("Cua."))
- {
- continue;
+
+ if crate::wayland::wayland_input_enabled() {
+ let (window_x, window_y, window_width, window_height) =
+ crate::wayland::window_geometry(xid)
+ .ok_or_else(|| anyhow::anyhow!("No Wayland geometry for window {xid}"))?;
+ if window_width == 0 || window_height == 0 {
+ anyhow::bail!("Wayland window {xid} has no usable geometry");
}
- covers |= over((popup.x, popup.y, popup.width, popup.height));
- out.popups.push(popup.to_json());
+ return Ok((
+ xid,
+ screen_cx - window_x as f64,
+ screen_cy - window_y as f64,
+ ));
}
- out.covers_window = covers;
- out
-}
-pub struct GetWindowStateTool {
- state: Arc<ToolState>,
+ use x11rb::connection::Connection;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
+ let (conn, screen_num) = RustConnection::connect(None)?;
+ let root = conn.setup().roots[screen_num].root;
+ let reply = conn
+ .translate_coordinates(xid as u32, root, 0, 0)?
+ .reply()?;
+ let local_x = screen_cx - reply.dst_x as f64;
+ let local_y = screen_cy - reply.dst_y as f64;
+ Ok((xid, local_x, local_y))
}
-const COLD_START_WALK_TIMEOUT_MS: u64 = 2_000;
-
-/// The AT-SPI walker's own node budget when the caller gives none.
-const LINUX_DEFAULT_MAX_ELEMENTS: usize = 5_000;
-
-fn linux_snapshot_timeout_ms(timeout: Option<&Value>, has_prior_snapshot: bool) -> u64 {
- cua_driver_core::tool_schema::resolve_timeout_ms_with_first_snapshot_grace(
- timeout,
- has_prior_snapshot,
- COLD_START_WALK_TIMEOUT_MS,
- )
+/// An element whose AT-SPI extents do not fall inside its own toplevel: GTK4
+/// reports empty extents for some widgets (a text view centre resolves to
+/// (0,0), the GNOME hot corner), and a real pointer click there would hit
+/// whatever is under that screen point. Refused before any input is sent.
+#[derive(Debug)]
+struct ElementBoundsUnusable {
+ idx: usize,
+ local: (f64, f64),
+ window: (u32, u32),
}
-static GWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetWindowStateTool {
- fn def(&self) -> &ToolDef {
- GWS_DEF.get_or_init(|| ToolDef {
- name: "get_window_state".into(),
- description: "Walk a running app's AT-SPI tree and return it ONCE, as \
- compact Markdown by default (`tree_format:\"markdown\"`). Every actionable \
- element is tagged `[N]` (an element_index); its element_token is \
- `<snapshot_id>:N`, where `snapshot_id` is printed in the response header and \
- returned in structuredContent. Pass `tree_format:\"elements\"` for the \
- structured `elements` array instead (explicit `element_token`, `role`, \
- `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when \
- AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for \
- both (about twice the size). Set `query` to project the tree to matching rows \
- plus their ancestor chain while preserving original indices; \
- `total_element_count` reports the complete snapshot.\n\n\
- Size: a read is bounded by default (`max_elements` 250 actionable nodes). \
- When the walk stops at that budget the response says `Tree truncated at \
- max_elements=…` and `truncated: true`; pass a larger `max_elements`, or \
- `query` / `max_depth`, to reach the rest. `_note` is omitted unless \
- `verbose:true`. `full_output:true` restores the previous full response: both \
- representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\n\
- DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
- to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
- (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
- their previous `[N]` unless a `reindexed:` line says otherwise. The response \
- carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
- other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
- back to a full read and `since_status` says why. The focused element is not \
- reported on Linux yet.\n\n\
- Always returns BOTH the element tree AND a screenshot — ground on \
- both and cross-check (the tree lies on some surfaces). Choose the \
- modality at ACTION time: an element ax action \
- (element_token → accessibility rung) or an element px \
- action (x,y → pixel rung off this screenshot). capture_mode is \
- deprecated and ignored. On Wayland, where output capture cannot prove \
- the requested surface's identity, the truthful tree is returned without \
- a screenshot and `screenshot_error.code` is \
- `surface_identity_unproven`.\n\n\
- The mirror image: pass `include_accessibility_tree:false` to SKIP \
- the AT-SPI walk entirely and return just the screenshot plus \
- window metadata (window_bounds, app_name, window_title) — the \
- capture-only path for a live window preview / picture-in-picture. \
- Setting BOTH `include_accessibility_tree:false` and \
- `include_screenshot:false` is an error. Optional `max_dimension` \
- caps the returned screenshot's long edge in pixels for a cheap \
- thumbnail.\n\n\
- Optional `max_elements` / `max_depth` bound the AT-SPI walk \
- (Electron / large web apps produce 10k+ element trees). Markdown and \
- structured elements are truncated identically. Default: `max_elements` 250.\n\n\
- TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI \
- walk. When omitted, a window's first snapshot gets a 2000 ms \
- cold-start budget so Chromium/Electron can finish publishing its \
- initial tree. Explicit values are always honored. Large apps \
- (LibreOffice, GIMP, file managers) can exceed the budget; \
- the call then returns the PARTIAL tree with `truncated: true`, \
- `truncation_reason`, `nodes_visited`/`nodes_pending` and \
- `elements_complete: false`. Every element listed is real and \
- clickable; elements after the cut are simply missing. Retry with a \
- larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` \
- when the element you need is absent.\n\n\
- SCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels \
- (long edge <= max_image_dimension, 1568 by default), because larger images \
- are downsized before a model reads them and its pixel coordinates would then \
- be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools \
- are pixels of the delivered screenshot (`frame` stays in screen coordinates, \
- as on every platform); `frame_scale` < 1 reports the downsizing. An \
- explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\n\
- POPUP MENUS: a context menu / popover / combo list is an \
- override-redirect window that list_windows never shows. A click or \
- right_click that opened one names it in its result (`popup: \
- {window_id, bounds, title}`); pass that window_id here to walk the \
- popup's own AT-SPI toplevel so its menu items get element indices \
- (then click them by element_token). Omitting window_id while a popup \
- of this pid is open walks that popup.".into(),
- input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Process ID that owns the window."},
- "window_id":{"type":"integer","description":"Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window."},
- "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
- "include_accessibility_tree":{"type":"boolean",
- "description":"Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
- "include_screenshot":{"type":"boolean",
- "description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
- "screenshot_out_file":{"type":"string",
- "description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},
- "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
- "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
- "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},
- "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
- "max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted."},
- "max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior."}
- },"additionalProperties":false})),
- read_only: true, destructive: false, idempotent: false, open_world: false,
- })
+impl std::fmt::Display for ElementBoundsUnusable {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(
+ f,
+ "element [{}] resolves to window-local ({:.0}, {:.0}), outside its {}x{} window; \
+ the toolkit reported unusable AT-SPI extents, so no pointer input was sent",
+ self.idx, self.local.0, self.local.1, self.window.0, self.window.1
+ )
}
+}
+impl std::error::Error for ElementBoundsUnusable {}
- async fn invoke(&self, args: Value) -> ToolResult {
- let display_only = match cua_driver_core::window_state_view::display_only(&args) {
- Ok(display_only) => display_only,
- Err(refusal) => return refusal,
- };
- use cua_driver_core::tool_args::ArgsExt;
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- // `window_id` omitted: the open popup menu of this pid when one is
- // mapped (a context menu the caller wants to read), else the pid's
- // focused / dialog / active / largest window.
- let xid = match args.opt_u64("window_id") {
- Some(v) => v,
- None => {
- let chosen = tokio::task::spawn_blocking(move || {
- if let Some(popup) = crate::input::mapped_popup_windows()
- .into_iter()
- .rev()
- .find(|p| p.pid == Some(pid))
- {
- return Some(popup.window);
- }
- let windows = crate::x11::list_windows(Some(pid));
- let candidates: Vec<u64> = windows.iter().map(|w| w.xid).collect();
- crate::x11::pick_pid_window(
- &windows,
- crate::x11::focused_window_among(&candidates),
- crate::x11::transient_for,
- crate::x11::active_window(),
- )
- })
- .await
- .ok()
- .flatten();
- match chosen {
- Some(v) => v,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- // Retain the legacy additive cap, but let the canonical per-call field
- // replace the configured ceiling entirely (including 0 = native size).
- let max_dimension = args
- .get("max_dimension")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as u32);
- let max_image_dimension = match args.get("max_image_dimension") {
- None => None,
- Some(value) => match value.as_u64().and_then(|value| u32::try_from(value).ok()) {
- Some(value) => Some(value),
- None => {
- return ToolResult::error(
- "get_window_state.max_image_dimension must be an integer from 0 through 4294967295.",
- )
- .with_structured(json!({ "code": "invalid_arguments" }))
- }
- },
- };
- let explicit_max_image_dimension = max_image_dimension.is_some();
- let max_dim = {
- let cfg = self.state.config.read().unwrap();
- cua_driver_core::image_utils::ImageDimensionLimits {
- configured: cfg.max_image_dimension,
- legacy_max_dimension: max_dimension,
- max_image_dimension,
- }
- .resolve()
- };
- // `capture_mode` is DEPRECATED and ignored — get_window_state always
- // returns BOTH the AT-SPI tree and a screenshot now, so the agent grounds
- // on both and cross-checks (the tree lies often enough that a grounding
- // screenshot should always be present). The modality is chosen at action
- // time: an element ax action (element_token) or element px action (x,y).
- // We don't even read the arg; it stays in the schema only so old callers
- // don't trip additionalProperties:false.
- let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
- Ok(view) => view,
- Err(refusal) => return refusal,
- };
- let query = args.opt_str("query");
- let session_id = args.opt_str("_session_id");
- // include_screenshot (default true) — the perf opt-out. The tree+screenshot
- // pair is the default; `include_screenshot:false` skips the grab and returns
- // tree only (the cheap re-index path before an element ax action). A
- // screenshot_out_file still forces a capture (to disk), regardless.
- let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
- // `include_accessibility_tree` (default true) mirrors include_screenshot:
- // set false to SKIP the AT-SPI walk and return just the screenshot +
- // window metadata (the capture-only / preview path).
- let want_tree = args
- .get("include_accessibility_tree")
- .and_then(|v| v.as_bool())
- != Some(false);
- // screenshot_out_file: when set, write the PNG to disk and surface the
- // path instead of embedding base64 in the response. `~` expands.
- let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {
- if let Some(rest) = s.strip_prefix("~/") {
- let home = std::env::var("HOME").unwrap_or_default();
- format!("{home}/{rest}")
- } else {
- s
- }
- });
- // Optional caps — when omitted, the AT-SPI walker uses its built-in
- // defaults (#22865).
- let max_elements = Some(
- view.max_elements(
- &args,
- LINUX_DEFAULT_MAX_ELEMENTS,
- display_only
- || args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true),
- ),
- );
- let max_depth = args
- .get("max_depth")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as usize);
- let timeout_ms = linux_snapshot_timeout_ms(
- args.get("timeout_ms"),
- self.state
- .snapshots
- .contains_semantic_window(pid as i32, xid),
- );
- let walk_timeout = std::time::Duration::from_millis(timeout_ms);
-
- let process_is_live = crate::proc_fs::is_process_live(pid);
- // Enumerate the pid's windows ONCE and reuse the result for both the
- // window-ownership check (Wayland) and the additive window metadata
- // below, instead of paying for the compositor/X11 enumeration twice.
- // `window_meta` also names the surface + its on-screen rectangle on the
- // capture-only path, where no AT-SPI tree identifies it.
- let popup_meta = (!crate::wayland::is_wayland())
- .then(|| crate::input::popup_window_info(xid))
- .flatten();
- let window_meta = crate::wayland::list_windows_dispatch(Some(pid))
- .into_iter()
- .find(|w| w.xid == xid)
- .or_else(|| {
- popup_meta.as_ref().map(|popup| crate::x11::WindowInfo {
- xid: popup.window,
- pid: popup.pid,
- app_name: String::new(),
- title: popup.title.clone(),
- is_on_screen: true,
- z_index: None,
- x: popup.x,
- y: popup.y,
- width: popup.width,
- height: popup.height,
- })
- });
- let window_matches = if crate::wayland::is_wayland() {
- window_meta.as_ref().is_some_and(|w| w.pid == Some(pid))
- || crate::wayland::window_was_listed_for_pid(pid, xid)
- } else {
- crate::x11::window_belongs_to_pid(xid, pid)
- || crate::input::popup_window_info(xid)
- .is_some_and(|popup| popup.pid.is_none_or(|owner| owner == pid))
- // A cross-process dialog/plugin window transient-for one of
- // pid's own windows (GIMP export dialogs, LibreOffice's
- // Document Recovery) legitimately belongs to a different pid
- // than the caller's target; accept it here too so the
- // get_window_state(window_id=...) follow-up named in the
- // dialogs[] overlay actually works.
- || crate::x11::list_cross_pid_transient_windows(pid)
- .iter()
- .any(|w| w.xid == xid)
- };
- if !process_is_live || !window_matches {
- return ToolResult::error(format!(
- "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows."
- ));
+/// `resolve_element_local_coords` plus the sanity check that the centre lies
+/// inside the window it belongs to. Blocking.
+pub(crate) fn checked_element_local_coords(
+ pid: u32,
+ token: &crate::atspi::TokenElement,
+ xid_hint: Option<u64>,
+) -> anyhow::Result<(u64, f64, f64)> {
+ let (xid, lx, ly) = resolve_element_local_coords(pid, token, xid_hint)?;
+ if crate::wayland::wayland_input_enabled() {
+ return Ok((xid, lx, ly));
+ }
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
+ let (conn, _) = RustConnection::connect(None)?;
+ let geom = conn.get_geometry(xid as u32)?.reply()?;
+ let (w, h) = (u32::from(geom.width), u32::from(geom.height));
+ if lx < 0.0 || ly < 0.0 || lx >= f64::from(w) || ly >= f64::from(h) {
+ return Err(ElementBoundsUnusable {
+ idx: token.index,
+ local: (lx, ly),
+ window: (w, h),
}
+ .into());
+ }
+ Ok((xid, lx, ly))
+}
- // Always walk the AT-SPI tree; capture the screenshot by default. The
- // tree+screenshot pair is the default so the agent grounds on both and
- // cross-checks the (sometimes-lying) tree against the frame. An explicit
- // `include_screenshot:false` skips the grab; an unproven Wayland surface
- // returns the tree with a typed screenshot error instead of unrelated pixels.
- let should_capture = include_screenshot != Some(false) || screenshot_out_file.is_some();
- if !want_tree && !should_capture {
- return ToolResult::error(
- "Nothing to return: both include_accessibility_tree:false and \
- include_screenshot:false. Set at least one to true, or pass \
- screenshot_out_file to force a capture.",
- );
- }
- // `display_only` is the public form of the internal observation-only
- // read: pixels for a preview, with no snapshot or capture change.
- let observation_only = display_only
- || args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true);
- let state = self.state.clone();
- let state_for_capture = state.clone();
- let query_for_walk = query.clone();
-
- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
- // Skip the AT-SPI walk on the capture-only path
- // (include_accessibility_tree:false).
- let tree_result = if want_tree {
- Some(crate::atspi::walk_tree_bounded_within(
- pid,
- xid,
- query_for_walk.as_deref(),
- max_elements,
- max_depth,
- walk_timeout,
- ))
- } else {
- None
- };
- let binds_window = tree_result.as_ref().is_some_and(|tree| {
- crate::atspi::snapshot_binds_window(pid, xid, tree.window_scoped, tree.trusted)
- });
- // Bounds and element indices come from the same captured AT-SPI
- // traversal. Joining two live walks by ordinal mis-associated
- // Chromium controls when its lazy subtree changed between walks.
- let bounds = tree_result
- .as_ref()
- .map(|tree| tree.bounds.clone())
- .unwrap_or_default();
- // Capture and DELIVER the screenshot alongside the tree by default — the
- // grounding frame the agent cross-checks the tree against. With
- // screenshot_out_file set, write to disk and surface the path instead
- // of embedding base64; otherwise embed base64. Skipped only when
- // include_screenshot:false and no disk path was requested.
- // Keep the exact delivered PNG for capture publication. The action
- // binding must identify the bytes returned to the caller, including
- // any max-dimension resize, rather than the backend's raw frame.
- let mut screenshot_error = None;
- // Same-pid popups (menus, combo lists) and transient dialogs mapped
- // over this window: listed for the caller, and when one overlaps
- // the window the screenshot is taken from the screen, since the
- // window's own drawable never shows them.
- let overlays = if crate::wayland::is_wayland() {
- WindowOverlays::default()
- } else {
- window_overlays(pid, xid)
- };
- let screenshot = if should_capture {
- // A screenshot-only read with a size cap (a live preview) asks
- // a GNOME helper with API 10 to downscale in the compositor,
- // and keeps the window's full size for frame_scale.
- let preview = if !want_tree && max_dim > 0 && crate::wayland::is_wayland() {
- crate::wayland::shell_helper::screenshot_window_preview(xid, pid, max_dim)
- } else {
- None
- };
- let captured = if let Some((png, width, height)) = preview {
- Ok((png, Some((width, height))))
- } else if overlays.covers_window {
- overlays
- .window_rect
- .ok_or_else(|| anyhow::anyhow!("window geometry unavailable"))
- .and_then(|(x, y, w, h)| {
- crate::capture::screenshot_root_region_png(x, y, w, h)
- })
- .or_else(|_| crate::wayland::screenshot_dispatch_with_pid(xid, pid))
- .map(|png| (png, None))
- } else {
- crate::wayland::screenshot_dispatch_with_pid(xid, pid).map(|png| (png, None))
- };
- match captured {
- Ok((raw, original_size)) => {
- let (orig_w, orig_h) = match original_size {
- Some(size) => size,
- None => crate::capture::png_dimensions_pub(&raw)?,
- };
- let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
- let (w, h) = crate::capture::png_dimensions_pub(&png)?;
- // An explicit per-call max_image_dimension (0 = native)
- // is authoritative; the model-safe cap applies otherwise.
- let cap = if explicit_max_image_dimension {
- None
- } else {
- megapixel_long_edge_cap(w, h, WINDOW_SCREENSHOT_MAX_PIXELS)
- };
- let png = match cap {
- Some(edge) => crate::capture::resize_png_if_needed(&png, edge)?,
- None => png,
- };
- let (w, h) = crate::capture::png_dimensions_pub(&png)?;
- let original_w = if w < orig_w { Some(orig_w) } else { None };
- let (b64, file_path) = if let Some(ref path) = screenshot_out_file {
- std::fs::write(path, &png)?;
- (None, Some(path.clone()))
- } else {
- use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
- (Some(B64.encode(&png)), None)
- };
- let capture_id = if observation_only {
- None
- } else {
- Some(crate::capture_action_frame::publish_window(
- &state_for_capture.capture_service,
- &args,
- &png,
- pid,
- xid,
- (w, h),
- (orig_w, orig_h),
- )?)
- };
- Some((b64, file_path, w, h, original_w, capture_id))
- }
- Err(error) if crate::wayland::is_surface_identity_unproven(&error) => {
- screenshot_error = Some(error.to_string());
- None
- }
- // The window passed the ownership check above and went
- // away during the AT-SPI walk (a document closed by the
- // action being observed): a stale target, not a capture
- // failure.
- Err(_) if !crate::wayland::is_wayland() && !crate::x11::window_exists(xid) => {
- return Err(anyhow::anyhow!(
- "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows."
- ));
- }
- Err(error) => {
- return Err(anyhow::anyhow!(
- "window screenshot failed for window {xid}: {error}"
- ));
- }
- }
- } else {
- None
- };
- Ok((
- tree_result,
- screenshot,
- bounds,
- screenshot_error,
- overlays,
- binds_window,
- ))
- })
- .await;
+pub(crate) fn element_screen_center(
+ pid: u32,
+ token: &crate::atspi::TokenElement,
+ xid: Option<u64>,
+) -> anyhow::Result<(f64, f64)> {
+ element_screen_center_and_rect(pid, token, xid).map(|(center, _)| center)
+}
- match result {
- Ok(Ok((tree_opt, shot_opt, bounds, screenshot_error, overlays, binds_window))) => {
- let mut content = Vec::new();
- let mut structured = json!({ "window_id": xid, "pid": pid });
- let screenshot_scale = shot_opt.as_ref().map(|(_, _, w, _, original_w, _)| {
- original_w.map_or(1.0, |ow| ow as f64 / *w as f64)
- });
- let mut published_snapshot = false;
- let mut invalidated = Vec::new();
-
- if let Some(tr) = tree_opt {
- let source_trusted = tr.trusted;
- let source_degraded_reason = tr.degraded_reason.clone();
- let count = tr
- .nodes
- .iter()
- .filter(|n| n.element_index.is_some())
- .count();
- let mut header = format!(
- "window_id={xid} pid={pid} elements={count} walk_ms={}\n",
- tr.elapsed_ms
- );
- if tr.truncated {
- header.push_str(&cua_driver_core::walk_budget::truncation_note(
- tr.truncation_reason.as_deref(),
- timeout_ms,
- tr.nodes_visited,
- tr.nodes_pending,
- ));
- header.push('\n');
- } else if !tr.bounds_complete {
- header.push_str(
- "⚠️ bounds phase ran out of time: some elements have no frame \
- (element_token clicks still work; pixel targeting may not). \
- Retry with a larger timeout_ms if you need frames.\n",
- );
- }
- header.push('\n');
- content.push(cua_driver_core::protocol::Content::text(
- header + &tr.tree_markdown,
- ));
- structured["element_count"] = json!(count);
- // `elements_complete` is a real claim now: a native walk that
- // finished without hitting the deadline / node budget saw
- // every node the application publishes. Truncated or
- // fallback trees keep negative existence unknown.
- structured["elements_complete"] = json!(tr.trusted && !tr.truncated);
- structured["truncated"] = json!(tr.truncated);
- if let Some(reason) = &tr.truncation_reason {
- structured["truncation_reason"] = json!(reason);
- }
- structured["nodes_visited"] = json!(tr.nodes_visited);
- structured["nodes_pending"] = json!(tr.nodes_pending);
- structured["bounds_complete"] = json!(tr.bounds_complete);
- structured["walk_elapsed_ms"] = json!(tr.elapsed_ms as u64);
- structured["timeout_ms"] = json!(timeout_ms);
- structured["tree_markdown"] = json!(tr.tree_markdown);
-
- // Elements bind to this window only when the walk proved
- // the window's scope (`snapshot_binds_window`).
- let target_scoped = binds_window;
- if !observation_only && !target_scoped {
- invalidated.extend(state.snapshots.remove(pid as i32, xid));
- crate::atspi::snapshot::forget_window(pid, xid);
- }
- let (snapshot_id, replaced) = (!observation_only && target_scoped)
- .then(|| {
- state.snapshots.publish_for_session(
- pid as i32,
- xid,
- crate::atspi::snapshot::update_snapshot(
- pid, xid, &tr.nodes, &tr.bounds,
- ),
- session_id.as_deref(),
- screenshot_scale,
- )
- })
- .flatten()
- .unzip();
- published_snapshot = snapshot_id.is_some();
- invalidated.extend(replaced.into_iter().flatten());
-
- // Structured `elements` array: one entry per actionable node.
- // Shape: `{element_index, element_token, role, label,
- // depth, actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is
- // included whenever AT-SPI Component.GetExtents(Screen)
- // reported usable bounds; omitted otherwise (some
- // toolkits leave bounds unset on hidden / virtual
- // elements).
- use std::collections::HashMap;
- // The walk produces screen extents (what the element
- // cache and hit-tests compare against). The published
- // `frame` is the element's screen rectangle, as on macOS and
- // Windows; `screenshot_frame` (added below) is the same
- // rectangle in pixels of the screenshot in this response,
- // the space of the pointer tools' window-local x/y. The
- // window-local origin is the X11 window's root-relative
- // origin, the one `window_bounds` and
- // `coordinate_frame:"desktop"` translation use. A popup
- // (override-redirect) window is not in the WM's client
- // list; when its origin cannot be read, the popup's own
- // screen rectangle is the origin.
- let local_origin = (!crate::wayland::is_wayland())
- .then(|| crate::atspi::native::x11_window_origin(xid))
- .flatten()
- .or_else(|| popup_meta.as_ref().map(|popup| (popup.x, popup.y)));
- let bounds_by_idx: HashMap<usize, (i32, i32, u32, u32)> = bounds
- .into_iter()
- .map(|(i, x, y, w, h)| (i, (x, y, w, h)))
- .collect();
- let elements: Vec<serde_json::Value> = tr
- .nodes
- .iter()
- .filter_map(|n| {
- build_element_entry(
- n,
- snapshot_id,
- bounds_by_idx.get(&n.element_index?).copied(),
- )
- })
- .collect();
- let elements = cua_driver_core::element_query::project_elements_for_query(
- elements,
- query.as_deref(),
- &tr.tree_markdown,
- );
- let elements = framed_elements_first(elements);
- // Screenshot pixels: the capture is the window's own X11
- // drawable (screen pixels), downsized by delivered/native.
- let elements = match (local_origin, shot_opt.as_ref()) {
- (Some((ox, oy)), Some((_, _, w, _, orig_w, _))) => {
- let scale = orig_w.map_or(1.0, |ow| *w as f64 / ow as f64);
- cua_driver_core::element_frame::with_screenshot_frames(
- elements,
- (f64::from(ox), f64::from(oy)),
- scale,
- )
- }
- _ => elements,
- };
- // A popup that has no AT-SPI frame of its own (LibreOffice
- // VCL menus live under the menubar's `menu` node): return
- // the open menu's items, the ones drawn inside the popup,
- // instead of the whole application.
- let elements = match (&popup_meta, tr.window_scoped) {
- (Some(popup), false) => popup_menu_elements(
- elements,
- (popup.x, popup.y),
- popup.width,
- popup.height,
- ),
- _ => elements,
- };
- structured["total_element_count"] = json!(count);
- structured["returned_element_count"] = json!(elements.len());
- structured["elements"] = json!(elements);
- // Surface 6: snapshot id mirror for debug correlation.
- if let Some(snapshot_id) = snapshot_id {
- structured["snapshot_id"] = json!(
- cua_driver_core::element_token::format_snapshot_id(snapshot_id)
- );
- }
- structured["_note"] = json!(
- "Prefer `elements` — `tree_markdown` will continue to work \
- but new fields will only be added to the structured side. \
- Use `max_elements` / `max_depth` to bound the \
- AT-SPI walk on apps with very large trees."
- );
- // Best-effort-background ladder parity with macOS/Windows: an
- // AT-SPI walk that ran but found zero actionable elements is
- // NOT a clean "this window has no controls" — far more often
- // the bridge wasn't ready (toolkit-accessibility off, or the
- // daemon isn't on the desktop session bus so the registry is
- // empty), or it's a non-AX surface (canvas/WebGL). Mark it
- // degraded so callers don't read `elements: []` as authoritative.
- if !target_scoped {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!("accessibility_window_identity_unproven: tree is application-scoped; exact-window element tokens and bounds are unavailable");
- } else if !source_trusted {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!(source_degraded_reason
- .unwrap_or_else(|| {
- "x11_property_fallback_partial: AT-SPI was unavailable and \
- Cua Driver only recovered window metadata. Treat it as \
- discovery evidence; it cannot prove checked state."
- .to_owned()
- }));
- } else if count == 0 && tr.truncated {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] =
- json!(cua_driver_core::walk_budget::truncation_note(
- tr.truncation_reason.as_deref(),
- timeout_ms,
- tr.nodes_visited,
- tr.nodes_pending,
- ));
- } else if count == 0 {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!(
- "atspi_tree_empty: the AT-SPI walk returned no actionable \
- elements. Common causes: the a11y bridge is off (enable \
- `gsettings set org.gnome.desktop.interface \
- toolkit-accessibility true`), the daemon is not on the \
- desktop session bus (DBUS_SESSION_BUS_ADDRESS unreachable — \
- run `cua-driver doctor`), or the window is a non-AX surface \
- (canvas/WebGL/custom-drawn). Do not treat element data as \
- authoritative — verify via the screenshot, and re-snapshot \
- after enabling a11y or if the app just launched."
- );
- // Point the agent at the next rung explicitly: an empty
- // tree means element_index has nothing to bind to. The
- // recommendation is session-dependent (X11 → px,
- // Wayland → foreground) — see non_ax_escalation.
- structured["escalation"] = non_ax_escalation();
- }
- }
+/// The element's screen centre plus its AT-SPI extents as the overlay's
+/// `[x, y, width, height]`, both from one bounds lookup (same screen space).
+pub(crate) fn element_screen_center_and_rect(
+ pid: u32,
+ token: &crate::atspi::TokenElement,
+ xid: Option<u64>,
+) -> anyhow::Result<((f64, f64), [f64; 4])> {
+ let (bx, by, bw, bh) = crate::atspi::token_bounds(pid, xid, token)?;
+ let rect = [bx as f64, by as f64, bw as f64, bh as f64];
+ Ok(((rect[0] + rect[2] / 2.0, rect[1] + rect[3] / 2.0), rect))
+}
- // A screenshot-only read adds an image but no element rows: it
- // refreshes the current snapshot's pixel frame in place and
- // keeps its tokens valid. Only a window with no element
- // snapshot yet gets a new capture-only one.
- let mut refreshed = None;
- if let (false, false, Some(scale)) =
- (observation_only, published_snapshot, screenshot_scale)
- {
- refreshed = state.snapshots.refresh_screenshot_for_session(
- pid as i32,
- xid,
- session_id.as_deref(),
- scale,
- None,
- );
- if refreshed.is_none() {
- if let Some((_, replaced)) = state.snapshots.publish_capture_for_session(
- pid as i32,
- xid,
- crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
- session_id.as_deref(),
- screenshot_scale,
- ) {
- invalidated.extend(replaced);
- }
- }
- }
- if let Some(id) = refreshed {
- let sid = cua_driver_core::element_token::format_snapshot_id(id);
- content.push(cua_driver_core::protocol::Content::text(format!(
- "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
- ({sid}:N) stay valid."
- )));
- structured["snapshot_id"] = json!(sid);
- structured["screenshot_refreshed_snapshot"] = json!(true);
- }
- if !invalidated.is_empty() {
- let ids: Vec<String> = invalidated
- .into_iter()
- .map(cua_driver_core::element_token::format_snapshot_id)
- .collect();
- content.push(cua_driver_core::protocol::Content::text(format!(
- "Invalidated snapshots {}: their element_tokens are stale.",
- ids.join(", ")
- )));
- structured["invalidated_snapshot_ids"] = json!(ids);
- }
+/// Shared schema for the optional `coordinate_frame` of pointer tools.
+pub(crate) fn coordinate_frame_schema() -> Value {
+ json!({
+ "type": "string",
+ "enum": ["window", "desktop"],
+ "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": \
+ window-local screenshot pixels as returned by get_window_state. \"desktop\": \
+ full-screen pixels as returned by get_desktop_state, translated to the target \
+ window. Passing scope:\"desktop\" together with pid/window_id means the same thing."
+ })
+}
- if let Some((b64_opt, file_path, w, h, orig_w, capture_id)) = shot_opt {
- // ax mode + screenshot_out_file writes the PNG to disk and
- // returns b64=None — never embed the image bytes in that case.
- // Keep a text content part when the image went to disk so the
- // response is never empty on the capture-only path (which has
- // no tree markdown either).
- if let Some(b64) = b64_opt {
- content.push(cua_driver_core::protocol::Content::image_png(b64));
- } else if let Some(fp) = &file_path {
- content.push(cua_driver_core::protocol::Content::text(format!(
- "window_id={xid} pid={pid} size={w}x{h} screenshot written to {fp}"
- )));
- }
- structured["screenshot_width"] = json!(w);
- structured["screenshot_height"] = json!(h);
- structured["screenshot_frame_valid"] = json!(true);
- if let Some(ow) = orig_w {
- if ow > 0 {
- structured["frame_scale"] = json!(w as f64 / ow as f64);
- structured["screenshot_original_width"] = json!(ow);
- }
- }
- // Surface 7: mirror the MCP image part's `mimeType` onto
- // the structured payload so consumers don't have to sniff
- // magic bytes off the base64 to know the format.
- structured["screenshot_mime_type"] = json!("image/png");
- if let Some(capture_id) = capture_id {
- structured["capture_id"] = json!(capture_id);
- }
- if let Some(fp) = file_path {
- structured["screenshot_file_path"] = json!(fp);
- }
- }
- if let Some(reason) = &screenshot_error {
- structured["screenshot_frame_valid"] = json!(false);
- structured["screenshot_error"] =
- surface_identity_unproven_error(xid, reason.clone());
- }
- // Window identity metadata (additive): app + title + on-screen
- // rectangle for the requested window_id, useful on the
- // capture-only path where no AT-SPI tree names the surface.
- if popup_meta.is_some() {
- structured["popup"] = json!(true);
- }
- if let Some(meta) = &window_meta {
- if !meta.app_name.is_empty() {
- structured["app_name"] = json!(meta.app_name);
- }
- if !meta.title.is_empty() {
- structured["window_title"] = json!(meta.title);
- }
- structured["window_bounds"] = json!({
- "x": meta.x, "y": meta.y, "width": meta.width, "height": meta.height
- });
- }
- // Transient dialogs and popups of this pid that are open over
- // the window, each with the call that targets it.
- if !overlays.dialogs.is_empty() {
- structured["dialogs"] = json!(overlays.dialogs);
- }
- if !overlays.popups.is_empty() {
- structured["popups"] = json!(overlays.popups);
- }
- if overlays.covers_window {
- structured["screenshot_composited"] = json!(true);
- }
- if let Some(note) = overlays.follow_up(pid) {
- structured["follow_up"] = json!(note);
- content.push(cua_driver_core::protocol::Content::text(note));
- }
- structured["coordinate_frame"] = json!("window");
- structured["frame_note"] = json!(
- "x/y for click / double_click / right_click / drag / scroll on this \
- window are pixels of THIS screenshot (window-local, 0..screenshot_width \
- x 0..screenshot_height), as are the element frames; the screenshot is \
- kept at or below 1.15 megapixels so the image you read is the image these \
- pixels index (frame_scale < 1 says the window was downsized to it). \
- window_bounds is where it sits on the screen. Pass scope:\"desktop\" only \
- for get_desktop_state pixels."
- );
- if display_only {
- cua_driver_core::window_state_view::mark_display_only(&mut structured);
- }
+pub(crate) fn coordinate_frame_is_desktop(args: &Value) -> bool {
+ args.get("coordinate_frame").and_then(Value::as_str) == Some("desktop")
+}
- // The capture-only path (include_accessibility_tree:false) leaves
- // `content` empty when the screenshot was also unavailable — most
- // often on Wayland, where per-window capture cannot prove surface
- // identity. Return a structured error rather than a "successful"
- // response with no content parts.
- if content.is_empty() {
- let reason_note = match &screenshot_error {
- Some(reason) => format!(" and no screenshot could be captured ({reason})"),
- None => " and no screenshot was returned".to_string(),
- };
- return ToolResult::error(format!(
- "No content produced for window_id {xid}: the accessibility tree was \
- skipped (include_accessibility_tree:false){reason_note}."
- ))
- .with_structured(structured);
- }
+pub(crate) fn window_scope_unproven_refusal(tool: &str, xid: u64, reason: &str) -> ToolResult {
+ let message = format!(
+ "{tool}: the element's membership in window_id {xid} is unproven ({reason}); \
+ call get_window_state again"
+ );
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": { "code": "window_scope_unproven", "message": message }
+ }))
+}
- if !observation_only {
- cua_driver_core::window_state_view::apply(
- &view,
- &cua_driver_core::window_state_view::ViewContext {
- pid: i64::from(pid),
- window_id: xid,
- key: cua_driver_core::window_state_view::ViewKey {
- query: query.clone(),
- max_elements: max_elements.unwrap_or(LINUX_DEFAULT_MAX_ELEMENTS),
- max_depth,
- },
- focus_probe: None,
- },
- &mut content,
- &mut structured,
- );
- }
+/// Prove from a fresh walk that the token's observed element is still in the
+/// token's window, then carry that element to dispatch. `Ok(None)` without a
+/// token. Refuses before any dispatch. X11, Hyprland and compositors without
+/// a GNOME Shell need no proof.
+pub(crate) async fn prove_token_element(
+ pid: u32,
+ resolved: &cua_driver_core::element_token::ResolvedElement<
+ crate::atspi::snapshot::CachedElement,
+ >,
+ tool: &str,
+) -> Result<Option<crate::atspi::TokenElement>, ToolResult> {
+ let cua_driver_core::element_token::ResolvedElement::Element {
+ window_id,
+ element_index,
+ element,
+ } = resolved
+ else {
+ return Ok(None);
+ };
+ let (xid, identity) = (*window_id, element.identity.clone());
+ let proof = tokio::task::spawn_blocking(move || {
+ crate::atspi::require_window_element(
+ pid,
+ xid,
+ crate::atspi::WindowElement::Identity(identity),
+ )
+ })
+ .await
+ .unwrap_or_else(|error| Err(anyhow::anyhow!("task error: {error}")));
+ token_element_for(tool, xid, *element_index, element.clone(), proof).map(Some)
+}
- ToolResult {
- content,
- is_error: None,
- structured_content: Some(structured),
- action_record: None,
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("Capture error: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
+pub(crate) fn token_element_for(
+ tool: &str,
+ xid: u64,
+ index: usize,
+ observed: crate::atspi::snapshot::CachedElement,
+ proof: anyhow::Result<crate::atspi::WindowProof>,
+) -> Result<crate::atspi::TokenElement, ToolResult> {
+ proof
+ .map(|proof| crate::atspi::TokenElement::new(index, observed, proof))
+ .map_err(|error| window_scope_unproven_refusal(tool, xid, &error.to_string()))
}
-/// One-line, model-facing explanation of a partial tree and what to do about
-/// it. Shared by the text header and the structured `degraded_reason`.
-fn surface_identity_unproven_error(xid: u64, reason: String) -> Value {
- json!({
- "code": "surface_identity_unproven",
- "window_id": xid,
- "reason": reason,
- "suggestion": "capture the full output explicitly, or retry on a compositor backend that supports identified per-window capture"
- })
+/// The object an element token observed no longer answers: the token is
+/// stale. Nothing reached that object, and it is never retargeted by index
+/// after a window proof.
+pub(crate) fn stale_observed_element(
+ tool: &str,
+ idx: usize,
+ error: &anyhow::Error,
+) -> Option<ToolResult> {
+ error
+ .is::<crate::atspi::native::CachedElementGone>()
+ .then(|| {
+ ToolResult::error(format!(
+ "{tool}: stale_element_token: observed AT-SPI object [{idx}] is no longer \
+ present ({error}); re-snapshot with get_window_state"
+ ))
+ .with_structured(json!({ "code": "stale_element_token", "effect": "none" }))
+ })
}
-// ── launch_app ───────────────────────────────────────────────────────────────
+pub(crate) fn window_local_to_screen(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {
+ use x11rb::connection::Connection;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
-#[cfg(test)]
-mod get_window_state_actions_tests;
-
-pub struct LaunchAppTool;
-static LAUNCH_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-/// Spawn a launcher command line (an executable plus arguments, e.g. an XDG
-/// `Exec=` value with field codes stripped) in the background and return the
-/// child pid.
-fn spawn_launch_command(cmd: &str, additional_arguments: &[String]) -> std::io::Result<u32> {
- let mut parts = cmd.split_whitespace();
- let prog = parts.next().unwrap_or(cmd);
- let mut rest: Vec<String> = parts.map(str::to_owned).collect();
- rest.extend(additional_arguments.iter().cloned());
- let mut launch = std::process::Command::new(prog);
- launch
- .args(&rest)
- // The child must not inherit the driver's stdio: in private-worker mode
- // stdout is the JSON-RPC stream to the SDK, and a chatty app (Chromium's
- // zygote logs, GTK warnings) writing there corrupts a response and shuts
- // the worker down.
- .stdin(std::process::Stdio::null())
- .stdout(std::process::Stdio::null())
- .stderr(std::process::Stdio::null())
- // Enable accessibility for this child without toggling GNOME's global
- // ScreenReaderEnabled setting (which can launch Orca). Native
- // toolkits ignore these when they do not need them.
- .env("ACCESSIBILITY_ENABLED", "1")
- .env("NO_AT_BRIDGE", "0");
- if chromium_family_program(prog)
- && !rest
- .iter()
- .any(|arg| arg == "--force-renderer-accessibility")
- {
- launch.arg("--force-renderer-accessibility");
- }
- let child = launch.spawn()?;
- let pid = child.id();
- reap_in_background(child);
- Ok(pid)
+ let (conn, screen_num) = RustConnection::connect(None)?;
+ let root = conn.setup().roots[screen_num].root;
+ let reply = conn
+ .translate_coordinates(xid as u32, root, 0, 0)?
+ .reply()?;
+ Ok((reply.dst_x as f64 + x, reply.dst_y as f64 + y))
}
-/// Collect a launched child's exit status in the background.
-///
-/// `std::process::Child` does not reap on drop and the driver never waits on
-/// an app it launches, so every launched app used to linger as a zombie for
-/// the daemon's lifetime: it held a pid slot, and — because the pid stays
-/// present — made "does this pid exist" liveness checks report a terminated
-/// app as still running.
-///
-/// Reaping is scoped to one thread per child rather than setting
-/// `SIGCHLD` to `SIG_IGN`, which is process-global and would break every
-/// caller that needs an exit status, including the `xdg-open` check below
-/// and any `Command::status()`/`output()` elsewhere in the daemon. The
-/// thread blocks in `wait` until the app exits, so it costs nothing while
-/// the app runs and never delays the launch itself.
-fn reap_in_background(mut child: std::process::Child) {
- let pid = child.id();
- let reaper = std::thread::Builder::new()
- .name(format!("cua-reap-{pid}"))
- .stack_size(64 * 1024)
- .spawn(move || {
- if let Err(e) = child.wait() {
- tracing::debug!(pid, "launched app could not be reaped: {e}");
- }
- });
- if let Err(e) = reaper {
- tracing::debug!(pid, "could not start reaper thread for launched app: {e}");
- }
+/// Either `coordinate_frame:"desktop"` or `scope:"desktop"` marks `x,y` as
+/// desktop (screen) pixels; both are translated exactly once by the caller.
+pub(crate) fn desktop_frame_requested(args: &Value) -> bool {
+ coordinate_frame_is_desktop(args)
+ || args.get("scope").and_then(Value::as_str) == Some("desktop")
}
-/// Match a launch_app `name` that failed direct exec against installed XDG
-/// .desktop applications: exact display name, desktop-file id, or `Exec=`
-/// basename first, then a display-name substring when it is unambiguous.
-fn match_installed_app<'a>(
- apps: &'a [crate::installed_apps::InstalledApp],
- query: &str,
-) -> Option<&'a crate::installed_apps::InstalledApp> {
- let q = query.to_ascii_lowercase();
- if q.is_empty() {
- return None;
+/// Convert desktop (screen) coordinates to `xid`'s window-local frame so the
+/// rest of the pointer pipeline (which is window-local by contract) is reused.
+pub(crate) fn desktop_to_window_local(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {
+ // A niri window id is no X11 window: never ask XWayland about it.
+ if crate::wayland::is_wayland() && crate::wayland::niri::is_session() {
+ return Err(crate::wayland::niri::desktop_frame_window_refusal(xid));
}
- apps.iter()
- .find(|a| {
- a.name.to_ascii_lowercase() == q
- || a.bundle_id.to_ascii_lowercase() == q
- || exec_basename(&a.launch_path) == q
- })
- .or_else(|| {
- let mut matches = apps
- .iter()
- .filter(|a| a.name.to_ascii_lowercase().contains(&q));
- match (matches.next(), matches.next()) {
- (Some(only), None) => Some(only),
- _ => None,
- }
- })
+ let (ox, oy) = window_local_to_screen(xid, 0.0, 0.0)?;
+ Ok((x - ox, y - oy))
}
-/// Watch a just-spawned `xdg-open` long enough to catch a fast failure.
-/// xdg-open's generic fallback can `exec` the target app and stay alive for
-/// its whole lifetime, so a child still running after the grace period counts
-/// as success; a quick non-zero exit (2 = file not found, 3 = no handler
-/// tool, 4 = action failed) is the only reliable failure signal.
-fn xdg_open_failure(mut child: std::process::Child) -> Option<String> {
- let deadline = std::time::Instant::now() + std::time::Duration::from_millis(1500);
- loop {
- // A branch that observes an exit status has already reaped the child;
- // one that gives up on it still owes that, so hand it to the reaper
- // instead of dropping it and leaving a zombie behind.
- match child.try_wait() {
- Ok(Some(status)) if status.success() => return None,
- Ok(Some(status)) => return Some(format!("xdg-open failed ({status})")),
- Ok(None) if std::time::Instant::now() >= deadline => {
- reap_in_background(child);
- return None;
- }
- Ok(None) => std::thread::sleep(std::time::Duration::from_millis(50)),
- Err(e) => {
- reap_in_background(child);
- return Some(format!("could not observe xdg-open: {e}"));
- }
- }
+/// A failed desktop-to-window mapping; a coded refusal keeps its structure.
+pub(crate) fn desktop_frame_unmapped(xid: u64, error: anyhow::Error) -> ToolResult {
+ if crate::input::foreground::error_code(&error).is_some() {
+ return input_error_result(error);
}
+ ToolResult::error(format!(
+ "desktop-frame coordinates could not be mapped into window {xid}: {error}"
+ ))
}
#[cfg(test)]
-mod launch_app_tests;
-
-#[async_trait]
-impl Tool for LaunchAppTool {
- fn def(&self) -> &ToolDef {
- LAUNCH_DEF.get_or_init(|| ToolDef {
- name: "launch_app".into(),
- description: "Launch a Linux app in the background. Provide launch_path (preferred — \
- round-trip the value from list_apps), name (tried as a direct command, then \
- matched against installed .desktop applications, then handed to xdg-open if it \
- is a URL or existing file path), bundle_id (ignored on Linux), or urls (list of \
- URLs to open). Resolution precedence: launch_path > name > bundle_id. Errors \
- when the name resolves to nothing launchable.".into(),
- input_schema: json!({"type":"object","properties":{
- "launch_path":{"type":"string","description":"Round-trip the `launch_path` returned by `list_apps` — the Exec= command from the .desktop file with XDG field codes already stripped. Highest precedence on Linux; spawned directly via the system shell."},
- "name":{"type":"string","description":"App name or command to launch. Tried as a direct command first, then matched against installed .desktop applications (exact display name, desktop-file id, or Exec basename; else an unambiguous display-name substring)."},
- "bundle_id":{"type":"string","description":"Ignored on Linux (macOS/Windows concept)."},
- "urls":{"type":"array","items":{"type":"string"},"description":"URLs to open via xdg-open."},
- "additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process."}
- },"additionalProperties":false}),
- read_only: false, destructive: false, idempotent: false, open_world: true,
- })
- }
+#[test]
+fn a_niri_desktop_frame_window_refusal_is_structured() {
+ let refused =
+ desktop_frame_unmapped(31, crate::wayland::niri::desktop_frame_window_refusal(31));
+ let structured = refused.structured_content.unwrap();
+ assert_eq!(structured["code"], "foreground_unavailable");
+ assert_eq!(structured["effect"], "none");
+ assert!(structured["suggestion"]
+ .as_str()
+ .unwrap()
+ .starts_with("Drop pid and window_id"));
+}
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let launch_path_opt = args.opt_str("launch_path");
- let name_opt = args.opt_str("name");
- let urls: Vec<String> = args.str_array("urls");
- let additional_arguments: Vec<String> = args.str_array("additional_arguments");
- if args.get("cdp_debugging_port").is_some() {
- return ToolResult::error(
- "cdp_debugging_port moved to browser_prepare so DevTools is never enabled on an unproven user profile",
- );
- }
- if launch_path_opt
- .as_deref()
- .into_iter()
- .chain(name_opt.as_deref())
- .chain(additional_arguments.iter().map(String::as_str))
- .any(cua_driver_core::launch_guard::contains_remote_debugging_flag)
- {
- return ToolResult::error(
- cua_driver_core::launch_guard::REMOTE_DEBUGGING_LAUNCH_REFUSAL,
- );
- }
-
- if launch_path_opt.is_none() && name_opt.is_none() && urls.is_empty() {
- return ToolResult::error("Provide at least one of: launch_path, name, or urls.");
- }
+/// The `steps` of a `drag` or `mouse_drag`, held to the advertised 1..=200
+/// before the button goes down: every step is a motion sent with the
+/// button held.
+pub(crate) fn drag_steps(args: &Value) -> usize {
+ args.u64_or("steps", 20).clamp(1, 200) as usize
+}
- // Snapshot the top-levels before spawning so a window created by an
- // already-running service (D-Bus activation) can be told apart.
- let windows_before: std::collections::HashSet<u64> =
- tokio::task::spawn_blocking(|| crate::wayland::list_windows_dispatch(None))
- .await
- .unwrap_or_default()
- .into_iter()
- .map(|w| w.xid)
- .collect();
- let result = tokio::task::spawn_blocking(
- move || -> anyhow::Result<(String, Option<u32>, String)> {
- // Open URLs via xdg-open.
- if !urls.is_empty() {
- let mut children = Vec::new();
- for url in &urls {
- children.push((
- url.clone(),
- std::process::Command::new("xdg-open").arg(url).spawn()?,
- ));
- }
- for (url, child) in children {
- if let Some(reason) = xdg_open_failure(child) {
- anyhow::bail!("could not open '{url}': {reason}");
- }
- }
- return Ok((
- format!("Opened {} URL(s) via xdg-open.", urls.len()),
- None,
- "xdg-open".to_owned(),
- ));
- }
- // launch_path > name. Both go through the same direct-exec path
- // (so XDG `Exec=` commands round-trip), but launch_path is the
- // canonical form preferred by list_apps callers.
- let command = launch_path_opt.as_deref().or(name_opt.as_deref());
- if let Some(cmd) = command {
- match spawn_launch_command(cmd, &additional_arguments) {
- Ok(pid) => {
- return Ok((
- format!("✅ Launched {cmd} (pid {pid}) in background."),
- Some(pid),
- cmd.to_owned(),
- ));
- }
- Err(_) => {
- // Not an executable on PATH. Resolve the name against
- // installed XDG .desktop applications — the same
- // source list_apps reads — and run the match's Exec=.
- let installed = crate::installed_apps::list_installed_apps();
- if let Some(app) = match_installed_app(&installed, cmd) {
- let pid =
- spawn_launch_command(&app.launch_path, &additional_arguments)
- .map_err(|e| {
- anyhow::anyhow!(
- "'{cmd}' matched installed app '{}' but its launcher \
- `{}` failed to start: {e}",
- app.name,
- app.launch_path
- )
- })?;
- return Ok((
- format!(
- "✅ Launched {} (`{}`, pid {pid}) in background — \
- resolved '{cmd}' via its desktop entry.",
- app.name, app.launch_path
- ),
- Some(pid),
- format!("{}\u{0}{cmd}\u{0}{}", app.name, app.launch_path),
- ));
- }
- // xdg-open handles URLs and file paths, not app
- // names — only fall through for something it can
- // plausibly open, and surface its fast non-zero
- // exit instead of reporting a launch that never
- // happened.
- if !cmd.contains("://") && !std::path::Path::new(cmd).exists() {
- anyhow::bail!(
- "'{cmd}' is not an executable on PATH and matches no \
- installed .desktop application. Call list_apps and \
- round-trip its launch_path."
- );
- }
- let child = std::process::Command::new("xdg-open").arg(cmd).spawn()?;
- if let Some(reason) = xdg_open_failure(child) {
- anyhow::bail!("could not open '{cmd}': {reason}");
- }
- // xdg-open may spawn a helper and exit, so do not
- // claim its pid is the app pid.
- return Ok((
- format!("Opened '{cmd}' via xdg-open."),
- None,
- cmd.to_owned(),
- ));
- }
- }
- }
- unreachable!()
- },
- )
+#[cfg(test)]
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_timed_out_call_holds_the_desktop_until_its_late_cleanup_is_done() {
+ use cua_driver_core::tool::DesktopCall;
+ // niri: call 1 focused its window, then waited in niri past its budget.
+ // Its worker gives focus back afterwards. Desktop typing that started in
+ // between would have focus moved away under it.
+ let events = Arc::new(std::sync::Mutex::new(Vec::new()));
+ let call_1 = {
+ let events = events.clone();
+ DesktopCall::acquire()
+ .await
+ .unwrap()
+ .scope(spawn_blocking_bounded(
+ "call 1",
+ std::time::Duration::from_millis(50),
+ move || {
+ std::thread::sleep(std::time::Duration::from_millis(300));
+ // The next key is refused, but focus still goes back.
+ assert!(crate::input::action::admit(|| ()).is_err());
+ events.lock().unwrap().push("call 1 gave focus back");
+ Ok(())
+ },
+ ))
+ .await
+ .unwrap()
+ };
+ assert!(call_1.is_err());
+ DesktopCall::acquire()
+ .await
+ .unwrap()
+ .scope(async { events.lock().unwrap().push("call 2 typed") })
.await;
-
- match result {
- Ok(Ok((message, pid_opt, name))) => {
- if let Some(launcher_pid) = pid_opt {
- // `name` may carry NUL-separated match keys (display name,
- // query, Exec=) for a desktop-entry launch.
- let query = name.clone();
- let name = name.split('\u{0}').next().unwrap_or("").to_owned();
- let resolved = tokio::task::spawn_blocking(move || {
- resolve_launched_windows(launcher_pid, &query, &windows_before)
- })
- .await
- .unwrap_or_else(|_| LaunchedWindows::default());
- let windows: Vec<Value> =
- resolved.windows.iter().map(window_record_json).collect();
- let mut text = message;
- let mut structured = json!({
- "pid": resolved.pid,
- "bundle_id": Value::Null,
- "name": name,
- "running": resolved.pid.is_some(),
- "active": false,
- "windows": windows,
- "launcher_pid": launcher_pid,
- });
- if resolved.handed_off {
- match resolved.pid {
- Some(pid) => {
- structured["handoff"] = json!("dbus_activation");
- text.push_str(&format!(
- " The launcher handed the request to another process \
- (a wrapper or D-Bus activation); the window belongs to pid {pid}."
- ));
- }
- None => {
- // The launcher exited (D-Bus activation or a failed
- // start) and the hand-off budget in
- // `resolve_launched_windows` passed with no new
- // window ever appearing. This is not a success: we
- // have no pid, no window, and no way to know the app
- // actually started. Report a typed refusal instead
- // of a `pid: null` shape that reads as success.
- return launch_handoff_timeout_result(
- &name,
- launcher_pid,
- LAUNCH_HANDOFF_BUDGET.as_secs(),
- );
- }
- }
- }
- ToolResult::text(text).with_structured(structured)
- } else {
- ToolResult::text(message).with_structured(json!({
- "pid": Value::Null,
- "bundle_id": Value::Null,
- "name": name,
- "running": Value::Null,
- "active": false,
- "windows": [],
- }))
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("Failed to launch: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
+ assert_eq!(
+ *events.lock().unwrap(),
+ ["call 1 gave focus back", "call 2 typed"]
+ );
}
-/// The `launch_app` D-Bus-activation hand-off timed out: the launcher exited
-/// (handing the request to a running service) but no new window of the app
-/// ever appeared within the deadline. This has no pid and no confirmed
-/// window, so it is a refusal (`code: "launch_handoff_timeout"`), never a
-/// `pid: null` shape that a caller could mistake for success.
-fn launch_handoff_timeout_result(name: &str, launcher_pid: u32, waited_secs: u64) -> ToolResult {
- ToolResult::error(format!(
- "'{name}' was handed off for D-Bus activation but no new window appeared within \
- {waited_secs}s; the launch could not be confirmed. Call list_windows to check \
- whether it started anyway, or retry."
- ))
- .with_structured(json!({
- "code": "launch_handoff_timeout",
- "effect": "refused",
- "handoff": "dbus_activation",
- "name": name,
- "launcher_pid": launcher_pid,
- "pid": Value::Null,
- "running": Value::Null,
- "waited_secs": waited_secs,
- }))
+#[cfg(test)]
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn input_that_cannot_be_cancelled_never_overlaps_the_next_desktop_action() {
+ use cua_driver_core::tool::DesktopCall;
+ // A wtype child does not read the call's action, and an abandoned XTest
+ // call still sends its releases: a worker whose call timed out, or whose
+ // caller went away, can still send input. That input must not reach
+ // whatever the next desktop action targets.
+ for budget in [50, 60_000] {
+ let events = Arc::new(std::sync::Mutex::new(Vec::new()));
+ let worker_events = events.clone();
+ let call = DesktopCall::acquire().await.unwrap();
+ let call_1 = async move {
+ call.scope(spawn_blocking_bounded(
+ "call 1",
+ std::time::Duration::from_millis(budget),
+ move || {
+ std::thread::sleep(std::time::Duration::from_millis(300));
+ worker_events.lock().unwrap().push("call 1 typed");
+ Ok(())
+ },
+ ))
+ .await
+ };
+ // A 50 ms budget times out; with 60 s the caller is dropped first.
+ let _ = tokio::time::timeout(std::time::Duration::from_millis(100), call_1).await;
+ DesktopCall::acquire()
+ .await
+ .unwrap()
+ .scope(async { events.lock().unwrap().push("call 2 typed") })
+ .await;
+ assert_eq!(
+ *events.lock().unwrap(),
+ ["call 1 typed", "call 2 typed"],
+ "budget {budget} ms"
+ );
+ }
}
-/// What `launch_app` could attribute to the launch after the spawn.
-#[derive(Default)]
-struct LaunchedWindows {
- /// The process that owns the app's window: the launcher itself, or the
- /// wrapper target / running service it handed off to.
- pid: Option<u32>,
- windows: Vec<crate::x11::WindowInfo>,
- /// The window belongs to another process than the launcher, or the
- /// launcher exited without one.
- handed_off: bool,
-}
-
-/// Whether the launched process is gone or a zombie (already reaped or being
-/// reaped by the launch thread).
-fn process_exited(pid: u32) -> bool {
- match std::fs::read_to_string(format!("/proc/{pid}/stat")) {
- Ok(stat) => stat
- .rsplit_once(')')
- .and_then(|(_, rest)| rest.split_whitespace().next())
- .is_some_and(|state| state == "Z" || state == "X"),
- Err(_) => true,
- }
-}
-
-/// Case-insensitive identity match between a launch query
-/// (`nautilus`, `org.gnome.Nautilus`, `gnome-control-center`, a display
-/// name) and a window's WM_CLASS / app id.
-fn window_matches_launch(window: &crate::x11::WindowInfo, query: &str) -> bool {
- query
- .split('\u{0}')
- .filter(|key| !key.trim().is_empty())
- .any(|key| window_matches_launch_key(window, key))
-}
-
-fn window_matches_launch_key(window: &crate::x11::WindowInfo, query: &str) -> bool {
- let class = window.app_name.to_ascii_lowercase();
- if class.is_empty() {
- return false;
- }
- let query = query.to_ascii_lowercase();
- let stem = query
- .rsplit('/')
- .next()
- .unwrap_or(&query)
- .trim_end_matches(".desktop")
- .to_owned();
- let last = stem.rsplit('.').next().unwrap_or(&stem).to_owned();
- let class_last = class.rsplit('.').next().unwrap_or(&class).to_owned();
- class == stem
- || class_last == last
- || class.contains(&last)
- || last.contains(&class_last)
- || stem
- .split_whitespace()
- .next()
- .is_some_and(|word| class.contains(word))
-}
-
-/// How long a launch may take to show a window while the launcher process is
-/// still alive (the app is starting; LibreOffice on a small VM needs well over
-/// ten seconds), and once it has exited (a wrapper or D-Bus activation has
-/// taken over, so the window is either imminent or never coming).
-const LAUNCH_STARTING_BUDGET: std::time::Duration = std::time::Duration::from_secs(20);
-const LAUNCH_HANDOFF_BUDGET: std::time::Duration = std::time::Duration::from_secs(8);
-
-/// Resolve the window(s) of a launch. Prefers the launcher's own window;
-/// otherwise adopts a newly mapped top-level whose WM_CLASS matches the
-/// launch, whoever owns it: a wrapper script that exec'd the real binary
-/// (`libreoffice` → `soffice.bin`), or the running D-Bus service GNOME apps
-/// hand their request to.
-fn resolve_launched_windows(
- launcher_pid: u32,
- query: &str,
- before: &std::collections::HashSet<u64>,
-) -> LaunchedWindows {
- let started = std::time::Instant::now();
- loop {
- let all = crate::wayland::list_windows_dispatch(None);
- if let Some(launched) = settle_launch(&all, launcher_pid, query, before, |w| {
- crate::x11::window_is_splash(w.xid)
- }) {
- return launched;
- }
- let launcher_exited = process_exited(launcher_pid);
- let budget = if launcher_exited {
- LAUNCH_HANDOFF_BUDGET
- } else {
- LAUNCH_STARTING_BUDGET
- };
- if started.elapsed() >= budget {
- return LaunchedWindows {
- pid: (!launcher_exited).then_some(launcher_pid),
- windows: Vec::new(),
- handed_off: launcher_exited,
- };
+#[cfg(test)]
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_cancelled_x11_press_is_recorded_before_the_next_call_on_its_cursor() {
+ use cua_driver_core::tool::DesktopCall;
+ let state = ToolState::new();
+ let hold = |xid| MouseHoldState {
+ pid: 1,
+ xid,
+ button: 1,
+ x: 0.0,
+ y: 0.0,
+ };
+ // Call 1: XTest is slow, and its client cancels the call.
+ let call = DesktopCall::acquire().await.unwrap();
+ let first = {
+ let state = state.clone();
+ async move {
+ call.scope(press_and_record_x11_hold(
+ state,
+ "default".to_owned(),
+ hold(1),
+ || {
+ std::thread::sleep(std::time::Duration::from_millis(300));
+ Ok(())
+ },
+ ))
+ .await
}
- std::thread::sleep(std::time::Duration::from_millis(100));
- }
+ };
+ let _ = tokio::time::timeout(std::time::Duration::from_millis(50), first).await;
+ // Call 2 on the same cursor checks for a hold once it has the desktop.
+ let _call = DesktopCall::acquire().await.unwrap();
+ let held = state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .get("default")
+ .map(|hold| hold.xid);
+ assert_eq!(
+ held,
+ Some(1),
+ "call 2 would find no hold, press window 2, and have call 1 overwrite its hold"
+ );
}
-fn settle_launch(
- all: &[crate::x11::WindowInfo],
- launcher_pid: u32,
- query: &str,
- before: &std::collections::HashSet<u64>,
- is_splash: impl Fn(&crate::x11::WindowInfo) -> bool,
-) -> Option<LaunchedWindows> {
- let own: Vec<_> = all
- .iter()
- .filter(|w| w.pid == Some(launcher_pid) && !is_splash(w))
- .cloned()
- .collect();
- if !own.is_empty() {
- return Some(LaunchedWindows {
- pid: Some(launcher_pid),
- windows: own,
- handed_off: false,
- });
- }
- let fresh: Vec<_> = all
- .iter()
- .filter(|w| !before.contains(&w.xid) && w.pid.is_some())
- .filter(|w| window_matches_launch(w, query) && !is_splash(w))
- .cloned()
- .collect();
- let pid = fresh.first()?.pid?;
- Some(LaunchedWindows {
- pid: Some(pid),
- windows: fresh.into_iter().filter(|w| w.pid == Some(pid)).collect(),
- handed_off: true,
- })
+#[cfg(test)]
+#[tokio::test]
+async fn a_late_x11_drag_press_is_followed_by_its_release() {
+ // The X server answers the press only after the call timed out. The
+ // drag then sends its own release, which arrives first.
+ let events = Arc::new(std::sync::Mutex::new(Vec::new()));
+ let pressed = press_x11_drag_start(
+ std::time::Duration::from_millis(50),
+ {
+ let events = events.clone();
+ move || {
+ std::thread::sleep(std::time::Duration::from_millis(200));
+ events.lock().unwrap().push("press");
+ Ok(())
+ }
+ },
+ {
+ let events = events.clone();
+ move || {
+ events.lock().unwrap().push("release");
+ Ok(())
+ }
+ },
+ )
+ .await
+ .unwrap();
+ assert!(pressed.is_err());
+ events.lock().unwrap().push("release");
+ tokio::time::sleep(std::time::Duration::from_millis(400)).await;
+ let events = events.lock().unwrap();
+ assert!(events.contains(&"press"), "{events:?}");
+ assert_eq!(events.last(), Some(&"release"), "{events:?}");
}
#[cfg(test)]
-mod launch_resolution_tests;
+#[test]
+fn drag_steps_stay_within_the_advertised_bounds() {
+ // Without the bound, steps = u32::MAX holds the button through about
+ // 4.3 billion motions.
+ assert_eq!(drag_steps(&json!({ "steps": u32::MAX })), 200);
+ assert_eq!(drag_steps(&json!({ "steps": 0 })), 1);
+ assert_eq!(drag_steps(&json!({ "steps": 20 })), 20);
+}
-fn chromium_family_program(program: &str) -> bool {
- let basename = std::path::Path::new(program)
- .file_name()
- .and_then(|name| name.to_str())
- .unwrap_or(program)
- .to_ascii_lowercase();
- ["chrome", "chromium", "electron", "brave", "edge"]
- .iter()
- .any(|needle| basename.contains(needle))
+/// The press of an X11 step-by-step drag. A press that goes down after its
+/// call ended is released here, in the same worker: the drag's own release
+/// runs in another worker on its own X connection and can reach the app
+/// first, which would leave the app's last event a press.
+pub(crate) async fn press_x11_drag_start(
+ budget: std::time::Duration,
+ press: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
+ release: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
+) -> Result<anyhow::Result<()>, tokio::task::JoinError> {
+ spawn_blocking_bounded("drag", budget, move || {
+ press()?;
+ if crate::input::action::admit(|| ()).is_err() {
+ release()?;
+ }
+ Ok(())
+ })
+ .await
}
-// ── shared helpers ────────────────────────────────────────────────────────────
+/// Press an X11 button and record its hold as soon as it is down, so a caller
+/// that goes away before the await resumes still leaves a hold that
+/// mouse_button_up can release. The worker keeps the call's desktop lease
+/// through both, so the next call on the cursor sees this hold instead of
+/// pressing and recording its own, which this one would then overwrite.
+pub(crate) async fn press_and_record_x11_hold(
+ state: Arc<ToolState>,
+ cursor_id: String,
+ hold: MouseHoldState,
+ press: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
+) -> Result<anyhow::Result<()>, tokio::task::JoinError> {
+ spawn_blocking_bounded("mouse_button_down", foreground_budget(0), move || {
+ press()?;
+ state.mouse_hold.lock().unwrap().insert(cursor_id, hold);
+ Ok(())
+ })
+ .await
+}
-/// Resolve a token element's center in window-local coordinates.
-///
-/// Returns `(xid, window_local_x, window_local_y)`.
-/// Looks up the element's bounds (see [`crate::atspi::token_bounds`]), finds
-/// the owning window (via `xid_hint` or the first window for `pid`), then
-/// converts screen-absolute → window-local coords via compositor metadata on
-/// Wayland or XTranslateCoordinates on X11.
-fn resolve_element_local_coords(
- pid: u32,
- token: &crate::atspi::TokenElement,
- xid_hint: Option<u64>,
-) -> anyhow::Result<(u64, f64, f64)> {
- let (bx, by, bw, bh) = crate::atspi::token_bounds(pid, xid_hint, token)?;
- let screen_cx = bx as f64 + bw as f64 / 2.0;
- let screen_cy = by as f64 + bh as f64 / 2.0;
+/// Upper bound for one foreground input transaction: focus confirmation plus
+/// the injection itself. Typing sleeps per character, so scale with `chars`.
+pub(crate) fn foreground_budget(chars: usize) -> std::time::Duration {
+ std::time::Duration::from_millis(12_000 + (chars as u64) * 40)
+}
- let xid = if let Some(x) = xid_hint {
- x
- } else if crate::wayland::wayland_input_enabled() {
- crate::wayland::list_windows_dispatch(Some(pid))
- .into_iter()
- .next()
- .map(|window| window.xid)
- .ok_or_else(|| anyhow::anyhow!("No Wayland windows for pid {pid}"))?
+/// The budget of a `type_text` call that types `chars` characters. niri's
+/// keyboard gives itself its own deadline, which starts once the window is
+/// focused and stops the call with how far it got; the call allows that
+/// plus the foreground setup, so that deadline ends it first.
+pub(crate) fn typing_budget(chars: usize, niri: bool) -> std::time::Duration {
+ let budget = foreground_budget(chars);
+ if niri {
+ budget.max(foreground_budget(0) + crate::wayland::niri_keyboard_budget(chars))
} else {
- crate::x11::list_windows(Some(pid))
- .into_iter()
- .next()
- .map(|w| w.xid)
- .ok_or_else(|| anyhow::anyhow!("No windows for pid {pid}"))?
- };
-
- if crate::wayland::wayland_input_enabled() {
- let (window_x, window_y, window_width, window_height) =
- crate::wayland::window_geometry(xid)
- .ok_or_else(|| anyhow::anyhow!("No Wayland geometry for window {xid}"))?;
- if window_width == 0 || window_height == 0 {
- anyhow::bail!("Wayland window {xid} has no usable geometry");
- }
- return Ok((
- xid,
- screen_cx - window_x as f64,
- screen_cy - window_y as f64,
- ));
+ budget
}
+}
- use x11rb::connection::Connection;
- use x11rb::protocol::xproto::ConnectionExt as _;
- use x11rb::rust_connection::RustConnection;
- let (conn, screen_num) = RustConnection::connect(None)?;
- let root = conn.setup().roots[screen_num].root;
- let reply = conn
- .translate_coordinates(xid as u32, root, 0, 0)?
- .reply()?;
- let local_x = screen_cx - reply.dst_x as f64;
- let local_y = screen_cy - reply.dst_y as f64;
- Ok((xid, local_x, local_y))
+#[cfg(test)]
+#[test]
+fn niri_typing_outlasts_the_keyboard_deadline() {
+ // niri's keyboard allows itself a deadline per key and reports how far
+ // it got when that passes. A call budget that ran out first would cut
+ // the text off with a bare timeout, and a retry would type its start
+ // again.
+ for chars in [0, 1, 150, 1_000, 100_000] {
+ assert!(
+ typing_budget(chars, true) > crate::wayland::niri_keyboard_budget(chars),
+ "{chars} characters"
+ );
+ assert_eq!(typing_budget(chars, false), foreground_budget(chars));
+ }
}
-/// An element whose AT-SPI extents do not fall inside its own toplevel: GTK4
-/// reports empty extents for some widgets (a text view centre resolves to
-/// (0,0), the GNOME hot corner), and a real pointer click there would hit
-/// whatever is under that screen point. Refused before any input is sent.
-#[derive(Debug)]
-struct ElementBoundsUnusable {
- idx: usize,
- local: (f64, f64),
- window: (u32, u32),
+/// [`foreground_budget`] for `chars` characters with `delay_ms` between them.
+/// The delay comes from the caller unbounded.
+pub(crate) fn paced_typing_budget(chars: usize, delay_ms: u64) -> std::time::Duration {
+ foreground_budget(chars)
+ + std::time::Duration::from_millis(delay_ms.saturating_mul(chars as u64))
}
-impl std::fmt::Display for ElementBoundsUnusable {
- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- write!(
- f,
- "element [{}] resolves to window-local ({:.0}, {:.0}), outside its {}x{} window; \
- the toolkit reported unusable AT-SPI extents, so no pointer input was sent",
- self.idx, self.local.0, self.local.1, self.window.0, self.window.1
- )
- }
+#[cfg(test)]
+#[test]
+fn any_typing_delay_gives_a_budget() {
+ assert_eq!(
+ paced_typing_budget(2, 30),
+ foreground_budget(2) + std::time::Duration::from_millis(60)
+ );
+ // Sizing the call must not panic, however long the caller's delay.
+ let longest = paced_typing_budget(3, u64::MAX);
+ assert!(longest >= std::time::Duration::from_millis(u64::MAX));
+ crate::input::action::Action::new(longest);
}
-impl std::error::Error for ElementBoundsUnusable {}
-/// `resolve_element_local_coords` plus the sanity check that the centre lies
-/// inside the window it belongs to. Blocking.
-fn checked_element_local_coords(
- pid: u32,
- token: &crate::atspi::TokenElement,
- xid_hint: Option<u64>,
-) -> anyhow::Result<(u64, f64, f64)> {
- let (xid, lx, ly) = resolve_element_local_coords(pid, token, xid_hint)?;
- if crate::wayland::wayland_input_enabled() {
- return Ok((xid, lx, ly));
- }
- use x11rb::protocol::xproto::ConnectionExt as _;
- use x11rb::rust_connection::RustConnection;
- let (conn, _) = RustConnection::connect(None)?;
- let geom = conn.get_geometry(xid as u32)?.reply()?;
- let (w, h) = (u32::from(geom.width), u32::from(geom.height));
- if lx < 0.0 || ly < 0.0 || lx >= f64::from(w) || ly >= f64::from(h) {
- return Err(ElementBoundsUnusable {
- idx: token.index,
- local: (lx, ly),
- window: (w, h),
+/// Map an input-path error to a tool error, attaching the structured
+/// foreground `code` (`foreground_unavailable` / `foreground_timeout`) when the
+/// message carries one so the caller can branch instead of parsing prose.
+pub(crate) fn input_error_result(e: anyhow::Error) -> ToolResult {
+ let text = e.to_string();
+ if crate::input::is_uinput_unavailable(&e) {
+ // The focus-free real-input route exists but this process cannot open
+ // /dev/uinput: an honest refusal the operator can act on, not a bare
+ // I/O error.
+ let mut refusal = with_uinput_hint(crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ ));
+ if let Some(structured) = refusal.structured_content.as_mut() {
+ structured["cause"] = json!(crate::input::UINPUT_UNAVAILABLE_CODE);
+ structured["detail"] = json!(text);
}
- .into());
+ return refusal;
+ }
+ match crate::input::foreground::error_code(&e) {
+ Some(code) => ToolResult::error(text.clone()).with_structured(json!({
+ "code": code,
+ "detail": text,
+ "effect": "none",
+ "suggestion": if code == crate::input::foreground::CODE_TIMEOUT {
+ "Take a screenshot (get_desktop_state) to confirm the desktop is responsive, then retry once."
+ } else {
+ crate::wayland::niri::refusal_suggestion(&text).unwrap_or(
+ "Call bring_to_front on the window (or its modal dialog), confirm with get_window_state, then retry.",
+ )
+ },
+ })),
+ None => linux_input_error(e),
}
- Ok((xid, lx, ly))
}
-fn element_screen_center(
- pid: u32,
- token: &crate::atspi::TokenElement,
- xid: Option<u64>,
-) -> anyhow::Result<(f64, f64)> {
- element_screen_center_and_rect(pid, token, xid).map(|(center, _)| center)
+#[cfg(test)]
+#[test]
+fn a_niri_refusal_suggests_its_own_remedy_instead_of_bring_to_front() {
+ let refused = input_error_result(anyhow::anyhow!(
+ "foreground_unavailable: the niri layout changed after the get_desktop_state screenshot, so its pixels may be elsewhere now; take a new screenshot"
+ ));
+ let structured = refused.structured_content.unwrap();
+ assert_eq!(structured["code"], "foreground_unavailable");
+ assert!(structured["suggestion"]
+ .as_str()
+ .unwrap()
+ .starts_with("Take a new screenshot"));
}
-/// The element's screen centre plus its AT-SPI extents as the overlay's
-/// `[x, y, width, height]`, both from one bounds lookup (same screen space).
-fn element_screen_center_and_rect(
- pid: u32,
- token: &crate::atspi::TokenElement,
- xid: Option<u64>,
-) -> anyhow::Result<((f64, f64), [f64; 4])> {
- let (bx, by, bw, bh) = crate::atspi::token_bounds(pid, xid, token)?;
- let rect = [bx as f64, by as f64, bw as f64, bh as f64];
- Ok(((rect[0] + rect[2] / 2.0, rect[1] + rect[3] / 2.0), rect))
+/// Structured payload for a completed foreground transaction.
+pub(crate) fn foreground_structured(
+ path: &str,
+ report: crate::input::ForegroundReport,
+ extra: serde_json::Map<String, Value>,
+) -> Value {
+ let mut v = json!({
+ "path": path,
+ "verified": false,
+ "delivery_mode": "foreground",
+ "effect": "unverifiable",
+ });
+ apply_foreground_report(&mut v, &report);
+ for (k, val) in extra {
+ v[k] = val;
+ }
+ v
}
-/// A snapshot frame `(x, y, w, h)` as the overlay target rect, only when it
-/// is non-empty and actually contains the glide point (a redirected press or
-/// a frame in another space gets no rect rather than a wrong one).
-fn frame_target_rect(bounds: Option<(i32, i32, u32, u32)>, sx: f64, sy: f64) -> Option<[f64; 4]> {
- let (x, y, w, h) = bounds?;
- let rect = [f64::from(x), f64::from(y), f64::from(w), f64::from(h)];
- (w > 0
- && h > 0
- && sx >= rect[0]
- && sy >= rect[1]
- && sx <= rect[0] + rect[2]
- && sy <= rect[1] + rect[3])
- .then_some(rect)
+/// Fold a foreground transaction's post-check into a result: the report
+/// itself, `focus_after` at the top level, and an `evidence` item so the
+/// observation survives the public-record reduction. A window of the target
+/// process that appeared or closed is `window_change` evidence and an
+/// `effect: confirmed` wherever the focus sits afterwards (the OK button of a
+/// dialog destroys the very window that held the focus, and closing the last
+/// window of a process leaves the focus to the window manager); a change
+/// among unattributed popups only counts while the focus stayed in the
+/// process (`ForegroundReport::window_change_confirms`). Otherwise, focus that
+/// left the process is a suspected no-op and anything else stays
+/// unverifiable with a `native_api_result` item.
+pub(crate) fn apply_foreground_report(v: &mut Value, report: &crate::input::ForegroundReport) {
+ v["foreground"] = report.to_json();
+ v["focus_after"] = json!(report.focus_after.as_str());
+ let confirming_change = report
+ .window_change
+ .as_ref()
+ .filter(|_| report.window_change_confirms());
+ let evidence = match confirming_change {
+ Some(change) => {
+ v["effect"] = json!("confirmed");
+ v["verified"] = json!(true);
+ json!({
+ "kind": "window_change",
+ "detail": format!("focus_after={}; {change}", report.focus_after.as_str()),
+ })
+ }
+ None if report.focus_after == crate::input::FocusAfter::Elsewhere => {
+ v["effect"] = json!("suspected_noop");
+ v["warning"] = json!(
+ "input focus left the target process before the post-check; the input \
+ may have reached another window. Verify with a screenshot."
+ );
+ return;
+ }
+ None => json!({
+ "kind": "native_api_result",
+ "detail": format!(
+ "real input delivered to the activated window (focus_after={}); {}",
+ report.focus_after.as_str(),
+ match &report.window_change {
+ Some(change) => format!("{change}, not attributed to the target process"),
+ None => "no window change observed".to_owned(),
+ }
+ ),
+ }),
+ };
+ let mut items = v
+ .get("evidence")
+ .and_then(Value::as_array)
+ .cloned()
+ .unwrap_or_default();
+ items.push(evidence);
+ v["evidence"] = json!(items);
}
-/// Shared schema for the optional `coordinate_frame` of pointer tools.
-fn coordinate_frame_schema() -> Value {
- json!({
- "type": "string",
- "enum": ["window", "desktop"],
- "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": \
- window-local screenshot pixels as returned by get_window_state. \"desktop\": \
- full-screen pixels as returned by get_desktop_state, translated to the target \
- window. Passing scope:\"desktop\" together with pid/window_id means the same thing."
- })
-}
-
-fn coordinate_frame_is_desktop(args: &Value) -> bool {
- args.get("coordinate_frame").and_then(Value::as_str) == Some("desktop")
-}
-
-fn window_scope_unproven_refusal(tool: &str, xid: u64, reason: &str) -> ToolResult {
- let message = format!(
- "{tool}: the element's membership in window_id {xid} is unproven ({reason}); \
- call get_window_state again"
- );
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "refusal": { "code": "window_scope_unproven", "message": message }
+/// Typed refusal for a pointer tool whose arguments name neither an element
+/// nor a complete point. Before this check a call with only a pid was
+/// executed at window-local (0, 0) — the top-left corner, i.e. the File menu.
+pub(crate) fn invalid_pointer_arguments(tool: &str, detail: &str) -> ToolResult {
+ let accepted = [
+ "pid + element_token from get_window_state",
+ "pid/window_id + x + y (window-local pixels of window_id; coordinate_frame:\"desktop\" for screen pixels)",
+ "x + y + scope:\"desktop\" (screen pixels, no pid)",
+ ];
+ ToolResult::error(format!(
+ "{tool}: invalid arguments: {detail}. Accepted forms: {}. No input was sent.",
+ accepted.join("; ")
+ ))
+ .with_structured(json!({
+ "code": "invalid_arguments",
+ "tool": tool,
+ "effect": "refused",
+ "detail": detail,
+ "accepted_forms": accepted,
}))
}
-/// Prove from a fresh walk that the token's observed element is still in the
-/// token's window, then carry that element to dispatch. `Ok(None)` without a
-/// token. Refuses before any dispatch. X11, Hyprland and compositors without
-/// a GNOME Shell need no proof.
-async fn prove_token_element(
- pid: u32,
- resolved: &cua_driver_core::element_token::ResolvedElement<
- crate::atspi::snapshot::CachedElement,
- >,
- tool: &str,
-) -> Result<Option<crate::atspi::TokenElement>, ToolResult> {
- let cua_driver_core::element_token::ResolvedElement::Element {
- window_id,
- element_index,
- element,
- } = resolved
- else {
- return Ok(None);
- };
- let (xid, identity) = (*window_id, element.identity.clone());
- let proof = tokio::task::spawn_blocking(move || {
- crate::atspi::require_window_element(
- pid,
- xid,
- crate::atspi::WindowElement::Identity(identity),
- )
- })
- .await
- .unwrap_or_else(|error| Err(anyhow::anyhow!("task error: {error}")));
- token_element_for(tool, xid, *element_index, element.clone(), proof).map(Some)
-}
-
-fn token_element_for(
- tool: &str,
- xid: u64,
- index: usize,
- observed: crate::atspi::snapshot::CachedElement,
- proof: anyhow::Result<crate::atspi::WindowProof>,
-) -> Result<crate::atspi::TokenElement, ToolResult> {
- proof
- .map(|proof| crate::atspi::TokenElement::new(index, observed, proof))
- .map_err(|error| window_scope_unproven_refusal(tool, xid, &error.to_string()))
-}
-
-/// The object an element token observed no longer answers: the token is
-/// stale. Nothing reached that object, and it is never retargeted by index
-/// after a window proof.
-fn stale_observed_element(tool: &str, idx: usize, error: &anyhow::Error) -> Option<ToolResult> {
- error
- .is::<crate::atspi::native::CachedElementGone>()
- .then(|| {
- ToolResult::error(format!(
- "{tool}: stale_element_token: observed AT-SPI object [{idx}] is no longer \
- present ({error}); re-snapshot with get_window_state"
- ))
- .with_structured(json!({ "code": "stale_element_token", "effect": "none" }))
- })
-}
-
-/// The cursor glides to an element before its click. A failed lookup skips
-/// only the glide, except a window-membership failure, which refuses.
-fn element_click_placement(
- placement: anyhow::Result<(u64, f64, f64)>,
- xid: Option<u64>,
-) -> Result<Option<(u64, f64, f64)>, ToolResult> {
- match placement {
- Ok(point) => Ok(Some(point)),
- Err(error) if crate::atspi::is_window_membership_unproven(&error) => Err(
- window_scope_unproven_refusal("click", xid.unwrap_or(0), &error.to_string()),
- ),
- Err(_) => Ok(None),
+/// `Some(refusal)` when a pointer tool reached its coordinate path without
+/// both `x` and `y` (and, by construction, without an element).
+pub(crate) fn require_point_args(tool: &str, args: &Value) -> Option<ToolResult> {
+ let present = |name: &str| args.get(name).is_some_and(|v| v.is_number());
+ match (present("x"), present("y")) {
+ (true, true) => None,
+ (false, false) => Some(invalid_pointer_arguments(
+ tool,
+ "no element_token and no x/y were given (a `target_id` is not an \
+ element handle)",
+ )),
+ (true, false) => Some(invalid_pointer_arguments(tool, "x was given without y")),
+ (false, true) => Some(invalid_pointer_arguments(tool, "y was given without x")),
}
}
#[cfg(test)]
-mod token_window_scope_tests;
-
-fn window_local_to_screen(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {
- use x11rb::connection::Connection;
- use x11rb::protocol::xproto::ConnectionExt as _;
- use x11rb::rust_connection::RustConnection;
-
- let (conn, screen_num) = RustConnection::connect(None)?;
- let root = conn.setup().roots[screen_num].root;
- let reply = conn
- .translate_coordinates(xid as u32, root, 0, 0)?
- .reply()?;
- Ok((reply.dst_x as f64 + x, reply.dst_y as f64 + y))
-}
-
-/// `scope:"desktop"` alongside a pid/window target: the caller read `x,y` off
-/// `get_desktop_state` (screen pixels) but still names the window to act on.
-/// Upper bound for one AT-SPI element operation (bounds lookup / action).
-/// The AT-SPI layer bounds single bus calls, but a full tree walk over a large
-/// GTK/VCL window chains many of them; observed stalls ran to minutes.
-const ELEMENT_AX_BUDGET: std::time::Duration = std::time::Duration::from_secs(30);
-
-/// Either `coordinate_frame:"desktop"` or `scope:"desktop"` marks `x,y` as
-/// desktop (screen) pixels; both are translated exactly once by the caller.
-fn desktop_frame_requested(args: &Value) -> bool {
- coordinate_frame_is_desktop(args)
- || args.get("scope").and_then(Value::as_str) == Some("desktop")
-}
+mod pointer_argument_tests {
+ use super::*;
-/// Convert desktop (screen) coordinates to `xid`'s window-local frame so the
-/// rest of the pointer pipeline (which is window-local by contract) is reused.
-fn desktop_to_window_local(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {
- // A niri window id is no X11 window: never ask XWayland about it.
- if crate::wayland::is_wayland() && crate::wayland::niri::is_session() {
- return Err(crate::wayland::niri::desktop_frame_window_refusal(xid));
+ #[test]
+ fn pid_only_pointer_call_is_refused_not_clicked_at_origin() {
+ let refusal = require_point_args("click", &json!({"target_id": "s1:147", "pid": 1867}))
+ .expect("refused");
+ assert_eq!(refusal.is_error, Some(true));
+ let structured = refusal.structured_content.expect("structured");
+ assert_eq!(structured["code"], "invalid_arguments");
+ assert_eq!(structured["effect"], "refused");
+ assert!(structured["accepted_forms"].as_array().unwrap().len() == 3);
}
- let (ox, oy) = window_local_to_screen(xid, 0.0, 0.0)?;
- Ok((x - ox, y - oy))
-}
-/// A failed desktop-to-window mapping; a coded refusal keeps its structure.
-fn desktop_frame_unmapped(xid: u64, error: anyhow::Error) -> ToolResult {
- if crate::input::foreground::error_code(&error).is_some() {
- return input_error_result(error);
+ #[test]
+ fn complete_point_passes() {
+ assert!(require_point_args(
+ "click",
+ &json!({"pid": 1, "window_id": 2, "x": 3, "y": 4.5})
+ )
+ .is_none());
+ assert!(require_point_args("click", &json!({"x": 1, "y": null})).is_some());
+ assert!(require_point_args("click", &json!({"y": 1})).is_some());
}
- ToolResult::error(format!(
- "desktop-frame coordinates could not be mapped into window {xid}: {error}"
- ))
-}
-#[cfg(test)]
-#[test]
-fn a_niri_desktop_frame_window_refusal_is_structured() {
- let refused =
- desktop_frame_unmapped(31, crate::wayland::niri::desktop_frame_window_refusal(31));
- let structured = refused.structured_content.unwrap();
- assert_eq!(structured["code"], "foreground_unavailable");
- assert_eq!(structured["effect"], "none");
- assert!(structured["suggestion"]
- .as_str()
- .unwrap()
- .starts_with("Drop pid and window_id"));
+ #[test]
+ fn foreground_report_becomes_evidence() {
+ let report = crate::input::ForegroundReport {
+ already_active: false,
+ retried_activation: false,
+ confirm_ms: 12,
+ focus_after: crate::input::FocusAfter::SamePid,
+ window_change: Some("appeared: window 9 \"Brightness-Contrast\"".into()),
+ window_change_owned: true,
+ };
+ let v = foreground_structured("x11_xtest_fg", report, serde_json::Map::new());
+ assert_eq!(v["effect"], "confirmed");
+ assert_eq!(v["focus_after"], "same_pid");
+ assert_eq!(v["evidence"][0]["kind"], "window_change");
+ let quiet = crate::input::ForegroundReport {
+ already_active: true,
+ retried_activation: false,
+ confirm_ms: 0,
+ focus_after: crate::input::FocusAfter::Target,
+ window_change: None,
+ window_change_owned: false,
+ };
+ let v = foreground_structured("x11_xtest_fg", quiet, serde_json::Map::new());
+ assert_eq!(v["effect"], "unverifiable");
+ assert_eq!(v["evidence"][0]["kind"], "native_api_result");
+ let closed = crate::input::ForegroundReport {
+ already_active: true,
+ retried_activation: false,
+ confirm_ms: 0,
+ focus_after: crate::input::FocusAfter::Unknown,
+ window_change: Some("closed: window 3 \"Format Cells\"".into()),
+ window_change_owned: true,
+ };
+ let v = foreground_structured("x11_xtest_fg", closed, serde_json::Map::new());
+ assert_eq!(v["effect"], "confirmed");
+ assert_eq!(v["evidence"][0]["kind"], "window_change");
+ // A popup that names no pid is listed in the target's window set (a VCL
+ // menu carries none), but it may be another client's: once the focus
+ // left the target process its change is no confirmation.
+ let unattributed = |focus_after| crate::input::ForegroundReport {
+ already_active: true,
+ retried_activation: false,
+ confirm_ms: 0,
+ focus_after,
+ window_change: Some("appeared: popup window 7 (200x300 at 1,2)".into()),
+ window_change_owned: false,
+ };
+ let v = foreground_structured(
+ "x11_xtest_fg",
+ unattributed(crate::input::FocusAfter::Elsewhere),
+ serde_json::Map::new(),
+ );
+ assert_eq!(v["effect"], "suspected_noop");
+ let v = foreground_structured(
+ "x11_xtest_fg",
+ unattributed(crate::input::FocusAfter::Unknown),
+ serde_json::Map::new(),
+ );
+ assert_eq!(v["effect"], "unverifiable");
+ assert_eq!(v["evidence"][0]["kind"], "native_api_result");
+ let v = foreground_structured(
+ "x11_xtest_fg",
+ unattributed(crate::input::FocusAfter::SamePid),
+ serde_json::Map::new(),
+ );
+ assert_eq!(v["effect"], "confirmed");
+ let lost = crate::input::ForegroundReport {
+ already_active: true,
+ retried_activation: false,
+ confirm_ms: 0,
+ focus_after: crate::input::FocusAfter::Elsewhere,
+ window_change: None,
+ window_change_owned: false,
+ };
+ let v = foreground_structured("x11_xtest_fg", lost, serde_json::Map::new());
+ assert_eq!(v["effect"], "suspected_noop");
+ }
}
-/// Upper bound for one foreground input transaction: focus confirmation plus
-/// the injection itself. Typing sleeps per character, so scale with `chars`.
-fn foreground_budget(chars: usize) -> std::time::Duration {
- std::time::Duration::from_millis(12_000 + (chars as u64) * 40)
+pub(crate) fn parse_mouse_button(name: &str) -> u8 {
+ match name {
+ "right" => 3,
+ "middle" => 2,
+ _ => 1,
+ }
}
-/// The budget of a `type_text` call that types `chars` characters. niri's
-/// keyboard gives itself its own deadline, which starts once the window is
-/// focused and stops the call with how far it got; the call allows that
-/// plus the foreground setup, so that deadline ends it first.
-fn typing_budget(chars: usize, niri: bool) -> std::time::Duration {
- let budget = foreground_budget(chars);
- if niri {
- budget.max(foreground_budget(0) + crate::wayland::niri_keyboard_budget(chars))
+/// Escalation hint for a non-AX / suspected-no-op surface — the cross-platform
+/// `escalation` field mirrored from macOS. The recommended next rung depends on
+/// the session type: X11 can pixel-target a specific window in the background,
+/// so the deliberate move is an element px action — click by pixel (x,y) off the
+/// screenshot already in this response (`px`). Native Wayland CANNOT
+/// background-target an unfocused window (libei injects to the compositor's input
+/// focus — see `input::delivery`), and the pixel path itself routes through the
+/// same coordinate-free AT-SPI actuation that just no-op'd, so the move there is
+/// to bring the window to the foreground first (`foreground`).
+pub(crate) fn non_ax_escalation() -> Value {
+ if crate::wayland::is_wayland() {
+ json!({
+ "recommended": "foreground",
+ "reason": "non-AX surface on Wayland: a specific unfocused window can't be \
+ pixel-targeted in the background (libei injects to the compositor's \
+ focus). bring_to_front, then click by pixel off the screenshot \
+ with delivery_mode:\"foreground\"."
+ })
} else {
- budget
+ json!({
+ "recommended": "px",
+ "reason": "non-AX surface — act by pixel (x,y) off the screenshot \
+ in this response (an element px action)."
+ })
}
}
-#[cfg(test)]
-#[test]
-fn niri_typing_outlasts_the_keyboard_deadline() {
- // niri's keyboard allows itself a deadline per key and reports how far
- // it got when that passes. A call budget that ran out first would cut
- // the text off with a bare timeout, and a retry would type its start
- // again.
- for chars in [0, 1, 150, 1_000, 100_000] {
- assert!(
- typing_budget(chars, true) > crate::wayland::niri_keyboard_budget(chars),
- "{chars} characters"
- );
- assert_eq!(typing_budget(chars, false), foreground_budget(chars));
+/// True when `pid` is a Chromium-based embedder — a Chrome/Chromium browser or
+/// any Electron/CEF app. On these surfaces an AT-SPI `EditableText.insertText`
+/// can succeed at the bridge while the Chromium *renderer* never observes it,
+/// so the AT-SPI "ax" rung must not be trusted as a confirmed insert (see
+/// [`type_text_ax_result`]).
+///
+/// This is the Linux analogue of macOS `ElectronJs::is_electron` (which checks
+/// for a bundled Electron Framework). Linux has no bundle, so the signal is
+/// Chromium's multiprocess fingerprint: the embedder forks sandboxed helpers
+/// whose argv carries `--type=renderer` / `--type=zygote` / `--type=gpu-process`.
+/// Native GTK/Qt apps never spawn such helpers, so this is conservative (very
+/// low false-positive). The single-process fallback also matches the embedder's
+/// own argv. Reads `/proc`; cheap and only invoked on the rare AT-SPI confirm.
+pub(crate) fn is_chromium_embedder(pid: u32) -> bool {
+ fn argv_is_chromium_helper(p: u32) -> bool {
+ fs::read(format!("/proc/{p}/cmdline"))
+ .map(|raw| crate::at_point_policy::cmdline_is_chromium_helper(&raw))
+ .unwrap_or(false)
}
-}
-
-/// [`foreground_budget`] for `chars` characters with `delay_ms` between them.
-/// The delay comes from the caller unbounded.
-fn paced_typing_budget(chars: usize, delay_ms: u64) -> std::time::Duration {
- foreground_budget(chars)
- + std::time::Duration::from_millis(delay_ms.saturating_mul(chars as u64))
-}
-
-#[cfg(test)]
-#[test]
-fn any_typing_delay_gives_a_budget() {
- assert_eq!(
- paced_typing_budget(2, 30),
- foreground_budget(2) + std::time::Duration::from_millis(60)
- );
- // Sizing the call must not panic, however long the caller's delay.
- let longest = paced_typing_budget(3, u64::MAX);
- assert!(longest >= std::time::Duration::from_millis(u64::MAX));
- crate::input::action::Action::new(longest);
-}
-
-use crate::input::action::spawn_blocking_bounded;
-
-#[cfg(test)]
-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn a_timed_out_call_holds_the_desktop_until_its_late_cleanup_is_done() {
- use cua_driver_core::tool::DesktopCall;
- // niri: call 1 focused its window, then waited in niri past its budget.
- // Its worker gives focus back afterwards. Desktop typing that started in
- // between would have focus moved away under it.
- let events = Arc::new(std::sync::Mutex::new(Vec::new()));
- let call_1 = {
- let events = events.clone();
- DesktopCall::acquire()
- .await
- .unwrap()
- .scope(spawn_blocking_bounded(
- "call 1",
- std::time::Duration::from_millis(50),
- move || {
- std::thread::sleep(std::time::Duration::from_millis(300));
- // The next key is refused, but focus still goes back.
- assert!(crate::input::action::admit(|| ()).is_err());
- events.lock().unwrap().push("call 1 gave focus back");
- Ok(())
- },
- ))
- .await
- .unwrap()
+ // Single-process / the embedder itself carrying a Chromium switch.
+ if argv_is_chromium_helper(pid) {
+ return true;
+ }
+ // Build PPid → children across /proc, then BFS the descendants of `pid`
+ // looking for a Chromium helper. Same /proc-walk shape as
+ // `terminal_descendant_ttys`.
+ let mut children: std::collections::HashMap<u32, Vec<u32>> = std::collections::HashMap::new();
+ let entries = match fs::read_dir("/proc") {
+ Ok(e) => e,
+ Err(_) => return false,
};
- assert!(call_1.is_err());
- DesktopCall::acquire()
- .await
- .unwrap()
- .scope(async { events.lock().unwrap().push("call 2 typed") })
- .await;
- assert_eq!(
- *events.lock().unwrap(),
- ["call 1 gave focus back", "call 2 typed"]
- );
-}
-
-#[cfg(test)]
-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn input_that_cannot_be_cancelled_never_overlaps_the_next_desktop_action() {
- use cua_driver_core::tool::DesktopCall;
- // A wtype child does not read the call's action, and an abandoned XTest
- // call still sends its releases: a worker whose call timed out, or whose
- // caller went away, can still send input. That input must not reach
- // whatever the next desktop action targets.
- for budget in [50, 60_000] {
- let events = Arc::new(std::sync::Mutex::new(Vec::new()));
- let worker_events = events.clone();
- let call = DesktopCall::acquire().await.unwrap();
- let call_1 = async move {
- call.scope(spawn_blocking_bounded(
- "call 1",
- std::time::Duration::from_millis(budget),
- move || {
- std::thread::sleep(std::time::Duration::from_millis(300));
- worker_events.lock().unwrap().push("call 1 typed");
- Ok(())
- },
- ))
- .await
+ for entry in entries.flatten() {
+ let name = entry.file_name();
+ let child: u32 = match name.to_string_lossy().parse() {
+ Ok(p) => p,
+ Err(_) => continue,
};
- // A 50 ms budget times out; with 60 s the caller is dropped first.
- let _ = tokio::time::timeout(std::time::Duration::from_millis(100), call_1).await;
- DesktopCall::acquire()
- .await
- .unwrap()
- .scope(async { events.lock().unwrap().push("call 2 typed") })
- .await;
- assert_eq!(
- *events.lock().unwrap(),
- ["call 1 typed", "call 2 typed"],
- "budget {budget} ms"
- );
+ let status = match fs::read_to_string(format!("/proc/{child}/status")) {
+ Ok(s) => s,
+ Err(_) => continue,
+ };
+ if let Some(ppid) = status
+ .lines()
+ .find(|l| l.starts_with("PPid:"))
+ .and_then(|l| l[5..].trim().parse::<u32>().ok())
+ {
+ children.entry(ppid).or_default().push(child);
+ }
+ }
+ let mut queue = std::collections::VecDeque::from([pid]);
+ let mut seen = std::collections::HashSet::new();
+ while let Some(cur) = queue.pop_front() {
+ if !seen.insert(cur) {
+ continue;
+ }
+ if let Some(kids) = children.get(&cur) {
+ for &kid in kids {
+ if argv_is_chromium_helper(kid) {
+ return true;
+ }
+ queue.push_back(kid);
+ }
+ }
}
+ false
}
-#[cfg(test)]
-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn a_cancelled_x11_press_is_recorded_before_the_next_call_on_its_cursor() {
- use cua_driver_core::tool::DesktopCall;
- let state = ToolState::new();
- let hold = |xid| MouseHoldState {
- pid: 1,
- xid,
- button: 1,
- x: 0.0,
- y: 0.0,
+pub(crate) fn is_webkitgtk_embedder(pid: u32) -> bool {
+ fn argv_is_webkit_helper(pid: u32) -> bool {
+ fs::read(format!("/proc/{pid}/cmdline"))
+ .map(|raw| {
+ let cmdline = String::from_utf8_lossy(&raw);
+ cmdline.contains("WebKitWebProcess") || cmdline.contains("WebKitNetworkProcess")
+ })
+ .unwrap_or(false)
+ }
+
+ let mut children: std::collections::HashMap<u32, Vec<u32>> = std::collections::HashMap::new();
+ let Ok(entries) = fs::read_dir("/proc") else {
+ return false;
};
- // Call 1: XTest is slow, and its client cancels the call.
- let call = DesktopCall::acquire().await.unwrap();
- let first = {
- let state = state.clone();
- async move {
- call.scope(press_and_record_x11_hold(
- state,
- "default".to_owned(),
- hold(1),
- || {
- std::thread::sleep(std::time::Duration::from_millis(300));
- Ok(())
- },
- ))
- .await
+ for entry in entries.flatten() {
+ let Ok(child) = entry.file_name().to_string_lossy().parse::<u32>() else {
+ continue;
+ };
+ let Ok(status) = fs::read_to_string(format!("/proc/{child}/status")) else {
+ continue;
+ };
+ if let Some(ppid) = status
+ .lines()
+ .find(|line| line.starts_with("PPid:"))
+ .and_then(|line| line[5..].trim().parse::<u32>().ok())
+ {
+ children.entry(ppid).or_default().push(child);
}
- };
- let _ = tokio::time::timeout(std::time::Duration::from_millis(50), first).await;
- // Call 2 on the same cursor checks for a hold once it has the desktop.
- let _call = DesktopCall::acquire().await.unwrap();
- let held = state
- .mouse_hold
- .lock()
- .unwrap()
- .get("default")
- .map(|hold| hold.xid);
- assert_eq!(
- held,
- Some(1),
- "call 2 would find no hold, press window 2, and have call 1 overwrite its hold"
- );
+ }
+ let mut queue = std::collections::VecDeque::from([pid]);
+ let mut seen = std::collections::HashSet::new();
+ while let Some(current) = queue.pop_front() {
+ if !seen.insert(current) {
+ continue;
+ }
+ if argv_is_webkit_helper(current) {
+ return true;
+ }
+ if let Some(descendants) = children.get(&current) {
+ queue.extend(descendants.iter().copied());
+ }
+ }
+ false
}
-#[cfg(test)]
-#[tokio::test]
-async fn a_late_x11_drag_press_is_followed_by_its_release() {
- // The X server answers the press only after the call timed out. The
- // drag then sends its own release, which arrives first.
- let events = Arc::new(std::sync::Mutex::new(Vec::new()));
- let pressed = press_x11_drag_start(
- std::time::Duration::from_millis(50),
- {
- let events = events.clone();
- move || {
- std::thread::sleep(std::time::Duration::from_millis(200));
- events.lock().unwrap().push("press");
- Ok(())
- }
- },
- {
- let events = events.clone();
- move || {
- events.lock().unwrap().push("release");
- Ok(())
- }
- },
- )
- .await
- .unwrap();
- assert!(pressed.is_err());
- events.lock().unwrap().push("release");
- tokio::time::sleep(std::time::Duration::from_millis(400)).await;
- let events = events.lock().unwrap();
- assert!(events.contains(&"press"), "{events:?}");
- assert_eq!(events.last(), Some(&"release"), "{events:?}");
+pub(crate) fn maps_indicate_gtk(maps: &str) -> bool {
+ maps.contains("libgtk-3.so") || maps.contains("libgtk-4.so")
}
-/// The `steps` of a `drag` or `mouse_drag`, held to the advertised 1..=200
-/// before the button goes down: every step is a motion sent with the
-/// button held.
-fn drag_steps(args: &Value) -> usize {
- args.u64_or("steps", 20).clamp(1, 200) as usize
+/// Toolkits known to discard synthetic (`send_event`) X11 pointer events:
+/// GTK3/4 (XInput2 only), LibreOffice VCL (all plugins), Qt5/6 (xcb, XI2).
+pub(crate) fn maps_indicate_synthetic_pointer_dropped(maps: &str) -> bool {
+ maps_indicate_gtk(maps)
+ || maps.contains("libvcl")
+ || maps.contains("libmergedlo")
+ || maps.contains("libQt5Gui")
+ || maps.contains("libQt6Gui")
}
-#[cfg(test)]
-#[test]
-fn drag_steps_stay_within_the_advertised_bounds() {
- // Without the bound, steps = u32::MAX holds the button through about
- // 4.3 billion motions.
- assert_eq!(drag_steps(&json!({ "steps": u32::MAX })), 200);
- assert_eq!(drag_steps(&json!({ "steps": 0 })), 1);
- assert_eq!(drag_steps(&json!({})), 20);
+pub(crate) fn synthetic_pointer_is_dropped(pid: u32) -> bool {
+ fs::read_to_string(format!("/proc/{pid}/maps"))
+ .map(|maps| maps_indicate_synthetic_pointer_dropped(&maps))
+ .unwrap_or(false)
}
-/// Map an input-path error to a tool error, attaching the structured
-/// foreground `code` (`foreground_unavailable` / `foreground_timeout`) when the
-/// message carries one so the caller can branch instead of parsing prose.
-fn input_error_result(e: anyhow::Error) -> ToolResult {
- let text = e.to_string();
- if crate::input::is_uinput_unavailable(&e) {
- // The focus-free real-input route exists but this process cannot open
- // /dev/uinput: an honest refusal the operator can act on, not a bare
- // I/O error.
- let mut refusal = with_uinput_hint(crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- ));
- if let Some(structured) = refusal.structured_content.as_mut() {
- structured["cause"] = json!(crate::input::UINPUT_UNAVAILABLE_CODE);
- structured["detail"] = json!(text);
- }
- return refusal;
- }
- match crate::input::foreground::error_code(&e) {
- Some(code) => ToolResult::error(text.clone()).with_structured(json!({
- "code": code,
- "detail": text,
- "effect": "none",
- "suggestion": if code == crate::input::foreground::CODE_TIMEOUT {
- "Take a screenshot (get_desktop_state) to confirm the desktop is responsive, then retry once."
- } else {
- crate::wayland::niri::refusal_suggestion(&text).unwrap_or(
- "Call bring_to_front on the window (or its modal dialog), confirm with get_window_state, then retry.",
- )
- },
- })),
- None => linux_input_error(e),
- }
-}
-
-#[cfg(test)]
-#[test]
-fn a_niri_refusal_suggests_its_own_remedy_instead_of_bring_to_front() {
- let refused = input_error_result(anyhow::anyhow!(
- "foreground_unavailable: the niri layout changed after the get_desktop_state screenshot, so its pixels may be elsewhere now; take a new screenshot"
- ));
- let structured = refused.structured_content.unwrap();
- assert_eq!(structured["code"], "foreground_unavailable");
- assert!(structured["suggestion"]
- .as_str()
- .unwrap()
- .starts_with("Take a new screenshot"));
-}
-
-/// Structured payload for a completed foreground transaction.
-fn foreground_structured(
- path: &str,
- report: crate::input::ForegroundReport,
- extra: serde_json::Map<String, Value>,
-) -> Value {
- let mut v = json!({
- "path": path,
- "verified": false,
- "delivery_mode": "foreground",
- "effect": "unverifiable",
- });
- apply_foreground_report(&mut v, &report);
- for (k, val) in extra {
- v[k] = val;
- }
- v
-}
-
-/// Fold a foreground transaction's post-check into a result: the report
-/// itself, `focus_after` at the top level, and an `evidence` item so the
-/// observation survives the public-record reduction. A window of the target
-/// process that appeared or closed is `window_change` evidence and an
-/// `effect: confirmed` wherever the focus sits afterwards (the OK button of a
-/// dialog destroys the very window that held the focus, and closing the last
-/// window of a process leaves the focus to the window manager); a change
-/// among unattributed popups only counts while the focus stayed in the
-/// process (`ForegroundReport::window_change_confirms`). Otherwise, focus that
-/// left the process is a suspected no-op and anything else stays
-/// unverifiable with a `native_api_result` item.
-fn apply_foreground_report(v: &mut Value, report: &crate::input::ForegroundReport) {
- v["foreground"] = report.to_json();
- v["focus_after"] = json!(report.focus_after.as_str());
- let confirming_change = report
- .window_change
- .as_ref()
- .filter(|_| report.window_change_confirms());
- let evidence = match confirming_change {
- Some(change) => {
- v["effect"] = json!("confirmed");
- v["verified"] = json!(true);
- json!({
- "kind": "window_change",
- "detail": format!("focus_after={}; {change}", report.focus_after.as_str()),
- })
- }
- None if report.focus_after == crate::input::FocusAfter::Elsewhere => {
- v["effect"] = json!("suspected_noop");
- v["warning"] = json!(
- "input focus left the target process before the post-check; the input \
- may have reached another window. Verify with a screenshot."
- );
- return;
- }
- None => json!({
- "kind": "native_api_result",
- "detail": format!(
- "real input delivered to the activated window (focus_after={}); {}",
- report.focus_after.as_str(),
- match &report.window_change {
- Some(change) => format!("{change}, not attributed to the target process"),
- None => "no window change observed".to_owned(),
- }
- ),
- }),
- };
- let mut items = v
- .get("evidence")
- .and_then(Value::as_array)
- .cloned()
- .unwrap_or_default();
- items.push(evidence);
- v["evidence"] = json!(items);
+fn is_gtk_process(pid: u32) -> bool {
+ fs::read_to_string(format!("/proc/{pid}/maps"))
+ .map(|maps| maps_indicate_gtk(&maps))
+ .unwrap_or(false)
}
-/// Typed refusal for a pointer tool whose arguments name neither an element
-/// nor a complete point. Before this check a call with only a pid was
-/// executed at window-local (0, 0) — the top-left corner, i.e. the File menu.
-fn invalid_pointer_arguments(tool: &str, detail: &str) -> ToolResult {
- let accepted = [
- "pid + element_token from get_window_state",
- "pid/window_id + x + y (window-local pixels of window_id; coordinate_frame:\"desktop\" for screen pixels)",
- "x + y + scope:\"desktop\" (screen pixels, no pid)",
- ];
- ToolResult::error(format!(
- "{tool}: invalid arguments: {detail}. Accepted forms: {}. No input was sent.",
- accepted.join("; ")
- ))
- .with_structured(json!({
- "code": "invalid_arguments",
- "tool": tool,
- "effect": "refused",
- "detail": detail,
- "accepted_forms": accepted,
- }))
+pub(crate) fn unavailable_webkit_background(
+ pid: u32,
+ delivery: crate::input::delivery::DeliveryMode,
+) -> Option<ToolResult> {
+ (!delivery.is_foreground()
+ && is_webkitgtk_embedder(pid)
+ && !crate::wayland::is_inject_mode()
+ && !crate::input::real_pointer_input_available())
+ .then(|| {
+ crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::WebKitSyntheticInput,
+ )
+ })
}
-/// `Some(refusal)` when a pointer tool reached its coordinate path without
-/// both `x` and `y` (and, by construction, without an element).
-fn require_point_args(tool: &str, args: &Value) -> Option<ToolResult> {
- let present = |name: &str| args.get(name).is_some_and(|v| v.is_number());
- match (present("x"), present("y")) {
- (true, true) => None,
- (false, false) => Some(invalid_pointer_arguments(
- tool,
- "no element_token and no x/y were given (a `target_id` is not an \
- element handle)",
- )),
- (true, false) => Some(invalid_pointer_arguments(tool, "x was given without y")),
- (false, true) => Some(invalid_pointer_arguments(tool, "y was given without x")),
- }
+pub(crate) fn unavailable_webkit_keyboard_background(
+ pid: u32,
+ delivery: crate::input::delivery::DeliveryMode,
+) -> Option<ToolResult> {
+ (!delivery.is_foreground() && is_webkitgtk_embedder(pid) && !crate::wayland::is_inject_mode())
+ .then(|| {
+ crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ )
+ })
}
-#[cfg(test)]
-mod pointer_argument_tests;
+/// Operator-facing hint attached to a background keyboard refusal on X11: the
+/// focus-free route exists (XI2 MPX virtual master keyboard) but needs a
+/// writable `/dev/uinput`, which is a one-time host setup step.
+const UINPUT_KEYBOARD_HINT: &str = "Background keyboard delivery on X11 needs a virtual master \
+ keyboard fed by a uinput device. Grant this process access to /dev/uinput \
+ (modprobe uinput; chmod 0666 /dev/uinput, or add the driver's user to the \
+ 'input' group and restart it) and retry. Xvfb/Xtigervnc cannot hot-add \
+ input devices, so the route is unavailable there.";
-fn parse_mouse_button(name: &str) -> u8 {
- match name {
- "right" => 3,
- "middle" => 2,
- _ => 1,
+fn with_uinput_hint(mut refusal: ToolResult) -> ToolResult {
+ refusal
+ .content
+ .push(cua_driver_core::protocol::Content::text(
+ UINPUT_KEYBOARD_HINT,
+ ));
+ if let Some(structured) = refusal.structured_content.as_mut() {
+ structured["hint"] = json!(UINPUT_KEYBOARD_HINT);
}
+ refusal
}
-/// Escalation hint for a non-AX / suspected-no-op surface — the cross-platform
-/// `escalation` field mirrored from macOS. The recommended next rung depends on
-/// the session type: X11 can pixel-target a specific window in the background,
-/// so the deliberate move is an element px action — click by pixel (x,y) off the
-/// screenshot already in this response (`px`). Native Wayland CANNOT
-/// background-target an unfocused window (libei injects to the compositor's input
-/// focus — see `input::delivery`), and the pixel path itself routes through the
-/// same coordinate-free AT-SPI actuation that just no-op'd, so the move there is
-/// to bring the window to the foreground first (`foreground`).
-fn non_ax_escalation() -> Value {
- if crate::wayland::is_wayland() {
- json!({
- "recommended": "foreground",
- "reason": "non-AX surface on Wayland: a specific unfocused window can't be \
- pixel-targeted in the background (libei injects to the compositor's \
- focus). bring_to_front, then click by pixel off the screenshot \
- with delivery_mode:\"foreground\"."
- })
+/// Honest refusal for a background keyboard action that has no focus-free
+/// actuator. Carries the uinput hint whenever the MPX route is missing only
+/// because `/dev/uinput` is not writable (the fixable case).
+pub(crate) fn background_keyboard_refusal(atspi_ok: bool) -> ToolResult {
+ let mut refusal = crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ let bus = crate::health_report::START_ATSPI_BUS_HINT;
+ let hint = format!("AT-SPI is unavailable. {bus} Or use delivery_mode \"foreground\".");
+ refusal
+ .content
+ .extend((!atspi_ok).then(|| cua_driver_core::protocol::Content::text(hint)));
+ if !crate::wayland::is_wayland() && !crate::input::uinput_accessible() {
+ with_uinput_hint(refusal)
} else {
- json!({
- "recommended": "px",
- "reason": "non-AX surface — act by pixel (x,y) off the screenshot \
- in this response (an element px action)."
- })
+ refusal
}
}
-/// Structured payload for a `type_text` response. Keeps the legacy
-/// `path`/`characters`/`verified` fields for back-compat and adds the cross-tool
-/// `effect` tri-state. Linux's AT-SPI `insertText` return value acknowledges
-/// the method call but does not read the widget value back, so it and every
-/// keystroke / XSendEvent / XTest / Wayland rung are `"unverifiable"` (the
-/// caller confirms through a separate observation). No escalation is
-/// attached to a successful delivery: the public projection renders one as
-/// `delivery_failed`, which is wrong for text that landed but was not read
-/// back. Mirrors the macOS `type_text` contract.
-fn type_text_structured(path: &str, characters: usize, verified: bool) -> Value {
- let s = json!({
- "path": path,
- "characters": characters,
- "verified": verified,
- "effect": if verified { "confirmed" } else { "unverifiable" },
- });
- // No escalation on a successful-but-unverified delivery: the public
- // projection renders a foreground escalation as `delivery_failed`, which
- // made models re-type text that had already landed. Refusals carry their own.
- let _ = verified;
- s
-}
-
-/// Structured payload for the Electron/Chromium AX-echo case: the AT-SPI
-/// `insertText` rung returned success on a Chromium embedder, but on those
-/// surfaces the a11y bridge can accept and echo the write while the renderer
-/// never observes it — so a path=="ax" "confirm" is a shim echo, not ground
-/// truth. Mirrors the macOS `type_text` `ax_echo_surface` branch: report
-/// effect:"unverifiable" + escalation:{recommended:"px"} (a renderer-focus
-/// problem — pixel-focus the field, not foreground-activate it).
-fn type_text_structured_electron(text_len: usize) -> Value {
- json!({
- "path": "ax",
- "characters": text_len,
- "verified": false,
- "effect": "unverifiable",
- "escalation": {
- "recommended": "px",
- "reason": "Electron/web surface — the AX write was echoed but the \
- renderer may not have observed it. Confirm via the screenshot; \
- if it didn't land, re-type with the element px action (pass x,y \
- to pixel-focus the field, then type)."
- }
- })
+pub(crate) fn atspi_up() -> bool {
+ let probe = std::thread::spawn(crate::health_report::probe_a11y_bus);
+ crate::wayland::is_wayland() || probe.join().unwrap_or(false)
}
-/// [`type_text_ax_result`] for an element-addressed insert: the snapshot
-/// object is read back (as `set_value` does) and, when the typed text is now
-/// part of its value, the result is `effect: confirmed` with a
-/// `value_readback` evidence item instead of `unverifiable`.
-async fn type_text_ax_result_verified(
+pub(crate) fn unavailable_gtk_keyboard_background(
pid: u32,
- token: crate::atspi::TokenElement,
- text: &str,
- route: &str,
-) -> ToolResult {
- let text_len = text.chars().count();
- if is_chromium_embedder(pid) {
- return type_text_ax_result(pid, text_len, route);
- }
- let idx = token.index;
- let readback = tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&token))
- .await
- .ok()
- .flatten();
- type_text_readback_result(pid, idx, text, route, readback)
-}
-
-/// The typed text is confirmed when the element's fresh value contains it
-/// (an insert at the caret) or agrees with it numerically (a spin button
-/// that reformats "-40" as "-40.0").
-fn type_text_readback_result(
- pid: u32,
- idx: usize,
- text: &str,
- route: &str,
- readback: Option<String>,
-) -> ToolResult {
- let text_len = text.chars().count();
- match readback {
- Some(seen)
- if !text.trim().is_empty() && (seen.contains(text) || values_agree(text, &seen)) =>
- {
- let shown: String = seen.chars().take(200).collect();
- ToolResult::text(format!(
- "Typed {text_len} character(s) into element [{idx}] ({route}); it reads back '{shown}'."
- ))
- .with_structured(json!({
- "path": "ax",
- "characters": text_len,
- "verified": true,
- "effect": "confirmed",
- "readback": shown,
- "evidence": [{
- "kind": "value_readback",
- "detail": format!("element [{idx}] reads back the typed text"),
- }],
- }))
- }
- Some(seen) => {
- let mut result = type_text_ax_result(pid, text_len, route);
- let shown: String = seen.chars().take(200).collect();
- if let Some(structured) = result.structured_content.as_mut() {
- structured["readback"] = json!(shown);
- }
- result
- .content
- .push(cua_driver_core::protocol::Content::text(format!(
- "The element reads back '{shown}', which does not contain the typed text; \
- confirm with get_window_state."
- )));
- result
- }
- None => type_text_ax_result(pid, text_len, route),
- }
-}
-
-#[cfg(test)]
-mod type_text_readback_tests;
-
-/// Build the success `ToolResult` for an AT-SPI insert. The EditableText
-/// method's boolean is a delivery acknowledgement, not a fresh value readback,
-/// so native widgets remain `unverifiable`. Chromium embedders additionally
-/// recommend the pixel rung because their accessibility bridge can acknowledge
-/// a write the renderer never observes.
-fn type_text_ax_result(pid: u32, text_len: usize, route: &str) -> ToolResult {
- if is_chromium_embedder(pid) {
- return ToolResult::text(format!(
- "📨 Sent (unverified) {text_len} character(s) ({route}). — Electron/web \
- surface: the AX layer accepts and echoes the write but the renderer may \
- not have observed it, so the driver cannot confirm via AX. Verify via the \
- screenshot; if it didn't land, re-type with the px form (pass x,y to \
- pixel-focus the field)."
- ))
- .with_structured(type_text_structured_electron(text_len));
- }
- ToolResult::text(format!("Typed {text_len} character(s) ({route})."))
- .with_structured(type_text_structured("ax", text_len, false))
-}
-
-/// True when `pid` is a Chromium-based embedder — a Chrome/Chromium browser or
-/// any Electron/CEF app. On these surfaces an AT-SPI `EditableText.insertText`
-/// can succeed at the bridge while the Chromium *renderer* never observes it,
-/// so the AT-SPI "ax" rung must not be trusted as a confirmed insert (see
-/// [`type_text_ax_result`]).
-///
-/// This is the Linux analogue of macOS `ElectronJs::is_electron` (which checks
-/// for a bundled Electron Framework). Linux has no bundle, so the signal is
-/// Chromium's multiprocess fingerprint: the embedder forks sandboxed helpers
-/// whose argv carries `--type=renderer` / `--type=zygote` / `--type=gpu-process`.
-/// Native GTK/Qt apps never spawn such helpers, so this is conservative (very
-/// low false-positive). The single-process fallback also matches the embedder's
-/// own argv. Reads `/proc`; cheap and only invoked on the rare AT-SPI confirm.
-fn is_chromium_embedder(pid: u32) -> bool {
- fn argv_is_chromium_helper(p: u32) -> bool {
- fs::read(format!("/proc/{p}/cmdline"))
- .map(|raw| crate::at_point_policy::cmdline_is_chromium_helper(&raw))
- .unwrap_or(false)
- }
- // Single-process / the embedder itself carrying a Chromium switch.
- if argv_is_chromium_helper(pid) {
- return true;
- }
- // Build PPid → children across /proc, then BFS the descendants of `pid`
- // looking for a Chromium helper. Same /proc-walk shape as
- // `terminal_descendant_ttys`.
- let mut children: std::collections::HashMap<u32, Vec<u32>> = std::collections::HashMap::new();
- let entries = match fs::read_dir("/proc") {
- Ok(e) => e,
- Err(_) => return false,
- };
- for entry in entries.flatten() {
- let name = entry.file_name();
- let child: u32 = match name.to_string_lossy().parse() {
- Ok(p) => p,
- Err(_) => continue,
- };
- let status = match fs::read_to_string(format!("/proc/{child}/status")) {
- Ok(s) => s,
- Err(_) => continue,
- };
- if let Some(ppid) = status
- .lines()
- .find(|l| l.starts_with("PPid:"))
- .and_then(|l| l[5..].trim().parse::<u32>().ok())
- {
- children.entry(ppid).or_default().push(child);
- }
- }
- let mut queue = std::collections::VecDeque::from([pid]);
- let mut seen = std::collections::HashSet::new();
- while let Some(cur) = queue.pop_front() {
- if !seen.insert(cur) {
- continue;
- }
- if let Some(kids) = children.get(&cur) {
- for &kid in kids {
- if argv_is_chromium_helper(kid) {
- return true;
- }
- queue.push_back(kid);
- }
- }
- }
- false
-}
-
-fn is_webkitgtk_embedder(pid: u32) -> bool {
- fn argv_is_webkit_helper(pid: u32) -> bool {
- fs::read(format!("/proc/{pid}/cmdline"))
- .map(|raw| {
- let cmdline = String::from_utf8_lossy(&raw);
- cmdline.contains("WebKitWebProcess") || cmdline.contains("WebKitNetworkProcess")
- })
- .unwrap_or(false)
- }
-
- let mut children: std::collections::HashMap<u32, Vec<u32>> = std::collections::HashMap::new();
- let Ok(entries) = fs::read_dir("/proc") else {
- return false;
- };
- for entry in entries.flatten() {
- let Ok(child) = entry.file_name().to_string_lossy().parse::<u32>() else {
- continue;
- };
- let Ok(status) = fs::read_to_string(format!("/proc/{child}/status")) else {
- continue;
- };
- if let Some(ppid) = status
- .lines()
- .find(|line| line.starts_with("PPid:"))
- .and_then(|line| line[5..].trim().parse::<u32>().ok())
- {
- children.entry(ppid).or_default().push(child);
- }
- }
- let mut queue = std::collections::VecDeque::from([pid]);
- let mut seen = std::collections::HashSet::new();
- while let Some(current) = queue.pop_front() {
- if !seen.insert(current) {
- continue;
- }
- if argv_is_webkit_helper(current) {
- return true;
- }
- if let Some(descendants) = children.get(&current) {
- queue.extend(descendants.iter().copied());
- }
- }
- false
-}
-
-fn maps_indicate_gtk(maps: &str) -> bool {
- maps.contains("libgtk-3.so") || maps.contains("libgtk-4.so")
-}
-
-/// Toolkits known to discard synthetic (`send_event`) X11 pointer events:
-/// GTK3/4 (XInput2 only), LibreOffice VCL (all plugins), Qt5/6 (xcb, XI2).
-fn maps_indicate_synthetic_pointer_dropped(maps: &str) -> bool {
- maps_indicate_gtk(maps)
- || maps.contains("libvcl")
- || maps.contains("libmergedlo")
- || maps.contains("libQt5Gui")
- || maps.contains("libQt6Gui")
-}
-
-fn synthetic_pointer_is_dropped(pid: u32) -> bool {
- fs::read_to_string(format!("/proc/{pid}/maps"))
- .map(|maps| maps_indicate_synthetic_pointer_dropped(&maps))
- .unwrap_or(false)
-}
-
-/// Qt5 maps its own GUI library; only then is the synthetic-FocusIn bridge
-/// workaround in `type_text` worth two extra AT-SPI walks.
-fn is_qt5_process(pid: u32) -> bool {
- fs::read_to_string(format!("/proc/{pid}/maps"))
- .map(|maps| maps.contains("libQt5Gui") || maps.contains("libQt5Widgets"))
- .unwrap_or(false)
-}
-
-fn is_gtk_process(pid: u32) -> bool {
- fs::read_to_string(format!("/proc/{pid}/maps"))
- .map(|maps| maps_indicate_gtk(&maps))
- .unwrap_or(false)
-}
-
-fn unavailable_webkit_background(
- pid: u32,
- delivery: crate::input::delivery::DeliveryMode,
-) -> Option<ToolResult> {
- (!delivery.is_foreground()
- && is_webkitgtk_embedder(pid)
- && !crate::wayland::is_inject_mode()
- && !crate::input::real_pointer_input_available())
- .then(|| {
- crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::WebKitSyntheticInput,
- )
- })
-}
-
-fn unavailable_webkit_keyboard_background(
- pid: u32,
- delivery: crate::input::delivery::DeliveryMode,
-) -> Option<ToolResult> {
- (!delivery.is_foreground() && is_webkitgtk_embedder(pid) && !crate::wayland::is_inject_mode())
- .then(|| {
- crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- )
- })
-}
-
-/// Operator-facing hint attached to a background keyboard refusal on X11: the
-/// focus-free route exists (XI2 MPX virtual master keyboard) but needs a
-/// writable `/dev/uinput`, which is a one-time host setup step.
-const UINPUT_KEYBOARD_HINT: &str = "Background keyboard delivery on X11 needs a virtual master \
- keyboard fed by a uinput device. Grant this process access to /dev/uinput \
- (modprobe uinput; chmod 0666 /dev/uinput, or add the driver's user to the \
- 'input' group and restart it) and retry. Xvfb/Xtigervnc cannot hot-add \
- input devices, so the route is unavailable there.";
-
-fn with_uinput_hint(mut refusal: ToolResult) -> ToolResult {
- refusal
- .content
- .push(cua_driver_core::protocol::Content::text(
- UINPUT_KEYBOARD_HINT,
- ));
- if let Some(structured) = refusal.structured_content.as_mut() {
- structured["hint"] = json!(UINPUT_KEYBOARD_HINT);
- }
- refusal
-}
-
-/// Honest refusal for a background keyboard action that has no focus-free
-/// actuator. Carries the uinput hint whenever the MPX route is missing only
-/// because `/dev/uinput` is not writable (the fixable case).
-fn background_keyboard_refusal(atspi_ok: bool) -> ToolResult {
- let mut refusal = crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- let bus = crate::health_report::START_ATSPI_BUS_HINT;
- let hint = format!("AT-SPI is unavailable. {bus} Or use delivery_mode \"foreground\".");
- refusal
- .content
- .extend((!atspi_ok).then(|| cua_driver_core::protocol::Content::text(hint)));
- if !crate::wayland::is_wayland() && !crate::input::uinput_accessible() {
- with_uinput_hint(refusal)
- } else {
- refusal
- }
-}
-
-fn atspi_up() -> bool {
- let probe = std::thread::spawn(crate::health_report::probe_a11y_bus);
- crate::wayland::is_wayland() || probe.join().unwrap_or(false)
-}
-
-fn unavailable_gtk_keyboard_background(
- pid: u32,
- delivery: crate::input::delivery::DeliveryMode,
-) -> Option<ToolResult> {
- (!delivery.is_foreground()
- && is_gtk_process(pid)
- && !crate::wayland::is_inject_mode()
- && !crate::input::real_keyboard_input_available())
- .then(|| background_keyboard_refusal(atspi_up()))
+ delivery: crate::input::delivery::DeliveryMode,
+) -> Option<ToolResult> {
+ (!delivery.is_foreground()
+ && is_gtk_process(pid)
+ && !crate::wayland::is_inject_mode()
+ && !crate::input::real_keyboard_input_available())
+ .then(|| background_keyboard_refusal(atspi_up()))
}
/// Outcome of one background/foreground keyboard delivery on X11, funnelled
/// through `spawn_blocking`'s success type so the tool can report the route.
-enum KeyRoute {
+pub(crate) enum KeyRoute {
/// Activated the window and injected via XTest.
Foreground(crate::input::ForegroundReport),
/// Real key events through the session's XI2 virtual master keyboard.
@@ -3560,7 +1867,7 @@ enum KeyRoute {
/// holds an active keyboard grab, so the virtual master keyboard route would
/// be silently lost; the core keyboard (XTest) reaches the grab holder, which
/// is the target itself, without any focus change.
-fn background_virtual_key(
+pub(crate) fn background_virtual_key(
cursor_id: &str,
pid: u32,
xid: u64,
@@ -3580,7 +1887,7 @@ fn background_virtual_key(
}
/// Text counterpart of [`background_virtual_key`].
-fn background_virtual_text(
+pub(crate) fn background_virtual_text(
cursor_id: &str,
pid: u32,
xid: u64,
@@ -3598,7 +1905,7 @@ fn background_virtual_text(
crate::input::send_virtual_keyboard_text(cursor_id, xid, text)
}
-fn is_popup_keyboard_grab(error: &anyhow::Error) -> bool {
+pub(crate) fn is_popup_keyboard_grab(error: &anyhow::Error) -> bool {
error
.downcast_ref::<crate::input::PopupKeyboardGrab>()
.is_some()
@@ -3610,7 +1917,7 @@ fn is_popup_keyboard_grab(error: &anyhow::Error) -> bool {
/// (the tool turns it into the refusal + hint); any other MPX failure falls
/// back to XSendEvent for the toolkits that still accept it, and is reported
/// as an error for the ones known to drop it.
-fn background_key_route(
+pub(crate) fn background_key_route(
cursor_id: &str,
pid: u32,
xid: u64,
@@ -3690,7 +1997,7 @@ fn is_grab_focus_failure(error: &anyhow::Error) -> bool {
/// Merge a background focus-guard report into a finished tool result: the
/// structured `focus_changed` / `focus_restored` / `grab_held_by` fields plus
/// one sentence of text when anything moved.
-fn attach_focus_guard(
+pub(crate) fn attach_focus_guard(
mut result: ToolResult,
report: Option<&crate::input::FocusGuardReport>,
) -> ToolResult {
@@ -3719,48 +2026,9 @@ fn attach_focus_guard(
result
}
-/// After a pointer press: name the grid cell that holds the widget focus now
-/// (LibreOffice reports "D2"), so a one-row miss shows in the click result
-/// instead of after six typed values. Read from the focus-event log only,
-/// so it costs nothing when the press did not land on a grid.
-async fn attach_focused_cell(mut result: ToolResult, pid: u32) -> ToolResult {
- if result.is_error == Some(true) {
- return result;
- }
- let focused = tokio::task::spawn_blocking(move || crate::atspi::focused_control(pid))
- .await
- .ok()
- .flatten();
- let Some((role, name)) = focused.filter(|(role, name)| {
- let role = role.to_ascii_lowercase();
- (role.contains("cell") || role == "table cell") && !name.trim().is_empty()
- }) else {
- return result;
- };
- result
- .content
- .push(cua_driver_core::protocol::Content::text(format!(
- "focus: cell {name}"
- )));
- let mut structured = result
- .structured_content
- .take()
- .unwrap_or_else(|| json!({}));
- structured["focused_cell"] = json!(name);
- push_evidence(
- &mut structured,
- json!({
- "kind": "native_api_result",
- "detail": format!("the accessible holding the focus after the press is {role} \"{name}\""),
- }),
- );
- result.structured_content = Some(structured);
- result
-}
-
/// Append an `evidence[]` item to a structured payload (the public action
/// contract keeps `evidence`, while flat diagnostic fields are reduced away).
-fn push_evidence(structured: &mut Value, item: Value) {
+pub(crate) fn push_evidence(structured: &mut Value, item: Value) {
match structured.get_mut("evidence").and_then(Value::as_array_mut) {
Some(items) => items.push(item),
None => structured["evidence"] = json!([item]),
@@ -3768,7 +2036,7 @@ fn push_evidence(structured: &mut Value, item: Value) {
}
/// Tool result for a completed keyboard route (`press_key` / `hotkey`).
-fn key_route_result(action: &str, route: KeyRoute, mode_label: &str) -> ToolResult {
+pub(crate) fn key_route_result(action: &str, route: KeyRoute, mode_label: &str) -> ToolResult {
match route {
KeyRoute::Foreground(report) => ToolResult::text(format!(
"{action} as a real key event (delivery_mode=foreground, focus_after={}); \
@@ -3839,7 +2107,7 @@ fn key_route_result(action: &str, route: KeyRoute, mode_label: &str) -> ToolResu
}
}
-fn unavailable_gtk_pointer_background(
+pub(crate) fn unavailable_gtk_pointer_background(
pid: u32,
delivery: crate::input::delivery::DeliveryMode,
) -> Option<ToolResult> {
@@ -3854,7 +2122,7 @@ fn unavailable_gtk_pointer_background(
})
}
-fn unavailable_wayland_focused_input_background(
+pub(crate) fn unavailable_wayland_focused_input_background(
delivery: crate::input::delivery::DeliveryMode,
focus_free_inject_supported: bool,
) -> Option<ToolResult> {
@@ -3869,55 +2137,21 @@ fn unavailable_wayland_focused_input_background(
}
#[derive(Debug)]
-enum CoordinateContext {
+pub(crate) enum CoordinateContext {
Zoom(ZoomContext),
Screenshot(f64),
}
-fn coordinate_click_context(
- native_refusal: Option<ToolResult>,
- resolve_context: impl FnOnce() -> Result<CoordinateContext, ToolResult>,
-) -> Result<CoordinateContext, ToolResult> {
- if let Some(refusal) = native_refusal {
- return Err(refusal);
- }
- resolve_context()
-}
-
-fn coordinate_drag_context(
- native_refusal: Option<ToolResult>,
- resolve_context: impl FnOnce() -> Result<CoordinateContext, ToolResult>,
-) -> Result<CoordinateContext, ToolResult> {
- if let Some(refusal) = native_refusal {
- return Err(refusal);
- }
- resolve_context()
-}
-
-/// Pixel scroll shares click and drag's refusal order: an unsupported native
-/// background delivery must report `background_unavailable` before the
-/// screenshot frame is consulted, because native Wayland may legitimately
-/// withhold the screenshot (`surface_identity_unproven`) that the frame needs.
-fn coordinate_scroll_scale(
- native_refusal: Option<ToolResult>,
- resolve_scale: impl FnOnce() -> Result<f64, ToolResult>,
-) -> Result<f64, ToolResult> {
- if let Some(refusal) = native_refusal {
- return Err(refusal);
- }
- resolve_scale()
-}
-
-/// Chromium's X11 renderer drops synthetic input sent to an occluded,
-/// unfocused toplevel. Returning success here would be a silent loss, so all
-/// input tools expose the same typed refusal and leave foreground activation
-/// as the explicit escalation.
-fn unavailable_chromium_background(
- pid: u32,
- delivery: crate::input::delivery::DeliveryMode,
-) -> Option<ToolResult> {
- if delivery.is_foreground() || !is_chromium_embedder(pid) {
- return None;
+/// Chromium's X11 renderer drops synthetic input sent to an occluded,
+/// unfocused toplevel. Returning success here would be a silent loss, so all
+/// input tools expose the same typed refusal and leave foreground activation
+/// as the explicit escalation.
+pub(crate) fn unavailable_chromium_background(
+ pid: u32,
+ delivery: crate::input::delivery::DeliveryMode,
+) -> Option<ToolResult> {
+ if delivery.is_foreground() || !is_chromium_embedder(pid) {
+ return None;
}
// Chromium is an XInput2 client: real (non-synthetic) events from the MPX
// virtual master pointer/keyboard reach its renderer like the private
@@ -3933,7 +2167,7 @@ fn unavailable_chromium_background(
}
}
-fn chromium_background_must_refuse(
+pub(crate) fn chromium_background_must_refuse(
foreground: bool,
focus_free_inject_mode: bool,
chromium: bool,
@@ -3944,7 +2178,7 @@ fn chromium_background_must_refuse(
/// Screen-absolute center of a window (top-left from translate_coordinates plus
/// half its geometry). Used to position the no-focus-steal scroll over the
/// window's content. Blocking — call inside spawn_blocking.
-fn window_screen_center(xid: u64) -> anyhow::Result<(i32, i32)> {
+pub(crate) fn window_screen_center(xid: u64) -> anyhow::Result<(i32, i32)> {
use x11rb::connection::Connection;
use x11rb::protocol::xproto::ConnectionExt as _;
use x11rb::rust_connection::RustConnection;
@@ -3964,7 +2198,7 @@ fn window_screen_center(xid: u64) -> anyhow::Result<(i32, i32)> {
/// How a background pointer action was delivered on X11, with the cheap
/// post-checks the real-pointer route can make.
#[derive(Debug)]
-enum PointerRoute {
+pub(crate) enum PointerRoute {
/// Real button events from the session's MPX virtual master pointer.
Mpx(crate::input::PointerEffect),
/// A background pixel click resolved to an accessible under the point and
@@ -3980,7 +2214,7 @@ enum PointerRoute {
impl PointerRoute {
/// Result `path`; the Wayland routes keep their own tool-level labels.
- fn path(&self) -> Option<&'static str> {
+ pub(crate) fn path(&self) -> Option<&'static str> {
match self {
Self::Mpx(_) => Some(crate::input::MPX_POINTER_PATH),
Self::Atspi(_) => Some("x11_atspi"),
@@ -3994,7 +2228,7 @@ impl PointerRoute {
/// `path`, `verified`, `effect`, and for the MPX route the post-checks
/// (`focus_unchanged`, `region_diff_pct`). `verified:true` + `effect:
/// "landed"` only when the screen around the point visibly reacted.
- fn structured(&self, mode_label: &str) -> Value {
+ pub(crate) fn structured(&self, mode_label: &str) -> Value {
let mut v = json!({
"verified": false,
"effect": "unverifiable",
@@ -4121,7 +2355,7 @@ impl PointerRoute {
}
/// Human-readable tail for the result text.
- fn text_suffix(&self, mode_label: &str) -> String {
+ pub(crate) fn text_suffix(&self, mode_label: &str) -> String {
let mut text = self.text_suffix_inner(mode_label);
if let Self::Mpx(effect) = self {
if let Some(popup) = effect.popups.last() {
@@ -4266,7 +2500,7 @@ fn popup_list(popups: &[crate::input::PopupWindow]) -> String {
/// a coordinate-frame mistake is visible: `window_point` is what the caller
/// passed (window-local pixels after zoom / resize translation),
/// `screen_point` is where that lands on the screen.
-fn with_pixel_points(
+pub(crate) fn with_pixel_points(
mut result: ToolResult,
screen: Option<(i32, i32)>,
window: (i32, i32),
@@ -4296,7 +2530,7 @@ fn with_pixel_points(
result
}
-fn x11_pixel_click_no_focus_steal(
+pub(crate) fn x11_pixel_click_no_focus_steal(
cursor_id: &str,
xid: u64,
lx: i32,
@@ -4361,7 +2595,7 @@ fn x11_pixel_click_no_focus_steal(
/// Typed tool error for a Linux input failure: uinput permission, an occluded
/// target, or a real-pointer delivery that did not go through. Everything
/// else keeps its full error chain in the text.
-fn linux_input_error(error: anyhow::Error) -> ToolResult {
+pub(crate) fn linux_input_error(error: anyhow::Error) -> ToolResult {
if crate::input::is_uinput_unavailable(&error) {
return ToolResult::error(error.to_string()).with_structured(json!({
"code": crate::input::UINPUT_UNAVAILABLE_CODE,
@@ -4467,11 +2701,11 @@ fn linux_input_error(error: anyhow::Error) -> ToolResult {
ToolResult::error(format!("{error:#}"))
}
-fn isolated_hyprland_background(delivery: crate::input::delivery::DeliveryMode) -> bool {
+pub(crate) fn isolated_hyprland_background(delivery: crate::input::delivery::DeliveryMode) -> bool {
!delivery.is_foreground() && crate::wayland::hyprland_input::enabled()
}
-fn hyprland_foreground(delivery: crate::input::delivery::DeliveryMode) -> bool {
+pub(crate) fn hyprland_foreground(delivery: crate::input::delivery::DeliveryMode) -> bool {
delivery.is_foreground()
&& crate::wayland::wayland_input_enabled()
&& crate::wayland::hyprland::is_session()
@@ -4480,12 +2714,12 @@ fn hyprland_foreground(delivery: crate::input::delivery::DeliveryMode) -> bool {
/// AT-SPI roles whose `activate`/`press` action does not move the widget
/// focus into them, so a following keystroke would land elsewhere; a real
/// pointer click at their centre is what a user does.
-fn element_needs_real_click(role: &str) -> bool {
+pub(crate) fn element_needs_real_click(role: &str) -> bool {
crate::atspi::is_focus_taking_role(role)
}
/// Menubar entries and the items of an open menu.
-fn element_is_menu_role(role: &str) -> bool {
+pub(crate) fn element_is_menu_role(role: &str) -> bool {
matches!(
role.trim().to_ascii_lowercase().as_str(),
"menu" | "menu item" | "check menu item" | "radio menu item"
@@ -4547,7 +2781,7 @@ fn wm_chord_kind(key: &str, modifiers: &[String]) -> Option<WmChord> {
/// Background delivery of a WM-level chord: close the window for Alt+F4,
/// refuse the rest (`wm_chord_unavailable`). `None` for ordinary chords.
-async fn wm_chord_background(
+pub(crate) async fn wm_chord_background(
pid: u32,
xid: u64,
key: &str,
@@ -4698,7 +2932,7 @@ async fn close_window_background(pid: u32, xid: u64, display: &str) -> ToolResul
/// `press_key` accepts `"alt+F4"` / `"ctrl+shift+t"` style keys: everything
/// before the last `+` is a modifier. A bare `"+"` stays the plus key.
-fn split_key_combo(key: &str) -> (Vec<String>, String) {
+pub(crate) fn split_key_combo(key: &str) -> (Vec<String>, String) {
if key.len() <= 1 || !key.contains('+') {
return (Vec::new(), key.to_owned());
}
@@ -4717,72 +2951,7 @@ fn split_key_combo(key: &str) -> (Vec<String>, String) {
}
}
-fn element_click_prefers_ax(
- hyprland_foreground: bool,
- button: u8,
- count: usize,
- has_modifiers: bool,
-) -> bool {
- !has_modifiers && (!hyprland_foreground || (button == 1 && count == 1))
-}
-
-fn foreground_ax_click_unknown(pid: u32, idx: usize) -> ToolResult {
- use cua_driver_core::action_record::{
- ActionEffect, ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
- };
- let record = ActionExecutionRecord::builder(
- ActionEffect::Unverifiable,
- ActionTransport::LinuxAtSpiAction,
- RequestedDelivery::Foreground,
- )
- .actual_delivery(ActualDelivery::Unknown)
- .build()
- .expect("uncertain semantic activation has no delivered count or replay");
- let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
- ToolResult::error(format!(
- "click: AT-SPI activation outcome is unknown for element [{idx}] (pid {pid}); refresh state before another action."
- ))
- .with_structured(public)
- .with_action_record(record)
-}
-
-#[cfg(test)]
-#[test]
-fn foreground_ax_click_error_does_not_claim_delivery_or_request_replay() {
- let result = foreground_ax_click_unknown(123, 4);
- assert_eq!(result.is_error, Some(true));
- let wire = serde_json::to_value(&result).unwrap();
- assert_eq!(wire["isError"], true);
- let public = &wire["structuredContent"];
- assert_eq!(
- *public,
- serde_json::to_value(result.action_record.unwrap().public_result().unwrap()).unwrap()
- );
- assert_eq!(public["effect"], "unverifiable");
- assert_eq!(public["route"], "accessibility");
- assert_eq!(public["delivery"]["mode"], "unknown");
- assert!(public.get("escalation").is_none());
- assert!(public["delivery"].get("delivered_count").is_none());
-}
-
-#[cfg(test)]
-#[test]
-fn hyprland_foreground_semantic_click_preserves_pointer_gestures() {
- assert!(element_click_prefers_ax(true, 1, 1, false));
- for (button, count) in [(2, 1), (3, 1), (1, 0), (1, 2), (1, 3), (2, 2), (3, 3)] {
- assert!(!element_click_prefers_ax(true, button, count, false));
- }
- // Preserve the existing AT-SPI eligibility for all other delivery routes.
- for button in [1, 2, 3] {
- for count in [0, 1, 2, 3] {
- assert!(element_click_prefers_ax(false, button, count, false));
- assert!(!element_click_prefers_ax(false, button, count, true));
- assert!(!element_click_prefers_ax(true, button, count, true));
- }
- }
-}
-
-fn isolated_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
+pub(crate) fn isolated_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
let detail = detail.into();
ToolResult::error(format!("background_unavailable: {detail}")).with_structured(json!({
"ok": false, "code": "background_unavailable", "reason": "unsupported_operation",
@@ -4790,11 +2959,11 @@ fn isolated_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
}))
}
-fn isolated_hyprland_result(result: anyhow::Result<Value>) -> ToolResult {
+pub(crate) fn isolated_hyprland_result(result: anyhow::Result<Value>) -> ToolResult {
hyprland_input_result(result, false)
}
-fn hyprland_input_result(result: anyhow::Result<Value>, foreground: bool) -> ToolResult {
+pub(crate) fn hyprland_input_result(result: anyhow::Result<Value>, foreground: bool) -> ToolResult {
use cua_driver_core::action_record::{
ActionEffect, ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
};
@@ -4944,14 +3113,17 @@ fn hyprland_input_result(result: anyhow::Result<Value>, foreground: bool) -> Too
}
}
-fn isolated_hyprland_task_error(error: tokio::task::JoinError, acknowledged: bool) -> ToolResult {
+pub(crate) fn isolated_hyprland_task_error(
+ error: tokio::task::JoinError,
+ acknowledged: bool,
+) -> ToolResult {
isolated_hyprland_result(Err(crate::wayland::hyprland_input::unknown_dispatch(
error.into(),
u32::from(acknowledged),
)))
}
-fn spawn_isolated_hyprland(
+pub(crate) fn spawn_isolated_hyprland(
args: &Value,
work: impl FnOnce(crate::wayland::hyprland_input::ActionCancellation) -> anyhow::Result<Value>
+ Send
@@ -4991,7 +3163,7 @@ fn spawn_isolated_hyprland(
Ok((guard, dispatch))
}
-async fn isolated_hyprland_action(
+pub(crate) async fn isolated_hyprland_action(
args: &Value,
pid: u32,
xid: u64,
@@ -5010,7 +3182,7 @@ async fn isolated_hyprland_action(
}
}
-fn foreground_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
+pub(crate) fn foreground_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
hyprland_input_result(
Ok(json!({
"ok": false, "code": "unsupported_operation", "detail": detail.into()
@@ -5019,7 +3191,7 @@ fn foreground_hyprland_refusal(detail: impl Into<String>) -> ToolResult {
)
}
-async fn foreground_hyprland_action(
+pub(crate) async fn foreground_hyprland_action(
args: &Value,
pid: u32,
xid: u64,
@@ -5047,41 +3219,7 @@ async fn foreground_hyprland_action(
)
}
-async fn foreground_hyprland_scroll(
- args: &Value,
- pid: u32,
- xid: u64,
- actions: Vec<crate::wayland::hyprland_input::Action>,
-) -> ToolResult {
- if !crate::wayland::hyprland_input::enabled() {
- return foreground_hyprland_refusal("production Hyprland input plugin is unavailable");
- }
- let owner = named_session_cursor_key(args);
- let (_cancellation, dispatch) = match spawn_isolated_hyprland(args, move |cancellation| {
- crate::wayland::hyprland_input::execute_foreground_scroll(
- owner,
- pid,
- xid,
- actions,
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(_) => return foreground_hyprland_refusal("authenticated admitted lifecycle required"),
- };
- hyprland_input_result(
- match dispatch.await {
- Ok(result) => result,
- Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
- error.into(),
- 0,
- )),
- },
- true,
- )
-}
-
-fn foreground_hyprland_key(
+pub(crate) fn foreground_hyprland_key(
key: String,
mut modifiers: Vec<String>,
) -> Result<crate::wayland::hyprland_input::Action, ToolResult> {
@@ -5106,7 +3244,7 @@ fn foreground_hyprland_key(
Ok(crate::wayland::hyprland_input::Action::Key { key, modifiers })
}
-async fn focus_hyprland_foreground(
+pub(crate) async fn focus_hyprland_foreground(
state: &Arc<ToolState>,
args: &Value,
pid: u32,
@@ -5193,25 +3331,12 @@ fn isolated_background_routes_do_not_reprobe_availability_before_primary_fallbac
// Structural regression for the availability-loss race: each tool keeps
// the decision that bypassed background refusal until its returning
// native branch. A second probe here could select primary-seat input.
- let source = include_str!("impl_.rs");
- for (start, end) in [
- ("impl Tool for ClickTool {", "impl Tool for TypeTextTool {"),
- (
- "impl Tool for ScrollTool {",
- "impl Tool for DoubleClickTool {",
- ),
- (
- "impl Tool for DragTool {",
- "impl Tool for MouseButtonDownTool {",
- ),
+ for (source, start) in [
+ (include_str!("click.rs"), "impl Tool for ClickTool {"),
+ (include_str!("scroll.rs"), "impl Tool for ScrollTool {"),
+ (include_str!("drag.rs"), "impl Tool for DragTool {"),
] {
- let body = source
- .rsplit_once(start)
- .unwrap()
- .1
- .split_once(end)
- .unwrap()
- .0;
+ let body = source.rsplit_once(start).unwrap().1;
let invoke = body.split_once("async fn invoke").unwrap().1;
assert_eq!(
invoke
@@ -5265,188 +3390,6 @@ fn background_keyboard_refusal_names_missing_atspi_bus() {
}
}
-#[cfg(test)]
-#[test]
-fn type_text_routes_isolated_background_before_generic_wayland_refusal() {
- let source = include_str!("impl_.rs");
- let invoke = source
- .rsplit_once("impl Tool for TypeTextTool {")
- .unwrap()
- .1
- .split_once("impl Tool for PressKeyTool {")
- .unwrap()
- .0
- .split_once("async fn invoke")
- .unwrap()
- .1;
- let isolated = invoke
- .find("isolated_hyprland_background(delivery)")
- .expect("type_text must select isolated Hyprland background delivery");
- let generic = invoke
- .find("unavailable_wayland_focused_input_background")
- .expect("type_text must retain the generic Wayland refusal");
- assert!(
- isolated < generic,
- "the plugin route must run before the generic focused-input refusal"
- );
- assert!(invoke.contains("execute_background_text"));
-}
-
-#[cfg(test)]
-#[test]
-fn coordinate_click_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
- use std::sync::atomic::{AtomicBool, Ordering};
-
- let resolved = AtomicBool::new(false);
- let native_refusal = crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- let result = coordinate_click_context(Some(native_refusal), || {
- resolved.store(true, Ordering::SeqCst);
- Ok(CoordinateContext::Screenshot(7.35))
- })
- .expect_err("unsupported Wayland background delivery must refuse");
-
- assert_eq!(
- result.structured_content.as_ref().unwrap()["code"],
- "background_unavailable"
- );
- assert!(
- !resolved.load(Ordering::SeqCst),
- "snapshot context must not be consulted before the native refusal"
- );
-}
-
-#[cfg(test)]
-#[test]
-fn coordinate_drag_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
- use std::sync::atomic::{AtomicBool, Ordering};
-
- let resolved = AtomicBool::new(false);
- let native_refusal = crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- let result = coordinate_drag_context(Some(native_refusal), || {
- resolved.store(true, Ordering::SeqCst);
- Ok(CoordinateContext::Screenshot(7.35))
- })
- .expect_err("unsupported Wayland background delivery must refuse drag");
-
- assert_eq!(
- result.structured_content.as_ref().unwrap()["code"],
- "background_unavailable"
- );
- assert!(
- !resolved.load(Ordering::SeqCst),
- "drag snapshot context must not be consulted before the native refusal"
- );
-}
-
-#[cfg(test)]
-#[test]
-fn coordinate_scroll_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
- use std::sync::atomic::{AtomicBool, Ordering};
-
- let resolved = AtomicBool::new(false);
- let native_refusal = crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- let result = coordinate_scroll_scale(Some(native_refusal), || {
- resolved.store(true, Ordering::SeqCst);
- Err(ToolResult::error(
- "screenshot context must not be consulted",
- ))
- })
- .expect_err("unsupported Wayland background delivery must refuse pixel scroll");
-
- assert_eq!(
- result.structured_content.as_ref().unwrap()["code"],
- "background_unavailable"
- );
- assert!(
- !resolved.load(Ordering::SeqCst),
- "scroll snapshot context must not be consulted before the native refusal"
- );
- assert_eq!(coordinate_scroll_scale(None, || Ok(1.5)).unwrap(), 1.5);
-
- let source = include_str!("impl_.rs");
- let invoke = source
- .rsplit_once("impl Tool for ScrollTool {")
- .unwrap()
- .1
- .split_once("async fn invoke")
- .unwrap()
- .1;
- let (before_scale, _) = invoke
- .split_once("coordinate_scroll_scale(native_refusal")
- .expect("pixel scroll must resolve its frame through the refusal-first helper");
- assert!(
- before_scale.contains("unavailable_wayland_focused_input_background(delivery, true)"),
- "pixel scroll must compute the native Wayland refusal before its frame"
- );
- assert!(
- !before_scale.contains("screenshot_scale("),
- "pixel scroll must not consult the screenshot frame before the native refusal"
- );
-}
-
-#[cfg(test)]
-#[test]
-fn coordinate_less_mouse_button_up_survives_snapshot_replacement() {
- let state = ToolState::new();
- let cursor_id = "held-after-replacement";
- let hold = MouseHoldState {
- pid: 42,
- xid: 7,
- button: 1,
- x: 120.0,
- y: 80.0,
- };
- state.snapshots.publish_for_session(
- hold.pid as i32,
- hold.xid,
- crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
- Some("press-owner"),
- Some(2.0),
- );
- state
- .mouse_hold
- .lock()
- .unwrap()
- .insert(cursor_id.to_owned(), hold.clone());
-
- state.snapshots.publish_for_session(
- hold.pid as i32,
- hold.xid,
- crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
- Some("replacement-owner"),
- Some(1.0),
- );
-
- let release_args = json!({"_session_id": "press-owner"});
- assert_eq!(
- mouse_button_up_coordinates(&state, &release_args, &hold).unwrap(),
- (hold.x, hold.y),
- "coordinate-less release must use the stored native hold point"
- );
-
- let stale_coordinate_args = json!({"_session_id": "press-owner", "x": 60.0, "y": 40.0});
- let refusal = mouse_button_up_coordinates(&state, &stale_coordinate_args, &hold)
- .expect_err("coordinate-bearing release must still reject a replaced snapshot");
- assert_eq!(
- refusal.structured_content.as_ref().unwrap()["code"],
- "screenshot_context_missing"
- );
-
- clear_mouse_hold_after_release(&state, cursor_id);
- assert!(state
- .mouse_hold
- .lock()
- .unwrap()
- .insert(cursor_id.to_owned(), hold)
- .is_none());
-}
-
#[cfg(test)]
#[test]
fn persistent_wayland_pointer_coordinates_include_window_origin() {
@@ -5556,9763 +3499,927 @@ fn failed_session_end_release_remains_retryable_and_bounded() {
assert_eq!(attempts, 2);
assert!(state
.mouse_hold
- .lock()
- .unwrap()
- .insert(session_id.to_owned(), hold)
- .is_none());
-}
-
-#[cfg(test)]
-#[test]
-fn isolated_hyprland_refused_partial_and_unknown_outcomes_stay_distinct() {
- let busy = isolated_hyprland_result(Err(crate::wayland::hyprland_input::LaneBusy.into()));
- let content = busy.structured_content.as_ref().unwrap();
- assert_eq!(content["reason"], "lane_busy");
- assert_eq!(content["effect"], "refused");
- assert!(content.get("delivery").is_none());
- assert!(busy
- .action_record
- .unwrap()
- .public_result()
- .unwrap()
- .delivery
- .is_none());
-
- let refused = isolated_hyprland_result(Ok(
- json!({"ok":false,"code":"stale_target","detail":"stale_target"}),
- ));
- assert_eq!(
- refused.structured_content.as_ref().unwrap()["effect"],
- "refused"
- );
- let public = refused.action_record.unwrap().public_result().unwrap();
- assert!(public.delivery.is_none());
- public.validate_invariants().unwrap();
-
- for foreground in [false, true] {
- let stale = hyprland_input_result(
- Ok(json!({"ok":false,"code":"stale_geometry","detail":"stale_geometry"})),
- foreground,
- );
- let value = stale.structured_content.as_ref().unwrap();
- assert_eq!(value["code"], "stale_observation");
- assert_eq!(value["reason"], "stale_geometry");
- assert_eq!(value["effect"], "refused");
- assert!(stale
- .action_record
- .unwrap()
- .public_result()
- .unwrap()
- .delivery
- .is_none());
- }
-
- let partial = isolated_hyprland_result(Ok(
- json!({"ok":false,"code":"cancelled","detail":"cancelled",
- "effect":"partial","delivery":{"mode":"background","delivered_count":1}}),
- ));
- let public = partial.action_record.unwrap().public_result().unwrap();
- assert_eq!(public.effect, cua_driver_contract::ActionEffect::Partial);
- assert_eq!(public.delivery.unwrap().delivered_count, Some(1));
-
- let unknown = isolated_hyprland_result(Err(anyhow::anyhow!("connection lost")));
- let public = unknown.action_record.unwrap().public_result().unwrap();
- assert_eq!(
- public.effect,
- cua_driver_contract::ActionEffect::Unverifiable
- );
- assert_eq!(
- public.delivery.as_ref().unwrap().mode,
- cua_driver_contract::ActionDeliveryMode::Unknown
- );
- assert_eq!(public.delivery.unwrap().delivered_count, None);
-
- let unknown_after_start = isolated_hyprland_result(Err(
- crate::wayland::hyprland_input::unknown_dispatch(anyhow::anyhow!("connection lost"), 1),
- ));
- let public = unknown_after_start
- .action_record
- .unwrap()
- .public_result()
- .unwrap();
- public.validate_invariants().unwrap();
- assert_eq!(public.effect, cua_driver_contract::ActionEffect::Partial);
- let delivery = public.delivery.unwrap();
- assert_eq!(
- delivery.mode,
- cua_driver_contract::ActionDeliveryMode::Unknown
- );
- assert_eq!(delivery.delivered_count, Some(1));
-}
-
-#[cfg(test)]
-#[test]
-fn experimental_pending_grant_is_an_error_with_operator_context() {
- let result = isolated_hyprland_result(Ok(json!({
- "ok": false, "code": "permission_required", "detail": "external approval pending",
- "epoch": "epoch", "challenge": "challenge", "target": "target", "revision": 3
- })));
- assert_eq!(result.is_error, Some(true));
- let value = result.structured_content.unwrap();
- assert_eq!(value["code"], "background_unavailable");
- assert_eq!(value["reason"], "permission_required");
- assert_eq!(value["challenge"], "challenge");
- assert_eq!(value["target"], "target");
- assert_eq!(value["revision"], 3);
- assert_eq!(value["verified"], false);
-}
-
-#[cfg(test)]
-#[test]
-fn experimental_dispatch_does_not_claim_application_success() {
- let result = isolated_hyprland_result(Ok(json!({
- "ok": true, "effect": "unverifiable", "route": "synthetic_events"
- })));
- assert_ne!(result.is_error, Some(true));
- let public = result.action_record.unwrap().public_result().unwrap();
- public.validate_invariants().unwrap();
- let value = serde_json::to_value(public).unwrap();
- assert_eq!(value["effect"], "unverifiable");
- assert_eq!(value["route"], "synthetic_events");
- assert!(value.get("verified").is_none());
-}
-
-#[cfg(test)]
-#[test]
-fn foreground_hyprland_reports_foreground_and_preserves_partial_progress() {
- use cua_driver_core::action_record::{ActionTransport, RequestedDelivery};
- for (reply, expected_effect, count) in [
- (json!({"ok":true}), "unverifiable", None),
- (
- json!({"ok":false,"code":"stale_target","detail":"stale"}),
- "refused",
- None,
- ),
- (
- json!({"ok":false,"code":"cancelled","detail":"cancelled","effect":"partial",
- "delivery":{"mode":"foreground","delivered_count":3}}),
- "partial",
- Some(3),
- ),
- ] {
- let result = hyprland_input_result(Ok(reply), true);
- let record = result.action_record.unwrap();
- assert_eq!(
- record.transport,
- ActionTransport::LinuxHyprlandForegroundInput
- );
- assert_eq!(record.requested_delivery, RequestedDelivery::Foreground);
- let public = record.public_result().unwrap();
- public.validate_invariants().unwrap();
- let value = serde_json::to_value(public).unwrap();
- assert_eq!(value["route"], "global_input");
- assert_eq!(value["effect"], expected_effect);
- if expected_effect == "refused" {
- assert!(value.get("delivery").is_none());
- } else {
- assert_eq!(value["delivery"]["mode"], "foreground");
- assert_eq!(value["delivery"]["delivered_count"].as_u64(), count);
- }
- }
- for reply in [
- Err(crate::wayland::hyprland_input::unknown_dispatch(
- anyhow::anyhow!("lost"),
- 2,
- )),
- Ok(json!({"ok":false,"code":"foreground_partial_unknown","detail":"focus changed"})),
- ] {
- let unknown = hyprland_input_result(reply, true);
- let value =
- serde_json::to_value(unknown.action_record.unwrap().public_result().unwrap()).unwrap();
- assert_eq!(value["delivery"]["mode"], "unknown");
- assert_ne!(value["effect"], "refused");
- }
-}
-
-#[cfg(test)]
-#[test]
-fn foreground_hyprland_chords_preserve_all_modifiers() {
- let action = foreground_hyprland_key("Ctrl+Shift+H".into(), vec!["alt".into()]).unwrap();
- match action {
- crate::wayland::hyprland_input::Action::Key { key, modifiers } => {
- assert_eq!(key, "H");
- assert_eq!(modifiers, ["alt", "ctrl", "shift"]);
- }
- _ => panic!("expected a key chord"),
- }
- assert!(foreground_hyprland_key("bogus+H".into(), vec![]).is_err());
-}
-
-#[cfg(test)]
-#[test]
-fn uinput_failure_has_a_stable_structured_code() {
- let result = linux_input_error(crate::input::uinput_unavailable("synthetic failure"));
- assert_eq!(result.is_error, Some(true));
- assert_eq!(
- result
- .structured_content
- .as_ref()
- .and_then(|value| value.get("code"))
- .and_then(Value::as_str),
- Some(crate::input::UINPUT_UNAVAILABLE_CODE)
- );
-}
-
-fn x11_pixel_click_no_focus_steal_modifiers(
- xid: u64,
- lx: i32,
- ly: i32,
- button: u8,
- count: usize,
- modifiers: &[&str],
-) -> anyhow::Result<()> {
- crate::input::send_click_with_modifiers(xid, lx, ly, count, button, modifiers)
-}
-
-fn mouse_button_name(button: u8) -> &'static str {
- match button {
- 3 => "right",
- 2 => "middle",
- _ => "left",
- }
-}
-
-fn resolve_cursor_key(args: &Value) -> String {
- cua_driver_core::tool_args::resolve_session_key(args)
-}
-
-/// Return the cursor key only for a lifecycle-owned session. Cursor positioning
-/// for keyboard actions deliberately does not opt anonymous calls or the
-/// legacy cursor_id-only path into session semantics. The proxy-minted
-/// `_session_id` is trusted lifecycle state and must behave like a public named
-/// session for cursor ownership.
-fn named_session_cursor_key(args: &Value) -> Option<String> {
- cursor_overlay::named_session_cursor_key(args)
-}
-
-fn mouse_hold_json(cursor_id: &str, hold: Option<&MouseHoldState>) -> Value {
- match hold {
- Some(hold) => json!({
- "cursor_id": cursor_id,
- "held": true,
- "pid": hold.pid,
- "window_id": hold.xid,
- "button": mouse_button_name(hold.button),
- "x": hold.x,
- "y": hold.y,
- }),
- None => json!({
- "cursor_id": cursor_id,
- "held": false,
- "pid": Value::Null,
- "window_id": Value::Null,
- "button": Value::Null,
- "x": Value::Null,
- "y": Value::Null,
- }),
- }
-}
-
-fn held_target_mismatch(
- args: &Value,
- cursor_id: &str,
- hold: &MouseHoldState,
-) -> Option<ToolResult> {
- match args.opt_u32("pid") {
- Ok(Some(pid)) if pid != hold.pid => {
- return Some(
- ToolResult::error(format!(
- "Cursor '{cursor_id}' is holding a button for pid {}, not pid {pid}.",
- hold.pid
- ))
- .with_structured(mouse_hold_json(cursor_id, Some(hold))),
- );
- }
- Err(err) => return Some(err.with_structured(mouse_hold_json(cursor_id, Some(hold)))),
- _ => {}
- }
-
- match args.opt_u64("window_id") {
- Some(xid) if xid != hold.xid => Some(
- ToolResult::error(format!(
- "Cursor '{cursor_id}' is holding a button for window_id {}, not {xid}.",
- hold.xid
- ))
- .with_structured(mouse_hold_json(cursor_id, Some(hold))),
- ),
- _ => None,
- }
-}
-
-fn mouse_button_up_coordinates(
- state: &ToolState,
- args: &Value,
- hold: &MouseHoldState,
-) -> Result<(f64, f64), ToolResult> {
- if args.get("x").is_none() && args.get("y").is_none() {
- return Ok((hold.x, hold.y));
- }
-
- let mut x = args.opt_f64("x").unwrap_or(hold.x);
- let mut y = args.opt_f64("y").unwrap_or(hold.y);
- if args.bool_or("from_zoom", false) {
- let context = state.zoom_context(args, hold.pid, Some(hold.xid))?;
- return Ok(context.zoom_to_window(x, y));
- }
-
- let ratio = screenshot_scale(state, args, hold.pid, Some(hold.xid))?;
- x *= ratio;
- y *= ratio;
- Ok((x, y))
-}
-
-fn clear_mouse_hold_after_release(state: &ToolState, cursor_id: &str) {
- state.mouse_hold.lock().unwrap().remove(cursor_id);
-}
-
-fn persistent_pointer_coordinates(
- is_wayland: bool,
- window_id: u64,
- x: i32,
- y: i32,
- window_local_to_output: impl FnOnce(u64, i32, i32) -> anyhow::Result<AimedPoint>,
-) -> anyhow::Result<AimedPoint> {
- if is_wayland {
- window_local_to_output(window_id, x, y)
- } else {
- Ok((x, y).into())
- }
-}
-
-fn release_mouse_hold_for_session<Release>(
- state: &ToolState,
- session_id: &str,
- mut release: Release,
-) -> Result<(), String>
-where
- Release: FnMut(&str, &MouseHoldState) -> anyhow::Result<()>,
-{
- let hold = state.mouse_hold.lock().unwrap().get(session_id).cloned();
- let Some(hold) = hold else {
- return Ok(());
- };
-
- release(session_id, &hold).map_err(|error| error.to_string())?;
- state.mouse_hold.lock().unwrap().remove(session_id);
- Ok(())
-}
-
-fn overlay_snap_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::SnapTo {
- x: sx,
- y: sy,
- heading_radians: heading,
- },
- );
-}
-
-fn overlay_move_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {
- overlay_move_to_target_for(cursor_id, sx, sy, heading, None);
-}
-
-/// [`overlay_move_to_for`] with the targeted element's screen rect
-/// `[x, y, width, height]` (same space as `sx`/`sy`).
-fn overlay_move_to_target_for(
- cursor_id: &str,
- sx: f64,
- sy: f64,
- heading: Option<f64>,
- target: Option<[f64; 4]>,
-) {
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::MoveTo {
- x: sx,
- y: sy,
- end_heading_radians: heading.unwrap_or(std::f64::consts::FRAC_PI_4),
- target,
- },
- );
-}
-
-async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {
- overlay_glide_to_target_for(cursor_id, sx, sy, None).await;
-}
-
-/// [`overlay_glide_to_for`] with the targeted element's screen rect
-/// `[x, y, width, height]` (same space as `sx`/`sy`); `None` for pixel actions.
-async fn overlay_glide_to_target_for(cursor_id: &str, sx: f64, sy: f64, target: Option<[f64; 4]>) {
- // Input always revives its agent cursor. Hiding it is useful while idle,
- // but a hidden cursor must not make pointer or keyboard control invisible.
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::SetEnabled(true),
- );
- // Every Wayland backend owns its animation loop. Send one destination and
- // let the Linux overlay layer select the semantic helper, layer-shell, or
- // older helper fallback without an interpolated stream fighting it.
- if crate::wayland::is_wayland() {
- overlay_move_to_target_for(cursor_id, sx, sy, None, target);
- return;
- }
- let pos = crate::overlay::current_position_for(cursor_id);
- if pos.is_none() {
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- return;
- }
- crate::overlay::animate_cursor_to_target_for(cursor_id.to_owned(), sx, sy, target).await;
-}
-
-/// The compositor snapshot a desktop-scope action is addressed in. One
-/// snapshot converts the action's points, places the overlay and sizes the
-/// virtual pointer, so they cannot disagree about the monitor layout.
-async fn desktop_input_space() -> anyhow::Result<crate::wayland::DesktopInputSpace> {
- tokio::task::spawn_blocking(|| {
- crate::interaction::admit_lock(
- crate::interaction::policy::Scope::DesktopInput,
- crate::interaction::facts::Fact::Unknown("desktop input addresses no window".into()),
- )?;
- if !crate::wayland::wayland_input_enabled() {
- return Ok(crate::wayland::DesktopInputSpace::default());
- }
- crate::wayland::DesktopInputSpace::current()
- })
- .await
- .map_err(|error| anyhow::anyhow!("task error: {error}"))?
-}
-
-/// Keep the logical cursor position in sync with every visibly targeted
-/// pointer action. Overlay delivery is intentionally best-effort: registry
-/// state is still updated when no renderer is running or its queue is closed.
-async fn reveal_pointer_action_for(
- state: &ToolState,
- cursor_id: &str,
- sx: f64,
- sy: f64,
- click_pulse: bool,
-) {
- reveal_pointer_action_target_for(state, cursor_id, sx, sy, click_pulse, None).await;
-}
-
-/// [`reveal_pointer_action_for`] with the targeted element's screen rect
-/// `[x, y, width, height]` (same space as `sx`/`sy`).
-async fn reveal_pointer_action_target_for(
- state: &ToolState,
- cursor_id: &str,
- sx: f64,
- sy: f64,
- click_pulse: bool,
- target: Option<[f64; 4]>,
-) {
- if !sx.is_finite() || !sy.is_finite() {
- return;
- }
- state.cursor_registry.set_enabled(cursor_id, true);
- state.cursor_registry.update_position(cursor_id, sx, sy);
- emit_cursor_hook(cursor_id, sx, sy, false);
- overlay_glide_to_target_for(cursor_id, sx, sy, target).await;
- if click_pulse {
- emit_cursor_hook(cursor_id, sx, sy, true);
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- }
-}
-
-/// Report a commanded cursor move (or a press) to the embedder's cursor
-/// hook, with the same guards as macOS's `CursorRegistry::emit_cursor_event`:
-/// nothing is built unless a hook is registered, and an ended session never
-/// reaches it. Emitted from the logical cursor write path, so it works with
-/// no overlay renderer (a headless Xvfb session).
-pub(crate) fn emit_cursor_hook(cursor_id: &str, x: f64, y: f64, pressed: bool) {
- if !cua_driver_core::cursor_hook::cursor_hook_enabled() {
- return;
- }
- if cursor_id.is_empty() || cua_driver_core::session::is_session_ended(cursor_id) {
- return;
- }
- cua_driver_core::cursor_hook::push_cursor_event(
- cua_driver_core::cursor_hook::CursorHookEvent {
- cursor_id: cursor_id.to_owned(),
- x,
- y,
- pressed,
- },
- );
-}
-
-fn keyboard_window_center(xid: u64) -> Option<(f64, f64)> {
- if xid == 0 {
- return None;
- }
- if crate::wayland::is_wayland() {
- return crate::wayland::window_geometry(xid).and_then(|(x, y, width, height)| {
- (width > 0 && height > 0).then_some((
- f64::from(x) + f64::from(width) / 2.0,
- f64::from(y) + f64::from(height) / 2.0,
- ))
- });
- }
- window_screen_center(xid)
- .ok()
- .map(|(x, y)| (f64::from(x), f64::from(y)))
-}
-
-fn current_pointer_position() -> Option<(f64, f64)> {
- if crate::wayland::is_wayland() {
- return crate::wayland::last_synth_cursor_pos().map(|(x, y)| (f64::from(x), f64::from(y)));
- }
-
- use x11rb::connection::Connection;
- use x11rb::protocol::xproto::ConnectionExt as _;
- use x11rb::rust_connection::RustConnection;
-
- let (connection, screen_num) = RustConnection::connect(None).ok()?;
- let root = connection.setup().roots[screen_num].root;
- let reply = connection.query_pointer(root).ok()?.reply().ok()?;
- Some((f64::from(reply.root_x), f64::from(reply.root_y)))
-}
-
-fn explicit_keyboard_cursor_target(
- pid: u32,
- xid: u64,
- element: Option<&crate::atspi::TokenElement>,
- pixel_target: Option<(f64, f64)>,
-) -> Option<(f64, f64)> {
- if let Some(element) = element {
- let (sx, sy) = element_screen_center(pid, element, (xid != 0).then_some(xid)).ok()?;
- return Some((sx, sy));
- }
-
- let (x, y) = pixel_target?;
- if crate::wayland::wayland_input_enabled() {
- return crate::wayland::window_geometry(xid)
- .map(|(wx, wy, _, _)| (f64::from(wx) + x.round(), f64::from(wy) + y.round()));
- }
- window_local_to_screen(xid, x, y).ok()
-}
-
-/// Position and reveal a named session's cursor before keyboard/value input.
-/// `preserve_legacy_element_visual` retains the existing element-only feedback
-/// for type_text/set_value anonymous calls without adding session-style fallback
-/// placement to them. Geometry and overlay failures are observational and never
-/// affect the tool's actual input result.
-/// Tell the cursor hook where an anonymous keyboard action types, before the
-/// keystrokes: the element or pixel target, else the window's center. The
-/// overlay is left alone (anonymous calls keep their legacy visuals); only an
-/// embedder publishing presence learns the position, so a viewer sees the
-/// agent's cursor go to the field it is about to type into.
-async fn announce_keyboard_target(
- args: &Value,
- pid: u32,
- xid: u64,
- element: Option<crate::atspi::TokenElement>,
- pixel_target: Option<(f64, f64)>,
-) {
- if !cua_driver_core::cursor_hook::cursor_hook_enabled() {
- return;
- }
- let cursor_id = resolve_cursor_key(args);
- let target = tokio::task::spawn_blocking(move || {
- explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
- .or_else(|| (xid != 0).then(|| keyboard_window_center(xid)).flatten())
- })
- .await
- .ok()
- .flatten();
- if let Some((x, y)) = target {
- emit_cursor_hook(&cursor_id, x, y, false);
- }
-}
-
-async fn position_named_session_keyboard_cursor(
- state: &ToolState,
- args: &Value,
- pid: u32,
- xid: u64,
- element: Option<crate::atspi::TokenElement>,
- pixel_target: Option<(f64, f64)>,
- preserve_legacy_element_visual: bool,
-) {
- let named_cursor_id = named_session_cursor_key(args);
- let cursor_id = match named_cursor_id {
- Some(ref cursor_id) => cursor_id.clone(),
- None if preserve_legacy_element_visual && element.is_some() => resolve_cursor_key(args),
- None => {
- announce_keyboard_target(args, pid, xid, element, pixel_target).await;
- return;
- }
- };
-
- let remembered = named_cursor_id.as_ref().and_then(|_| {
- state
- .cursor_registry
- .get(&cursor_id)
- .and_then(|cursor| cursor.x.zip(cursor.y))
- });
- let explicit = tokio::task::spawn_blocking(move || {
- explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
- })
- .await
- .ok()
- .flatten();
-
- let fallback = if named_cursor_id.is_some()
- && explicit.is_none()
- && cursor_overlay::keyboard_cursor_target(None, remembered, None, None).is_none()
- {
- tokio::task::spawn_blocking(move || {
- let center = keyboard_window_center(xid);
- let pointer = center.is_none().then(current_pointer_position).flatten();
- (center, pointer)
- })
- .await
- .unwrap_or((None, None))
- } else {
- (None, None)
- };
-
- let Some((sx, sy)) =
- cursor_overlay::keyboard_cursor_target(explicit, remembered, fallback.0, fallback.1)
- else {
- return;
- };
- if xid != 0 {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- }
- reveal_pointer_action_for(state, &cursor_id, sx, sy, false).await;
-}
-
-async fn track_overlay_drag_for(
- cursor_id: String,
- from: (f64, f64),
- to: (f64, f64),
- duration_ms: u64,
- steps: usize,
-) {
- track_overlay_drag_with(
- |command| crate::overlay::send_command_for(cursor_id.clone(), command),
- from,
- to,
- duration_ms,
- steps,
- )
- .await;
-}
-
-async fn track_overlay_drag_with(
- send: impl Fn(cursor_overlay::OverlayCommand),
- from: (f64, f64),
- to: (f64, f64),
- duration_ms: u64,
- steps: usize,
-) {
- send(cursor_overlay::OverlayCommand::SetEnabled(true));
- let _pressed = cursor_overlay::PressedVisualGuard::new(&send);
- let steps = steps.max(1);
- let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
- for index in 0..=steps {
- let t = index as f64 / steps as f64;
- let x = from.0 + (to.0 - from.0) * t;
- let y = from.1 + (to.1 - from.1) * t;
- send(cursor_overlay::track_pointer_command(x, y));
- if index < steps && !step_delay.is_zero() {
- tokio::time::sleep(step_delay).await;
- }
- }
-}
-
-#[cfg(test)]
-#[tokio::test]
-async fn cancelled_overlay_drag_releases_visual_press_without_native_commands() {
- use std::{cell::RefCell, future::Future, task::Context};
- let commands = RefCell::new(Vec::new());
- let mut drag = Box::pin(track_overlay_drag_with(
- |command| commands.borrow_mut().push(command),
- (10.0, 20.0),
- (30.0, 40.0),
- 2000,
- 20,
- ));
- let mut context = Context::from_waker(std::task::Waker::noop());
- assert!(drag.as_mut().poll(&mut context).is_pending());
- assert!(matches!(
- commands.borrow()[1],
- cursor_overlay::OverlayCommand::SetPressed(true)
- ));
- drop(drag);
- let commands = commands.borrow();
- assert_eq!(commands.len(), 4);
- assert!(matches!(
- commands.last(),
- Some(cursor_overlay::OverlayCommand::SetPressed(false))
- ));
-}
-
-#[cfg(test)]
-#[tokio::test]
-async fn completed_overlay_drag_balances_visual_press_once() {
- use std::cell::RefCell;
- let pressed = RefCell::new(Vec::new());
- track_overlay_drag_with(
- |command| {
- if let cursor_overlay::OverlayCommand::SetPressed(value) = command {
- pressed.borrow_mut().push(value);
- }
- },
- (10.0, 20.0),
- (30.0, 40.0),
- 0,
- 2,
- )
- .await;
- assert_eq!(*pressed.borrow(), vec![true, false]);
-}
-
-fn process_name(pid: u32) -> Option<String> {
- let cmdline = fs::read(format!("/proc/{pid}/cmdline")).ok()?;
- let first = String::from_utf8_lossy(&cmdline)
- .split('\0')
- .next()
- .unwrap_or("")
- .trim()
- .to_owned();
- if !first.is_empty() {
- return std::path::Path::new(&first)
- .file_name()
- .map(|s| s.to_string_lossy().into_owned())
- .or(Some(first));
- }
-
- let status = fs::read_to_string(format!("/proc/{pid}/status")).ok()?;
- status
- .lines()
- .find(|l| l.starts_with("Name:"))
- .map(|l| l[5..].trim().to_owned())
-}
-
-fn is_terminal_process(pid: u32) -> bool {
- // Canonical list lives in `crate::terminal::TERMINAL_PROCESS_NAMES`
- // — keeps the additive contract centralised. Adding a new terminal
- // here means appending one string in `crate::terminal`.
- match process_name(pid).as_deref() {
- Some(name) => crate::terminal::is_terminal_process_name(name),
- None => false,
- }
-}
-
-fn terminal_descendant_ttys(pid: u32) -> Vec<PathBuf> {
- let mut parent_to_children: std::collections::HashMap<u32, Vec<u32>> =
- std::collections::HashMap::new();
- let proc_dir = std::path::Path::new("/proc");
- let entries = match fs::read_dir(proc_dir) {
- Ok(entries) => entries,
- Err(_) => return Vec::new(),
- };
-
- for entry in entries.flatten() {
- let pid_str = entry.file_name();
- let pid_str = pid_str.to_string_lossy();
- let child_pid: u32 = match pid_str.parse() {
- Ok(pid) => pid,
- Err(_) => continue,
- };
- let status = match fs::read_to_string(proc_dir.join(&*pid_str).join("status")) {
- Ok(status) => status,
- Err(_) => continue,
- };
- let parent_pid = status
- .lines()
- .find(|l| l.starts_with("PPid:"))
- .and_then(|l| l[5..].trim().parse::<u32>().ok());
- if let Some(parent_pid) = parent_pid {
- parent_to_children
- .entry(parent_pid)
- .or_default()
- .push(child_pid);
- }
- }
-
- let mut descendants = Vec::new();
- let mut queue = std::collections::VecDeque::from([pid]);
- while let Some(current) = queue.pop_front() {
- if let Some(children) = parent_to_children.get(&current) {
- for &child in children {
- descendants.push(child);
- queue.push_back(child);
- }
- }
- }
- descendants.sort_unstable();
-
- let mut ttys = Vec::new();
- for child in descendants {
- let tty = match fs::read_link(format!("/proc/{child}/fd/0")) {
- Ok(path) => path,
- Err(_) => continue,
- };
- if tty.starts_with("/dev/pts/") {
- ttys.push(tty);
- }
- }
- ttys
-}
-
-fn terminal_tty_for_window(pid: u32, xid: u64) -> Option<PathBuf> {
- if !is_terminal_process(pid) {
- return None;
- }
- let mut windows = crate::x11::list_windows(Some(pid));
- windows.sort_by_key(|w| w.xid);
- let window_index = windows.iter().position(|w| w.xid == xid)?;
- let ttys = terminal_descendant_ttys(pid);
- ttys.get(window_index).cloned()
-}
-
-/// True when `key` names the Enter key in the shared X keysym vocabulary
-/// (`key_name_to_keysym`). The terminal pty short-circuit below applies to
-/// every spelling of that physical key — `enter`, `return`, any case — so an
-/// agent following the documented key names cannot silently lose the keypress
-/// on a terminal window (terminals discard synthetic XSendEvent keys).
-fn is_enter_key(key: &str) -> bool {
- crate::input::key_name_to_keysym(key).ok() == Some(0xFF0D)
-}
-
-/// Type into a terminal window without touching X focus. Resolves the window's
-/// pty, then borrows the emulator's master fd and writes to it (see
-/// `crate::tty`). Returns `Ok(false)` when the target isn't a terminal we can
-/// reach this way so the caller falls back to the generic XSendEvent path.
-fn inject_terminal_input(pid: u32, xid: u64, text: &str) -> anyhow::Result<bool> {
- let Some(tty) = terminal_tty_for_window(pid, xid) else {
- return Ok(false);
- };
- // tty is `/dev/pts/<N>`; the emulator (pid) holds the master for the same N.
- let Some(ptn) = tty
- .file_name()
- .and_then(|s| s.to_str())
- .and_then(|s| s.parse::<u32>().ok())
- else {
- return Ok(false);
- };
- crate::tty::inject_via_master(pid, ptn, text)
-}
-
-// ── click ─────────────────────────────────────────────────────────────────────
-
-pub struct ClickTool {
- state: Arc<ToolState>,
-}
-
-fn capture_admission_error(error: anyhow::Error) -> ToolResult {
- let code = cua_driver_core::capture_runtime::admission_error_code(&error);
- ToolResult::error(format!("capture-bound click refused: {error}"))
- .with_structured(json!({ "code": code, "effect": "refused" }))
-}
-static CLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-/// Wheel detents for a compositor scroll: one per line. On niri a page is
-/// the detent approximation the Hyprland route documents; other compositors
-/// keep one detent per page, which no run has measured against.
-fn wheel_ticks(amount: usize, by: cua_driver_contract::ScrollBy, niri: bool) -> u32 {
- let detents = match by {
- cua_driver_contract::ScrollBy::Page if niri => {
- amount * crate::wayland::hyprland_input::PAGE_WHEEL_DETENTS
- }
- _ => amount,
- };
- detents as u32
-}
-
-#[cfg(test)]
-#[test]
-fn a_niri_page_scroll_sends_more_than_one_detent_per_page() {
- // `by: "page"` used to send `amount` detents, the same as lines: two
- // pages moved a WebKitGTK scroller 80 px under niri, less than its
- // 128 px viewport.
- use cua_driver_contract::ScrollBy;
- assert_eq!(wheel_ticks(2, ScrollBy::Line, true), 2);
- assert_eq!(
- wheel_ticks(2, ScrollBy::Page, true),
- 2 * crate::wayland::hyprland_input::PAGE_WHEEL_DETENTS as u32
- );
- // Sway, KWin and GNOME keep their earlier page scroll.
- assert_eq!(wheel_ticks(2, ScrollBy::Page, false), 2);
-}
-
-/// Presses in one click train: 1 to 3, as the contract states. A niri click
-/// train is one gated action, so a larger count would mean seconds of
-/// presses no recheck covers.
-fn click_train(requested: u64) -> usize {
- requested.clamp(1, 3) as usize
-}
-
-#[cfg(test)]
-#[test]
-fn click_trains_stop_at_the_advertised_triple_click() {
- // A niri click train is rechecked once, before its first press, so a
- // `count: 20` must not become seconds of ungated presses.
- let tool = ClickTool {
- state: ToolState::new(),
- };
- let count = &tool.def().input_schema["properties"]["count"];
- assert_eq!(
- (count["minimum"].as_u64(), count["maximum"].as_u64()),
- (Some(1), Some(3))
- );
- assert_eq!(click_train(20), 3);
- assert_eq!(click_train(2), 2);
-}
-
-/// Where a foreground element click may put the real pointer.
-enum ForegroundElementPlacement {
- /// The element centre lies inside the target window (or a popup of the
- /// same process): XTest-click it there.
- Point { xid: u64, sx: f64, sy: f64 },
- /// The element has no usable on-screen bounds (its menu is closed, its
- /// page is unmapped, its centre is outside every window of the process):
- /// only the AT-SPI action can reach it.
- NoPoint { reason: String },
-}
-
-impl ClickTool {
- /// X11 foreground element click. Behave like a user when the element is
- /// really on screen: activate the window and XTest-click the element's
- /// centre. A degenerate centre — the walk answering (0,0) for a menu item
- /// whose menu is closed used to glide the real pointer into the GNOME hot
- /// corner — is never clicked; the observed object's AT-SPI action is fired
- /// instead (`path: "ax_fg"`, a label the core's action record maps to
- /// the AT-SPI transport), with the foreground activation kept, and when
- /// that is not possible either the call is refused
- /// (`element_bounds_unavailable`).
- #[allow(clippy::too_many_arguments)]
- async fn foreground_element_click(
- &self,
- cursor_id: &str,
- pid: u32,
- idx: usize,
- observed: &crate::atspi::snapshot::CachedElement,
- xid_hint: Option<u64>,
- placement: Option<(u64, f64, f64)>,
- button: u8,
- count: usize,
- modifiers: &[String],
- ) -> ToolResult {
- let observed_for_state = observed.clone();
- let decided = tokio::task::spawn_blocking(move || {
- decide_foreground_element_placement(pid, &observed_for_state, xid_hint, placement)
- })
- .await;
- let (xid, decision) = match decided {
- Ok(v) => v,
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- };
- match decision {
- ForegroundElementPlacement::Point { xid, sx, sy } => {
- // Ask the toolkit who actually owns this screen point before
- // pressing it: a snapshot frame is where the toolkit *said*
- // the element is, and a hidden sibling view (a QFileDialog's
- // off-page QListView mirroring the visible QTreeView) can
- // report a frame nothing is drawn at. See atspi::native::hit_test.
- // Web content is exempt: Chromium's GetAccessibleAtPoint
- // answers with an unrelated `panel` for web elements, so it
- // cannot vouch for (or against) the point.
- let owned_ref = observed
- .object_ref
- .clone()
- .filter(|_| !observed.in_web_content);
- let (sx, sy, redirect_note) = if let Some(oref) = owned_ref {
- let (pid_c, oref_c, sx_i, sy_i) =
- (pid, oref, sx.round() as i32, sy.round() as i32);
- let ownership = tokio::task::spawn_blocking(move || {
- crate::atspi::native::hit_test::point_ownership(
- pid_c, xid, &oref_c, sx_i, sy_i,
- )
- })
- .await
- .unwrap_or(crate::atspi::native::hit_test::PointOwnership::Unknown);
- match ownership {
- crate::atspi::native::hit_test::PointOwnership::Redirect {
- target,
- screen_point,
- ..
- } => (
- screen_point.0 as f64,
- screen_point.1 as f64,
- Some(format!("redirected the press to {}", target.describe())),
- ),
- crate::atspi::native::hit_test::PointOwnership::Mismatch { owner } => {
- return ToolResult::error(format!(
- "click: element [{idx}] (pid {pid}) has a snapshot frame at \
- ({sx_i}, {sy_i}), but that screen point is owned by {} rather \
- than the observed element; a real pointer click there would hit \
- the wrong control. Re-snapshot with get_window_state, or click by \
- pixel from a screenshot.",
- owner.describe()
- ))
- .with_structured(json!({
- "code": "element_bounds_unavailable",
- "effect": "none",
- "element_index": idx,
- "reason": "point_owned_by_another_element",
- "owner": owner.describe(),
- }));
- }
- _ => (sx, sy, None),
- }
- } else {
- (sx, sy, None)
- };
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- // A redirected press targets another control: no rect.
- let target_rect = redirect_note
- .is_none()
- .then(|| frame_target_rect(observed.bounds, sx, sy))
- .flatten();
- reveal_pointer_action_target_for(&self.state, cursor_id, sx, sy, true, target_rect)
- .await;
- let modifier_owned = modifiers.to_vec();
- let result = spawn_blocking_bounded(
- "foreground element click",
- foreground_budget(0),
- move || {
- let modifier_refs: Vec<&str> =
- modifier_owned.iter().map(String::as_str).collect();
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- crate::input::send_click_xtest_desktop_with_modifiers(
- sx.round() as i32,
- sy.round() as i32,
- button,
- count,
- &modifier_refs,
- )
- },
- )
- },
- )
- .await;
- let window_point = crate::x11::window_info(xid)
- .map(|w| (sx.round() as i32 - w.x, sy.round() as i32 - w.y))
- .unwrap_or((sx.round() as i32, sy.round() as i32));
- match result {
- Ok(Ok(((), report))) => {
- let confirmed = report.window_change_confirms();
- let result = ToolResult::text(format!(
- "Clicked element [{idx}] (pid {pid}) with a real pointer click \
- (delivery_mode=foreground, focus_after={}){}{}",
- report.focus_after.as_str(),
- if confirmed {
- format!("; {}.", report.window_change.as_deref().unwrap_or(""))
- } else {
- "; not verified — confirm with a screenshot.".to_owned()
- },
- redirect_note
- .as_deref()
- .map(|n| format!(" ({n})."))
- .unwrap_or_default(),
- ))
- .with_structured(foreground_structured(
- "x11_xtest_fg",
- report,
- serde_json::Map::new(),
- ));
- with_pixel_points(
- result,
- Some((sx.round() as i32, sy.round() as i32)),
- window_point,
- )
- }
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
- ForegroundElementPlacement::NoPoint { reason } => {
- let Some(xid) = xid else {
- return ToolResult::error(format!(
- "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
- ({reason}) and pid {pid} has no window to activate; no input was sent."
- ))
- .with_structured(json!({
- "code": "element_bounds_unavailable",
- "effect": "none",
- "element_index": idx,
- "reason": reason,
- }));
- };
- if observed.object_ref.is_none()
- || button != 1
- || count != 1
- || !modifiers.is_empty()
- {
- return ToolResult::error(format!(
- "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
- ({reason}); a real pointer click cannot reach it and this click \
- shape (button {button} × {count}{}) has no AT-SPI action form. No \
- input was sent. Open the containing menu first, re-snapshot with \
- get_window_state, or click by pixel from a screenshot.",
- if modifiers.is_empty() { String::new() } else { format!(", modifiers {modifiers:?}") }
- ))
- .with_structured(json!({
- "code": "element_bounds_unavailable",
- "effect": "none",
- "element_index": idx,
- "reason": reason,
- "suggestion": "open the containing menu first, or re-snapshot with get_window_state and click by x/y",
- }));
- }
- crate::overlay::send_command_for(
- cursor_id.to_owned(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- let observed_for_ax = observed.clone();
- let result = spawn_blocking_bounded(
- "foreground element action",
- foreground_budget(0),
- move || {
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || crate::atspi::perform_action_observed(&observed_for_ax),
- )
- },
- )
- .await;
- match result {
- Ok(Ok(((action, suspected_noop, unacknowledged), report))) => {
- let mut extra = serde_json::Map::new();
- extra.insert("ax_action".into(), json!(action));
- extra.insert("bounds_reason".into(), json!(reason));
- if suspected_noop {
- extra.insert("effect".into(), json!("suspected_noop"));
- extra.insert("escalation".into(), non_ax_escalation());
- } else if unacknowledged {
- // No doAction reply arrived in time: never let a
- // coincidental window-change observation promote
- // this to `confirmed` — the AT-SPI layer itself
- // never confirmed dispatch, let alone effect.
- extra.insert("effect".into(), json!("unverifiable"));
- extra.insert("verified".into(), json!(false));
- extra.insert(
- "note".into(),
- json!(
- "no acknowledgement received for the doAction call within \
- the AT-SPI timeout; its effect is unknown"
- ),
- );
- }
- ToolResult::text(format!(
- "Clicked element [{idx}] (pid {pid}) through its AT-SPI action \
- \"{action}\" with the window activated (delivery_mode=foreground, \
- path=ax_fg, focus_after={}): the element had no usable \
- on-screen bounds ({reason}), so no pointer was moved.{}",
- report.focus_after.as_str(),
- match &report.window_change {
- Some(change) => format!(" {change}."),
- None => " Not verified — confirm with a screenshot.".to_owned(),
- }
- ))
- .with_structured(foreground_structured("ax_fg", report, extra))
- }
- Ok(Err(error)) => {
- if error.is::<crate::atspi::native::CachedElementGone>() {
- return ToolResult::error(format!(
- "click: stale_element_token: observed AT-SPI object [{idx}] is no \
- longer present ({error}); re-snapshot with get_window_state"
- ))
- .with_structured(json!({
- "code": "stale_element_token",
- "effect": "none",
- }));
- }
- if crate::input::foreground::error_code(&error).is_some() {
- return input_error_result(error);
- }
- ToolResult::error(format!(
- "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \
- ({reason}) and its AT-SPI action failed ({error:#}); no input was \
- sent. Open the containing menu first, re-snapshot with \
- get_window_state, or click by pixel from a screenshot."
- ))
- .with_structured(json!({
- "code": "element_bounds_unavailable",
- "effect": "none",
- "element_index": idx,
- "reason": reason,
- "ax_error": format!("{error:#}"),
- }))
- }
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
- }
- }
-}
-
-/// Decide whether a foreground element click has a real point to press.
-/// Returns the window to activate (when the process has one) and the
-/// placement decision. Runs blocking X11 / AT-SPI queries.
-fn decide_foreground_element_placement(
- pid: u32,
- observed: &crate::atspi::snapshot::CachedElement,
- xid_hint: Option<u64>,
- placement: Option<(u64, f64, f64)>,
-) -> (Option<u64>, ForegroundElementPlacement) {
- let xid = xid_hint
- .or_else(|| placement.map(|(xid, ..)| xid).filter(|xid| *xid != 0))
- .or_else(|| {
- crate::x11::list_windows(Some(pid))
- .into_iter()
- .next()
- .map(|w| w.xid)
- });
- // The object's own state decides whether it is on screen at all: a menu
- // item in a closed menu still answers a (stale or zero) extent.
- let showing = match crate::atspi::element_showing_observed(observed) {
- Ok(showing) => Some(showing),
- Err(error) => {
- tracing::debug!(
- "foreground element [{}] showing-state unavailable: {error:#}",
- observed.key
- );
- None
- }
- };
- if showing == Some(false) {
- return (
- xid,
- ForegroundElementPlacement::NoPoint {
- reason: format!(
- "the {} is not showing (its menu or page is closed)",
- observed.role
- ),
- },
- );
- }
- let Some((placed_xid, sx, sy)) = placement.filter(|(xid, ..)| *xid != 0) else {
- return (
- xid,
- ForegroundElementPlacement::NoPoint {
- reason: "no usable on-screen bounds were resolved for it".to_owned(),
- },
- );
- };
- let (px, py) = (sx.round() as i32, sy.round() as i32);
- let inside = |w: &crate::x11::WindowInfo| {
- px >= w.x && py >= w.y && px < w.x + w.width as i32 && py < w.y + w.height as i32
- };
- if crate::x11::window_info(placed_xid).is_some_and(|w| inside(&w)) {
- return (
- xid,
- ForegroundElementPlacement::Point {
- xid: placed_xid,
- sx,
- sy,
- },
- );
- }
- // The centre is outside the named window: a popup menu of the same
- // process (an open menu's item) is the one legitimate case.
- if let Some(popup) = crate::input::popup_under_screen_point(px, py) {
- if popup.pid.is_none_or(|p| p == pid) {
- return (
- xid,
- ForegroundElementPlacement::Point {
- xid: placed_xid,
- sx,
- sy,
- },
- );
- }
- }
- (
- xid,
- ForegroundElementPlacement::NoPoint {
- reason: format!(
- "its centre ({px}, {py}) lies outside window {placed_xid} and every popup of \
- pid {pid}"
- ),
- },
- )
-}
-
-#[async_trait]
-impl Tool for ClickTool {
- fn def(&self) -> &ToolDef {
- CLICK_DEF.get_or_init(|| ToolDef {
- name: "click".into(),
- description: "Click against a target pid. **Prefer `element_token` over pixel \
- coordinates** — element_token works on backgrounded / hidden windows, surfaces \
- a stable handle, and tells you what you're clicking via the cached AT-SPI \
- element's role + label. Reach for `x, y` only when the target is a canvas / \
- custom-drawn surface that doesn't appear in the AT-SPI tree.\n\n\
- Provide either (window_id + x/y) or (pid + element_token). x/y are \
- WINDOW-LOCAL pixels of window_id (the get_window_state screenshot frame); \
- a point outside the window is refused (point_outside_window) and every \
- pixel result reports the resolved window_point / screen_point. Background \
- delivery first resolves the accessible under the point and fires its action \
- (the result says `hit: <role> \"<name>\" action=...`; a file / list item is \
- selected through its container: `selected: ...`), else presses the real \
- virtual pointer there. A click that opens a popup menu names it (`popup: \
- {window_id, bounds}`): pass that window_id to get_window_state to index its \
- items. The token is stale once the next get_window_state of the same \
- window replaces its snapshot — \
- re-snapshot every turn before clicking.\n\n\
- After a zoom call, pass from_zoom=true to auto-translate zoom-image coords \
- back to full-window space.\n\n\
- `count`: 1 (default), 2 for a double-click, 3 for a triple-click (selects a \
- line / paragraph in editors) — one press train with real double-click cadence, \
- in background and foreground alike; there is no separate triple_click tool.\n\n\
- button: \"left\" (default), \"right\", or \"middle\". Defaults to left so the \
- field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release \
- with the matching button code. Native Wayland: only left-button is supported \
- via the virtual-pointer protocol — right/middle return an error rather than \
- silently degrading to left. `modifier` holds ctrl/shift/alt/super for the \
- click on X11. Native Wayland refuses modified pointer clicks until its input \
- protocol can carry keyboard modifier state.".into(),
- input_schema: json!({
- // `pid` is conditionally required (validated in code: needed for
- // window/element clicks, omitted for windowless scope="desktop"),
- // so it is NOT pinned in `required` — matches the click→[] canon
- // in cua_driver_core::tool_schema.
- "type":"object","required":[],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid)."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "capture_id":{"type":"string","description":"Optional one-shot binding to the exact PNG returned by get_window_state or get_desktop_state. When present, x/y are interpreted in that capture and admitted before native dispatch."},
- // Shape matches the shared button_schema() canon (string +
- // [left,right,middle]); kept inline to carry the Linux/Wayland
- // back-compat prose the click button-schema test asserts on.
- "button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},
- "count":{"type":"integer","minimum":1,"maximum":3,"description":"Number of clicks in one press train: 1 (default), 2 = double-click, 3 = triple-click (line/paragraph selection). There is no separate triple_click tool — pass count: 3 here."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Coordinate frame (default \"window\"). Pass \"desktop\" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates."},
- "coordinate_frame": coordinate_frame_schema(),
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let modifiers: Vec<String> = args.str_array("modifier");
-
- // ── Window-less screen-absolute branch (desktop target) ───────────────
- // x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,
- // desktop-scope click (the Linux peer of the Windows WindowFromPoint /
- // macOS global-HID path). The core registry has already enforced the
- // session policy; this branch validates the explicit action form.
- let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
- let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
- let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
- && args.get("y").map(|v| v.is_number()).unwrap_or(false);
- if args.get("capture_id").is_some() && !has_xy {
- return ToolResult::error("click.capture_id requires both x and y coordinates.")
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- if has_xy && !has_pid && !has_window_id {
- if args.get("scope").and_then(Value::as_str) != Some("desktop") {
- return ToolResult::error(
- "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
- use get_desktop_state to read true screen pixels first."
- .to_string(),
- )
- .with_structured(json!({
- "code": "desktop_coordinate_scope_required",
- "suggestion": "pass scope=desktop",
- }));
- }
- let input = match parse_legacy_click_input(&args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let button = parse_mouse_button(input.button.unwrap_or(ClickButton::Left).as_str());
- let n = input.count.unwrap_or(1);
- if n == 0 {
- return ToolResult::error("click.count must be at least 1.")
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- let n = click_train(u64::from(n));
- let space = match desktop_input_space().await {
- Ok(space) => space,
- Err(error) => {
- return ToolResult::error(format!("desktop-scope click failed: {error}"))
- }
- };
- let (action_x, action_y) = if let Some(capture_id) = args.opt_str("capture_id") {
- match crate::capture_action_frame::admit_desktop_click(
- &self.state.capture_service,
- &args,
- &capture_id,
- input.x,
- input.y,
- ) {
- Ok(point) => point,
- Err(error) => return capture_admission_error(error),
- }
- } else {
- (input.x, input.y)
- };
- let sx = action_x.round() as i32;
- let sy = action_y.round() as i32;
- // Glide the agent-cursor overlay to the click point first (the macOS
- // / Windows desktop paths already do this). Without it the overlay
- // sits idle elsewhere while only the real pointer warps, so a viewer
- // sees the cursor "click somewhere else."
- // The overlay draws in layout coordinates, like the input below.
- let (overlay_x, overlay_y) = space.to_layout(sx, sy);
- reveal_pointer_action_for(
- &self.state,
- &cursor_id,
- f64::from(overlay_x),
- f64::from(overlay_y),
- true,
- )
- .await;
- let r = spawn_blocking_bounded("desktop click", foreground_budget(0), move || {
- if crate::wayland::wayland_input_enabled() {
- if !modifiers.is_empty() {
- anyhow::bail!(
- "modified desktop clicks are unavailable on native Wayland: \
- the virtual-pointer route cannot carry keyboard modifier state"
- );
- }
- crate::wayland::click_desktop(&space, sx, sy, n as u32, button)
- } else {
- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::send_click_xtest_desktop_with_modifiers(
- sx,
- sy,
- button,
- n,
- &modifier_refs,
- )
- }
- })
- .await;
- return match r {
- // Screen-absolute click — never driver-verifiable (no
- // read-back); the caller confirms via screenshot.
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent screen-absolute click at ({sx},{sy}) (desktop scope)."
- ))
- .with_structured(
- json!({ "path": if crate::wayland::wayland_input_enabled() { "wayland_desktop" } else { "xtest_desktop" }, "verified": false, "effect": "unverifiable" }),
- ),
- Ok(Err(e)) if crate::input::foreground::error_code(&e).is_some() => {
- input_error_result(e)
- }
- Ok(Err(e)) => ToolResult::error(format!("desktop-scope click failed: {e}")),
- Err(e) => ToolResult::error(format!("task error: {e}")),
- };
- }
-
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- let count = click_train(args.u64_or("count", 1));
- let isolated_background = isolated_hyprland_background(delivery);
- // Surface 5: reject unknown buttons so a typo can't silently fall through
- // to a left-click. Empty string keeps back-compat with old clients.
- let button_str_raw = args.str_or("button", "left").to_lowercase();
- if !matches!(button_str_raw.as_str(), "" | "left" | "right" | "middle") {
- return ToolResult::error(format!(
- "click: unknown button \"{button_str_raw}\" — expected one of left, right, middle."
- ));
- }
- let button_str = if button_str_raw.is_empty() {
- "left"
- } else {
- button_str_raw.as_str()
- };
- let button = parse_mouse_button(button_str);
-
- // Surface 6: element_token / element_index precedence resolution.
- // We resolve before the legacy `opt_u64("element_index")` branch
- // so a token-only call (no integer arg) still takes the element path.
- let capture_id_arg = args.opt_str("capture_id");
- let window_id_arg = args.opt_u64("window_id");
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "click").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let elem_idx_resolved: Option<usize> = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- Some(*element_index)
- }
- cua_driver_core::element_token::ResolvedElement::None => None,
- };
- let window_id_resolved: Option<u64> = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
- };
-
- if capture_id_arg.is_some() && elem_idx_resolved.is_some() {
- return ToolResult::error(
- "click.capture_id is only valid with x/y coordinates, not an element target.",
- )
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
-
- if let Some(token) = token {
- // The observed element: its proven AT-SPI identity and the frame
- // the snapshot recorded. Every route below acts on that identity
- // (or its cached frame); the live ordinal `idx` is only an
- // address within the snapshot, never used to retarget.
- let idx = token.index;
- let observed = token.observed.clone();
- let xid_hint = window_id_resolved;
- // Resolve the element's screen center + its window FIRST, so the
- // agent cursor glides to the target *before* the click fires —
- // matching the coordinate path below and the macOS/Windows backends.
- // Previously perform_action ran inside this spawn_blocking, so the
- // app updated before the cursor visibly arrived.
- // Bounded: resolving bounds re-walks the AT-SPI tree, which on a
- // large GTK/VCL tree can stall for minutes; a missing placement
- // degrades to the AX action (background) or a refusal (foreground).
- let placed = token.clone();
- let placement = tokio::time::timeout(
- ELEMENT_AX_BUDGET,
- tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
- let (cx, cy) = element_screen_center(pid, &placed, xid_hint)?;
- let xid = xid_hint
- .or_else(|| {
- crate::x11::list_windows(Some(pid))
- .into_iter()
- .next()
- .map(|w| w.xid)
- })
- .unwrap_or(0);
- Ok((xid, cx, cy))
- }),
- )
- .await;
- let placement = match placement {
- Ok(Ok(placement)) => placement,
- Ok(Err(error)) => Err(anyhow::anyhow!("task error: {error}")),
- Err(_elapsed) => Err(anyhow::anyhow!("element placement timed out")),
- };
- let placement = match element_click_placement(placement, xid_hint) {
- Ok(placement) => placement,
- Err(refusal) => return refusal,
- };
- // X11 foreground element click: behave like a user — activate the
- // window and XTest-click the element's screen centre. No AT-SPI
- // action, no second tree walk. Element bounds come from the
- // (bounded) placement above.
- if delivery.is_foreground()
- && !crate::wayland::wayland_input_enabled()
- && !hyprland_foreground(delivery)
- {
- return self
- .foreground_element_click(
- &cursor_id, pid, idx, &observed, xid_hint, placement, button, count,
- &modifiers,
- )
- .await;
- }
- if let Some((xid, sx, sy)) = placement {
- if xid != 0 {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- }
- reveal_pointer_action_target_for(
- &self.state,
- &cursor_id,
- sx,
- sy,
- true,
- frame_target_rect(observed.bounds, sx, sy),
- )
- .await;
- }
-
- // Chromium can execute a genuine AT-SPI action without focus. Try
- // that route before applying its background synthetic-input gate.
- // Plain Hyprland foreground clicks also retain semantic activation
- // for offscreen controls; other pointer gestures need raw input.
- let foreground_hyprland = hyprland_foreground(delivery);
- if foreground_hyprland && window_id_resolved.is_none() {
- return foreground_hyprland_refusal(
- "an exact window_id or window-bound element token is required",
- );
- }
- // An AT-SPI action can open a menu or press a button, but it cannot
- // give an entry / spin button / slider the widget focus a following
- // type_text needs (GIMP spin scales, VS Code settings inputs). With
- // the MPX real pointer available, click those like a user would.
- let needs_real_click = element_needs_real_click(&observed.role);
- // A menu / menu item's `doAction` is declined or silently ignored
- // by LibreOffice VCL (and opens GTK menus without the pointer
- // grab a following item click expects): with a real pointer, press
- // it like a user would and let the popup be listed. That holds
- // for an entry that is on screen (SHOWING: a menubar entry, an
- // item of an open GTK3 / VCL popup). A menu item that is not
- // showing has no point to press: gail (GTK2, GIMP) never marks
- // menu items SHOWING, open or closed, and an item of a closed
- // menu is still reached through its AT-SPI `click` action (the
- // same rule the foreground route applies), so those keep the
- // action route.
- let real_pointer = !delivery.is_foreground()
- && !crate::wayland::wayland_input_enabled()
- && crate::input::real_pointer_input_available();
- let menu_entry_on_screen = real_pointer && element_is_menu_role(&observed.role) && {
- // A bounded D-Bus round-trip: off the runtime thread.
- let observed_for_state = observed.clone();
- let showing = tokio::task::spawn_blocking(move || {
- crate::atspi::element_showing_observed(&observed_for_state)
- })
- .await;
- !matches!(showing, Ok(Ok(false)))
- };
- let real_click_role = real_pointer && (needs_real_click || menu_entry_on_screen);
- // Without a real pointer, an editable / table cell cannot take a
- // background click that gives it focus: refuse instead of firing
- // an AT-SPI action whose effect is not a click.
- if !delivery.is_foreground()
- && !crate::wayland::is_wayland()
- && needs_real_click
- && !crate::input::real_pointer_input_available()
- {
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- return crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- }
- if !real_click_role
- && element_click_prefers_ax(
- foreground_hyprland,
- button,
- count,
- !modifiers.is_empty(),
- )
- {
- let observed_for_ax = observed.clone();
- let guard_pid = (!delivery.is_foreground()).then_some(pid);
- let ax_result = match tokio::time::timeout(
- ELEMENT_AX_BUDGET,
- tokio::task::spawn_blocking(move || {
- // Background: an AT-SPI action can open a menu (GTK/VCL
- // grab the keyboard and activate the toplevel) or map a
- // dialog mutter then focuses. Snapshot, act, restore.
- match guard_pid {
- Some(pid) => crate::input::focus_guard::guarded(Some(pid), || {
- crate::atspi::perform_action_observed(&observed_for_ax)
- }),
- None => crate::atspi::perform_action_observed(&observed_for_ax)
- .map(|value| (value, None)),
- }
- }),
- )
- .await
- {
- Ok(joined) => joined,
- Err(_elapsed) => {
- return ToolResult::error(format!(
- "click: the AT-SPI action for element [{idx}] did not complete within {ELEMENT_AX_BUDGET:?}; its effect is unknown. Take a screenshot, then retry by pixel (x/y) or with delivery_mode:\"foreground\"."
- ))
- .with_structured(json!({
- "code": "ax_timeout",
- "effect": "unknown",
- "escalation": non_ax_escalation(),
- }));
- }
- };
- if let Ok(Ok(((_action, suspected_noop, unacknowledged), guard))) = ax_result {
- let mut structured = json!({
- "path": "ax",
- "verified": false,
- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },
- });
- if suspected_noop {
- structured["escalation"] = non_ax_escalation();
- } else if unacknowledged {
- structured["note"] = json!(
- "no acknowledgement received for the doAction call within the \
- AT-SPI timeout; its effect is unknown"
- );
- }
- return attach_focus_guard(
- ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))
- .with_structured(structured),
- guard.as_ref(),
- );
- }
- // The observed object is gone: the snapshot is stale. Refuse
- // rather than retarget whatever now lives at that index.
- if let Ok(Err(error)) = &ax_result {
- if error.is::<crate::atspi::native::CachedElementGone>() {
- return ToolResult::error(format!(
- "AT-SPI element click failed: stale_element_token: observed \
- AT-SPI object [{idx}] is no longer present ({error}); \
- re-snapshot with get_window_state"
- ))
- .with_structured(json!({
- "code": "stale_element_token",
- "effect": "none",
- }));
- }
- }
- // AT-SPI errors can arrive after dispatch. Do not introduce a
- // primary-seat replay when its semantic outcome is uncertain.
- if foreground_hyprland {
- return foreground_ax_click_unknown(pid, idx);
- }
- }
- if foreground_hyprland {
- if !modifiers.is_empty() {
- return foreground_hyprland_refusal("modified clicks are unsupported");
- }
- let Some(xid) = window_id_resolved else {
- return foreground_hyprland_refusal(
- "an exact window_id or window-bound element token is required",
- );
- };
- let point = tokio::task::spawn_blocking(move || {
- resolve_element_local_coords(pid, &token, Some(xid))
- })
- .await;
- let (x, y) = match point {
- Ok(Ok((_, x, y))) => (x, y),
- _ => {
- return foreground_hyprland_refusal(
- "could not resolve exact element coordinates",
- )
- }
- };
- return foreground_hyprland_action(
- &args,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Click {
- x,
- y,
- button: u32::from(button),
- count,
- },
- )
- .await;
- }
- if isolated_background {
- if !modifiers.is_empty() {
- return isolated_hyprland_refusal(
- "modified element clicks are unavailable on native Wayland: \
- isolated input does not carry pointer modifier state",
- );
- }
- let Some(exact_xid) = window_id_resolved else {
- return isolated_hyprland_refusal(
- "an exact window_id or window-bound element token is required",
- );
- };
- let owner = named_session_cursor_key(&args);
- let (_cancellation, dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- let (_, x, y) = resolve_element_local_coords(pid, &token, Some(exact_xid))?;
- crate::wayland::hyprland_input::execute(
- owner,
- pid,
- exact_xid,
- crate::wayland::hyprland_input::Action::Click {
- x,
- y,
- button: u32::from(button),
- count,
- },
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(refusal) => return refusal,
- };
- return match dispatch.await {
- Ok(result) => isolated_hyprland_result(result),
- Err(error) => isolated_hyprland_task_error(error, false),
- };
- }
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
-
- // The AX route was unavailable. Fall back to a real MPX pointer
- // click at the element (no focus steal), then to a target-addressed
- // X11 event for toolkits that accept it.
- let cursor_id_for_fallback = cursor_id.clone();
- let result = spawn_blocking_bounded(
- "element click",
- foreground_budget(0),
- move || -> anyhow::Result<PointerRoute> {
- // Refuse before resolving coordinates: the refusal does not
- // depend on where the element is, and a window without
- // compositor geometry must not turn it into a lookup error.
- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {
- anyhow::bail!(
- "modified element clicks are unavailable on native Wayland: \
- the pointer route cannot carry keyboard modifier state"
- );
- }
- let (xid2, lx, ly) = checked_element_local_coords(pid, &token, xid_hint)?;
- // An explicit X11 foreground request needs the real XTest path
- // even for a plain click. Some native widgets (notably GTK
- // selectable rows) expose bounds but no AT-SPI Action and
- // ignore a targeted XSendEvent; limiting XTest to modified
- // clicks made those rows addressable but not selectable.
- if delivery.is_foreground() && !crate::wayland::wayland_input_enabled() {
- let (_, sx, sy) = placement
- .ok_or_else(|| anyhow::anyhow!("element screen bounds unavailable"))?;
- crate::interaction::with_x11_foreground(
- xid2,
- crate::input::ForegroundOptions::pointer(),
- || {
- crate::input::send_click_xtest_desktop_with_modifiers(
- sx.round() as i32,
- sy.round() as i32,
- button,
- count,
- &modifier_refs,
- )
- },
- )
- .map(|(value, _report)| value)
- .map(|()| PointerRoute::Foreground)
- } else if modifier_refs.is_empty() && !crate::wayland::wayland_input_enabled() {
- x11_pixel_click_no_focus_steal(
- &cursor_id_for_fallback,
- xid2,
- lx as i32,
- ly as i32,
- button,
- count,
- )
- } else {
- crate::input::send_click_with_modifiers(
- xid2,
- lx as i32,
- ly as i32,
- count,
- button,
- &modifier_refs,
- )
- .map(|()| PointerRoute::Synthetic)
- }
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- return match result {
- // The MPX route reports its cheap post-checks (screen region
- // around the element, focus untouched); the other routes stay
- // unverifiable and the caller confirms via screenshot.
- Ok(Ok(route)) => {
- attach_focused_cell(
- ToolResult::text(format!(
- "Clicked element [{idx}] (pid {pid}) with a real pointer click {}",
- route.text_suffix(mode_label)
- ))
- .with_structured(route.structured(mode_label)),
- pid,
- )
- .await
- }
- Ok(Err(e)) => input_error_result(e.context("AT-SPI element click failed")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- if !isolated_background {
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- }
-
- // Coordinate-based path.
- if let Some(refusal) = require_point_args("click", &args) {
- return refusal;
- }
- let xid = match args.opt_u64("window_id") {
- Some(v) => v,
- None => return ToolResult::error("Provide either element_token or window_id + x/y."),
- };
- let desktop_frame = desktop_frame_requested(&args);
- let from_zoom = args.bool_or("from_zoom", false);
- let mut x = args.f64_or("x", 0.0);
- let mut y = args.f64_or("y", 0.0);
- if capture_id_arg.is_some() && from_zoom {
- return ToolResult::error(
- "click.capture_id cannot be combined with from_zoom; use coordinates from the published capture.",
- )
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- if capture_id_arg.is_some() && desktop_frame {
- return ToolResult::error(
- "click.capture_id cannot be combined with desktop-frame coordinates; use coordinates from the published capture.",
- )
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- let native_refusal = (!isolated_background)
- .then(|| unavailable_wayland_focused_input_background(delivery, true))
- .flatten();
- if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated once below.
- if let Some(refusal) = native_refusal {
- return refusal;
- }
- } else if let Some(capture_id) = capture_id_arg {
- if let Some(refusal) = native_refusal {
- return refusal;
- }
- match crate::capture_action_frame::admit_window_click(
- &self.state.capture_service,
- &args,
- &capture_id,
- pid,
- xid,
- x,
- y,
- ) {
- Ok(point) => (x, y) = point,
- Err(error) => return capture_admission_error(error),
- }
- } else {
- let context = match coordinate_click_context(native_refusal, || {
- if from_zoom {
- self.state
- .zoom_context(&args, pid, Some(xid))
- .map(CoordinateContext::Zoom)
- } else {
- screenshot_scale(&self.state, &args, pid, Some(xid))
- .map(CoordinateContext::Screenshot)
- }
- }) {
- Ok(context) => context,
- Err(refusal) => return refusal,
- };
- match context {
- CoordinateContext::Zoom(context) => {
- (x, y) = context.zoom_to_window(x, y);
- }
- CoordinateContext::Screenshot(scale) => {
- x *= scale;
- y *= scale;
- }
- }
- }
- if desktop_frame {
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
- Ok(Ok((lx, ly))) => {
- x = lx;
- y = ly;
- }
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
-
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- // Resolve the screen point the cursor glides to. Tool coordinates are
- // always window-local screenshot pixels; native Wayland translates
- // through compositor/AT-SPI geometry while X11 uses XTranslateCoordinates.
- let wayland_output_point = if crate::wayland::wayland_input_enabled() {
- match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
- Ok(point) => Some(point),
- Err(error) => return input_error_result(error),
- }
- } else {
- None
- };
- let glide_target = if let Some(AimedPoint {
- point: (sx, sy), ..
- }) = wayland_output_point
- {
- Some((sx as f64, sy as f64))
- } else {
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
- .await
- .ok()
- .and_then(|r| r.ok())
- };
- if let Some((sx, sy)) = glide_target {
- reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
- }
-
- let (xi, yi) = (x as i32, y as i32);
- let pixel_screen_point =
- glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32));
- let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
- let (output_x, output_y) = aimed.point;
- if hyprland_foreground(delivery) {
- if !modifiers.is_empty() {
- return foreground_hyprland_refusal("modified clicks are unsupported");
- }
- if window_id_resolved.is_none() {
- return foreground_hyprland_refusal("an exact window_id is required");
- }
- return foreground_hyprland_action(
- &args,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Click {
- x,
- y,
- button: u32::from(button),
- count,
- },
- )
- .await;
- }
- if isolated_background {
- if !modifiers.is_empty() {
- return isolated_hyprland_refusal(
- "modified clicks are unsupported by isolated input",
- );
- }
- if button == 1 && count == 1 {
- let semantic = tokio::task::spawn_blocking(move || {
- crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
- })
- .await;
- if matches!(semantic, Ok(Ok(Some(_)))) {
- return ToolResult::text("Dispatched AT-SPI click.").with_structured(json!({
- "path": "wayland_atspi", "verified": false, "effect": "unverifiable"
- }));
- }
- }
- return isolated_hyprland_action(
- &args,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Click {
- x,
- y,
- button: u32::from(button),
- count,
- },
- )
- .await;
- }
- let cursor_id_for_task = cursor_id.clone();
- let modifiers_for_task = modifiers.clone();
- // delivery_mode: background (default) = no-focus-steal injection;
- // foreground = activate the target window (EWMH) first, then inject,
- // then restore prior active. Mirrors macOS/Windows.
- let fg_budget = foreground_budget(0);
- let result = spawn_blocking_bounded("foreground click", fg_budget, move || -> anyhow::Result<(&'static str, Option<crate::input::ForegroundReport>, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {
- if crate::wayland::wayland_input_enabled() {
- if !modifiers_for_task.is_empty() {
- anyhow::bail!(
- "modified coordinate clicks are unavailable on native Wayland: \
- the pointer route cannot carry keyboard modifier state"
- );
- }
- // Vision/pixel click on native Wayland. Mutter drops synthetic
- // virtual-pointer events (the `wayland::click` warp doesn't land),
- // so for a plain left single click resolve the screen pixel to the
- // covering accessible element and fire its action by
- // `element_index` — the coordinate-free path already verified
- // working. (x,y) are screen coords here, matching the frames in
- // `get_window_state`. Miss → fall through to the injection paths.
- if !delivery.is_foreground() && button == 1 && count == 1 {
- if let Ok(Some(_)) =
- crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
- {
- return Ok(("wayland_atspi", None, None, None));
- }
- }
- if crate::wayland::is_inject_mode() {
- crate::wayland::inject_click(pid, xid, x, y, count as u32, button)?;
- return Ok(("wayland_cua_compositor", None, None, None));
- }
- if !delivery.is_foreground() {
- return Ok(("background_unavailable", None, None, None));
- }
- // Native Wayland: focus+raise the target toplevel
- // (foreign-toplevel `activate`), then drive `count` virtual-pointer
- // button events. Wayland injection routes to the compositor focus.
- crate::interaction::with_target_foreground(pid, xid, crate::interaction::RestoreMode::RestorePrevious, || {
- crate::wayland::click_focused(aimed, count as u32, button)
- })?;
- return Ok(("wayland_activate", None, None, None));
- }
- // X11 injection. Tiered no-focus-steal delivery (background):
- // 1. Plain left single-click → AT-SPI doAction at that point.
- // 2. Right / middle / double click → real MPX uinput pointer + XI2
- // shield grab (real Xorg only; skipped on Xvfb/Wayland).
- // 3. Fallback → synthetic XSendEvent.
- // Foreground skips the AT-SPI shortcut and does a real activated pixel
- // click (the agent's escalation when background didn't land).
- let inject = |fg: bool| -> anyhow::Result<(&'static str, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {
- if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {
- // The accessible action under the point may open a menu or
- // a dialog that takes the focus: guard and restore. With a
- // real pointer available, an entry / spin button / cell
- // under the point is left to the MPX click below (its
- // `doAction` would not focus it for a following type_text).
- let real_click_for_focus_roles =
- crate::input::real_pointer_input_available();
- let (hit, guard) = crate::input::focus_guard::guarded(Some(pid), || {
- Ok(crate::atspi::perform_action_at_point_in(
- pid,
- xid,
- xi,
- yi,
- real_click_for_focus_roles,
- )
- .ok()
- .flatten())
- })?;
- if let Some(hit) = hit {
- return Ok(("x11_atspi", Some(PointerRoute::Atspi(hit)), guard));
- }
- }
- if !fg
- && !crate::input::real_pointer_input_available()
- && synthetic_pointer_is_dropped(pid)
- {
- // No accessible actuator covers the point and the only
- // route left is a synthetic XSendEvent that this toolkit
- // (GTK3/4 XInput2, LibreOffice VCL) discards. Say so
- // instead of reporting a click that changed nothing.
- return Ok(("background_unavailable_pointer", None, None));
- }
- if fg {
- // Foreground: the window is already activated. Deliver a REAL
- // XTest warp+button click. Synthetic XSendEvent button events
- // are dropped by GTK/Qt/Chromium/Firefox, and the MPX uinput
- // path needs /dev/uinput, which headless X servers
- // (Xvfb/Xtigervnc) lack — so neither focuses the clicked
- // widget. XTest is accepted as real input and gives the
- // widget keyboard focus, so a following type lands.
- if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64) {
- let modifier_refs: Vec<&str> =
- modifiers_for_task.iter().map(String::as_str).collect();
- crate::input::send_click_xtest_desktop_with_modifiers(
- sx.round() as i32,
- sy.round() as i32,
- button,
- count,
- &modifier_refs,
- )?;
- return Ok(("x11_xtest_fg", None, None));
- }
- }
- if modifiers_for_task.is_empty() {
- // The MPX press train runs under its own (settled) focus
- // guard inside the helper; its report rides on the route.
- let route = x11_pixel_click_no_focus_steal(
- &cursor_id_for_task,
- xid,
- xi,
- yi,
- button,
- count,
- )?;
- return Ok((route.path().unwrap_or("x11_pixel"), Some(route), None));
- } else {
- let modifier_refs: Vec<&str> =
- modifiers_for_task.iter().map(String::as_str).collect();
- x11_pixel_click_no_focus_steal_modifiers(
- xid,
- xi,
- yi,
- button,
- count,
- &modifier_refs,
- )?;
- }
- Ok((if fg { "x11_pixel_fg" } else { "x11_pixel" }, None, None))
- };
- if delivery.is_foreground() {
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || inject(true),
- )
- .map(|((path, route, guard), report)| (path, Some(report), route, guard))
- } else {
- inject(false).map(|(path, route, guard)| (path, None, route, guard))
- }
- })
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- let outcome = match result {
- Ok(Ok(("background_unavailable", _, _, _))) => {
- crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- )
- }
- Ok(Ok(("background_unavailable_pointer", _, _, _))) => {
- let mut refusal = crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- let hint = "No accessible control covers this point and the toolkit drops \
- synthetic pointer events, so a background pixel click here would \
- change nothing. Use get_window_state and click by element_token \
- (AT-SPI action), or retry with delivery_mode='foreground'.";
- refusal
- .content
- .push(cua_driver_core::protocol::Content::text(hint));
- if let Some(structured) = refusal.structured_content.as_mut() {
- structured["hint"] = json!(hint);
- structured["path"] = json!("background_unavailable_pointer");
- }
- refusal
- }
- // A pixel/coordinate click is never driver-verifiable (no read-back) —
- // verified:false, effect:"unverifiable"; the caller confirms via
- // screenshot. path reports the rung taken; a foreground click also
- // reports the activation/focus transaction it confirmed first.
- Ok(Ok((path, report, route, guard))) => {
- let mut structured = match &route {
- Some(route) => route.structured(mode_label),
- None => json!({
- "path": path,
- "verified": false,
- "effect": "unverifiable",
- "delivery_mode": mode_label,
- }),
- };
- if let Some(report) = report {
- apply_foreground_report(&mut structured, &report);
- }
- let suffix = match &route {
- Some(route) => route.text_suffix(mode_label),
- None => format!(
- "(delivery_mode={mode_label}, path={path}); not verified — confirm \
- with a screenshot."
- ),
- };
- attach_focus_guard(
- ToolResult::text(format!("Clicked at ({x:.1}, {y:.1}) × {count} {suffix}"))
- .with_structured(structured),
- guard.as_ref(),
- )
- }
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- let outcome = attach_focused_cell(outcome, pid).await;
- with_pixel_points(outcome, pixel_screen_point, (xi, yi))
- }
-}
-
-// ── px-focus helper (keyboard family) ───────────────────────────────────────
-
-/// px-focus for the keyboard family (type_text / press_key / hotkey): pixel-click
-/// at (x,y) to establish real renderer focus before a keystroke — the *element px
-/// action* form of a keyboard tool. Reuses ClickTool's exact coordinate
-/// translation + delivery_mode so it lands on the same pixel a px-click would.
-/// `Ok(())` on success; `Err(ToolResult)` short-circuits the caller.
-///
-/// Retains the parent's authenticated session admission for the nested click.
-/// `point` is the window-local pixel to click for focus.
-async fn focus_by_pixel(
- state: &Arc<ToolState>,
- pid: u32,
- window_id: Option<u64>,
- (x, y): (f64, f64),
- foreground: bool,
- parent_args: &Value,
- from_zoom: bool,
-) -> Result<(), ToolResult> {
- let mut click_args = json!({
- "pid": pid, "x": x, "y": y,
- "delivery_mode": if foreground { "foreground" } else { "background" },
- });
- if let Some(wid) = window_id {
- click_args["window_id"] = json!(wid);
- }
- for field in [
- "session",
- "_session_id",
- "_transport_session_id",
- "cursor_id",
- "scope",
- "coordinate_frame",
- ] {
- if let Some(value) = parent_args.get(field) {
- click_args[field] = value.clone();
- }
- }
- if from_zoom {
- click_args["from_zoom"] = json!(true);
- }
- let focus = ClickTool {
- state: state.clone(),
- }
- .invoke(click_args)
- .await;
- if focus.is_error == Some(true) {
- return Err(focus);
- }
- // Brief settle so the renderer registers focus before the keystrokes.
- tokio::time::sleep(std::time::Duration::from_millis(120)).await;
- Ok(())
-}
-
-/// Establish widget-local focus inside a nested-compositor target without
-/// changing the compositor's focused toplevel. AX uses Component.GrabFocus;
-/// PX sends a private per-surface left click at the requested local point.
-async fn focus_nested_inject_target(
- pid: u32,
- window_id: u64,
- element: Option<crate::atspi::TokenElement>,
- pixel: Option<(f64, f64)>,
-) -> Result<(), ToolResult> {
- if let Some(element) = element {
- let index = element.index;
- return match tokio::task::spawn_blocking(move || crate::atspi::focus_token(pid, &element))
- .await
- {
- Ok(Ok(true)) => Ok(()),
- Ok(Ok(false)) => Err(ToolResult::error(format!(
- "AT-SPI Component.GrabFocus returned false for element {index}"
- ))),
- Ok(Err(error)) => Err(stale_observed_element("focus", index, &error)
- .unwrap_or_else(|| ToolResult::error(error.to_string()))),
- Err(error) => Err(ToolResult::error(format!("Task error: {error}"))),
- };
- }
- if let Some((x, y)) = pixel {
- return match tokio::task::spawn_blocking(move || {
- crate::wayland::inject_click(pid, window_id, x, y, 1, 1)
- })
- .await
- {
- Ok(Ok(())) => Ok(()),
- Ok(Err(error)) => Err(ToolResult::error(error.to_string())),
- Err(error) => Err(ToolResult::error(format!("Task error: {error}"))),
- };
- }
- Ok(())
-}
-
-// ── type_text ─────────────────────────────────────────────────────────────────
-
-pub struct TypeTextTool {
- state: Arc<ToolState>,
-}
-static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for TypeTextTool {
- fn def(&self) -> &ToolDef {
- TYPE_DEF.get_or_init(|| ToolDef {
- name: "type_text".into(),
- description: "Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.".into(),
- input_schema: json!({
- "type":"object","required":["text"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to type into the focused application."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y; optional with element_token (the token carries it)."},
- "text":{"type":"string","description":"Text to type."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\") — a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget."},
- "y":{"type":"number","description":"Pixel Y of the field (see x)."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to type into the focused application. Default \"window\"."},
- "coordinate_frame": coordinate_frame_schema(),
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if (args.opt_str("scope").as_deref() == Some("desktop")
- || coordinate_frame_is_desktop(&args))
- && args.get("pid").is_none()
- {
- let input = match parse_typed_projection::<TypeTextInput>("type_text", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let text =
- cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&input.text)
- .into_owned();
- position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
- .await;
- let wayland = crate::wayland::wayland_input_enabled();
- let path = if wayland { "wayland_focused" } else { "xtest" };
- let result = spawn_blocking_bounded(
- "desktop type_text",
- typing_budget(text.chars().count(), crate::wayland::niri::is_session()),
- move || {
- if wayland {
- crate::wayland::type_text_focused(&text)
- } else {
- crate::input::send_type_text_xtest(&text)
- }
- },
- )
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text("Typed text into the focused desktop application.")
- .with_structured(
- json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let mut pid = args.u64_or("pid", 0) as u32;
- let text_raw = match args.require_str("text") {
- Ok(v) => v,
- Err(e) => return e,
- };
- // Strip trailing agent-protocol closing tags — see
- // cua_driver_core::text_sanitize docs for rationale.
- let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
- .into_owned();
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "type_text").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let (resolved_elem_idx, resolved_window_id) = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element {
- element_index,
- window_id,
- ..
- } => (Some(*element_index), Some(*window_id)),
- cua_driver_core::element_token::ResolvedElement::None => (None, None),
- };
- let xid_opt = resolved_window_id.or(args.opt_u64("window_id"));
-
- // Resolve XID: use window_id if given, else first window for pid.
- let xid = match xid_opt {
- Some(x) => x,
- None => {
- let windows = tokio::task::spawn_blocking(move || {
- crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
- })
- .await
- .unwrap_or_default();
- // pid omitted: the keys go to the active window, like a
- // physical keyboard would, and the action adopts its pid.
- let chosen = if pid == 0 {
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active && w.pid.is_some())
- .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
- } else {
- // The pid's active window, else its largest on-screen
- // toplevel: LibreOffice and GIMP own hidden/utility
- // toplevels that a plain `first()` could pick, and an
- // unmapped window cannot take the virtual keyboard focus.
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active)
- .or_else(|| {
- windows
- .iter()
- .filter(|w| w.is_on_screen)
- .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
- })
- .or_else(|| windows.first())
- };
- match chosen {
- Some(w) => {
- if pid == 0 {
- pid = w.pid.unwrap_or(0);
- }
- w.xid
- }
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- if isolated_hyprland_background(delivery)
- && resolved_elem_idx.is_none()
- && args.get("x").is_none()
- && args.get("y").is_none()
- {
- if xid_opt.is_none() {
- return isolated_hyprland_refusal(
- "an exact window_id is required for isolated text",
- );
- }
- match crate::wayland::hyprland_input::text_actions(&text) {
- Ok(actions) if !actions.is_empty() => {}
- Ok(_) => return isolated_hyprland_refusal("background text must not be empty"),
- Err(error) => return isolated_hyprland_refusal(error.to_string()),
- }
- position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, true)
- .await;
- let owner = named_session_cursor_key(&args);
- let (_cancellation, dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- crate::wayland::hyprland_input::execute_background_text(
- owner,
- pid,
- xid,
- &text,
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(refusal) => return refusal,
- };
- return match dispatch.await {
- Ok(result) => isolated_hyprland_result(result),
- Err(error) => isolated_hyprland_task_error(error, false),
- };
- }
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- if resolved_elem_idx.is_none() {
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
- return refusal;
- }
- }
-
- let px = args.get("x").and_then(|value| value.as_f64());
- let py = args.get("y").and_then(|value| value.as_f64());
- if px.is_some() != py.is_some() {
- return invalid_pointer_arguments(
- "type_text",
- "x and y must be passed together (a pixel focus point), or neither",
- );
- }
- if px.is_some() && resolved_elem_idx.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to type_text, not both.",
- );
- }
-
- if hyprland_foreground(delivery) {
- if xid_opt.is_none() {
- return foreground_hyprland_refusal("an exact window_id is required");
- }
- // Native editables retain their verifiable, addressed AT-SPI route.
- if !is_chromium_embedder(pid) && !is_webkitgtk_embedder(pid) {
- if let Some(element) = token.clone() {
- let text_ax = text.clone();
- if matches!(
- tokio::task::spawn_blocking(move || crate::atspi::type_into_token(
- pid, &element, &text_ax
- ))
- .await,
- Ok(Ok(()))
- ) {
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- None,
- true,
- )
- .await;
- return type_text_ax_result(
- pid,
- text.chars().count(),
- "via targeted AT-SPI",
- );
- }
- }
- }
- match crate::wayland::hyprland_input::text_actions(&text) {
- Ok(actions) if !actions.is_empty() => {}
- Ok(_) => return foreground_hyprland_refusal("foreground text must not be empty"),
- Err(error) => return foreground_hyprland_refusal(error.to_string()),
- }
- if let Err(error) =
- focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
- .await
- {
- return error;
- }
- if named_session_cursor_key(&args).is_some() {
- if px.is_none() {
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- None,
- true,
- )
- .await;
- }
- } else {
- announce_keyboard_target(&args, pid, xid, token.clone(), px.zip(py)).await;
- }
- let owner = named_session_cursor_key(&args);
- let (_cancellation, dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- crate::wayland::hyprland_input::execute_foreground_text(
- owner,
- pid,
- xid,
- &text,
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(_) => {
- return foreground_hyprland_refusal(
- "authenticated admitted lifecycle required",
- )
- }
- };
- return hyprland_input_result(
- match dispatch.await {
- Ok(result) => result,
- Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
- error.into(),
- 0,
- )),
- },
- true,
- );
- }
-
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- px.zip(py),
- true,
- )
- .await;
-
- let text_len = text.chars().count();
- // Native toolkit editables have a stronger focus-free route than raw
- // compositor keyboard injection. Keep Chromium/WebKit on real key events
- // because their accessibility bridges may echo a write that never reaches
- // renderer-owned state.
- if let Some(element) = token.clone().filter(|_| {
- crate::wayland::is_inject_mode()
- && !is_chromium_embedder(pid)
- && !is_webkitgtk_embedder(pid)
- }) {
- let text_at = text.clone();
- let target = element.clone();
- let targeted = tokio::task::spawn_blocking(move || {
- crate::atspi::type_into_token(pid, &target, &text_at)
- })
- .await;
- if let Ok(Ok(())) = targeted {
- return type_text_ax_result_verified(pid, element, &text, "via targeted AT-SPI")
- .await;
- }
- }
- // The private nested compositor can target the owning Wayland client
- // directly. Establish widget-local focus first, without changing the
- // compositor's focused toplevel, so keys reach the addressed control.
- if crate::wayland::is_inject_mode() {
- if let Err(error) =
- focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
- {
- return error;
- }
- let text_w = text.clone();
- let result = tokio::task::spawn_blocking(move || {
- crate::wayland::inject_type_text(pid, xid, &text_w)
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) (focus-free via cua-compositor)."
- ))
- .with_structured(type_text_structured(
- "key_events",
- text_len,
- false,
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // ── px form: focus by pixel-click, then type into the now-focused element ──
- // Pass x,y (no element_token) for an *element px action*: pixel-click
- // the field to give the renderer the real keyboard focus the AT-SPI path
- // can't, then fall through to the focused-element type path below (it
- // escalates AT-SPI → key events and lands once focused). Reuses ClickTool's
- // exact coordinate translation + delivery_mode.
- if let (Some(cx), Some(cy)) = (px, py) {
- if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
- return refusal;
- }
- let from_zoom = args.bool_or("from_zoom", false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- Some(xid),
- (cx, cy),
- delivery.is_foreground(),
- &args,
- from_zoom,
- )
- .await
- {
- return e;
- }
- // resolved_elem_idx stays None → the type path below writes to the now-
- // focused element via the background key / AT-SPI rung.
- }
-
- if resolved_elem_idx.is_some() {
- if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
- return refusal;
- }
- }
-
- // Renderer EditableText writes can update the accessible value without
- // emitting the DOM input event. For an explicit foreground request on
- // native Wayland, focus the named field and send real keyboard input so
- // Chromium/WebKit observe the same event sequence as a user.
- if delivery.is_foreground()
- && crate::wayland::wayland_input_enabled()
- && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid))
- {
- if let Some(element) = token.clone() {
- let idx = element.index;
- let text_w = text.clone();
- let result = spawn_blocking_bounded(
- "type_text",
- typing_budget(text_w.chars().count(), crate::wayland::niri::is_session()),
- move || {
- crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::RestoreMode::RestorePrevious,
- || {
- if !crate::atspi::focus_token(pid, &element)? {
- anyhow::bail!(
- "AT-SPI Component.GrabFocus returned false for element {idx}"
- );
- }
- crate::wayland::type_text_focused(&text_w)
- },
- )
- },
- )
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) (via Wayland virtual-keyboard)."
- ))
- .with_structured(type_text_structured(
- "key_events",
- text_len,
- false,
- )),
- Ok(Err(error)) => stale_observed_element("type_text", idx, &error)
- .unwrap_or_else(|| ToolResult::error(error.to_string())),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- }
-
- // AX addressing names one exact editable. Try this focus-free route
- // before native Wayland keyboard injection, which can only target the
- // compositor's globally focused surface.
- if let Some(element) = token.clone() {
- let text_at = text.clone();
- let target = element.clone();
- let targeted = tokio::task::spawn_blocking(move || {
- crate::atspi::type_into_token(pid, &target, &text_at)
- })
- .await;
- if let Ok(Err(error)) = &targeted {
- // The observed object is gone: nothing was written to it.
- if let Some(stale) = stale_observed_element("type_text", element.index, error) {
- return stale;
- }
- }
- match targeted {
- Ok(Ok(())) => {
- return type_text_ax_result_verified(
- pid,
- element,
- &text,
- "via targeted AT-SPI",
- )
- .await;
- }
- Ok(Err(_)) | Err(_)
- if !delivery.is_foreground() && crate::wayland::wayland_input_enabled() =>
- {
- return crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- }
- _ => {}
- }
- }
-
- // Native Wayland: keys go to the *focused* surface (no pid/window
- // targeting in the protocol). Type via the virtual-keyboard tool; pair
- // with a prior `click`/`activate` to focus the intended window.
- if crate::wayland::wayland_input_enabled() {
- if !delivery.is_foreground() {
- return crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- }
- let text_w = text.clone();
- let element = token.clone();
- let result = spawn_blocking_bounded(
- "type_text",
- typing_budget(text_w.chars().count(), crate::wayland::niri::is_session()),
- move || {
- crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::RestoreMode::RestorePrevious,
- || {
- if let Some(element) = &element {
- if !crate::atspi::focus_token(pid, element)? {
- anyhow::bail!(
- "AT-SPI Component.GrabFocus returned false for element {}",
- element.index
- );
- }
- }
- crate::wayland::type_text_focused(&text_w)
- },
- )
- },
- )
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) (via Wayland virtual-keyboard)."
- ))
- .with_structured(type_text_structured(
- "key_events",
- text_len,
- false,
- )),
- Ok(Err(e)) => resolved_elem_idx
- .and_then(|idx| stale_observed_element("type_text", idx, &e))
- .unwrap_or_else(|| ToolResult::error(e.to_string())),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Terminal short-circuit: when the target window's WM_CLASS marks it
- // as a terminal emulator (Ghostty / Alacritty / kitty / …), skip the
- // AT-SPI path entirely. Terminals expose an editable text area that
- // AT-SPI `insertText` would aim at, but the write never reaches the
- // pty, so the user sees "type acknowledged but nothing appeared".
- // Try pty-master injection first (most reliable), then XTest key
- // synthesis. Either way the structured response reports
- // `path: "key_events"` so callers can verify the route taken.
- let pid_is_terminal = is_terminal_process(pid);
- let wm_class_is_terminal =
- tokio::task::spawn_blocking(move || crate::terminal::is_terminal_window(xid))
- .await
- .unwrap_or(false);
- if pid_is_terminal || wm_class_is_terminal {
- let text_len = text.chars().count();
- let text_t = text.clone();
- let foreground = delivery.is_foreground();
- let cursor_id = resolve_cursor_key(&args);
- let result = spawn_blocking_bounded(
- "type_text terminal",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {
- // pty-master injection is preferred — it skips the X event
- // queue entirely. Falls through to key events if the terminal
- // isn't reachable that way (descendant pty unresolvable).
- if inject_terminal_input(pid, xid, &text_t)? {
- return Ok(("pty", None));
- }
- if foreground {
- crate::interaction::with_x11_foreground(xid, crate::input::ForegroundOptions::pointer(), || {
- crate::input::send_type_text_xtest(&text_t)
- }).map(|(value, _report)| value)?;
- return Ok(("key_events_fg", None));
- }
- // Background: real key events through the XI2 virtual master
- // keyboard (xterm/gnome-terminal accept them like physical input).
- match background_text_route(&cursor_id, pid, xid, &text_t)? {
- Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),
- _ => Ok(("background_unavailable", None)),
- }
- })
- .await;
- return match result {
- Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(atspi_up()),
- Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),
- Ok(Ok((path, None))) => ToolResult::text(format!(
- "Typed {text_len} character(s) (terminal emulator: pty/XTest key events)."
- ))
- .with_structured(type_text_structured(path, text_len, false)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // Foreground means the caller explicitly permits activation: behave like
- // a user. Activate the window and confirm it holds the input focus,
- // GrabFocus the indexed element when one was given, then type REAL key
- // events via XTest. The AT-SPI EditableText probes below are skipped on
- // purpose: on large trees (LibreOffice) they stall for the full bus
- // timeout, Chromium/WebKitGTK echo them without the renderer seeing the
- // input, and for a spreadsheet cell / canvas they would target the
- // wrong widget anyway. The whole transaction is deadline-bounded.
- if delivery.is_foreground() {
- let text_f = text.clone();
- let element = token.clone();
- let result = spawn_blocking_bounded(
- "foreground type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || {
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::keyboard(),
- || {
- if let Some(element) = &element {
- if !crate::atspi::focus_token(pid, element)? {
- anyhow::bail!(
- "AT-SPI Component.GrabFocus returned false for element {}",
- element.index
- );
- }
- }
- crate::input::send_type_text_xtest(&text_f)
- },
- )
- },
- )
- .await;
- return match result {
- Ok(Ok(((), report))) => {
- let mut extra = serde_json::Map::new();
- extra.insert("characters".into(), json!(text_len));
- ToolResult::text(format!(
- "Typed {text_len} character(s) as real key events \
- (delivery_mode=foreground, focus_after={}); not verified — \
- confirm with a screenshot.",
- report.focus_after.as_str()
- ))
- .with_structured(foreground_structured(
- "key_events_fg",
- report,
- extra,
- ))
- }
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // With a focus-free real keyboard available, type where the widget
- // focus is — exactly what a physical keyboard does after the click that
- // focused the field. The blind AT-SPI editable search below can pick a
- // different editable of the app (it "succeeded" into GIMP's wrong entry
- // and VS Code's search box); it stays as the fallback for hosts without
- // the MPX route and for explicit element targets.
- if !delivery.is_foreground()
- && resolved_elem_idx.is_none()
- && crate::input::real_keyboard_input_available()
- {
- let text_b = text.clone();
- let cursor_id = resolve_cursor_key(&args);
- let result = spawn_blocking_bounded(
- "background type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || background_text_route(&cursor_id, pid, xid, &text_b),
- )
- .await;
- match result {
- Ok(Ok(Some(KeyRoute::Mpx(report)))) => {
- return type_text_mpx_result(text_len, report)
- }
- Ok(Err(e)) => return input_error_result(e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- Ok(Ok(_)) => {}
- }
- }
-
- // Prefer the focused widget — the element the user just clicked. If a
- // NON-editable input holds keyboard focus (a spreadsheet cell, a
- // terminal, a canvas), the focus-free AT-SPI editable search below would
- // grab the wrong field (e.g. gnumeric's name box instead of the selected
- // cell, or skip a terminal entirely), so synth-type into the focused
- // widget instead: terminals via pty injection, everything else via
- // XSendEvent to the focused window. A focused *editable* (Some(true)) or
- // nothing focused (None) falls through to the existing AT-SPI-first flow.
- let focus_kind = tokio::task::spawn_blocking(move || {
- crate::atspi::focused_is_editable(pid).ok().flatten()
- })
- .await
- .ok()
- .flatten();
- if focus_kind == Some(false) {
- if !delivery.is_foreground() {
- // A spreadsheet cell / canvas / terminal holds the widget focus:
- // real key events aimed at this window are the only thing that
- // types there. The XI2 virtual master keyboard delivers them
- // without touching the user's focus; terminals take the pty.
- let text_b = text.clone();
- let cursor_id = resolve_cursor_key(&args);
- let result = spawn_blocking_bounded(
- "background type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || -> anyhow::Result<Option<KeyRoute>> {
- if inject_terminal_input(pid, xid, &text_b)? {
- return Ok(Some(KeyRoute::Terminal));
- }
- background_text_route(&cursor_id, pid, xid, &text_b)
- },
- )
- .await;
- return match result {
- Ok(Ok(Some(KeyRoute::Terminal))) => ToolResult::text(format!(
- "Typed {text_len} character(s) into the focused terminal (pty)."
- ))
- .with_structured(type_text_structured("key_events", text_len, false)),
- Ok(Ok(Some(KeyRoute::Mpx(report)))) => type_text_mpx_result(text_len, report),
- Ok(Ok(_)) => background_keyboard_refusal(atspi_up()),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- let text_f = text.clone();
- let result = spawn_blocking_bounded(
- "foreground type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || {
- if inject_terminal_input(pid, xid, &text_f)? {
- return Ok(());
- }
- // XTest (real input to the focused window), NOT XSendEvent: GTK/Qt
- // drop synthetic key events, so a spreadsheet cell / canvas would
- // stay empty. The click that gave this widget focus already put it
- // under the X input focus, so XTest-to-focus lands correctly.
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || crate::input::send_type_text_xtest(&text_f),
- )
- .map(|(value, _report)| value)
- },
- )
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) into the focused widget."
- ))
- .with_structured(type_text_structured(
- "key_events",
- text_len,
- false,
- )),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Try AT-SPI EditableText first (focus-free, works for Qt6/GTK4).
- let text_clone = text.clone();
- let atspi_result =
- tokio::task::spawn_blocking(move || crate::atspi::type_into_editable(pid, &text_clone))
- .await;
-
- match atspi_result {
- Ok(Ok(())) => {
- // AT-SPI succeeded — focus-free typing worked (Qt6, GTK4, etc.)!
- // Electron/Chromium can echo this write without the renderer
- // observing it, so the confirm is suppressed there (mirrors macOS).
- return type_text_ax_result(pid, text_len, "via AT-SPI");
- }
- _ => {
- // AT-SPI failed (no editable exposed). Qt5 doesn't expose widgets
- // when unfocused, so try the synthetic-focus workaround.
- }
- }
-
- // Qt5 workaround: send synthetic FocusIn to make Qt5's AT-SPI bridge
- // expose the widget tree, type via AT-SPI, then send FocusOut.
- // This doesn't change the X11 active window, so the test's focus check passes.
- let text_clone2 = text.clone();
- let qt5_result = spawn_blocking_bounded(
- "Qt5 type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || {
- if !is_qt5_process(pid) {
- anyhow::bail!("not a Qt5 process; synthetic-FocusIn bridge workaround skipped");
- }
- // Send FocusIn to trigger Qt5's bridge
- crate::input::send_focus_in(xid)?;
- std::thread::sleep(std::time::Duration::from_millis(100));
-
- // Try AT-SPI again now that widgets should be exposed
- let result = crate::atspi::type_into_editable(pid, &text_clone2);
-
- // Restore state with FocusOut
- crate::input::send_focus_out(xid)?;
-
- result
- },
- )
- .await;
-
- match qt5_result {
- Ok(Ok(())) => {
- return type_text_ax_result(pid, text_len, "via AT-SPI with focus workaround");
- }
- _ => {
- // AT-SPI still didn't work. Fall back to X11 XSendEvent.
- }
- }
-
- // Track which path the final fallback chain took, so the
- // structured response stays honest. The closure can't borrow
- // a local mutably across `spawn_blocking`, so funnel the
- // decision through the success type instead.
- // delivery_mode: background (default) = focus-free AT-SPI / XSendEvent;
- // foreground = activate the window (EWMH), then synthesize REAL key
- // events to it via XTest — the escalation when background didn't land
- // (e.g. a GTK dialog whose widget ignores synthetic XSendEvent keys).
- let cursor_id = resolve_cursor_key(&args);
- let result = spawn_blocking_bounded(
- "type_text",
- typing_budget(text_len, crate::wayland::niri::is_session()),
- move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {
- // Terminals: write to the pty master (focus-free, below the toolkit).
- if inject_terminal_input(pid, xid, &text)? {
- return Ok(("key_events", None));
- }
- // GUI apps: X11 only routes keystrokes to the *focused* toplevel's
- // focused widget, so background XSendEvent typing doesn't land. Fill
- // the editable field via AT-SPI instead — focus-free and toolkit-
- // agnostic. Fall back to Tk send or XSendEvent when no a11y field is exposed.
- if crate::atspi::insert_text(pid, &text).unwrap_or(false) {
- return Ok(("ax", None));
- }
- // Tk apps: use Tk's `send` command (no AT-SPI bridge, so AT-SPI above
- // returned false). This is the Tk-specific override, like CDP for Chromium.
- if crate::input::inject_tk_send(&text).unwrap_or(false) {
- return Ok(("key_events", None));
- }
- if delivery.is_foreground() {
- crate::interaction::with_x11_foreground(xid, crate::input::ForegroundOptions::pointer(), || {
- crate::input::send_type_text_xtest(&text)
- }).map(|(value, _report)| value)?;
- return Ok(("key_events_fg", None));
- }
- // No a11y editable, no pty: real key events through the XI2 virtual
- // master keyboard are the last focus-free route.
- match background_text_route(&cursor_id, pid, xid, &text)? {
- Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),
- _ => Ok(("background_unavailable", None)),
- }
- })
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- match result {
- // AT-SPI's boolean acknowledges the EditableText call; it is not a
- // fresh value readback. Keep the result unverifiable and apply the
- // stricter Chromium escalation where appropriate.
- Ok(Ok(("ax", _))) => type_text_ax_result(
- pid,
- text_len,
- &format!("via X11, delivery_mode={mode_label}"),
- ),
- Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(atspi_up()),
- Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),
- Ok(Ok((path, None))) => ToolResult::text(format!(
- "Typed {text_len} character(s) (via X11, delivery_mode={mode_label})."
- ))
- .with_structured(type_text_structured(path, text_len, false)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-/// Background text delivery through the XI2 virtual master keyboard. `None`
-/// when the host has no such route (Xvfb, no `/dev/uinput`); a uinput
-/// permission error propagates so the tool can attach the operator hint.
-fn background_text_route(
- cursor_id: &str,
- pid: u32,
- xid: u64,
- text: &str,
-) -> anyhow::Result<Option<KeyRoute>> {
- if !crate::input::real_keyboard_input_available() {
- return Ok(None);
- }
- match crate::input::focus_guard::guarded(Some(pid), || {
- background_virtual_text(cursor_id, pid, xid, text)
- }) {
- Ok((mut report, guard)) => {
- report.focus_guard = guard;
- Ok(Some(KeyRoute::Mpx(report)))
- }
- Err(error) if crate::input::is_uinput_unavailable(&error) => Err(error),
- Err(error) if is_popup_keyboard_grab(&error) => Err(error),
- Err(error) => {
- tracing::warn!(pid, "MPX keyboard text delivery failed: {error}");
- Err(error.context("virtual master keyboard delivery failed"))
- }
- }
-}
-
-/// `type_text` result for a delivery through the virtual master keyboard.
-fn type_text_mpx_result(
- text_len: usize,
- report: crate::input::KeyboardDeliveryReport,
-) -> ToolResult {
- let mut structured = type_text_structured(report.path, text_len, false);
- for (key, value) in report.to_json().as_object().into_iter().flatten() {
- structured[key] = value.clone();
- }
- structured["delivery_mode"] = json!("background");
- let outcome = if report.skipped_characters.is_empty() {
- format!("Typed {text_len} character(s)")
- } else {
- "Partial typing result".to_owned()
- };
- let mut text = format!(
- "{outcome} as real key events {} (delivery_mode=background); \
- not verified — confirm with a screenshot.",
- report.route_phrase()
- );
- if !report.skipped_characters.is_empty() {
- text.push_str(&format!(
- " {} character(s) had no keycode in the current keymap and were skipped: {:?}.",
- report.skipped_characters.len(),
- report.skipped_characters.iter().collect::<String>()
- ));
- }
- if !report.virtual_focus_held || !report.delivery_confirmed {
- structured["effect"] = json!("suspected_noop");
- text.push_str(
- " Warning: the target lost the virtual keyboard's focus or the server did \
- not confirm the last key; the text may not have landed.",
- );
- }
- text.push_str(&report.delivery_notes());
- let result = if report.skipped_characters.is_empty() {
- ToolResult::text(text)
- } else {
- ToolResult::error(text)
- };
- result.with_structured(structured)
-}
-
-#[cfg(test)]
-#[test]
-fn mpx_type_text_with_skipped_characters_is_an_error() {
- let report = |skipped_characters: Vec<char>| crate::input::KeyboardDeliveryReport {
- virtual_focus_held: true,
- core_focus_unchanged: true,
- delivery_confirmed: true,
- key_events: 2,
- skipped_characters,
- focus_guard: None,
- released_stuck: Vec::new(),
- path: crate::input::MPX_UINPUT_PATH,
- grab_popup: None,
- };
- assert_eq!(type_text_mpx_result(2, report(Vec::new())).is_error, None);
- let skipped = type_text_mpx_result(2, report(vec!['約']));
- assert_eq!(skipped.is_error, Some(true));
- assert!(!format!("{:?}", skipped.content).contains("Typed 2 character(s)"));
- assert_eq!(
- skipped.structured_content.unwrap()["skipped_characters"],
- "約"
- );
-}
-
-// ── press_key ─────────────────────────────────────────────────────────────────
-
-pub struct PressKeyTool {
- state: Arc<ToolState>,
-}
-static PRESS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-/// Turn a modified `press_key` request into the chord an underlying keyboard
-/// path expects when it takes a single key and no separate modifier list.
-///
-/// Returns `None` when there is nothing to promote, so the caller keeps the
-/// plain single-key route. Dropping the modifiers instead is not a silent
-/// no-op: the bare key still reaches the app and is typed as a character.
-fn press_key_chord(mods: &[String], key: &str) -> Option<Vec<String>> {
- if mods.is_empty() {
- return None;
- }
- let mut chord = mods.to_vec();
- chord.push(key.to_owned());
- Some(chord)
-}
-
-#[cfg(test)]
-mod press_key_tests;
-
-#[async_trait]
-impl Tool for PressKeyTool {
- fn def(&self) -> &ToolDef {
- PRESS_DEF.get_or_init(|| ToolDef {
- name: "press_key".into(),
- description: "Press a key via XSendEvent to a window. No focus steal.".into(),
- input_schema: json!({
- "type":"object","required":["key"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to send the key to the focused application."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
- "key":{"type":"string","description":"Key name: enter/return, tab, escape, space, backspace, delete, insert, home, end, pageup, pagedown, up, down, left, right, f1-f12, or any single ASCII character."},
- "modifiers":{"type":"array","items":{"type":"string"},"description":"Modifier keys held while the key is pressed: ctrl, shift, alt, super."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Pixel Y (see x)."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to send the key to the focused application. Default \"window\"."},
- "coordinate_frame": coordinate_frame_schema(),
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if (args.opt_str("scope").as_deref() == Some("desktop")
- || coordinate_frame_is_desktop(&args))
- && args.get("pid").is_none()
- {
- let input = match parse_typed_projection::<PressKeyInput>("press_key", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let key = input.key;
- let display = key.clone();
- let modifiers = input.modifiers.unwrap_or_default();
- position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
- .await;
- let wayland = crate::wayland::wayland_input_enabled();
- let path = if wayland { "wayland_focused" } else { "xtest" };
- let result =
- spawn_blocking_bounded("desktop press_key", foreground_budget(1), move || {
- if wayland && modifiers.is_empty() {
- crate::wayland::press_key_focused(&key)
- } else if wayland {
- let mut keys = modifiers;
- keys.push(key);
- crate::wayland::hotkey_focused(&keys)
- } else {
- let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::send_key_xtest(&key, &refs)
- }
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!("Pressed desktop key '{display}'."))
- .with_structured(
- json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let mut pid = args.u64_or("pid", 0) as u32;
- let key = match args.require_str("key") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let mut mods: Vec<String> = args.str_array("modifiers");
- let (combo_mods, key) = split_key_combo(&key);
- mods.extend(combo_mods);
-
- // Resolve the element token into both its owning window and exact
- // child. Foreground delivery establishes child focus inside the
- // verified top-level activation transaction; background XSendEvent
- // retains the historical direct-window path.
- let window_id_arg = args.opt_u64("window_id");
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "press_key").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let resolved_element_index = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- Some(*element_index)
- }
- cua_driver_core::element_token::ResolvedElement::None => None,
- };
- let xid_opt = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
- };
- let xid = match xid_opt {
- Some(x) => x,
- None => {
- let windows = tokio::task::spawn_blocking(move || {
- crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
- })
- .await
- .unwrap_or_default();
- // pid omitted: the keys go to the active window, like a
- // physical keyboard would, and the action adopts its pid.
- let chosen = if pid == 0 {
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active && w.pid.is_some())
- .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
- } else {
- // The pid's active window, else its largest on-screen
- // toplevel: LibreOffice and GIMP own hidden/utility
- // toplevels that a plain `first()` could pick, and an
- // unmapped window cannot take the virtual keyboard focus.
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active)
- .or_else(|| {
- windows
- .iter()
- .filter(|w| w.is_on_screen)
- .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
- })
- .or_else(|| windows.first())
- };
- match chosen {
- Some(w) => {
- if pid == 0 {
- pid = w.pid.unwrap_or(0);
- }
- w.xid
- }
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // ── px form: pixel-click to focus, then the key goes to the focused element ──
- // Reuses click's translation + delivery_mode; after it, deliver via the plain
- // background path (the focus-click already handled fronting when fg). Pass x,y
- // (no element_token) for Chromium/Electron surfaces the AX path can't focus.
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- if isolated_hyprland_background(delivery) {
- if xid_opt.is_none() {
- return isolated_hyprland_refusal(
- "an exact window_id is required for isolated keys",
- );
- }
- if resolved_element_index.is_some()
- || args.get("x").is_some()
- || args.get("y").is_some()
- {
- return isolated_hyprland_refusal(
- "isolated keys address the exact top-level; first click the child explicitly",
- );
- }
- position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, false)
- .await;
- return isolated_hyprland_action(
- &args,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Key {
- key,
- modifiers: mods,
- },
- )
- .await;
- }
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
-
- if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_keyboard_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
- return refusal;
- }
- if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {
- return result;
- }
-
- let px = args.get("x").and_then(|value| value.as_f64());
- let py = args.get("y").and_then(|value| value.as_f64());
- if px.is_some() != py.is_some() {
- return invalid_pointer_arguments(
- "press_key",
- "x and y must be passed together (a pixel focus point), or neither",
- );
- }
- if px.is_some() && resolved_element_index.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to press_key, not both.",
- );
- }
-
- if hyprland_foreground(delivery) {
- if xid_opt.is_none() {
- return foreground_hyprland_refusal("an exact window_id is required");
- }
- let action = match foreground_hyprland_key(key, mods) {
- Ok(action) => action,
- Err(error) => return error,
- };
- if let Err(error) =
- focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
- .await
- {
- return error;
- }
- if px.is_none() {
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- None,
- false,
- )
- .await;
- }
- return foreground_hyprland_action(&args, pid, xid, action).await;
- }
-
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- px.zip(py),
- false,
- )
- .await;
-
- // Nested cua-compositor addresses the owning Wayland client directly.
- // Preserve legacy modifiers by promoting the request to a chord.
- if crate::wayland::is_inject_mode() {
- if let Err(error) =
- focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
- {
- return error;
- }
- let result = match press_key_chord(&mods, &key) {
- None => {
- let key_w = key.clone();
- tokio::task::spawn_blocking(move || {
- crate::wayland::inject_press_key(pid, xid, &key_w)
- })
- .await
- }
- Some(chord) => {
- tokio::task::spawn_blocking(move || {
- crate::wayland::inject_hotkey(pid, xid, &chord)
- })
- .await
- }
- };
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Pressed key '{key}' (focus-free via cua-compositor)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- let px_target = {
- if let (Some(cx), Some(cy)) = (px, py) {
- let from_zoom = args.bool_or("from_zoom", false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- Some(xid),
- (cx, cy),
- delivery.is_foreground(),
- &args,
- from_zoom,
- )
- .await
- {
- return e;
- }
- Some((cx.round() as i32, cy.round() as i32))
- } else {
- None
- }
- };
-
- // Native Wayland: send the key to the focused surface via virtual-keyboard.
- // `wayland::press_key` carries no modifier list, so a modified request has
- // to become a chord here — otherwise the modifiers are dropped and the
- // bare key is typed as a character (Ctrl+S inserts a literal "s").
- if crate::wayland::wayland_input_enabled() {
- let key_w = key.clone();
- let chord = press_key_chord(&mods, &key);
- let element = token.clone();
- let result = spawn_blocking_bounded("press_key", foreground_budget(1), move || {
- crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::RestoreMode::RestorePrevious,
- || {
- if let Some(element) = &element {
- if !crate::atspi::focus_token(pid, element)? {
- anyhow::bail!(
- "AT-SPI Component.GrabFocus returned false for element {}",
- element.index
- );
- }
- }
- match chord {
- Some(keys) => crate::wayland::hotkey_focused(&keys),
- None => crate::wayland::press_key_focused(&key_w),
- }
- },
- )
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Pressed key '{key}' (via Wayland virtual-keyboard)."
- )),
- Ok(Err(e)) => resolved_element_index
- .and_then(|idx| stale_observed_element("press_key", idx, &e))
- .unwrap_or_else(|| ToolResult::error(e.to_string())),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- let key_for_task = key.clone();
- let cursor_id = resolve_cursor_key(&args);
- // Foreground delivery is one atomic activate-and-XTest transaction.
- // A preceding PX click establishes internal widget focus, but that
- // click restores the prior top-level before returning.
- let deliver_fg = delivery.is_foreground();
- let result = spawn_blocking_bounded(
- "press_key",
- foreground_budget(1),
- move || -> anyhow::Result<KeyRoute> {
- if resolved_element_index.is_none()
- && mods.is_empty()
- && is_enter_key(&key_for_task)
- && inject_terminal_input(pid, xid, "\n")?
- {
- return Ok(KeyRoute::Terminal);
- }
- let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- // foreground: activate the window first, then inject a REAL key via
- // XTest. Synthetic XSendEvent keys (`send_key`) are dropped by
- // GTK/Qt/Chromium/Firefox, so the foreground rung must use XTest —
- // it delivers to the now-focused window. background = real key
- // events through the XI2 virtual master keyboard (no focus steal),
- // else direct XSendEvent for apps that accept it.
- if deliver_fg {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::keyboard(),
- || {
- if let Some(element) = &token {
- let element_index = element.index;
- if !crate::atspi::focus_token(pid, element)? {
- anyhow::bail!(
- "AT-SPI Component.GrabFocus returned false for element {element_index}"
- );
- }
- }
- crate::input::send_key_xtest(&key_for_task, &m)
- },
- )
- .map(|((), report)| KeyRoute::Foreground(report));
- }
- background_key_route(
- &cursor_id,
- pid,
- xid,
- px_target,
- &key_for_task,
- &m,
- token.as_ref(),
- )
- })
- .await;
- let mode_label = if deliver_fg {
- "foreground"
- } else {
- "background"
- };
- match result {
- Ok(Ok(route)) => key_route_result(&format!("Pressed key '{key}'"), route, mode_label),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── hotkey ────────────────────────────────────────────────────────────────────
-
-fn is_modifier(k: &str) -> bool {
- matches!(
- k.to_lowercase().as_str(),
- "ctrl"
- | "control"
- | "shift"
- | "alt"
- | "super"
- | "meta"
- | "cmd"
- | "command"
- | "win"
- | "windows"
- )
-}
-
-pub struct HotkeyTool {
- state: Arc<ToolState>,
-}
-static HOTKEY_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for HotkeyTool {
- fn def(&self) -> &ToolDef {
- HOTKEY_DEF.get_or_init(|| ToolDef {
- name: "hotkey".into(),
- description: "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. \
- Delivered to the target window without focus steal (real key events from a virtual \
- master keyboard on X11); target does NOT need to be frontmost. Window-manager chords \
- cannot reach the WM that way: alt+F4 on a window target closes that window through \
- the WM (_NET_CLOSE_WINDOW, effect confirmed when it unmaps); super+*, alt+tab, \
- ctrl+alt+* and other WM bindings are refused with code wm_chord_unavailable — use \
- bring_to_front / list_windows / kill_app or delivery_mode:\"foreground\" instead.".into(),
- input_schema: json!({
- "type":"object","required":["keys"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to send the chord to the focused application."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
- "keys":{"type":"array","items":{"type":"string"},"minItems":2,
- "description":"Modifier(s) + one non-modifier key, e.g. [\"ctrl\",\"c\"]."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
- "y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to send the chord to the focused application. Default \"window\"."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {
- let input = match parse_typed_projection::<HotkeyInput>("hotkey", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let keys = input.keys;
- if keys.len() < 2 {
- return ToolResult::error("hotkey.keys must contain at least two keys.")
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- let modifiers: Vec<String> = keys
- .iter()
- .filter(|key| is_modifier(key))
- .cloned()
- .collect();
- let Some(key) = keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {
- return ToolResult::error("keys must include at least one non-modifier key.");
- };
- let display = keys.join("+");
- position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
- .await;
- let wayland = crate::wayland::wayland_input_enabled();
- let path = if wayland { "wayland_focused" } else { "xtest" };
- let result =
- spawn_blocking_bounded("desktop hotkey", foreground_budget(1), move || {
- if wayland {
- crate::wayland::hotkey_focused(&keys)
- } else {
- let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::send_key_xtest(&key, &refs)
- }
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))
- .with_structured(
- json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let mut pid = args.u64_or("pid", 0) as u32;
- let window_id_arg = args.opt_u64("window_id");
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(resolved) => resolved,
- Err(error) => return error,
- };
- let token = match prove_token_element(pid, &resolved, "hotkey").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let resolved_element_index = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- Some(*element_index)
- }
- cua_driver_core::element_token::ResolvedElement::None => None,
- };
- let xid_opt = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
- };
-
- // Resolve XID: use window_id if given, else first window for pid.
- let xid = match xid_opt {
- Some(x) => x,
- None => {
- let windows = tokio::task::spawn_blocking(move || {
- crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
- })
- .await
- .unwrap_or_default();
- // pid omitted: the keys go to the active window, like a
- // physical keyboard would, and the action adopts its pid.
- let chosen = if pid == 0 {
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active && w.pid.is_some())
- .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
- } else {
- // The pid's active window, else its largest on-screen
- // toplevel: LibreOffice and GIMP own hidden/utility
- // toplevels that a plain `first()` could pick, and an
- // unmapped window cannot take the virtual keyboard focus.
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active)
- .or_else(|| {
- windows
- .iter()
- .filter(|w| w.is_on_screen)
- .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
- })
- .or_else(|| windows.first())
- };
- match chosen {
- Some(w) => {
- if pid == 0 {
- pid = w.pid.unwrap_or(0);
- }
- w.xid
- }
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // Parse keys array (preferred) or fall back to legacy key+modifiers.
- let (key, mods) = if let Some(arr) = args.get("keys").and_then(|v| v.as_array()) {
- let keys: Vec<String> = arr
- .iter()
- .filter_map(|v| v.as_str().map(str::to_owned))
- .collect();
- let modifiers: Vec<String> = keys.iter().filter(|k| is_modifier(k)).cloned().collect();
- let non_mods: Vec<String> = keys.iter().filter(|k| !is_modifier(k)).cloned().collect();
- if non_mods.is_empty() {
- return ToolResult::error("keys must include at least one non-modifier key.");
- }
- (non_mods.last().unwrap().clone(), modifiers)
- } else if let Some(k) = args.opt_str("key") {
- let mods: Vec<String> = args.str_array("modifiers");
- (k, mods)
- } else {
- return ToolResult::error(
- "Provide 'keys' array (e.g. [\"ctrl\",\"c\"]) or 'key'+'modifiers' parameters.",
- );
- };
-
- let key_display = format!("{}+{}", mods.join("+"), key);
- let key_for_wayland = key.clone();
- let mods_for_wayland = mods.clone();
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
-
- if isolated_hyprland_background(delivery) {
- if xid_opt.is_none() {
- return isolated_hyprland_refusal(
- "an exact window_id is required for isolated hotkeys",
- );
- }
- if resolved_element_index.is_some()
- || args.get("x").is_some()
- || args.get("y").is_some()
- {
- return isolated_hyprland_refusal("isolated hotkeys address the exact top-level; first click the child explicitly");
- }
- if let Some(keys) = args.get("keys").and_then(Value::as_array) {
- if keys
- .iter()
- .filter(|value| value.as_str().is_some_and(|key| !is_modifier(key)))
- .count()
- != 1
- || keys.iter().any(|value| !value.is_string())
- {
- return isolated_hyprland_refusal(
- "isolated hotkeys require exactly one non-modifier key",
- );
- }
- }
- position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, false)
- .await;
- return isolated_hyprland_action(
- &args,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Key {
- key,
- modifiers: mods,
- },
- )
- .await;
- }
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_keyboard_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
- return refusal;
- }
- if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {
- return result;
- }
-
- let px = args.get("x").and_then(|value| value.as_f64());
- let py = args.get("y").and_then(|value| value.as_f64());
- if px.is_some() != py.is_some() {
- return invalid_pointer_arguments(
- "hotkey",
- "x and y must be passed together (a pixel focus point), or neither",
- );
- }
- if px.is_some() && resolved_element_index.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to hotkey, not both.",
- );
- }
-
- if hyprland_foreground(delivery) {
- if xid_opt.is_none() {
- return foreground_hyprland_refusal("an exact window_id is required");
- }
- if let Some(keys) = args.get("keys").and_then(Value::as_array) {
- if keys.iter().any(|key| !key.is_string())
- || keys
- .iter()
- .filter(|key| key.as_str().is_some_and(|key| !is_modifier(key)))
- .count()
- != 1
- {
- return foreground_hyprland_refusal(
- "hotkeys require exactly one non-modifier key",
- );
- }
- }
- let action = match foreground_hyprland_key(key, mods) {
- Ok(action) => action,
- Err(error) => return error,
- };
- if let Err(error) =
- focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
- .await
- {
- return error;
- }
- if px.is_none() {
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- None,
- false,
- )
- .await;
- }
- return foreground_hyprland_action(&args, pid, xid, action).await;
- }
-
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- token.clone(),
- px.zip(py),
- false,
- )
- .await;
-
- if crate::wayland::is_inject_mode() {
- if let Err(error) =
- focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
- {
- return error;
- }
- let mut chord = mods.clone();
- chord.push(key.clone());
- let result = tokio::task::spawn_blocking(move || {
- crate::wayland::inject_hotkey(pid, xid, &chord)
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Pressed hotkey '{key_display}' (focus-free via cua-compositor)."
- )),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
-
- // Chromium renderer shortcuts need DOM focus established by a real
- // pointer event. Component.GrabFocus alone can report success while
- // the renderer still drops the virtual-keyboard chord.
- if delivery.is_foreground()
- && crate::wayland::wayland_input_enabled()
- && is_chromium_embedder(pid)
- {
- if let Some(element) = token.clone() {
- let coordinates = tokio::task::spawn_blocking(move || {
- resolve_element_local_coords(pid, &element, Some(xid))
- })
- .await;
- let (_, x, y) = match coordinates {
- Ok(Ok(value)) => value,
- Ok(Err(error)) => return ToolResult::error(error.to_string()),
- Err(error) => return ToolResult::error(format!("Task error: {error}")),
- };
- if let Err(error) =
- focus_by_pixel(&self.state, pid, Some(xid), (x, y), true, &args, false).await
- {
- return error;
- }
- }
- }
-
- // Foreground: GrabFocus the addressed element up front. Background
- // does it inside `background_key_route`, under the focus guard.
- if let Some(element) = token.clone().filter(|_| delivery.is_foreground()) {
- let element_index = element.index;
- let focused =
- tokio::task::spawn_blocking(move || crate::atspi::focus_token(pid, &element)).await;
- match focused {
- Ok(Ok(true)) => {}
- Ok(Ok(false)) => {
- return ToolResult::error(format!(
- "AT-SPI Component.GrabFocus returned false for element {element_index}"
- ))
- }
- Ok(Err(error)) => {
- return stale_observed_element("hotkey", element_index, &error)
- .unwrap_or_else(|| ToolResult::error(error.to_string()))
- }
- Err(error) => return ToolResult::error(format!("Task error: {error}")),
- }
- }
-
- // ── px form: pixel-click to focus, then the combo acts on the focused field ──
- // (e.g. Ctrl+V into a Chromium input). Reuses click's translation +
- // delivery_mode; after it, deliver the combo via the plain background path
- // (the focus-click already fronted when fg).
- let px_target = {
- if let (Some(cx), Some(cy)) = (px, py) {
- let from_zoom = args.bool_or("from_zoom", false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- Some(xid),
- (cx, cy),
- delivery.is_foreground(),
- &args,
- from_zoom,
- )
- .await
- {
- return e;
- }
- Some((cx.round() as i32, cy.round() as i32))
- } else {
- None
- }
- };
- let deliver_fg = delivery.is_foreground();
- let cursor_id = resolve_cursor_key(&args);
-
- let result = spawn_blocking_bounded(
- "hotkey",
- foreground_budget(1),
- move || -> anyhow::Result<KeyRoute> {
- if crate::wayland::wayland_input_enabled() {
- // Native Wayland: route the modifier combo through wtype's
- // -M/-k/-m sequence — the closest equivalent to the X11
- // state-mask path. window_id is irrelevant once focused.
- let mut combo: Vec<String> = mods_for_wayland.clone();
- combo.push(key_for_wayland.clone());
- return crate::wayland::hotkey(xid, &combo).map(|()| KeyRoute::Synthetic);
- }
- let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- // foreground: activate the target first, then inject the accelerator
- // as REAL key events via XTest. Synthetic XSendEvent keys
- // (`send_key`) are dropped by GTK/Qt/Chromium/Firefox, so the
- // foreground rung must use XTest, which reaches the focused window.
- if deliver_fg {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::keyboard(),
- || crate::input::send_key_xtest(&key, &m),
- )
- .map(|((), report)| KeyRoute::Foreground(report));
- }
- background_key_route(&cursor_id, pid, xid, px_target, &key, &m, token.as_ref())
- },
- )
- .await;
- let mode_label = if deliver_fg {
- "foreground"
- } else {
- "background"
- };
- match result {
- Ok(Ok(route)) => key_route_result(
- &format!("Pressed {key_display} on pid {pid}"),
- route,
- mode_label,
- ),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── set_value ─────────────────────────────────────────────────────────────────
-
-pub struct SetValueTool {
- state: Arc<ToolState>,
-}
-static SV_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetValueTool {
- fn def(&self) -> &ToolDef {
- SV_DEF.get_or_init(|| ToolDef {
- name: "set_value".into(),
- description: "Set value of an AT-SPI element via SetValue action.".into(),
- input_schema: json!({
- "type":"object","required":["value"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"Omit when element_token is supplied (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "value":{"type":"string","description":"New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let value = match args.get("value") {
- Some(Value::String(text)) => text.clone(),
- Some(Value::Number(number)) => number.to_string(),
- _ => {
- return ToolResult::error("set_value: 'value' must be a string or a number.")
- .with_structured(json!({ "code": "invalid_arguments" }))
- }
- };
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "set_value").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let (token, xid) = match (token, &resolved) {
- (
- Some(token),
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. },
- ) => (token, *window_id),
- _ => {
- return ToolResult::error(
- "set_value requires element_token to address the target element.",
- )
- }
- };
- let idx = token.index;
- let value_for_task = value.clone();
- let xid_opt = Some(xid);
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- let cursor_id = resolve_cursor_key(&args);
- position_named_session_keyboard_cursor(
- &self.state,
- &args,
- pid,
- xid,
- Some(token.clone()),
- None,
- true,
- )
- .await;
- // 1. Focus-free accessibility write on the exact snapshot object.
- let target = token.clone();
- let ax = spawn_blocking_bounded("set_value", foreground_budget(0), move || {
- crate::atspi::set_value_token(pid, &target, &value_for_task)
- })
- .await;
- let ax_error = match ax {
- Ok(Ok(())) => {
- let target = token.clone();
- let readback =
- tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&target))
- .await
- .ok()
- .flatten();
- return set_value_result(idx, &value, "ax", readback, None);
- }
- Ok(Err(error)) if crate::atspi::native::is_no_value_route(&error) => error,
- Ok(Err(error)) if crate::atspi::native::is_no_such_option(&error) => {
- return no_such_option_result(idx, &error)
- }
- Ok(Err(error)) => {
- return stale_observed_element("set_value", idx, &error)
- .unwrap_or_else(|| input_error_result(error))
- }
- Err(error) => return ToolResult::error(format!("Task error: {error}")),
- };
- // 2. No accessibility write route. A combo box is opened with a click
- // and the option picked from its popup (whose items only exist and
- // act while it is open), then the popup is closed. Anything else is
- // typed like a user would: click the field (real pointer), select
- // all, type, commit with Tab.
- let is_combo = token.observed.role.eq_ignore_ascii_case("combo box");
- let wayland = crate::wayland::wayland_input_enabled();
- let background_route = !delivery.is_foreground()
- && !wayland
- && crate::input::real_pointer_input_available()
- && crate::input::real_keyboard_input_available();
- let foreground_route = delivery.is_foreground() && !wayland;
- if !background_route && !foreground_route {
- let detail = format!(
- "{ax_error}; the keyboard fallback needs {}",
- if wayland {
- "an X11 session"
- } else {
- "the focus-free real-input route (a writable /dev/uinput), or delivery_mode:\"foreground\""
- }
- );
- let mut structured = json!({
- "code": "set_value_unavailable",
- "detail": detail,
- "effect": "none",
- });
- // Only X11 can run the foreground keyboard fallback.
- if !wayland {
- structured["escalation"] = json!({ "recommended": "foreground", "reason": "click the field and type the value with delivery_mode:\"foreground\"" });
- }
- return ToolResult::error(format!(
- "set_value: element [{idx}] has no accessibility write route and no keyboard \
- fallback is available: {detail}."
- ))
- .with_structured(structured);
- }
- let value_for_keys = value.clone();
- let target = token.clone();
- let typed = spawn_blocking_bounded(
- "set_value keyboard fallback",
- foreground_budget(value.len()),
- move || -> anyhow::Result<Option<crate::input::FocusGuardReport>> {
- let (win, lx, ly) = resolve_element_local_coords(pid, &target, xid_opt)?;
- if is_combo {
- return pick_from_popup(
- &cursor_id,
- pid,
- &target,
- (win, lx, ly),
- &value_for_keys,
- foreground_route,
- );
- }
- if foreground_route {
- let (sx, sy) = window_local_to_screen(win, lx, ly)?;
- crate::interaction::with_x11_foreground(
- win,
- crate::input::ForegroundOptions::pointer(),
- || {
- crate::input::send_click_xtest_desktop(
- sx.round() as i32,
- sy.round() as i32,
- 1,
- 1,
- )?;
- std::thread::sleep(std::time::Duration::from_millis(80));
- crate::input::send_key_xtest("a", &["ctrl"])?;
- crate::input::send_type_text_xtest(&value_for_keys)?;
- crate::input::send_key_xtest("Tab", &[])
- },
- )?;
- return Ok(None);
- }
- let (_, guard) = crate::input::focus_guard::guarded(Some(pid), || {
- x11_pixel_click_no_focus_steal(&cursor_id, win, lx as i32, ly as i32, 1, 1)?;
- std::thread::sleep(std::time::Duration::from_millis(80));
- background_virtual_key(&cursor_id, pid, win, "a", &["ctrl"])?;
- background_virtual_text(&cursor_id, pid, win, &value_for_keys)?;
- background_virtual_key(&cursor_id, pid, win, "Tab", &[])?;
- Ok(())
- })?;
- Ok(guard)
- },
- )
- .await;
- match typed {
- Ok(Ok(guard)) => {
- // Let the toolkit commit on focus-out before reading back.
- tokio::time::sleep(std::time::Duration::from_millis(150)).await;
- let readback =
- tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&token))
- .await
- .ok()
- .flatten();
- set_value_result(
- idx,
- &value,
- match (is_combo, foreground_route) {
- (true, true) => "click_select_fg",
- (true, false) => "click_select_mpx",
- (false, true) => "click_type_fg",
- (false, false) => "click_type_mpx",
- },
- readback,
- guard.as_ref(),
- )
- }
- Ok(Err(error)) if crate::atspi::native::is_no_such_option(&error) => {
- no_such_option_result(idx, &error)
- }
- Ok(Err(error)) => input_error_result(error),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- }
- }
-}
-
-/// `set_value` named an option the combo box or list does not have.
-fn no_such_option_result(idx: usize, error: &anyhow::Error) -> ToolResult {
- let detail = error.to_string();
- let detail = detail
- .strip_prefix(crate::atspi::native::NO_SUCH_OPTION)
- .map(|rest| rest.trim_start_matches(':').trim())
- .unwrap_or(&detail)
- .to_owned();
- ToolResult::error(format!(
- "set_value: element [{idx}] has {detail}; nothing was changed."
- ))
- .with_structured(json!({ "code": "option_not_found", "detail": detail, "effect": "none" }))
-}
-
-/// How long `set_value` waits for an opened combo box popup to list its
-/// options.
-const POPUP_OPTIONS_WAIT: std::time::Duration = std::time::Duration::from_millis(1500);
-
-/// Keyboard-fallback `set_value` on a combo box whose options cannot be
-/// selected while it is closed: click it open (the real pointer: XTest in
-/// the foreground, the session's MPX pointer in the background), activate
-/// the option `value` names from the popup, and press Escape if the popup
-/// is still open afterwards (an unknown option, or a toolkit that keeps it
-/// open), so no popup is left behind.
-fn pick_from_popup(
- cursor_id: &str,
- pid: u32,
- token: &crate::atspi::TokenElement,
- (win, lx, ly): (u64, f64, f64),
- value: &str,
- foreground: bool,
-) -> anyhow::Result<Option<crate::input::FocusGuardReport>> {
- let close_popup = || -> anyhow::Result<()> {
- if !crate::atspi::options_showing_token(token) {
- return Ok(());
- }
- if foreground {
- crate::interaction::with_x11_foreground(
- win,
- crate::input::ForegroundOptions::keyboard(),
- || crate::input::send_key_xtest("Escape", &[]),
- )
- .map(|_| ())
- } else {
- background_virtual_key(cursor_id, pid, win, "Escape", &[]).map(|_| ())
- }
- };
- let pick = || -> anyhow::Result<String> {
- let deadline = std::time::Instant::now() + POPUP_OPTIONS_WAIT;
- loop {
- match crate::atspi::activate_option_token(token, value) {
- Ok(Some(name)) => return Ok(name),
- Ok(None) if std::time::Instant::now() < deadline => {
- std::thread::sleep(std::time::Duration::from_millis(100));
- }
- Ok(None) => anyhow::bail!("the combo box popup listed no options"),
- Err(error) => return Err(error),
- }
- }
- };
- let open_and_pick = || -> anyhow::Result<()> {
- let picked = pick();
- // Let the toolkit close the popup on activation before checking.
- std::thread::sleep(std::time::Duration::from_millis(150));
- let closed = close_popup();
- picked?;
- closed
- };
- if foreground {
- let (sx, sy) = window_local_to_screen(win, lx, ly)?;
- crate::interaction::with_x11_foreground(
- win,
- crate::input::ForegroundOptions::pointer(),
- || crate::input::send_click_xtest_desktop(sx.round() as i32, sy.round() as i32, 1, 1),
- )?;
- open_and_pick()?;
- return Ok(None);
- }
- let (_, guard) = crate::input::focus_guard::guarded(Some(pid), || {
- x11_pixel_click_no_focus_steal(cursor_id, win, lx as i32, ly as i32, 1, 1)?;
- open_and_pick()
- })?;
- Ok(guard)
-}
-
-/// Numeric-aware equality for the `set_value` read-back ("40" == "40.0").
-fn values_agree(wanted: &str, observed: &str) -> bool {
- let wanted = wanted.trim();
- let observed = observed.trim();
- if wanted == observed {
- return true;
- }
- match (wanted.parse::<f64>(), observed.parse::<f64>()) {
- (Ok(a), Ok(b)) => (a - b).abs() <= 1e-6 * a.abs().max(1.0),
- _ => false,
- }
-}
-
-/// `set_value` result with an honest verification: `verified` only when the
-/// element read back the requested value.
-fn set_value_result(
- idx: usize,
- value: &str,
- path: &str,
- readback: Option<String>,
- guard: Option<&crate::input::FocusGuardReport>,
-) -> ToolResult {
- let normalised_path = readback
- .as_deref()
- .is_some_and(|seen| path_normalised_to_basename(value, seen));
- let verified = readback
- .as_deref()
- .is_some_and(|seen| values_agree(value, seen))
- || normalised_path;
- let mut structured = json!({
- "path": path,
- "verified": verified,
- "effect": if verified { "confirmed" } else if readback.is_some() { "suspected_noop" } else { "unverifiable" },
- });
- let mut text = format!("Set value of element [{idx}] to '{value}' (path={path})");
- match readback {
- Some(seen) if normalised_path => {
- structured["readback"] = json!(seen);
- text.push_str(&format!(
- "; the field reads back '{seen}': a file chooser navigated to the path's \
- directory and kept the file name (the write committed)."
- ));
- }
- Some(seen) if verified => text.push_str(&format!("; read back '{seen}'.")),
- Some(seen) => {
- structured["readback"] = json!(seen);
- structured["escalation"] = non_ax_escalation();
- text.push_str(&format!(
- "; but the element reads back '{seen}' — the write may not have committed. \
- Confirm with get_window_state or a screenshot."
- ));
- }
- None => {
- structured["readback"] = Value::Null;
- text.push_str("; not verified — confirm with get_window_state.");
- }
- }
- attach_focus_guard(ToolResult::text(text).with_structured(structured), guard)
-}
-
-/// A Qt/GTK file chooser given an absolute path in its "File name" field
-/// moves to that directory and shows only the final component: the readback
-/// is the basename, and the write committed.
-fn path_normalised_to_basename(value: &str, seen: &str) -> bool {
- let value = value.trim();
- let seen = seen.trim();
- value.contains('/')
- && !seen.is_empty()
- && value
- .rsplit('/')
- .next()
- .is_some_and(|base| !base.is_empty() && base == seen)
-}
-
-#[cfg(test)]
-mod set_value_tests;
-
-// ── scroll ────────────────────────────────────────────────────────────────────
-
-pub struct ScrollTool {
- state: Arc<ToolState>,
-}
-
-fn atspi_scroll_result(progress: crate::atspi::ScrollProgress, foreground: bool) -> ToolResult {
- use cua_driver_core::action_record::{
- ActionEffect, ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
- };
- let effect = if !progress.complete && progress.acknowledged > 0 {
- ActionEffect::Partial
- } else {
- ActionEffect::Unverifiable
- };
- let mut record = ActionExecutionRecord::builder(
- effect,
- ActionTransport::LinuxAtSpiAction,
- if foreground {
- RequestedDelivery::Foreground
- } else {
- RequestedDelivery::Background
- },
- )
- .actual_delivery(if !progress.complete {
- ActualDelivery::Unknown
- } else if foreground {
- ActualDelivery::Foreground
- } else {
- ActualDelivery::Background
- });
- if progress.acknowledged > 0 {
- record = record.delivered_count(progress.acknowledged);
- }
- let text = if progress.complete {
- format!(
- "AT-SPI acknowledged {} scroll action(s); effect remains unverified.",
- progress.acknowledged
- )
- } else {
- format!("AT-SPI scroll stopped after {} acknowledged action(s): {}. Refresh state before another action.", progress.acknowledged, progress.detail.as_deref().unwrap_or("unknown outcome"))
- };
- let record = record
- .detail(text.clone())
- .build()
- .expect("AT-SPI scroll preserves acknowledged progress");
- let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
- let result = if progress.complete {
- ToolResult::text(text)
- } else {
- ToolResult::error(text)
- };
- result.with_structured(public).with_action_record(record)
-}
-
-#[test]
-fn atspi_scroll_outcomes_preserve_uncertainty_and_acknowledged_count() {
- use cua_driver_core::action_record::ActionEffect;
- for (complete, acknowledged, effect) in [
- (false, 0, ActionEffect::Unverifiable),
- (false, 1, ActionEffect::Partial),
- (true, 2, ActionEffect::Unverifiable),
- ] {
- let result = atspi_scroll_result(
- crate::atspi::ScrollProgress {
- acknowledged,
- complete,
- detail: None,
- },
- false,
- );
- let record = result.action_record.unwrap();
- assert_eq!(record.effect, effect);
- let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
- assert_eq!(
- public["delivery"]["mode"],
- if complete { "background" } else { "unknown" }
- );
- assert_eq!(
- public["delivery"]["delivered_count"],
- if acknowledged == 0 {
- Value::Null
- } else {
- json!(acknowledged)
- }
- );
- }
-}
-static SCROLL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ScrollTool {
- fn def(&self) -> &ToolDef {
- SCROLL_DEF.get_or_init(|| ToolDef {
- name: "scroll".into(),
- description: "Scroll the target pid's focused region via XSendEvent Button4/5. \
- direction required; by defaults to line, amount defaults to 3.".into(),
- input_schema: json!({
- // `pid` is conditionally required (validated in code), so only
- // `direction` is pinned — matches the scroll→["direction"] canon
- // in cua_driver_core::tool_schema.
- "type":"object","required":["direction"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid)."},
- "direction":{"type":"string","enum":["up","down","left","right"],"description":"Scroll direction."},
- "by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},
- "amount":{"type":"integer","minimum":1,"maximum":50,"description":"Number of scroll steps. Default: 3."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y; optional with element_token (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_token."},
- "y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_token."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with x,y and no pid/window_id for get_desktop_state screen coordinates. Default \"window\"."},
- "coordinate_frame": coordinate_frame_schema(),
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: false, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- if args.opt_str("scope").as_deref() == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<ScrollInput>("scroll", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let direction = input.direction.as_str().to_owned();
- let (x, y) = (input.x, input.y);
- let x = x.round() as i32;
- let y = y.round() as i32;
- let amount = input.amount.unwrap_or(3).clamp(1, 50) as usize;
- let ticks = wheel_ticks(
- amount,
- input.by.unwrap_or(cua_driver_contract::ScrollBy::Line),
- crate::wayland::niri::is_session(),
- );
- let display = direction.clone();
- let wayland = crate::wayland::wayland_input_enabled();
- let path = if wayland { "wayland_desktop" } else { "xtest" };
- let space = match desktop_input_space().await {
- Ok(space) => space,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- if named_session_cursor_key(&args).is_some() {
- let (overlay_x, overlay_y) = space.to_layout(x, y);
- reveal_pointer_action_for(
- &self.state,
- &cursor_id,
- f64::from(overlay_x),
- f64::from(overlay_y),
- false,
- )
- .await;
- }
- let result = spawn_blocking_bounded(
- "desktop scroll",
- foreground_budget(ticks as usize),
- move || {
- if wayland {
- crate::wayland::scroll_desktop(&space, x, y, &direction, ticks)
- } else {
- crate::input::send_scroll_xtest_desktop(x, y, &direction, amount)
- }
- },
- )
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!("Scrolled desktop {display} × {amount}."))
- .with_structured(
- json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
- input_error_result(error)
- }
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let mut pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let direction = match args.require_str("direction") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;
- let by = match serde_json::from_value::<cua_driver_contract::ScrollBy>(
- args.get("by").cloned().unwrap_or(json!("line")),
- ) {
- Ok(by) => by,
- Err(error) => return ToolResult::error(format!("Invalid scroll by: {error}")),
- };
- // The Linux scroll implementation doesn't pre-focus the element (X11
- // scroll buttons go to the window root), but the token still needs to
- // be validated so a stale token surfaces an error instead of silently
- // no-op'ing.
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "scroll").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let xid_opt: Option<u64> = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
- };
-
- // Resolve XID: use window_id if given, else first window for pid.
- let xid = match xid_opt {
- Some(x) => x,
- None => {
- let windows = tokio::task::spawn_blocking(move || {
- crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
- })
- .await
- .unwrap_or_default();
- // pid omitted: the keys go to the active window, like a
- // physical keyboard would, and the action adopts its pid.
- let chosen = if pid == 0 {
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active && w.pid.is_some())
- .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
- } else {
- // The pid's active window, else its largest on-screen
- // toplevel: LibreOffice and GIMP own hidden/utility
- // toplevels that a plain `first()` could pick, and an
- // unmapped window cannot take the virtual keyboard focus.
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active)
- .or_else(|| {
- windows
- .iter()
- .filter(|w| w.is_on_screen)
- .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
- })
- .or_else(|| windows.first())
- };
- match chosen {
- Some(w) => {
- if pid == 0 {
- pid = w.pid.unwrap_or(0);
- }
- w.xid
- }
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- let isolated_background = isolated_hyprland_background(delivery);
- let pixel_target = match (
- args.get("x").and_then(|value| value.as_f64()),
- args.get("y").and_then(|value| value.as_f64()),
- ) {
- (Some(x), Some(y)) if desktop_frame_requested(&args) => {
- // Desktop-frame pixels (from get_desktop_state) against a named
- // window: map into the window-local frame the pipeline expects.
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await
- {
- Ok(Ok(local)) => Some(local),
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
- (Some(x), Some(y)) => {
- // Pixel targets use the latest screenshot's coordinate frame.
- // Apply the same buffer-to-window ratio as click/drag before
- // positioning either the agent cursor or the input device.
- // An unsupported native background delivery refuses first.
- let native_refusal = (!isolated_background)
- .then(|| unavailable_wayland_focused_input_background(delivery, true))
- .flatten();
- let ratio = match coordinate_scroll_scale(native_refusal, || {
- screenshot_scale(&self.state, &args, pid, Some(xid))
- }) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- Some((x * ratio, y * ratio))
- }
- (None, None) => None,
- (Some(_), None) => return invalid_pointer_arguments("scroll", "x was given without y"),
- (None, Some(_)) => return invalid_pointer_arguments("scroll", "y was given without x"),
- };
- let resolved_element_index = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- Some(*element_index)
- }
- cua_driver_core::element_token::ResolvedElement::None => None,
- };
- if pixel_target.is_some() && resolved_element_index.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to scroll, not both.",
- );
- }
-
- if named_session_cursor_key(&args).is_some() {
- let element = token.clone();
- let visual_target = tokio::task::spawn_blocking(move || {
- explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
- .or_else(|| keyboard_window_center(xid))
- })
- .await
- .ok()
- .flatten();
- if let Some((sx, sy)) = visual_target {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, false).await;
- }
- }
-
- if !isolated_background {
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- }
- if let Some(element) = token.clone() {
- let idx = element.index;
- // WebKitGTK acknowledges the AT-SPI scroll action without moving
- // the DOM scroller. On native Wayland, use a real compositor wheel
- // event at the resolved element instead of reporting a silent
- // success. Other AX targets keep their semantic route before any
- // coordinate-based compositor fallback.
- if !(crate::wayland::wayland_input_enabled() && is_webkitgtk_embedder(pid)) {
- let direction_for_ax = direction.clone();
- let ax_result = tokio::task::spawn_blocking(move || {
- crate::atspi::scroll_token(pid, &element, &direction_for_ax, amount, by)
- })
- .await;
- if let Ok(Err(error)) = &ax_result {
- // The observed element is gone: nothing was scrolled.
- if let Some(stale) = stale_observed_element("scroll", idx, error) {
- return stale;
- }
- }
- match ax_result {
- Ok(Ok(progress)) => {
- return atspi_scroll_result(progress, delivery.is_foreground())
- }
- Err(error) => {
- return atspi_scroll_result(
- crate::atspi::ScrollProgress {
- acknowledged: 0,
- complete: false,
- detail: Some(error.to_string()),
- },
- delivery.is_foreground(),
- )
- }
- Ok(Err(_)) => {}
- }
- }
- }
- if hyprland_foreground(delivery) {
- if xid_opt.is_none() {
- return foreground_hyprland_refusal(
- "an exact window_id or window-bound element token is required",
- );
- }
- let point = if let Some(element) = token.clone() {
- match tokio::task::spawn_blocking(move || {
- resolve_element_local_coords(pid, &element, Some(xid))
- })
- .await
- {
- Ok(Ok((_, x, y))) => Some((x, y)),
- _ => {
- return foreground_hyprland_refusal(
- "could not resolve exact element coordinates",
- )
- }
- }
- } else {
- pixel_target
- };
- // `by: "page"` is the documented 3-detent approximation, not an
- // exact viewport; large requests become several wheel packets.
- let actions =
- match crate::wayland::hyprland_input::scroll_actions(point, &direction, amount, by)
- {
- Ok(actions) => actions,
- Err(error) => return foreground_hyprland_refusal(error.to_string()),
- };
- return foreground_hyprland_scroll(&args, pid, xid, actions).await;
- }
- if isolated_background {
- if xid_opt.is_none() {
- return isolated_hyprland_refusal(
- "an exact window_id or window-bound element token is required",
- );
- }
- let owner = named_session_cursor_key(&args);
- let element = token.clone();
- let (_cancellation, dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- let point = if let Some(element) = &element {
- let (_, x, y) = resolve_element_local_coords(pid, element, Some(xid))?;
- Some((x, y))
- } else {
- pixel_target
- };
- let actions = crate::wayland::hyprland_input::scroll_actions(
- point, &direction, amount, by,
- )?;
- crate::wayland::hyprland_input::execute_scroll(
- owner,
- pid,
- xid,
- actions,
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(refusal) => return refusal,
- };
- return match dispatch.await {
- Ok(result) => isolated_hyprland_result(result),
- Err(error) => isolated_hyprland_task_error(error, false),
- };
- }
-
- if crate::wayland::is_inject_mode() {
- let Some((x, y)) = pixel_target else {
- return crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- };
- let direction_for_inject = direction.clone();
- let result = tokio::task::spawn_blocking(move || {
- crate::wayland::inject_scroll(pid, xid, x, y, &direction_for_inject, amount as u32)
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Scrolled {direction} {amount} ticks (focus-free via cua-compositor)."
- ))
- .with_structured(json!({
- "verified": false,
- "delivery_mode": if delivery.is_foreground() { "foreground" } else { "background" },
- "route": "cua_compositor_inject"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
-
- if crate::wayland::wayland_input_enabled() {
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, false) {
- return refusal;
- }
- let direction_for_wayland = direction.clone();
- let local_point = match (pixel_target, token.clone()) {
- (Some(point), _) => Some(point),
- (None, Some(element)) => {
- let idx = element.index;
- match tokio::task::spawn_blocking(move || {
- resolve_element_local_coords(pid, &element, Some(xid))
- })
- .await
- {
- Ok(Ok((_resolved_xid, x, y))) => Some((x, y)),
- Ok(Err(error)) => {
- return ToolResult::error(format!(
- "Could not resolve scroll element [{idx}] coordinates: {error}"
- ))
- }
- Err(error) => {
- return ToolResult::error(format!(
- "Scroll element coordinate task failed: {error}"
- ))
- }
- }
- }
- (None, None) => None,
- };
- let output_point = match local_point
- .map(|(x, y)| {
- crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32)
- })
- .transpose()
- {
- Ok(point) => point,
- Err(error) => return input_error_result(error),
- };
- let ticks = wheel_ticks(amount, by, crate::wayland::niri::is_session());
- let result =
- spawn_blocking_bounded("scroll", foreground_budget(ticks as usize), move || {
- crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::pointer_gesture_restore(),
- || {
- crate::wayland::scroll_at(
- xid,
- output_point,
- &direction_for_wayland,
- ticks,
- )
- },
- )
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Scrolled {direction} {ticks} ticks (delivery_mode=foreground)."
- ))
- .with_structured(json!({ "verified": false, "delivery_mode": "foreground" })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
-
- if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
- return refusal;
- }
-
- // An element-addressed scroll must land over the element. The old
- // fallback used (0, 0) in the window, which can report success while
- // Chromium/GTK routes the wheel to the toplevel instead of the
- // requested scroll region.
- let element_point = match token {
- Some(element) => {
- let idx = element.index;
- match tokio::task::spawn_blocking(move || {
- resolve_element_local_coords(pid, &element, Some(xid))
- })
- .await
- {
- Ok(Ok((_resolved_xid, local_x, local_y))) => {
- let screen = window_local_to_screen(xid, local_x, local_y).ok();
- Some(((local_x, local_y), screen))
- }
- Ok(Err(e)) => {
- return ToolResult::error(format!(
- "Could not resolve scroll element [{idx}] coordinates: {e}"
- ))
- }
- Err(e) => {
- return ToolResult::error(format!(
- "Scroll element coordinate task failed: {e}"
- ))
- }
- }
- }
- None => pixel_target.map(|local| {
- let screen = window_local_to_screen(xid, local.0, local.1).ok();
- (local, screen)
- }),
- };
-
- // X11 scroll buttons: 4=up, 5=down, 6=left, 7=right
- // Note: "page" scroll is still per-click on X11; send more ticks for page.
- let button: u8 = match direction.as_str() {
- "up" => 4,
- "left" => 6,
- "right" => 7,
- _ => 5,
- };
- let cursor_id_for_task = cursor_id.clone();
- let direction_for_wayland = direction.clone();
- let amount_u32 = amount as u32;
- let result = spawn_blocking_bounded(
- "scroll",
- foreground_budget(amount_u32 as usize),
- move || -> anyhow::Result<()> {
- if crate::wayland::wayland_input_enabled() {
- return crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::pointer_gesture_restore(),
- || crate::wayland::scroll(xid, &direction_for_wayland, amount_u32),
- );
- }
- // foreground: activate the window, then scroll, then restore — for
- // surfaces that only route wheel events to the active window.
- let x11_scroll = || -> anyhow::Result<()> {
- // X11: synthetic Button4-7 XSendEvents are dropped by XInput2
- // toolkits (GTK never scrolls). On a real Xorg host, drive a real
- // wheel detent through the MPX uinput pointer over the window's
- // center — libinput turns it into the XI2 smooth-scroll GTK reads —
- // without stealing focus. Falls back to the legacy XSendEvent
- // Button4-7 path on Xvfb / unsupported servers.
- // Button → axis/sign: 4=up(+v) 5=down(-v) 6=left(-h) 7=right(+h).
- if crate::input::real_pointer_input_available() {
- let pointer_point = element_point
- .and_then(|(_, screen)| screen)
- .map(|(x, y)| (x as i32, y as i32))
- .or_else(|| window_screen_center(xid).ok());
- if let Some((cx, cy)) = pointer_point {
- let horizontal = matches!(button, 6 | 7);
- let ticks = match button {
- 4 | 7 => amount as i32,
- _ => -(amount as i32), // 5 (down) and 6 (left)
- };
- match crate::input::send_virtual_pointer_scroll(
- &cursor_id_for_task,
- &crate::input::VirtualPointerScroll {
- target_window: xid,
- x: cx,
- y: cy,
- horizontal,
- ticks,
- },
- ) {
- Ok(()) => return Ok(()),
- Err(error) if crate::input::is_uinput_unavailable(&error) => {
- return Err(error)
- }
- Err(e) => tracing::warn!("MPX scroll fell back to XSendEvent: {e}"),
- }
- }
- }
- let (local_x, local_y) =
- element_point.map(|(local, _)| local).unwrap_or((0.0, 0.0));
- crate::input::send_click(xid, local_x as i32, local_y as i32, amount, button)
- };
- if delivery.is_foreground() {
- let point = element_point
- .and_then(|(_, screen)| screen)
- .or_else(|| {
- window_screen_center(xid)
- .ok()
- .map(|(x, y)| (x as f64, y as f64))
- })
- .ok_or_else(|| {
- anyhow::anyhow!("could not resolve foreground scroll point")
- })?;
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- crate::input::send_click_xtest_desktop(
- point.0 as i32,
- point.1 as i32,
- button,
- amount,
- )
- },
- )
- .map(|(value, _report)| value)
- } else {
- x11_scroll()
- }
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Scrolled {direction} {amount} ticks (delivery_mode={mode_label})."
- ))
- .with_structured(json!({ "verified": false, "delivery_mode": mode_label })),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// `ScreenshotTool` and `ScreenshotCompatTool` removed in PR #1692 — see the
-// matching note in platform-windows/src/tools/impl_.rs. `get_window_state` is
-// the canonical screenshot path (it always returns a screenshot now); the
-// underlying capture machinery (XGetImage / `import` shell-out / etc.)
-// stays reachable through GetWindowStateTool.
-
-// ── double_click ──────────────────────────────────────────────────────────────
-
-pub struct DoubleClickTool {
- state: Arc<ToolState>,
-}
-static DCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for DoubleClickTool {
- fn def(&self) -> &ToolDef {
- DCLICK_DEF.get_or_init(|| ToolDef {
- name: "double_click".into(),
- description: "Double-click at (x,y) or an element_token (AT-SPI bounds). Background \
- delivery on X11 sends real button events from the session's virtual master pointer \
- (path mpx_pointer); a point covered by another application's window is refused \
- (target_occluded), one covered by this app's own dialog is retargeted to it \
- (retargeted_to). No focus steal. Provide either (window_id + x/y) or (pid + element_token). \
- After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),
- input_schema: json!({"type":"object","required":[],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "coordinate_frame": coordinate_frame_schema(),
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
- return refusal;
- }
- if hyprland_foreground(delivery) {
- // Share exact-target validation and the admitted native click lifecycle.
- let mut args = args;
- args["button"] = json!("left");
- args["count"] = json!(2);
- return ClickTool {
- state: self.state.clone(),
- }
- .invoke(args)
- .await;
- }
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "double_click").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let window_id_resolved: Option<u64> = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
- };
- if let Some(token) = token {
- let idx = token.index;
- let xid_hint = window_id_resolved;
- let target = token.clone();
- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
- checked_element_local_coords(pid, &target, xid_hint)
- })
- .await;
- return match result {
- Ok(Ok((xid, lx, ly))) => {
- if let Ok(Ok(((sx, sy), rect))) = tokio::task::spawn_blocking(move || {
- element_screen_center_and_rect(pid, &token, Some(xid))
- })
- .await
- {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- reveal_pointer_action_target_for(
- &self.state,
- &cursor_id,
- sx,
- sy,
- true,
- Some(rect),
- )
- .await;
- }
- let lxi = lx as i32;
- let lyi = ly as i32;
- let wayland_point = match crate::wayland::wayland_input_enabled()
- .then(|| crate::wayland::window_local_to_output(xid, lxi, lyi))
- .transpose()
- {
- Ok(point) => point,
- Err(error) => return input_error_result(error),
- };
- let cursor_id_for_task = cursor_id.clone();
- let click_result = spawn_blocking_bounded(
- "double_click",
- foreground_budget(0),
- move || -> anyhow::Result<PointerRoute> {
- if crate::wayland::is_inject_mode() {
- return crate::wayland::inject_click(pid, xid, lx, ly, 2, 1)
- .map(|()| PointerRoute::Wayland);
- }
- if crate::wayland::wayland_input_enabled() {
- let aimed = wayland_point.unwrap_or_else(|| (lxi, lyi).into());
- return crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::pointer_gesture_restore(),
- || crate::wayland::click(xid, aimed, 2, 1),
- )
- .map(|()| PointerRoute::Wayland);
- }
- if delivery.is_foreground() {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- let (sx, sy) =
- window_local_to_screen(xid, lxi as f64, lyi as f64)?;
- crate::input::send_click_xtest_desktop(
- sx.round() as i32,
- sy.round() as i32,
- 1,
- 2,
- )
- },
- )
- .map(|(value, _report)| value)
- .map(|()| PointerRoute::Foreground);
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 1, 2)
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- match click_result {
- Ok(Ok(route)) => ToolResult::text(format!(
- "Double-clicked element [{idx}] {}",
- route.text_suffix(mode_label)
- ))
- .with_structured(route.structured(mode_label)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
- Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- if let Some(refusal) = require_point_args("double_click", &args) {
- return refusal;
- }
- let xid = match window_id_resolved {
- Some(v) => v,
- None => return ToolResult::error("Provide either element_token or window_id + x/y."),
- };
- let desktop_frame = desktop_frame_requested(&args);
- let from_zoom = args.bool_or("from_zoom", false);
- let mut x = args.f64_or("x", 0.0);
- let mut y = args.f64_or("y", 0.0);
- if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated once below.
- } else if from_zoom {
- match self.state.zoom_context(&args, pid, Some(xid)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(x, y);
- x = wx;
- y = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- x *= ratio;
- y *= ratio;
- }
- if desktop_frame {
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
- Ok(Ok((lx, ly))) => {
- x = lx;
- y = ly;
- }
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- let wayland_output_point = if crate::wayland::wayland_input_enabled() {
- match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
- Ok(point) => Some(point),
- Err(error) => return input_error_result(error),
- }
- } else {
- None
- };
- let glide_target = if let Some(AimedPoint {
- point: (sx, sy), ..
- }) = wayland_output_point
- {
- Some((sx as f64, sy as f64))
- } else {
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
- .await
- .ok()
- .and_then(|r| r.ok())
- };
- if let Some((sx, sy)) = glide_target {
- reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
- }
- let (xi, yi) = (x as i32, y as i32);
- let cursor_id_for_task = cursor_id.clone();
- let result = spawn_blocking_bounded(
- "double_click",
- foreground_budget(0),
- move || -> anyhow::Result<PointerRoute> {
- if crate::wayland::is_inject_mode() {
- return crate::wayland::inject_click(pid, xid, x, y, 2, 1)
- .map(|()| PointerRoute::Wayland);
- }
- if crate::wayland::wayland_input_enabled() {
- let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
- return crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::pointer_gesture_restore(),
- || crate::wayland::click(xid, aimed, 2, 1),
- )
- .map(|()| PointerRoute::Wayland);
- }
- if delivery.is_foreground() {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- // Real XTest double-click at the screen point — synthetic
- // XSendEvent button events are dropped by GTK/Qt and the MPX
- // uinput path needs /dev/uinput (absent on Xvfb/Xtigervnc), so
- // neither lands. Mirrors the single-click foreground path.
- if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64)
- {
- crate::input::send_click_xtest_desktop(
- sx.round() as i32,
- sy.round() as i32,
- 1,
- 2,
- )?;
- return Ok(());
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)
- .map(|_| ())
- },
- )
- .map(|(value, _report)| value)
- .map(|()| PointerRoute::Foreground);
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- let outcome = match result {
- Ok(Ok(route)) => ToolResult::text(format!(
- "Double-clicked at ({x:.1}, {y:.1}) {}",
- route.text_suffix(mode_label)
- ))
- .with_structured(route.structured(mode_label)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- with_pixel_points(
- outcome,
- glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),
- (xi, yi),
- )
- }
-}
-
-// ── right_click ───────────────────────────────────────────────────────────────
-
-pub struct RightClickTool {
- state: Arc<ToolState>,
-}
-static RCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for RightClickTool {
- fn def(&self) -> &ToolDef {
- RCLICK_DEF.get_or_init(|| ToolDef {
- name: "right_click".into(),
- description: "Right-click at (x,y) or an element_token (AT-SPI bounds). Background \
- delivery on X11 sends real button events from the session's virtual master pointer \
- (path mpx_pointer); a point covered by another application's window is refused \
- (target_occluded), one covered by this app's own dialog is retargeted to it \
- (retargeted_to), one outside the window is refused (point_outside_window; x/y are \
- window-local pixels, the result reports window_point / screen_point). No focus \
- steal. When a context menu opens, the result names it (`popup: {window_id, bounds, \
- title}`): call get_window_state(pid, window_id=<that id>) to index its menu items \
- and click them by element_token. Provide either (window_id + x/y) or (pid + element_token). \
- After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),
- input_schema: json!({"type":"object","required":[],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "coordinate_frame": coordinate_frame_schema(),
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
- return refusal;
- }
- if hyprland_foreground(delivery) {
- // Share exact-target validation and the admitted native click lifecycle.
- let mut args = args;
- args["button"] = json!("right");
- args["count"] = json!(1);
- return ClickTool {
- state: self.state.clone(),
- }
- .invoke(args)
- .await;
- }
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let token = match prove_token_element(pid, &resolved, "right_click").await {
- Ok(token) => token,
- Err(refusal) => return refusal,
- };
- let window_id_resolved: Option<u64> = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
- Some(*window_id)
- }
- cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
- };
- if let Some(token) = token {
- let idx = token.index;
- let xid_hint = window_id_resolved;
- let target = token.clone();
- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
- checked_element_local_coords(pid, &target, xid_hint)
- })
- .await;
- return match result {
- Ok(Ok((xid, lx, ly))) => {
- if let Ok(Ok(((sx, sy), rect))) = tokio::task::spawn_blocking(move || {
- element_screen_center_and_rect(pid, &token, Some(xid))
- })
- .await
- {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- reveal_pointer_action_target_for(
- &self.state,
- &cursor_id,
- sx,
- sy,
- true,
- Some(rect),
- )
- .await;
- }
- let lxi = lx as i32;
- let lyi = ly as i32;
- let wayland_point = match crate::wayland::wayland_input_enabled()
- .then(|| crate::wayland::window_local_to_output(xid, lxi, lyi))
- .transpose()
- {
- Ok(point) => point,
- Err(error) => return input_error_result(error),
- };
- let cursor_id_for_task = cursor_id.clone();
- let click_result = spawn_blocking_bounded(
- "right_click",
- foreground_budget(0),
- move || -> anyhow::Result<PointerRoute> {
- if crate::wayland::is_inject_mode() {
- return crate::wayland::inject_click(pid, xid, lx, ly, 1, 3)
- .map(|()| PointerRoute::Wayland);
- }
- if crate::wayland::wayland_input_enabled() {
- let aimed = wayland_point.unwrap_or_else(|| (lxi, lyi).into());
- return crate::wayland::click(xid, aimed, 1, 3)
- .map(|()| PointerRoute::Wayland);
- }
- if delivery.is_foreground() {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- let (sx, sy) =
- window_local_to_screen(xid, lxi as f64, lyi as f64)?;
- crate::input::send_click_xtest_desktop(
- sx.round() as i32,
- sy.round() as i32,
- 3,
- 1,
- )
- },
- )
- .map(|(value, _report)| value)
- .map(|()| PointerRoute::Foreground);
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 3, 1)
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- match click_result {
- Ok(Ok(route)) => ToolResult::text(format!(
- "Right-clicked element [{idx}] {}",
- route.text_suffix(mode_label)
- ))
- .with_structured(route.structured(mode_label)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
- Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- if let Some(refusal) = require_point_args("right_click", &args) {
- return refusal;
- }
- let xid = match window_id_resolved {
- Some(v) => v,
- None => return ToolResult::error("Provide either element_token or window_id + x/y."),
- };
- let desktop_frame = desktop_frame_requested(&args);
- let from_zoom = args.bool_or("from_zoom", false);
- let mut x = args.f64_or("x", 0.0);
- let mut y = args.f64_or("y", 0.0);
- if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated once below.
- } else if from_zoom {
- match self.state.zoom_context(&args, pid, Some(xid)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(x, y);
- x = wx;
- y = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- x *= ratio;
- y *= ratio;
- }
- if desktop_frame {
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
- Ok(Ok((lx, ly))) => {
- x = lx;
- y = ly;
- }
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- let wayland_output_point = if crate::wayland::wayland_input_enabled() {
- match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
- Ok(point) => Some(point),
- Err(error) => return input_error_result(error),
- }
- } else {
- None
- };
- let glide_target = if let Some(AimedPoint {
- point: (sx, sy), ..
- }) = wayland_output_point
- {
- Some((sx as f64, sy as f64))
- } else {
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
- .await
- .ok()
- .and_then(|r| r.ok())
- };
- if let Some((sx, sy)) = glide_target {
- reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
- }
- let (xi, yi) = (x as i32, y as i32);
- let cursor_id_for_task = cursor_id.clone();
- let result = spawn_blocking_bounded(
- "right_click",
- foreground_budget(0),
- move || -> anyhow::Result<PointerRoute> {
- if crate::wayland::is_inject_mode() {
- return crate::wayland::inject_click(pid, xid, x, y, 1, 3)
- .map(|()| PointerRoute::Wayland);
- }
- if crate::wayland::wayland_input_enabled() {
- let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
- return crate::wayland::click(xid, aimed, 1, 3).map(|()| PointerRoute::Wayland);
- }
- if delivery.is_foreground() {
- return crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- // Real XTest right-click at the screen point (synthetic
- // XSendEvent is dropped by GTK/Qt; MPX needs /dev/uinput).
- // Mirrors the single-click foreground path.
- if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64)
- {
- crate::input::send_click_xtest_desktop(
- sx.round() as i32,
- sy.round() as i32,
- 3,
- 1,
- )?;
- return Ok(());
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)
- .map(|_| ())
- },
- )
- .map(|(value, _report)| value)
- .map(|()| PointerRoute::Foreground);
- }
- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)
- },
- )
- .await;
- let mode_label = if delivery.is_foreground() {
- "foreground"
- } else {
- "background"
- };
- let outcome = match result {
- Ok(Ok(route)) => ToolResult::text(format!(
- "Right-clicked at ({x:.1}, {y:.1}) {}",
- route.text_suffix(mode_label)
- ))
- .with_structured(route.structured(mode_label)),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- with_pixel_points(
- outcome,
- glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),
- (xi, yi),
- )
- }
-}
-
-// ── drag ─────────────────────────────────────────────────────────────────────
-
-/// Typed refusal for a modified drag on native Wayland. The Wayland pointer
-/// routes (virtual pointer, libei, and the nested cua-compositor socket) do not
-/// hold keyboard modifier state across a pointer gesture, so the drag is
-/// refused before any input is dispatched rather than degraded to an
-/// unmodified drag. The stable `code` lets callers branch without parsing text.
-fn wayland_modified_drag_refusal() -> ToolResult {
- const DETAIL: &str = "the pointer route cannot carry keyboard modifier state";
- ToolResult::error(format!(
- "modified drags are unavailable on native Wayland: {DETAIL}"
- ))
- .with_structured(json!({
- "code": "modified_pointer_unavailable",
- "effect": "refused",
- "verified": false,
- "detail": DETAIL,
- }))
-}
-
-pub struct DragTool {
- state: Arc<ToolState>,
-}
-static DRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for DragTool {
- fn has_independent_input_lane(&self, args: &Value) -> bool {
- // This opt-in branch below either uses its own leased synthetic seat
- // or refuses. It never falls back to primary-seat input. All ordinary
- // Linux routes retain the common process-wide input coordinator.
- // Socket availability can change between admission and invocation.
- // Native Hyprland window drags remain fail-closed on their own lane.
- args.opt_str("scope").as_deref() != Some("desktop")
- && !crate::input::delivery::DeliveryMode::from_args(args).is_foreground()
- && crate::wayland::wayland_input_enabled()
- && crate::wayland::hyprland::is_session()
- }
-
- fn def(&self) -> &ToolDef {
- DRAG_DEF.get_or_init(|| ToolDef {
- name: "drag".into(),
- description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in \
- window-local screenshot pixels. Background delivery on X11 is one real held \
- gesture on the session's virtual master pointer (path mpx_pointer: press, \
- interpolated motion, release) — it lands on GTK/VCL/Qt/Chromium; a point \
- covered by another application's window is refused (target_occluded), one \
- covered by this app's own dialog is retargeted to it. Headless servers fall \
- back to XSendEvent. duration_ms (default 500), steps (default 20). No focus steal.".into(),
- input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\"."},
- "window_id":{"type":"integer","description":"Target window XID. Required."},
- "from_x":{"type":"number","description":"Drag start X: window-local pixels of the target window's own get_window_state screenshot; pass scope:\"desktop\" for get_desktop_state pixels."},
- "from_y":{"type":"number","description":"Drag start Y (same frame as from_x)."},
- "to_x":{"type":"number","description":"Drag end X (same frame as from_x)."},
- "to_y":{"type":"number","description":"Drag end Y (same frame as from_x)."},
- "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},
- "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "button": cua_driver_core::tool_schema::button_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid/window_id for get_desktop_state screen coordinates. Default \"window\"."},
- "coordinate_frame": coordinate_frame_schema(),
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let modifiers: Vec<String> = args.str_array("modifier");
- if args.opt_str("scope").as_deref() == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<DragInput>("drag", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let (from_x, from_y) = (input.from_x, input.from_y);
- let (to_x, to_y) = (input.to_x, input.to_y);
- let button = parse_mouse_button(input.button.unwrap_or(ClickButton::Left).as_str());
- let duration_ms = input.duration_ms.unwrap_or(500).min(10_000);
- let steps = input.steps.unwrap_or(20).clamp(1, 200) as usize;
- let modifiers = input.modifier.unwrap_or_default();
- let wayland = crate::wayland::wayland_input_enabled();
- if wayland && !modifiers.is_empty() {
- return wayland_modified_drag_refusal();
- }
- let path = if wayland { "wayland_desktop" } else { "xtest" };
- let space = match desktop_input_space().await {
- Ok(space) => space,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- // The overlay draws in layout coordinates, like the input below.
- let overlay_point = |x: f64, y: f64| {
- let (lx, ly) = space.to_layout(x.round() as i32, y.round() as i32);
- (f64::from(lx), f64::from(ly))
- };
- let (overlay_from, overlay_to) =
- (overlay_point(from_x, from_y), overlay_point(to_x, to_y));
- let result = spawn_blocking_bounded(
- "desktop drag",
- foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
- move || {
- if wayland {
- crate::wayland::drag_desktop(
- &space,
- from_x.round() as i32,
- from_y.round() as i32,
- to_x.round() as i32,
- to_y.round() as i32,
- steps as u32,
- duration_ms,
- button,
- )
- } else {
- let modifier_refs: Vec<&str> =
- modifiers.iter().map(String::as_str).collect();
- crate::input::send_drag_xtest_desktop_with_modifiers(
- from_x.round() as i32,
- from_y.round() as i32,
- to_x.round() as i32,
- to_y.round() as i32,
- button,
- duration_ms,
- steps,
- &modifier_refs,
- )
- }
- },
- );
- let visual_drag = track_overlay_drag_for(
- cursor_id.clone(),
- overlay_from,
- overlay_to,
- duration_ms,
- steps,
- );
- let (result, ()) = tokio::join!(result, visual_drag);
- if matches!(&result, Ok(Ok(()))) {
- self.state
- .cursor_registry
- .update_position(&cursor_id, overlay_to.0, overlay_to.1);
- }
- return match result {
- Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(
- json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
- input_error_result(error)
- }
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let xid = match args.opt_u64("window_id") {
- Some(v) => v,
- None => return ToolResult::error("window_id is required on Linux."),
- };
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- let isolated_background = isolated_hyprland_background(delivery);
- let native_refusal = if !isolated_background {
- if crate::wayland::wayland_input_enabled() && !modifiers.is_empty() {
- return wayland_modified_drag_refusal();
- }
- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
- return refusal;
- }
- if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
- return refusal;
- }
- unavailable_wayland_focused_input_background(delivery, true)
- } else {
- None
- };
-
- let from_zoom = args.bool_or("from_zoom", false);
- let desktop_frame = desktop_frame_requested(&args);
- let context = match coordinate_drag_context(native_refusal, || {
- if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated below.
- Ok(CoordinateContext::Screenshot(1.0))
- } else if from_zoom {
- self.state
- .zoom_context(&args, pid, Some(xid))
- .map(CoordinateContext::Zoom)
- } else {
- screenshot_scale(&self.state, &args, pid, Some(xid))
- .map(CoordinateContext::Screenshot)
- }
- }) {
- Ok(context) => context,
- Err(refusal) => return refusal,
- };
-
- let coerce = |key: &str| -> Option<f64> {
- args.opt_f64(key)
- .or_else(|| args.opt_i64(key).map(|i| i as f64))
- };
- let mut from_x = match coerce("from_x") {
- Some(v) => v,
- None => {
- return invalid_pointer_arguments(
- "drag",
- "from_x is missing (from_x/from_y/to_x/to_y are all required)",
- )
- }
- };
- let mut from_y = match coerce("from_y") {
- Some(v) => v,
- None => {
- return invalid_pointer_arguments(
- "drag",
- "from_y is missing (from_x/from_y/to_x/to_y are all required)",
- )
- }
- };
- let mut to_x = match coerce("to_x") {
- Some(v) => v,
- None => return ToolResult::error("Missing: to_x"),
- };
- let mut to_y = match coerce("to_y") {
- Some(v) => v,
- None => return ToolResult::error("Missing: to_y"),
- };
-
- let duration_ms = args.u64_or("duration_ms", 500);
- let steps = drag_steps(&args);
- let button_str = args.str_or("button", "left");
- let button = parse_mouse_button(button_str.as_str());
- match context {
- CoordinateContext::Zoom(context) => {
- (from_x, from_y) = context.zoom_to_window(from_x, from_y);
- (to_x, to_y) = context.zoom_to_window(to_x, to_y);
- }
- CoordinateContext::Screenshot(scale) => {
- from_x *= scale;
- from_y *= scale;
- to_x *= scale;
- to_y *= scale;
- }
- }
- if desktop_frame {
- let mapped = tokio::task::spawn_blocking(move || {
- Ok::<_, anyhow::Error>((
- desktop_to_window_local(xid, from_x, from_y)?,
- desktop_to_window_local(xid, to_x, to_y)?,
- ))
- })
- .await;
- match mapped {
- Ok(Ok(((fx, fy), (tx, ty)))) => {
- from_x = fx;
- from_y = fy;
- to_x = tx;
- to_y = ty;
- }
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
-
- if hyprland_foreground(delivery) {
- if button_str != "left" || args.get("modifier").is_some_and(|value| !value.is_null()) {
- return foreground_hyprland_refusal(
- "foreground drag supports only an unmodified left button",
- );
- }
- if !crate::wayland::hyprland_input::enabled() {
- return foreground_hyprland_refusal(
- "production Hyprland input plugin is unavailable",
- );
- }
- let owner = named_session_cursor_key(&args);
- let points = crate::wayland::window_local_to_output(
- xid,
- from_x.round() as i32,
- from_y.round() as i32,
- )
- .and_then(|from| {
- crate::wayland::window_local_to_output(
- xid,
- to_x.round() as i32,
- to_y.round() as i32,
- )
- .map(|to| (from.point, to.point))
- });
- let (from, to) = match points {
- Ok(points) => points,
- Err(error) => return input_error_result(error),
- };
- let (started, acknowledged) = tokio::sync::oneshot::channel();
- let (_cancellation, mut dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- crate::wayland::hyprland_input::execute_foreground_with_started(
- owner,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Drag {
- from: (from_x, from_y),
- to: (to_x, to_y),
- duration_ms,
- },
- Some(started),
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(_) => {
- return foreground_hyprland_refusal(
- "authenticated admitted lifecycle required",
- )
- }
- };
- let acknowledged = acknowledged.await.is_ok();
- let result = if acknowledged {
- overlay_snap_to_for(&cursor_id, from.0 as f64, from.1 as f64, None);
- tokio::select! {
- result = &mut dispatch => result,
- () = track_overlay_drag_for(cursor_id.clone(), (from.0 as f64, from.1 as f64),
- (to.0 as f64, to.1 as f64), duration_ms, steps) => dispatch.await,
- }
- } else {
- dispatch.await
- };
- return hyprland_input_result(
- match result {
- Ok(result) => result,
- Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
- error.into(),
- u32::from(acknowledged),
- )),
- },
- true,
- );
- }
-
- if isolated_background {
- if button_str != "left" || args.get("modifier").is_some_and(|value| !value.is_null()) {
- return isolated_hyprland_refusal(
- "isolated drag supports only an unmodified left button",
- );
- }
- let owner = named_session_cursor_key(&args);
- let points = crate::wayland::window_local_to_output(
- xid,
- from_x.round() as i32,
- from_y.round() as i32,
- )
- .and_then(|from| {
- crate::wayland::window_local_to_output(
- xid,
- to_x.round() as i32,
- to_y.round() as i32,
- )
- .map(|to| (from.point, to.point))
- });
- let (from, to) = match points {
- Ok(points) => points,
- Err(error) => return input_error_result(error),
- };
- let (started, acknowledged) = tokio::sync::oneshot::channel();
- let (_cancellation, mut dispatch) =
- match spawn_isolated_hyprland(&args, move |cancellation| {
- crate::wayland::hyprland_input::execute_with_started(
- owner,
- pid,
- xid,
- crate::wayland::hyprland_input::Action::Drag {
- from: (from_x, from_y),
- to: (to_x, to_y),
- duration_ms,
- },
- Some(started),
- cancellation,
- )
- }) {
- Ok(dispatch) => dispatch,
- Err(refusal) => return refusal,
- };
- // Do not animate a drag that was refused. The compositor starts
- // the overlay clock only after accepting and pressing the button.
- let acknowledged = acknowledged.await.is_ok();
- if acknowledged {
- overlay_snap_to_for(&cursor_id, from.0 as f64, from.1 as f64, None);
- tokio::select! {
- result = &mut dispatch => {
- return match result { Ok(result) => isolated_hyprland_result(result), Err(error) => isolated_hyprland_task_error(error, acknowledged) };
- }
- () = track_overlay_drag_for(cursor_id.clone(), (from.0 as f64, from.1 as f64),
- (to.0 as f64, to.1 as f64), duration_ms, steps) => {}
- }
- }
- return match dispatch.await {
- Ok(result) => isolated_hyprland_result(result),
- Err(error) => isolated_hyprland_task_error(error, acknowledged),
- };
- }
-
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- let wayland_points = match crate::wayland::wayland_input_enabled()
- .then(|| {
- crate::wayland::window_local_to_output(
- xid,
- from_x.round() as i32,
- from_y.round() as i32,
- )
- .and_then(|from| {
- crate::wayland::window_local_to_output(
- xid,
- to_x.round() as i32,
- to_y.round() as i32,
- )
- .map(|to| (from, to))
- })
- })
- .transpose()
- {
- Ok(points) => points,
- Err(error) => return input_error_result(error),
- };
- let screen_from = if let Some((from, _)) = wayland_points {
- Some((from.point.0 as f64, from.point.1 as f64))
- } else {
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, from_x, from_y))
- .await
- .ok()
- .and_then(|result| result.ok())
- };
- if let Some((sx_from, sy_from)) = screen_from {
- overlay_glide_to_for(&cursor_id, sx_from, sy_from).await;
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx_from, sy_from);
- overlay_snap_to_for(&cursor_id, sx_from, sy_from, None);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx_from,
- y: sy_from,
- },
- );
- }
- // Native Wayland: emit press + interpolated motion + release as one
- // virtual-pointer (wlroots) or libei (GNOME/KDE) sequence, output-relative
- // coords. Returns early so we don't fall into the X11 XSendEvent loop below.
- if crate::wayland::wayland_input_enabled() {
- let steps_u32 = steps as u32;
- let inject = crate::wayland::is_inject_mode();
- let (from_aimed, to_aimed) = wayland_points.unwrap_or_else(|| {
- (
- (from_x.round() as i32, from_y.round() as i32).into(),
- (to_x.round() as i32, to_y.round() as i32).into(),
- )
- });
- let drag_result = spawn_blocking_bounded(
- "drag",
- foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
- move || {
- if inject {
- return crate::wayland::inject_drag(
- pid,
- xid,
- (from_x, from_y),
- (to_x, to_y),
- steps,
- button as u32,
- );
- }
- crate::interaction::with_target_foreground(
- pid,
- xid,
- crate::interaction::pointer_gesture_restore(),
- || {
- crate::wayland::drag(
- xid,
- from_aimed,
- to_aimed,
- steps_u32,
- duration_ms,
- button,
- )
- },
- )
- },
- );
- let ((from_output_x, from_output_y), (to_output_x, to_output_y)) =
- (from_aimed.point, to_aimed.point);
- let visual_drag = track_overlay_drag_for(
- cursor_id.clone(),
- (f64::from(from_output_x), f64::from(from_output_y)),
- (f64::from(to_output_x), f64::from(to_output_y)),
- duration_ms,
- steps,
- );
- let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
- if matches!(&drag_result, Ok(Ok(()))) {
- self.state.cursor_registry.update_position(
- &cursor_id,
- f64::from(to_output_x),
- f64::from(to_output_y),
- );
- }
- return match drag_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Posted drag ({button_str}) to pid {pid} \
- from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \
- in {duration_ms}ms / {steps} steps."
- )),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- if delivery.is_foreground() {
- let screen_points = tokio::task::spawn_blocking(move || {
- Ok::<_, anyhow::Error>((
- window_local_to_screen(xid, from_x, from_y)?,
- window_local_to_screen(xid, to_x, to_y)?,
- ))
- })
- .await;
- let ((screen_from_x, screen_from_y), (screen_to_x, screen_to_y)) = match screen_points {
- Ok(Ok(points)) => points,
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- };
- let drag_result = spawn_blocking_bounded(
- "foreground drag",
- foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
- move || {
- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::interaction::with_x11_foreground(
- xid,
- crate::input::ForegroundOptions::pointer(),
- || {
- crate::input::send_drag_xtest_desktop_with_modifiers(
- screen_from_x.round() as i32,
- screen_from_y.round() as i32,
- screen_to_x.round() as i32,
- screen_to_y.round() as i32,
- button,
- duration_ms,
- steps,
- &modifier_refs,
- )
- },
- )
- .map(|(value, _report)| value)
- },
- );
- let visual_drag = track_overlay_drag_for(
- cursor_id.clone(),
- (screen_from_x, screen_from_y),
- (screen_to_x, screen_to_y),
- duration_ms,
- steps,
- );
- let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
- if matches!(&drag_result, Ok(Ok(()))) {
- self.state
- .cursor_registry
- .update_position(&cursor_id, screen_to_x, screen_to_y);
- }
- return match drag_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Dragged ({button_str}) to pid {pid} from ({from_x:.0}, {from_y:.0}) \
- to ({to_x:.0}, {to_y:.0}) in {duration_ms}ms / {steps} steps \
- (delivery_mode=foreground)."
- ))
- .with_structured(json!({
- "path": "x11_xtest_fg",
- "verified": false,
- "delivery_mode": "foreground"
- })),
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Background X11 drag. With a real Xorg server and /dev/uinput the
- // gesture is a held press + interpolated motion + release on the
- // session's MPX virtual master pointer (GTK/VCL/Qt/Chromium drop the
- // synthetic XSendEvent gesture below); the target's screen points are
- // resolved once and the agent cursor tracks the glide.
- if crate::input::real_pointer_input_available() {
- let screen_points = tokio::task::spawn_blocking(move || {
- Ok::<_, anyhow::Error>((
- window_local_to_screen(xid, from_x, from_y)?,
- window_local_to_screen(xid, to_x, to_y)?,
- ))
- })
- .await;
- let ((sfx, sfy), (stx, sty)) = match screen_points {
- Ok(Ok(points)) => points,
- Ok(Err(e)) => return ToolResult::error(format!("{e:#}")),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- };
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
- let cursor_id_for_task = cursor_id.clone();
- let drag_result = spawn_blocking_bounded(
- "drag",
- foreground_budget(steps) + std::time::Duration::from_millis(duration_ms),
- move || {
- let (outcome, guard) =
- crate::input::focus_guard::guarded_settled(Some(pid), || {
- Ok(crate::input::send_virtual_pointer_drag(
- &cursor_id_for_task,
- &crate::input::VirtualPointerDrag {
- target_window: xid,
- button,
- path: vec![
- (sfx.round() as i32, sfy.round() as i32),
- (stx.round() as i32, sty.round() as i32),
- ],
- duration_ms,
- steps,
- },
- ))
- })?;
- outcome.map(|mut effect| {
- effect.focus_guard = guard;
- effect
- })
- },
- )
- .await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- if matches!(&drag_result, Ok(Ok(_))) {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::track_pointer_command(stx, sty),
- );
- self.state
- .cursor_registry
- .update_position(&cursor_id, stx, sty);
- }
- return match drag_result {
- Ok(Ok(effect)) => {
- let route = PointerRoute::Mpx(effect);
- ToolResult::text(format!(
- "Dragged ({button_str}) pid {pid} from ({from_x:.0}, {from_y:.0}) to \
- ({to_x:.0}, {to_y:.0}) in {duration_ms}ms / {steps} steps with a real \
- held pointer gesture {}",
- route.text_suffix("background")
- ))
- .with_structured(route.structured("background"))
- }
- Ok(Err(e)) => input_error_result(e),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Legacy target-addressed XSendEvent gesture (headless servers / no
- // uinput): only core-protocol toolkits accept it.
- if synthetic_pointer_is_dropped(pid) {
- return crate::input::delivery::background_unavailable_error(
- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
- );
- }
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
- let press_result = press_x11_drag_start(
- foreground_budget(0),
- move || {
- crate::input::send_button_down(
- xid,
- from_x.round() as i32,
- from_y.round() as i32,
- button,
- )
- },
- move || {
- crate::input::send_button_up(
- xid,
- from_x.round() as i32,
- from_y.round() as i32,
- button,
- )
- },
- )
- .await;
- let mut result: anyhow::Result<()> = match press_result {
- Ok(Ok(())) => Ok(()),
- Ok(Err(e)) => Err(e),
- Err(e) => Err(anyhow::anyhow!("Task error: {e}")),
- };
-
- if result.is_ok() {
- let step_delay_ms = if steps > 1 {
- duration_ms / steps as u64
- } else {
- duration_ms
- };
- for i in 1..=steps {
- let t = i as f64 / steps.max(1) as f64;
- let ix = from_x + (to_x - from_x) * t;
- let iy = from_y + (to_y - from_y) * t;
- let motion_result =
- spawn_blocking_bounded("drag", foreground_budget(0), move || {
- crate::input::send_motion(
- xid,
- ix.round() as i32,
- iy.round() as i32,
- Some(button),
- )
- })
- .await;
- match motion_result {
- Ok(Ok(())) => {
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy))
- .await
- {
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::track_pointer_command(sx, sy),
- );
- }
- if step_delay_ms > 0 {
- tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms))
- .await;
- }
- }
- Ok(Err(e)) => {
- result = Err(e);
- break;
- }
- Err(e) => {
- result = Err(anyhow::anyhow!("Task error: {e}"));
- break;
- }
- }
- }
- }
-
- // XSendEvent does not read the call's action, so the release still
- // goes out if the call ends first.
- let release_result = spawn_blocking_bounded("drag", foreground_budget(0), move || {
- crate::input::send_button_up(xid, to_x.round() as i32, to_y.round() as i32, button)
- })
- .await;
- if result.is_ok() {
- result = match release_result {
- Ok(Ok(())) => Ok(()),
- Ok(Err(e)) => Err(e),
- Err(e) => Err(anyhow::anyhow!("Task error: {e}")),
- };
- }
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
-
- if result.is_ok() {
- if let Ok(Ok((sx_to, sy_to))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y)).await
- {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::track_pointer_command(sx_to, sy_to),
- );
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx_to, sy_to);
- }
- }
-
- let route = PointerRoute::Synthetic;
- match result {
- Ok(()) => ToolResult::text(format!(
- "Posted drag ({button_str}) to pid {pid} \
- from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \
- in {duration_ms}ms / {steps} steps {}",
- route.text_suffix("background")
- ))
- .with_structured(route.structured("background")),
- Err(e) => input_error_result(e),
- }
- }
-}
-
-// ── mouse_button_down / mouse_drag / mouse_button_up ────────────────────────
-
-pub struct MouseButtonDownTool {
- state: Arc<ToolState>,
-}
-static MDOWN_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for MouseButtonDownTool {
- fn def(&self) -> &ToolDef {
- MDOWN_DEF.get_or_init(|| ToolDef {
- name: "mouse_button_down".into(),
- description: "Press and hold a mouse button at (x,y) via background X11 delivery. \
- Does not release the button; pair with mouse_drag / mouse_button_up. \
- Returns the current held-button state.".into(),
- input_schema: json!({"type":"object","required":["pid","window_id","x","y"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema_with(
- "When both are present, session takes precedence over cursor_id."
- ),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Target process ID."},
- "window_id":{"type":"integer","description":"Window id from list_windows."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "coordinate_frame": coordinate_frame_schema(),
- "button": cua_driver_core::tool_schema::button_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- if let Some(held) = self
- .state
- .mouse_hold
- .lock()
- .unwrap()
- .get(&cursor_id)
- .cloned()
- {
- return ToolResult::error(format!(
- "Cursor '{cursor_id}' already has a held mouse button. Call mouse_button_up first."
- ))
- .with_structured(mouse_hold_json(&cursor_id, Some(&held)));
- }
-
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let xid = match args.opt_u64("window_id") {
- Some(v) => v,
- None => return ToolResult::error("window_id is required on Linux."),
- };
- let button_name = args.str_or("button", "left");
- let button = parse_mouse_button(button_name.as_str());
- let mut x = args.f64_or("x", 0.0);
- let mut y = args.f64_or("y", 0.0);
- let desktop_frame = desktop_frame_requested(&args);
- if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated once below.
- } else if args.bool_or("from_zoom", false) {
- match self.state.zoom_context(&args, pid, Some(xid)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(x, y);
- x = wx;
- y = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- x *= ratio;
- y *= ratio;
- }
- if desktop_frame {
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
- Ok(Ok((lx, ly))) => {
- x = lx;
- y = ly;
- }
- Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- }
-
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
- {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- }
-
- let xi = x as i32;
- let yi = y as i32;
- let is_wl = crate::wayland::is_wayland();
- // Native Wayland: route through the persistent virtual-pointer module
- // so the held button survives across tool calls; the X11 path keeps
- // the existing input::send_button_down behaviour.
- let result = if is_wl {
- let cid = cursor_id.clone();
- let aimed = match persistent_pointer_coordinates(
- is_wl,
- xid,
- xi,
- yi,
- crate::wayland::window_local_to_output,
- ) {
- Ok(point) => point,
- Err(error) => return input_error_result(error),
- };
- let state = self.state.clone();
- let hold = MouseHoldState {
- pid,
- xid,
- button,
- x,
- y,
- };
- spawn_blocking_bounded("mouse_button_down", foreground_budget(0), move || {
- crate::wayland::persistent_vptr::press(&cid, xid, aimed, button, || {
- state.mouse_hold.lock().unwrap().insert(cid.clone(), hold);
- })
- })
- .await
- } else {
- let hold = MouseHoldState {
- pid,
- xid,
- button,
- x,
- y,
- };
- press_and_record_x11_hold(self.state.clone(), cursor_id.clone(), hold, move || {
- crate::input::send_button_down(xid, xi, yi, button)
- })
- .await
- };
- match result {
- Ok(Ok(())) => {
- let hold = MouseHoldState {
- pid,
- xid,
- button,
- x,
- y,
- };
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
- {
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- overlay_snap_to_for(&cursor_id, sx, sy, None);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
- }
- ToolResult::text(format!(
- "✅ Cursor '{cursor_id}' held {} button down at ({x:.1}, {y:.1}).",
- mouse_button_name(button),
- ))
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()).with_structured(mouse_hold_json(
- &cursor_id,
- self.state.mouse_hold.lock().unwrap().get(&cursor_id),
- )),
- Err(e) => {
- ToolResult::error(format!("Task error: {e}")).with_structured(mouse_hold_json(
- &cursor_id,
- self.state.mouse_hold.lock().unwrap().get(&cursor_id),
- ))
- }
- }
- }
-}
-
-/// The press of an X11 step-by-step drag. A press that goes down after its
-/// call ended is released here, in the same worker: the drag's own release
-/// runs in another worker on its own X connection and can reach the app
-/// first, which would leave the app's last event a press.
-async fn press_x11_drag_start(
- budget: std::time::Duration,
- press: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
- release: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
-) -> Result<anyhow::Result<()>, tokio::task::JoinError> {
- spawn_blocking_bounded("drag", budget, move || {
- press()?;
- if crate::input::action::admit(|| ()).is_err() {
- release()?;
- }
- Ok(())
- })
- .await
-}
-
-/// Press an X11 button and record its hold as soon as it is down, so a caller
-/// that goes away before the await resumes still leaves a hold that
-/// mouse_button_up can release. The worker keeps the call's desktop lease
-/// through both, so the next call on the cursor sees this hold instead of
-/// pressing and recording its own, which this one would then overwrite.
-async fn press_and_record_x11_hold(
- state: Arc<ToolState>,
- cursor_id: String,
- hold: MouseHoldState,
- press: impl FnOnce() -> anyhow::Result<()> + Send + 'static,
-) -> Result<anyhow::Result<()>, tokio::task::JoinError> {
- spawn_blocking_bounded("mouse_button_down", foreground_budget(0), move || {
- press()?;
- state.mouse_hold.lock().unwrap().insert(cursor_id, hold);
- Ok(())
- })
- .await
-}
-
-pub struct MouseDragTool {
- state: Arc<ToolState>,
-}
-static MDRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for MouseDragTool {
- fn def(&self) -> &ToolDef {
- MDRAG_DEF.get_or_init(|| ToolDef {
- name: "mouse_drag".into(),
- description: "Move a previously-held mouse button to a new point via background X11 delivery. \
- Requires an active mouse_button_down state; does not release the button. \
- Returns the updated held-button state.".into(),
- input_schema: json!({"type":"object","required":["x","y"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema_with(
- "When both are present, session takes precedence over cursor_id."
- ),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Optional process ID; must match the held button's target."},
- "window_id":{"type":"integer","description":"Optional window id; must match the held button's target."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},
- "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let Some(mut hold) = self
- .state
- .mouse_hold
- .lock()
- .unwrap()
- .get(&cursor_id)
- .cloned()
- else {
- return ToolResult::error(format!(
- "No mouse button is currently held for cursor '{cursor_id}'. Call mouse_button_down first."
- ))
- .with_structured(mouse_hold_json(&cursor_id, None));
- };
- if let Some(err) = held_target_mismatch(&args, &cursor_id, &hold) {
- return err;
- }
-
- if let Some(refusal) = require_point_args("mouse_drag", &args) {
- return refusal;
- }
- let mut to_x = args.f64_or("x", 0.0);
- let mut to_y = args.f64_or("y", 0.0);
- if args.bool_or("from_zoom", false) {
- match self.state.zoom_context(&args, hold.pid, Some(hold.xid)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(to_x, to_y);
- to_x = wx;
- to_y = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, hold.pid, Some(hold.xid)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- to_x *= ratio;
- to_y *= ratio;
- }
-
- let xid = hold.xid;
-
- let from_x = hold.x;
- let from_y = hold.y;
- let duration_ms = args.u64_or("duration_ms", 500);
- let steps = drag_steps(&args);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, from_x, from_y)).await
- {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- overlay_snap_to_for(&cursor_id, sx, sy, None);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
- }
-
- let button = hold.button;
- let step_delay_ms = if steps > 1 {
- duration_ms / steps as u64
- } else {
- duration_ms
- };
- let mut result: anyhow::Result<()> = Ok(());
- let mut moved = 0;
- let mut prev_x = from_x;
- let mut prev_y = from_y;
- let is_wl = crate::wayland::is_wayland();
- for i in 1..=steps {
- let t = i as f64 / steps as f64;
- let ix = from_x + (to_x - from_x) * t;
- let iy = from_y + (to_y - from_y) * t;
- // Native Wayland: route motion through the persistent virtual-
- // pointer so the held button stays held across the entire drag;
- // X11 keeps the existing input::send_motion path.
- let cid_inner = cursor_id.clone();
- let move_result = if is_wl {
- match persistent_pointer_coordinates(
- is_wl,
- xid,
- ix.round() as i32,
- iy.round() as i32,
- crate::wayland::window_local_to_output,
- ) {
- Ok(AimedPoint {
- point: (output_x, output_y),
- ..
- }) => {
- spawn_blocking_bounded("mouse_drag", foreground_budget(0), move || {
- crate::wayland::persistent_vptr::move_to(&cid_inner, output_x, output_y)
- })
- .await
- }
- Err(error) => Ok(Err(error)),
- }
- } else {
- spawn_blocking_bounded("mouse_drag", foreground_budget(0), move || {
- crate::input::send_motion(
- xid,
- ix.round() as i32,
- iy.round() as i32,
- Some(button),
- )
- })
- .await
- };
- match move_result {
- Ok(Ok(())) => {
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy))
- .await
- {
- let heading = if (ix - prev_x).abs() > f64::EPSILON
- || (iy - prev_y).abs() > f64::EPSILON
- {
- Some((iy - prev_y).atan2(ix - prev_x))
- } else {
- None
- };
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- overlay_move_to_for(&cursor_id, sx, sy, heading);
- }
- prev_x = ix;
- prev_y = iy;
- moved = i;
- if step_delay_ms > 0 {
- tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms)).await;
- }
- }
- Ok(Err(e)) => {
- result = Err(e);
- break;
- }
- Err(e) => {
- result = Err(anyhow::anyhow!("Task error: {e}"));
- break;
- }
- }
- }
-
- match result {
- Ok(()) => {
- hold.x = to_x;
- hold.y = to_y;
- self.state
- .mouse_hold
- .lock()
- .unwrap()
- .insert(cursor_id.clone(), hold.clone());
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y))
- .await
- {
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- overlay_snap_to_for(
- &cursor_id,
- sx,
- sy,
- Some((to_y - from_y).atan2(to_x - from_x)),
- );
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- }
- ToolResult::text(format!(
- "✅ Cursor '{cursor_id}' dragged held {} button to ({to_x:.1}, {to_y:.1}).",
- mouse_button_name(hold.button),
- ))
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
- }
- Err(e) if e.is::<crate::wayland::persistent_vptr::HeldPointerEnded>() => {
- // The persistent pointer released the button and discarded
- // the cursor: it holds nothing any more.
- self.state.mouse_hold.lock().unwrap().remove(&cursor_id);
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- let mut structured = mouse_hold_json(&cursor_id, None);
- structured["effect"] = json!("partial");
- ToolResult::error(format!(
- "mouse_drag stopped after {moved} of {steps} steps: {e}"
- ))
- .with_structured(structured)
- }
- Err(e) => ToolResult::error(e.to_string())
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
- }
- }
-}
-
-pub struct MouseButtonUpTool {
- state: Arc<ToolState>,
-}
-static MUP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for MouseButtonUpTool {
- fn def(&self) -> &ToolDef {
- MUP_DEF.get_or_init(|| ToolDef {
- name: "mouse_button_up".into(),
- description: "Release a previously-held mouse button via background X11 delivery. \
- If x/y are omitted, releases at the last held position. Returns the current held-button state.".into(),
- input_schema: json!({"type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema_with(
- "When both are present, session takes precedence over cursor_id."
- ),
- "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
- "pid":{"type":"integer","description":"Optional process ID; must match the held button's target."},
- "window_id":{"type":"integer","description":"Optional window id; must match the held button's target."},
- "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
- "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
- "coordinate_frame": coordinate_frame_schema(),
- "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_id = resolve_cursor_key(&args);
- let Some(hold) = self
- .state
- .mouse_hold
- .lock()
- .unwrap()
- .get(&cursor_id)
- .cloned()
- else {
- // A press whose recording failed and whose release niri did not
- // confirm stays with the persistent pointer: release it there.
- if crate::wayland::is_wayland() {
- let cid = cursor_id.clone();
- let released =
- spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
- crate::wayland::persistent_vptr::release_unrecorded(&cid)
- })
- .await;
- let unheld = mouse_hold_json(&cursor_id, None);
- match released {
- Ok(Ok(true)) => {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- return ToolResult::text(format!(
- "✅ Cursor '{cursor_id}' released a button an earlier call pressed but could not record."
- ))
- .with_structured(unheld);
- }
- Ok(Ok(false)) => {}
- Ok(Err(e)) => return ToolResult::error(e.to_string()).with_structured(unheld),
- Err(e) => {
- return ToolResult::error(format!("Task error: {e}"))
- .with_structured(unheld)
- }
- }
- }
- return ToolResult::error(format!(
- "No mouse button is currently held for cursor '{cursor_id}'."
- ))
- .with_structured(mouse_hold_json(&cursor_id, None));
- };
- if let Some(err) = held_target_mismatch(&args, &cursor_id, &hold) {
- return err;
- }
-
- let xid = hold.xid;
-
- let desktop_frame = args.opt_f64("x").is_some()
- && args.opt_f64("y").is_some()
- && desktop_frame_requested(&args);
- let (mut x, mut y) = if desktop_frame {
- // Desktop pixels: no zoom/screenshot scaling; translated below.
- (args.f64_or("x", hold.x), args.f64_or("y", hold.y))
- } else {
- match mouse_button_up_coordinates(&self.state, &args, &hold) {
- Ok(point) => point,
- Err(refusal) => return refusal,
- }
- };
- if desktop_frame {
- match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
- Ok(Ok((lx, ly))) => {
- x = lx;
- y = ly;
- }
- Ok(Err(e)) => {
- return ToolResult::error(format!(
- "desktop-frame coordinates could not be mapped into window {xid}: {e}"
- ))
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
- }
- Err(e) => {
- return ToolResult::error(format!("Task error: {e}"))
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
- }
- }
- }
-
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
- {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- }
-
- let button = hold.button;
- let xi = x as i32;
- let yi = y as i32;
- // Native Wayland: release through the persistent virtual-pointer so
- // the same vptr device that emitted the press also emits the release
- // (single logical drag rather than a click pair).
- let result = if crate::wayland::is_wayland() {
- let cid = cursor_id.clone();
- spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
- crate::wayland::persistent_vptr::release(&cid, button)
- })
- .await
- } else {
- // XSendEvent does not read the call's action, so the release
- // still goes out if the call ends first.
- spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
- crate::input::send_button_up(xid, xi, yi, button)
- })
- .await
- };
- match result {
- Ok(Ok(())) => {
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
- {
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx, sy);
- overlay_snap_to_for(&cursor_id, sx, sy, None);
- }
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- clear_mouse_hold_after_release(&self.state, &cursor_id);
- let cleared = mouse_hold_json(&cursor_id, None);
- ToolResult::text(format!(
- "✅ Cursor '{cursor_id}' released held {} button at ({x:.1}, {y:.1}).",
- mouse_button_name(button),
- ))
- .with_structured(cleared)
- }
- Ok(Err(e)) => ToolResult::error(e.to_string())
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
- Err(e) => ToolResult::error(format!("Task error: {e}"))
- .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
- }
- }
-}
-
-pub struct ParallelMouseDragTool {
- state: Arc<ToolState>,
-}
-static PMDRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-/// cua-compositor path for parallel_mouse_drag: build window-local drag paths
-/// and run them as concurrent multi-cursor injections over the control socket.
-/// Coordinates stay window-local (the compositor maps them per app_id), so no
-/// X11 geometry/MPX is needed — the X11 path's hard blocker on Wayland.
-async fn parallel_drag_inject(args: &Value) -> ToolResult {
- let Some(items) = args.get("drags").and_then(|v| v.as_array()) else {
- return ToolResult::error("drags[] is required.");
- };
- if items.len() < 2 {
- return ToolResult::error("parallel_mouse_drag requires at least two drag items.");
- }
- let mut drags = Vec::with_capacity(items.len());
- for (i, item) in items.iter().enumerate() {
- let Some(xid) = item.get("window_id").and_then(|v| v.as_u64()) else {
- return ToolResult::error("each drag item requires window_id.");
- };
- let local: Vec<(f64, f64)> = if let Some(pts) = item.get("path").and_then(|v| v.as_array())
- {
- pts.iter()
- .filter_map(|p| {
- let a = p.as_array()?;
- Some((a.first()?.as_f64()?, a.get(1)?.as_f64()?))
- })
- .collect()
- } else {
- let g = |k: &str| item.get(k).and_then(|v| v.as_f64());
- match (g("from_x"), g("from_y"), g("to_x"), g("to_y")) {
- (Some(fx), Some(fy), Some(tx), Some(ty)) => vec![(fx, fy), (tx, ty)],
- _ => {
- return ToolResult::error(
- "each drag item requires path[] or from_x/from_y/to_x/to_y.",
- )
- }
- }
- };
- if local.len() < 2 {
- return ToolResult::error("drag path needs at least 2 points.");
- }
- let steps = item
- .get("steps")
- .and_then(|v| v.as_u64())
- .unwrap_or(60)
- .clamp(1, 300) as usize;
- let x_button = parse_mouse_button(
- item.get("button")
- .and_then(|v| v.as_str())
- .unwrap_or("left"),
- ) as u32;
- let app = match tokio::task::spawn_blocking(move || {
- crate::wayland::inject_target_for_window(xid)
- })
- .await
- {
- Ok(Ok(target)) => target,
- Ok(Err(error)) => return ToolResult::error(error.to_string()),
- Err(error) => return ToolResult::error(format!("Task error: {error}")),
- };
- drags.push(crate::wayland::InjectDrag {
- app_id: app,
- idx: i,
- x_button,
- path: local,
- steps,
- });
- }
- let n = drags.len();
- match tokio::task::spawn_blocking(move || crate::wayland::inject_parallel_drags(&drags)).await {
- Ok(Ok(())) => ToolResult::text(format!(
- "Ran {n} concurrent drags (multi-cursor via cua-compositor)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
-}
-
-#[async_trait]
-impl Tool for ParallelMouseDragTool {
- fn def(&self) -> &ToolDef {
- PMDRAG_DEF.get_or_init(|| ToolDef {
- name: "parallel_mouse_drag".into(),
- description: "Run multiple mouse drag gestures concurrently via Linux MPX/XI2 virtual master pointers. \
- Each drag item runs on its own session-scoped master pointer (true same-window concurrent draws on X11). \
- Each item presses once, glides continuously through its whole path, and releases once — one smooth held \
- drag, not a chain of clicks. A path is given either as a straight segment (from_x/from_y → to_x/to_y) or \
- as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels (e.g. fn:\"x\" is a diagonal, \
- fn:\"300+120*sin(x/40)\" a sine wave). Functions support + - * / ^, sin/cos/tan, sqrt, abs, exp, ln, pi, e.".into(),
- input_schema: json!({"type":"object","required":["drags"],"properties":{
- "drags":{"type":"array","minItems":2,"description":"Two or more drag gestures to run concurrently, each on its own virtual master pointer.","items":{"type":"object","required":["session","window_id"],"properties":{
- "session":{"type":"string","description":"Session/cursor id; also keys the virtual master pointer."},
- "window_id":{"type":"integer","description":"Window id from list_windows."},
- "path":{"type":"array","items":{"type":"array","items":{"type":"number"}},"description":"Explicit window-local waypoints [[x,y],...] (>=2); pressed once, glided through, released once. Takes precedence over fn/from-to."},
- "fn":{"type":"string","description":"Expression y(x) in window-local pixels; sampled over [x_from,x_to]. Mutually exclusive with from_x/to_x."},
- "x_from":{"type":"number","description":"Domain start (window-local x) when `fn` is used."},
- "x_to":{"type":"number","description":"Domain end (window-local x) when `fn` is used."},
- "samples":{"type":"integer","minimum":2,"maximum":400,"description":"Waypoints sampled along `fn`. Default: 80."},
- "from_x":{"type":"number","description":"Straight-segment start X (window-local pixels)."},
- "from_y":{"type":"number","description":"Straight-segment start Y (window-local pixels)."},
- "to_x":{"type":"number","description":"Straight-segment end X (window-local pixels)."},
- "to_y":{"type":"number","description":"Straight-segment end Y (window-local pixels)."},
- "button": cua_driver_core::tool_schema::button_schema(),
- "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Default: 1500 for fn paths, 500 for straight."},
- "steps":{"type":"integer","minimum":1,"maximum":300,"description":"Motion sub-steps along the whole path. Default: scaled to path length."}
- },"additionalProperties":false}}
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- // Nested cua-compositor: run the drags as concurrent multi-cursor
- // injections (window-local, no X11 MPX/geometry needed).
- if crate::wayland::is_inject_mode() {
- return parallel_drag_inject(&args).await;
- }
- // Native Wayland without the inject socket: MPX/XI2 + uinput master
- // pointers don't exist on Wayland. Surface a typed error instead of
- // silently calling the X11 path that's guaranteed to fail.
- if crate::wayland::is_wayland() {
- return ToolResult::error(
- "parallel_mouse_drag requires the cua-compositor inject socket on Wayland \
- (set CUA_INJECT_SOCKET to the cua-compositor control socket), \
- or run the target under X11.",
- );
- }
- match tokio::task::spawn_blocking(crate::input::check_parallel_pointer_support).await {
- Ok(Ok(())) => {}
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
-
- let Some(items) = args.get("drags").and_then(|v| v.as_array()) else {
- return ToolResult::error("drags[] is required.");
- };
- if items.len() < 2 {
- return ToolResult::error("parallel_mouse_drag requires at least two drag items.");
- }
-
- let mut drags = Vec::with_capacity(items.len());
- for item in items {
- let Some(session) = item.get("session").and_then(|v| v.as_str()) else {
- return ToolResult::error("each drag item requires session.");
- };
- let Some(xid) = item.get("window_id").and_then(|v| v.as_u64()) else {
- return ToolResult::error("each drag item requires window_id.");
- };
-
- // Build the window-local waypoint path from one of: an explicit
- // `path` of [x,y] points, a function `fn` (y = f(x) sampled over
- // [x_from, x_to]), or a straight from→to segment.
- let is_fn = item.get("fn").and_then(|v| v.as_str()).is_some();
- let local: Vec<(f64, f64)> = if let Some(pts) =
- item.get("path").and_then(|v| v.as_array())
- {
- let mut out = Vec::with_capacity(pts.len());
- for p in pts {
- let a = p.as_array();
- let (Some(px), Some(py)) = (
- a.and_then(|a| a.first()).and_then(|v| v.as_f64()),
- a.and_then(|a| a.get(1)).and_then(|v| v.as_f64()),
- ) else {
- return ToolResult::error("each `path` entry must be [x, y].");
- };
- out.push((px, py));
- }
- if out.len() < 2 {
- return ToolResult::error("`path` needs at least 2 points.");
- }
- out
- } else if let Some(expr_str) = item.get("fn").and_then(|v| v.as_str()) {
- let Some(x_from) = item.get("x_from").and_then(|v| v.as_f64()) else {
- return ToolResult::error("`fn` requires x_from.");
- };
- let Some(x_to) = item.get("x_to").and_then(|v| v.as_f64()) else {
- return ToolResult::error("`fn` requires x_to.");
- };
- let samples = item
- .get("samples")
- .and_then(|v| v.as_u64())
- .unwrap_or(80)
- .clamp(2, 400);
- match crate::input::sample_function(expr_str, x_from, x_to, samples) {
- Ok(pts) => pts,
- Err(e) => return ToolResult::error(e.to_string()),
- }
- } else {
- let coerce = |k: &str| item.get(k).and_then(|v| v.as_f64());
- match (coerce("from_x"), coerce("from_y"), coerce("to_x"), coerce("to_y")) {
- (Some(fx), Some(fy), Some(tx), Some(ty)) => vec![(fx, fy), (tx, ty)],
- _ => return ToolResult::error("each drag item requires either `fn`+x_from+x_to, or from_x/from_y/to_x/to_y."),
- }
- };
-
- let button = parse_mouse_button(
- item.get("button")
- .and_then(|v| v.as_str())
- .unwrap_or("left"),
- );
- let duration_ms = item
- .get("duration_ms")
- .and_then(|v| v.as_u64())
- .unwrap_or(if is_fn { 1500 } else { 500 });
-
- // One translate gives the window origin; the path is a pure offset.
- let origin =
- match tokio::task::spawn_blocking(move || window_local_to_screen(xid, 0.0, 0.0))
- .await
- {
- Ok(Ok(o)) => o,
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- };
- let path: Vec<(i32, i32)> = local
- .iter()
- .map(|(lx, ly)| {
- (
- (origin.0 + lx).round() as i32,
- (origin.1 + ly).round() as i32,
- )
- })
- .collect();
-
- // Default sub-step count scaled to path length (smooth glide),
- // overridable via `steps`.
- let total_len: f64 = path
- .windows(2)
- .map(|w| {
- (((w[1].0 - w[0].0) as f64).powi(2) + ((w[1].1 - w[0].1) as f64).powi(2)).sqrt()
- })
- .sum();
- let steps = item
- .get("steps")
- .and_then(|v| v.as_u64())
- .map(|s| (s as usize).clamp(1, 300))
- .unwrap_or_else(|| ((total_len / 3.0).round() as usize).clamp(24, 300));
-
- let start = path[0];
- self.state
- .cursor_registry
- .update_position(session, start.0 as f64, start.1 as f64);
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SnapTo {
- x: start.0 as f64,
- y: start.1 as f64,
- heading_radians: None,
- },
- );
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
-
- drags.push((
- session.to_owned(),
- crate::input::VirtualPointerDrag {
- target_window: xid,
- button,
- path,
- duration_ms,
- steps,
- },
- ));
- }
-
- let drags_for_task = drags.clone();
- // The drags run side by side, so the longest one bounds the call.
- let budget = drags
- .iter()
- .map(|(_, drag)| {
- foreground_budget(drag.steps) + std::time::Duration::from_millis(drag.duration_ms)
- })
- .max()
- .unwrap_or_default();
- let result = spawn_blocking_bounded("parallel_mouse_drag", budget, move || {
- crate::input::send_parallel_virtual_pointer_drags(&drags_for_task)
- })
- .await;
- match result {
- Ok(Ok(())) => {
- for (session, drag) in &drags {
- let n = drag.path.len();
- let end = drag.path[n - 1];
- let prev = drag.path[n.saturating_sub(2)];
- self.state
- .cursor_registry
- .update_position(session, end.0 as f64, end.1 as f64);
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SnapTo {
- x: end.0 as f64,
- y: end.1 as f64,
- heading_radians: Some(
- ((end.1 - prev.1) as f64).atan2((end.0 - prev.0) as f64),
- ),
- },
- );
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: end.0 as f64,
- y: end.1 as f64,
- },
- );
- }
- ToolResult::text(format!(
- "✅ Ran {} MPX drag gesture(s) concurrently.",
- drags.len()
- ))
- .with_structured(json!({"count": drags.len()}))
- }
- Ok(Err(e)) => {
- for (session, _) in &drags {
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- }
- linux_input_error(e)
- }
- Err(e) => {
- for (session, _) in &drags {
- crate::overlay::send_command_for(
- session.to_owned(),
- cursor_overlay::OverlayCommand::SetPressed(false),
- );
- }
- ToolResult::error(format!("Task error: {e}"))
- }
- }
- }
-}
-
-// ── get_screen_size ───────────────────────────────────────────────────────────
-
-pub struct GetScreenSizeTool;
-static GSS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetScreenSizeTool {
- fn def(&self) -> &ToolDef {
- GSS_DEF.get_or_init(|| ToolDef {
- name: "get_screen_size".into(),
- description: "Return the logical size of the main display in points plus its backing \
- scale factor. Agents click in points; Retina displays have scale_factor 2.0. \
- Requires no TCC permissions."
- .into(),
- input_schema: json!({"type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema()
- },"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Err(result) = parse_typed_input::<GetScreenSizeInput>("get_screen_size", args) {
- return result;
- }
- let result = tokio::task::spawn_blocking(|| {
- if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
- // Shared manifest admission needs content-free display
- // metadata even when this native desktop has no X11 DISPLAY.
- // The adapter attests the IPC/Wayland compositor peer and
- // refuses layouts its logical frame cannot represent: no
- // powered output, or a rotated powered output.
- // Scaled outputs report logical pixels plus the output scale.
- // The size and the monitor list come from one snapshot.
- let (frame, monitors) = crate::wayland::hyprland::screen_report()?;
- return Ok((frame.width, frame.height, frame.scale, Some(monitors)));
- }
- if crate::wayland::is_wayland() && crate::wayland::niri::is_session() {
- // niri reports logical output geometry and its fractional
- // scale over attested IPC, for a single-output layout.
- let (frame, monitors) = crate::wayland::niri::screen_report()?;
- return Ok((frame.width, frame.height, frame.scale, Some(monitors)));
- }
- // X11 reports pixel dimensions; scale factor on X11 is not
- // well-defined per-monitor, so report 1.0 (matches DPI-unaware
- // assumption). Other Wayland compositors retain their existing
- // limitation; do not infer native metadata support there.
- let (w, h) = x11_screen_size()?;
- Ok::<(u32, u32, f64, Option<Value>), anyhow::Error>((w, h, 1.0, None))
- })
- .await;
- match result {
- Ok(Ok((w, h, scale, Some(monitors)))) => {
- // Tell the agent which monitors are actually on: the desktop
- // frame covers powered monitors only and changes as they turn
- // on or off, so re-read it after any display change.
- let describe = |m: &Value| {
- let name = m["name"].as_str().unwrap_or("?");
- match (
- m["powered"].as_bool(),
- m["frame_x"].as_i64(),
- m["frame_y"].as_i64(),
- ) {
- (Some(true), Some(x), Some(y)) => format!(
- "{name} {}x{} on, at ({x},{y}) in the desktop frame",
- m["width"], m["height"]
- ),
- _ => format!("{name} off (standby), not in the desktop frame"),
- }
- };
- let lines: Vec<String> = monitors
- .as_array()
- .into_iter()
- .flatten()
- .map(describe)
- .collect();
- ToolResult::text(format!(
- "✅ Desktop frame: {w}x{h} points @ {scale}x, spanning every monitor that is on.\n{}",
- lines.join("\n")
- ))
- .with_structured(json!({
- "width": w, "height": h, "scale_factor": scale, "monitors": monitors,
- }))
- }
- // Matches Swift text format 1:1.
- Ok(Ok((w, h, scale, None))) => {
- ToolResult::text(format!("✅ Main display: {w}x{h} points @ {scale}x"))
- .with_structured(json!({ "width": w, "height": h, "scale_factor": scale }))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-/// Read the true X11 root-window size in pixels: (width, height).
-/// Shared by `get_screen_size` and `get_desktop_state`.
-fn x11_screen_size() -> anyhow::Result<(u32, u32)> {
- use x11rb::connection::Connection;
- use x11rb::rust_connection::RustConnection;
- let (conn, screen_num) = RustConnection::connect(None)
- .map_err(|e| anyhow::anyhow!("{e}{}", crate::no_display_hint()))?;
- let setup = conn.setup();
- let screen = &setup.roots[screen_num];
- let w = screen.width_in_pixels as u32;
- let h = screen.height_in_pixels as u32;
- // WSLg / headless XWayland quirk: the X server connects but the
- // root screen advertises a 0-px geometry until a real output is
- // attached. Returning {width:0,height:0} here would propagate a
- // success with zero dimensions to the client, which then either
- // divides by zero when scaling or feeds the value into `int(...)`
- // after the missing key collapses to None. Fail loudly with an
- // actionable, typed error instead (never emit a 0/null where the
- // client expects a usable int). See issue #2005.
- if w == 0 || h == 0 {
- anyhow::bail!(
- "X11 connected but reports a 0x0 root screen — no usable \
- display geometry.{}",
- crate::no_display_hint()
- );
- }
- Ok((w, h))
+ .lock()
+ .unwrap()
+ .insert(session_id.to_owned(), hold)
+ .is_none());
}
-/// Put the desktop image in the exact coordinate frame consumed by desktop
-/// actions. Native Wayland capture buffers may use backing pixels while the
-/// compositor's pointer protocol and reported screen geometry use logical
-/// pixels. Returning the backing image alongside logical dimensions violates
-/// the screenshot-to-action contract and makes every vision-grounded action
-/// miss by the output scale.
-fn normalize_desktop_capture_for_action_frame(
- png: Vec<u8>,
- action_width: u32,
- action_height: u32,
-) -> anyhow::Result<(Vec<u8>, u32, u32, f64)> {
- if action_width == 0 || action_height == 0 {
- anyhow::bail!("desktop action frame is empty: {action_width}x{action_height}");
- }
+#[cfg(test)]
+#[test]
+fn isolated_hyprland_refused_partial_and_unknown_outcomes_stay_distinct() {
+ let busy = isolated_hyprland_result(Err(crate::wayland::hyprland_input::LaneBusy.into()));
+ let content = busy.structured_content.as_ref().unwrap();
+ assert_eq!(content["reason"], "lane_busy");
+ assert_eq!(content["effect"], "refused");
+ assert!(content.get("delivery").is_none());
+ assert!(busy
+ .action_record
+ .unwrap()
+ .public_result()
+ .unwrap()
+ .delivery
+ .is_none());
- let (capture_width, capture_height) = crate::capture::png_dimensions_pub(&png)?;
- let scale_x = f64::from(capture_width) / f64::from(action_width);
- let scale_y = f64::from(capture_height) / f64::from(action_height);
- if (scale_x - scale_y).abs() > 0.01 {
- anyhow::bail!(
- "desktop capture {capture_width}x{capture_height} cannot be mapped uniformly to \
- action frame {action_width}x{action_height} (scale {scale_x:.4}x{scale_y:.4})"
+ let refused = isolated_hyprland_result(Ok(
+ json!({"ok":false,"code":"stale_target","detail":"stale_target"}),
+ ));
+ assert_eq!(
+ refused.structured_content.as_ref().unwrap()["effect"],
+ "refused"
+ );
+ let public = refused.action_record.unwrap().public_result().unwrap();
+ assert!(public.delivery.is_none());
+ public.validate_invariants().unwrap();
+
+ for foreground in [false, true] {
+ let stale = hyprland_input_result(
+ Ok(json!({"ok":false,"code":"stale_geometry","detail":"stale_geometry"})),
+ foreground,
);
+ let value = stale.structured_content.as_ref().unwrap();
+ assert_eq!(value["code"], "stale_observation");
+ assert_eq!(value["reason"], "stale_geometry");
+ assert_eq!(value["effect"], "refused");
+ assert!(stale
+ .action_record
+ .unwrap()
+ .public_result()
+ .unwrap()
+ .delivery
+ .is_none());
}
- if capture_width == action_width && capture_height == action_height {
- return Ok((png, action_width, action_height, 1.0));
- }
+ let partial = isolated_hyprland_result(Ok(
+ json!({"ok":false,"code":"cancelled","detail":"cancelled",
+ "effect":"partial","delivery":{"mode":"background","delivered_count":1}}),
+ ));
+ let public = partial.action_record.unwrap().public_result().unwrap();
+ assert_eq!(public.effect, cua_driver_contract::ActionEffect::Partial);
+ assert_eq!(public.delivery.unwrap().delivered_count, Some(1));
- let image = image::load_from_memory_with_format(&png, image::ImageFormat::Png)?;
- let resized = image.resize_exact(
- action_width,
- action_height,
- image::imageops::FilterType::Lanczos3,
+ let unknown = isolated_hyprland_result(Err(anyhow::anyhow!("connection lost")));
+ let public = unknown.action_record.unwrap().public_result().unwrap();
+ assert_eq!(
+ public.effect,
+ cua_driver_contract::ActionEffect::Unverifiable
);
- let mut encoded = std::io::Cursor::new(Vec::new());
- resized.write_to(&mut encoded, image::ImageFormat::Png)?;
- Ok((encoded.into_inner(), action_width, action_height, scale_x))
-}
-
-// ── get_desktop_state ─────────────────────────────────────────────────────────
+ assert_eq!(
+ public.delivery.as_ref().unwrap().mode,
+ cua_driver_contract::ActionDeliveryMode::Unknown
+ );
+ assert_eq!(public.delivery.unwrap().delivered_count, None);
-pub struct GetDesktopStateTool {
- state: Arc<ToolState>,
+ let unknown_after_start = isolated_hyprland_result(Err(
+ crate::wayland::hyprland_input::unknown_dispatch(anyhow::anyhow!("connection lost"), 1),
+ ));
+ let public = unknown_after_start
+ .action_record
+ .unwrap()
+ .public_result()
+ .unwrap();
+ public.validate_invariants().unwrap();
+ assert_eq!(public.effect, cua_driver_contract::ActionEffect::Partial);
+ let delivery = public.delivery.unwrap();
+ assert_eq!(
+ delivery.mode,
+ cua_driver_contract::ActionDeliveryMode::Unknown
+ );
+ assert_eq!(delivery.delivered_count, Some(1));
}
-static GDS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetDesktopStateTool {
- fn def(&self) -> &ToolDef {
- GDS_DEF.get_or_init(|| ToolDef {
- name: "get_desktop_state".into(),
- description: "Capture the full display in the desktop action coordinate frame. \
- Use the returned PNG directly as the coordinate source for actions whose target is \
- {kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.".into(),
- input_schema: json!({"type":"object","properties":{
- "session":{"type":"string","description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session."},
- "screenshot_out_file":{"type":"string","description":"Write PNG here instead of base64."},
- "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()
- },"additionalProperties":false}),
- read_only: true, destructive: false, idempotent: false, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- let capture_args = args.clone();
- let input = match parse_typed_input::<GetDesktopStateInput>("get_desktop_state", args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let out_file = input.screenshot_out_file;
- let max_image_dimension = input.max_image_dimension;
- let capture_service = self.state.capture_service.clone();
-
- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
- // Capture the full display at native size first. When the
- // compositor consumes logical input coordinates, normalize the
- // image below so screenshot pixels still land exactly.
- //
- // The agent reads this image, so the Driver's own cursor and
- // session pill are hidden around the grab (or the limitation is
- // reported) instead of being baked over the controls it reads.
- // Hyprland: read the desktop frame once. The screenshot, its action
- // size and the window geometry below all use it.
- let hyprland_plan =
- if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
- Some(crate::wayland::hyprland::desktop_capture_plan()?)
- } else {
- None
- };
- // niri: the single output's frame, read before the capture and
- // compared after it.
- let niri_frame = (crate::wayland::is_wayland() && crate::wayland::niri::is_session())
- .then(|| crate::wayland::niri::desktop_snapshot().map(|snapshot| snapshot.frame))
- .transpose()?;
- let (native_png, overlay_capture) =
- cursor_overlay::capture_exclusion::capture_excluding_overlays(
- &crate::overlay_capture::OverlayExcluder,
- |hidden| {
- // A multi-monitor layout is composed from the frame
- // read above, so the image is exactly that frame.
- let png = match &hyprland_plan {
- Some(plan) if plan.composite => {
- crate::wayland::hyprland::compose_desktop_capture(&plan.frame)?
- }
- _ => crate::capture::screenshot_display_bytes()?,
- };
- Ok::<_, anyhow::Error>(crate::overlay_capture::verify_hidden_capture(
- png, hidden,
- ))
- },
- )?;
- let (native_w, native_h) = crate::capture::png_dimensions_pub(&native_png)?;
- // True screen size. On a pure-Wayland session (native backend
- // opted in, no X11 DISPLAY) the capture above came from the
- // wlroots `zwlr_screencopy` cascade, whose full-display buffer is
- // the whole output at native (physical) pixels — so the PNG
- // dimensions ARE the true screen size. Querying the X11 root
- // window here would fail with "$DISPLAY variable not set" and
- // abort the tool even though the screenshot already succeeded.
- // Only fall back to the X11 root-window geometry off Wayland, so
- // the X11 / XWayland path is unchanged. See #2017 / Sway testing.
- let hyprland_frame = hyprland_plan.map(|plan| plan.frame);
- let (screen_w, screen_h) = if let Some(frame) = &hyprland_frame {
- (frame.width, frame.height)
- } else if crate::wayland::is_wayland() {
- crate::capture_action_frame::desktop_action_dimensions((native_w, native_h))?
- } else {
- x11_screen_size()?
- };
- let (png, shot_w, shot_h, scale_factor) =
- normalize_desktop_capture_for_action_frame(native_png, screen_w, screen_h)?;
- // Opt-in long-edge cap: the full-size capture is the desktop
- // action frame; a capped image is mapped back at dispatch
- // (cua_driver_core::desktop_capture_scale) and by its capture_id.
- let (png, shot_w, shot_h) = match max_image_dimension.filter(|cap| *cap > 0) {
- Some(cap) if shot_w.max(shot_h) > cap => {
- let png = crate::capture::resize_png_if_needed(&png, cap)?;
- let (w, h) = crate::capture::png_dimensions_pub(&png)?;
- (png, w, h)
- }
- _ => (png, shot_w, shot_h),
- };
- // Optional: write PNG to disk instead of returning base64.
- let written = if let Some(path) = out_file.as_deref() {
- std::fs::write(path, &png)?;
- Some(path.to_string())
- } else {
- None
- };
- use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
- let b64 = if written.is_some() {
- None
- } else {
- Some(B64.encode(&png))
- };
- // Which pid owns which window is what the agent needs before its
- // first click; without it the first action guesses `pid: 1`.
- let mut windows = crate::wayland::list_windows_dispatch(None);
- windows
- .retain(|w| w.is_on_screen && w.pid.is_some_and(crate::proc_fs::is_process_live));
- if let Some(frame) = &hyprland_frame {
- // Window geometry is in layout coordinates; screenshot pixel
- // (0, 0) is the frame origin. Refuse rather than pair this
- // capture with a layout that changed meanwhile.
- anyhow::ensure!(
- crate::wayland::hyprland::desktop_frame()? == *frame,
- "the Hyprland monitor layout changed during get_desktop_state; call it again"
- );
- frame.rebase_windows(&mut windows);
- } else if let Some(frame) = &niri_frame {
- // niri window geometry is in layout coordinates; screenshot
- // pixel (0, 0) is the single output's logical origin.
- anyhow::ensure!(
- crate::wayland::niri::desktop_snapshot()?.frame == *frame,
- "the niri output layout changed during get_desktop_state; call it again"
- );
- frame.rebase_windows(&mut windows);
- }
- let capture_id = crate::capture_action_frame::publish_desktop(
- &capture_service,
- &capture_args,
- &png,
- (shot_w, shot_h),
- (screen_w, screen_h),
- )?;
- Ok((
- b64,
- shot_w,
- shot_h,
- screen_w,
- screen_h,
- scale_factor,
- written,
- windows,
- capture_id,
- overlay_capture,
- ))
- })
- .await;
+#[cfg(test)]
+#[test]
+fn experimental_pending_grant_is_an_error_with_operator_context() {
+ let result = isolated_hyprland_result(Ok(json!({
+ "ok": false, "code": "permission_required", "detail": "external approval pending",
+ "epoch": "epoch", "challenge": "challenge", "target": "target", "revision": 3
+ })));
+ assert_eq!(result.is_error, Some(true));
+ let value = result.structured_content.unwrap();
+ assert_eq!(value["code"], "background_unavailable");
+ assert_eq!(value["reason"], "permission_required");
+ assert_eq!(value["challenge"], "challenge");
+ assert_eq!(value["target"], "target");
+ assert_eq!(value["revision"], 3);
+ assert_eq!(value["verified"], false);
+}
- match result {
- Ok(Ok((
- b64_opt,
- shot_w,
- shot_h,
- screen_w,
- screen_h,
- scale_factor,
- written,
- windows,
- capture_id,
- overlay_capture,
- ))) => {
- let frame_scale = if shot_w > 0 {
- f64::from(screen_w) / f64::from(shot_w)
- } else {
- 1.0
- };
- let mut content = Vec::new();
- let mut structured = json!({
- "platform": "linux",
- "display": "primary",
- "screenshot_width": shot_w,
- "screenshot_height": shot_h,
- "screen_width": screen_w,
- "screen_height": screen_h,
- "scale_factor": scale_factor,
- "frame_scale": frame_scale,
- "screenshot_mime_type": "image/png",
- "windows": windows.iter().map(window_record_json).collect::<Vec<_>>(),
- "capture_id": capture_id,
- "agent_overlay_capture": overlay_capture,
- });
- if (frame_scale - 1.0).abs() > 0.001 {
- // Capped: the uncapped capture is the action frame.
- structured["screenshot_original_width"] = json!(screen_w);
- structured["screenshot_original_height"] = json!(screen_h);
- }
- if let Some(b64) = b64_opt {
- content.push(cua_driver_core::protocol::Content::image_png(b64));
- }
- let window_lines = desktop_window_lines(&windows, frame_scale);
- let mut frame_note = if (frame_scale - 1.0).abs() > 0.001 {
- format!(
- "; x/y for scope:\"desktop\" actions are pixels of THIS screenshot \
- (mapped ×{frame_scale:.2} back to the screen automatically)"
- )
- } else {
- String::new()
- };
- cursor_overlay::capture_exclusion::append_summary_note(
- &mut frame_note,
- &overlay_capture,
- );
- if let Some(path) = written {
- structured["screenshot_file_path"] = json!(path);
- content.push(cua_driver_core::protocol::Content::text(format!(
- "✅ Desktop screenshot {shot_w}x{shot_h} written to {path} (screen {screen_w}x{screen_h}{frame_note}){window_lines}"
- )));
- } else {
- content.push(cua_driver_core::protocol::Content::text(format!(
- "✅ Desktop screenshot {shot_w}x{shot_h} (screen {screen_w}x{screen_h}{frame_note}){window_lines}"
- )));
- }
- ToolResult {
- content,
- is_error: None,
- structured_content: Some(structured),
- action_record: None,
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("Capture error: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
+#[cfg(test)]
+#[test]
+fn experimental_dispatch_does_not_claim_application_success() {
+ let result = isolated_hyprland_result(Ok(json!({
+ "ok": true, "effect": "unverifiable", "route": "synthetic_events"
+ })));
+ assert_ne!(result.is_error, Some(true));
+ let public = result.action_record.unwrap().public_result().unwrap();
+ public.validate_invariants().unwrap();
+ let value = serde_json::to_value(public).unwrap();
+ assert_eq!(value["effect"], "unverifiable");
+ assert_eq!(value["route"], "synthetic_events");
+ assert!(value.get("verified").is_none());
}
-/// Text block naming every visible window with the `pid` / `window_id` pair
-/// that every other tool takes, so the model's first action targets a real
-/// process instead of guessing.
-fn desktop_window_lines(windows: &[crate::x11::WindowInfo], frame_scale: f64) -> String {
- let px = |v: i32| (f64::from(v) / frame_scale).round() as i32;
- let sz = |v: u32| (f64::from(v) / frame_scale).round() as u32;
- if windows.is_empty() {
- return "\nVisible windows: none (use list_windows / launch_app).".to_owned();
- }
- let mut out =
- String::from("\nVisible windows (use these pid + window_id values in every tool call):");
- for w in windows {
- let title = if w.title.is_empty() {
- "(no title)".to_owned()
+#[cfg(test)]
+#[test]
+fn foreground_hyprland_reports_foreground_and_preserves_partial_progress() {
+ use cua_driver_core::action_record::{ActionTransport, RequestedDelivery};
+ for (reply, expected_effect, count) in [
+ (json!({"ok":true}), "unverifiable", None),
+ (
+ json!({"ok":false,"code":"stale_target","detail":"stale"}),
+ "refused",
+ None,
+ ),
+ (
+ json!({"ok":false,"code":"cancelled","detail":"cancelled","effect":"partial",
+ "delivery":{"mode":"foreground","delivered_count":3}}),
+ "partial",
+ Some(3),
+ ),
+ ] {
+ let result = hyprland_input_result(Ok(reply), true);
+ let record = result.action_record.unwrap();
+ assert_eq!(
+ record.transport,
+ ActionTransport::LinuxHyprlandForegroundInput
+ );
+ assert_eq!(record.requested_delivery, RequestedDelivery::Foreground);
+ let public = record.public_result().unwrap();
+ public.validate_invariants().unwrap();
+ let value = serde_json::to_value(public).unwrap();
+ assert_eq!(value["route"], "global_input");
+ assert_eq!(value["effect"], expected_effect);
+ if expected_effect == "refused" {
+ assert!(value.get("delivery").is_none());
} else {
- format!("\"{}\"", w.title)
- };
- out.push_str(&format!(
- "\n- pid={} window_id={} {} {}x{} at ({},{}){}",
- w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
- w.xid,
- title,
- sz(w.width),
- sz(w.height),
- px(w.x),
- px(w.y),
- if w.app_name.is_empty() {
- String::new()
- } else {
- format!(" app={}", w.app_name)
- }
- ));
+ assert_eq!(value["delivery"]["mode"], "foreground");
+ assert_eq!(value["delivery"]["delivered_count"].as_u64(), count);
+ }
}
- out.push_str("\n→ get_window_state(pid, window_id) lists clickable elements; click/type_text take the same pid (+ window_id for x,y).");
- out
-}
-
-// ── get_cursor_position ───────────────────────────────────────────────────────
-
-pub struct GetCursorPositionTool;
-static GCP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetCursorPositionTool {
- fn def(&self) -> &ToolDef {
- GCP_DEF.get_or_init(|| ToolDef {
- name: "get_cursor_position".into(),
- description:
- "Return the current mouse cursor position in screen points (origin top-left)."
- .into(),
- input_schema: json!({"type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema()
- },"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
+ for reply in [
+ Err(crate::wayland::hyprland_input::unknown_dispatch(
+ anyhow::anyhow!("lost"),
+ 2,
+ )),
+ Ok(json!({"ok":false,"code":"foreground_partial_unknown","detail":"focus changed"})),
+ ] {
+ let unknown = hyprland_input_result(reply, true);
+ let value =
+ serde_json::to_value(unknown.action_record.unwrap().public_result().unwrap()).unwrap();
+ assert_eq!(value["delivery"]["mode"], "unknown");
+ assert_ne!(value["effect"], "refused");
}
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Err(result) =
- parse_typed_input::<GetCursorPositionInput>("get_cursor_position", args)
- {
- return result;
- }
- // Hyprland reports the real pointer over IPC; return it in the
- // desktop-frame coordinates that get_desktop_state and desktop actions use.
- if crate::wayland::is_wayland() && crate::wayland::hyprland::is_session() {
- let position = tokio::task::spawn_blocking(|| -> anyhow::Result<(i32, i32)> {
- let (x, y) = crate::wayland::hyprland::cursor_position()?;
- Ok(crate::wayland::hyprland::desktop_frame()?
- .from_layout(x.round() as i32, y.round() as i32))
- })
- .await;
- // The synthetic registry below holds layout coordinates, not the
- // desktop frame, so a failed compositor query is an error here.
- return match position {
- Ok(Ok((x, y))) => {
- ToolResult::text(format!("✅ Cursor at ({x}, {y}) in the desktop frame"))
- .with_structured(json!({ "x": x, "y": y, "source": "compositor" }))
- }
- Ok(Err(e)) => ToolResult::error(format!(
- "Could not read the Hyprland pointer position: {e:#}"
- )),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // Native Wayland: there's no protocol for clients to query the real
- // global cursor position. Fall back to the synthetic registry that
- // records every `motion_absolute` this process emits.
- if crate::wayland::is_wayland() {
- return match crate::wayland::last_synth_cursor_pos() {
- Some((x, y)) => ToolResult::text(
- format!("✅ Cursor at ({x}, {y}) (synthetic — last move_cursor in this process)")
- ).with_structured(json!({
- "x": x, "y": y, "source": "synthetic"
- })),
- None => ToolResult::text(
- "Cursor position unknown on Wayland — no move_cursor has been issued in this process yet.".to_string()
- ).with_structured(json!({ "source": "synthetic", "available": false })),
- };
- }
- let result = tokio::task::spawn_blocking(|| {
- use x11rb::connection::Connection;
- use x11rb::protocol::xproto::ConnectionExt as _;
- use x11rb::rust_connection::RustConnection;
- let (conn, screen_num) = RustConnection::connect(None)?;
- let root = conn.setup().roots[screen_num].root;
- let reply = conn.query_pointer(root)?.reply()?;
- Ok::<(i32, i32), anyhow::Error>((reply.root_x as i32, reply.root_y as i32))
- })
- .await;
- match result {
- // Text format matches Swift `GetCursorPositionTool` 1:1.
- Ok(Ok((x, y))) => ToolResult::text(format!("✅ Cursor at ({x}, {y})"))
- .with_structured(json!({ "x": x, "y": y, "source": "x11" })),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
+}
+
+#[cfg(test)]
+#[test]
+fn foreground_hyprland_chords_preserve_all_modifiers() {
+ let action = foreground_hyprland_key("Ctrl+Shift+H".into(), vec!["alt".into()]).unwrap();
+ match action {
+ crate::wayland::hyprland_input::Action::Key { key, modifiers } => {
+ assert_eq!(key, "H");
+ assert_eq!(modifiers, ["alt", "ctrl", "shift"]);
}
+ _ => panic!("expected a key chord"),
}
+ assert!(foreground_hyprland_key("bogus+H".into(), vec![]).is_err());
+}
+
+#[cfg(test)]
+#[test]
+fn uinput_failure_has_a_stable_structured_code() {
+ let result = linux_input_error(crate::input::uinput_unavailable("synthetic failure"));
+ assert_eq!(result.is_error, Some(true));
+ assert_eq!(
+ result
+ .structured_content
+ .as_ref()
+ .and_then(|value| value.get("code"))
+ .and_then(Value::as_str),
+ Some(crate::input::UINPUT_UNAVAILABLE_CODE)
+ );
}
-// ── move_cursor ───────────────────────────────────────────────────────────────
-
-pub struct MoveCursorTool {
- state: Arc<ToolState>,
+pub(crate) fn mouse_button_name(button: u8) -> &'static str {
+ match button {
+ 3 => "right",
+ 2 => "middle",
+ _ => "left",
+ }
}
-static MCURSOR_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+pub(crate) fn resolve_cursor_key(args: &Value) -> String {
+ cua_driver_core::tool_args::resolve_session_key(args)
+}
-#[derive(Clone, Copy, Debug, Eq, PartialEq)]
-enum CursorControlScope {
- Agent,
- Desktop,
+/// Return the cursor key only for a lifecycle-owned session. Cursor positioning
+/// for keyboard actions deliberately does not opt anonymous calls or the
+/// legacy cursor_id-only path into session semantics. The proxy-minted
+/// `_session_id` is trusted lifecycle state and must behave like a public named
+/// session for cursor ownership.
+pub(crate) fn named_session_cursor_key(args: &Value) -> Option<String> {
+ cursor_overlay::named_session_cursor_key(args)
}
-fn cursor_control_scope(args: &Value) -> CursorControlScope {
- match args.get("scope").and_then(Value::as_str) {
- Some("desktop") => CursorControlScope::Desktop,
- _ => CursorControlScope::Agent,
+pub(crate) fn mouse_hold_json(cursor_id: &str, hold: Option<&MouseHoldState>) -> Value {
+ match hold {
+ Some(hold) => json!({
+ "cursor_id": cursor_id,
+ "held": true,
+ "pid": hold.pid,
+ "window_id": hold.xid,
+ "button": mouse_button_name(hold.button),
+ "x": hold.x,
+ "y": hold.y,
+ }),
+ None => json!({
+ "cursor_id": cursor_id,
+ "held": false,
+ "pid": Value::Null,
+ "window_id": Value::Null,
+ "button": Value::Null,
+ "x": Value::Null,
+ "y": Value::Null,
+ }),
}
}
-#[async_trait]
-impl Tool for MoveCursorTool {
- fn def(&self) -> &ToolDef {
- MCURSOR_DEF.get_or_init(|| ToolDef {
- name: "move_cursor".into(),
- description: "Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates.".into(),
- input_schema: json!({"type":"object","required":["x","y"],"properties":{
- "x":{"type":"number","description":"Destination X. Window scope: screen coordinates of the agent cursor overlay. Desktop scope: get_desktop_state screenshot pixels."},
- "y":{"type":"number","description":"Destination Y, in the same space as x."},
- "session": cua_driver_core::tool_schema::session_schema(),
- "cursor_id":{"type":"string","description":"Cursor instance to move. Default: 'default'."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"\"window\" (default) moves only the agent cursor overlay; \"desktop\" moves the real OS pointer."}
- },"additionalProperties":false}),
- read_only: false, destructive: false, idempotent: true, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if cursor_control_scope(&args) == CursorControlScope::Desktop {
- let input = match parse_typed_projection::<MoveCursorInput>("move_cursor", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let (x, y) = (input.x, input.y);
- let xi = x.round() as i32;
- let yi = y.round() as i32;
- let wayland = crate::wayland::wayland_input_enabled();
- let path = if wayland {
- "wayland_desktop"
- } else {
- "xtest_desktop"
- };
- let result = if wayland {
- spawn_blocking_bounded("move_cursor", foreground_budget(0), move || {
- let space = crate::wayland::DesktopInputSpace::current()?;
- crate::wayland::move_cursor_desktop(&space, xi, yi)
- })
- .await
- } else {
- spawn_blocking_bounded("move_cursor", foreground_budget(0), move || {
- crate::input::send_move_xtest_desktop(xi, yi)
- })
- .await
- };
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Moved the real desktop pointer to ({xi}, {yi})."
+pub(crate) fn held_target_mismatch(
+ args: &Value,
+ cursor_id: &str,
+ hold: &MouseHoldState,
+) -> Option<ToolResult> {
+ match args.opt_u32("pid") {
+ Ok(Some(pid)) if pid != hold.pid => {
+ return Some(
+ ToolResult::error(format!(
+ "Cursor '{cursor_id}' is holding a button for pid {}, not pid {pid}.",
+ hold.pid
))
- .with_structured(
- json!({"scope":"desktop","path":path,"x":xi,"y":yi,"effect":"unverifiable"}),
- ),
- Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
- input_error_result(error)
- }
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
+ .with_structured(mouse_hold_json(cursor_id, Some(hold))),
+ );
}
- let x = args.f64_or("x", 0.0);
- let y = args.f64_or("y", 0.0);
- let cursor_id = resolve_cursor_key(&args);
- // End pointing upper-left (45°) — matches Swift's
- // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so the
- // overlay arrow settles to the natural macOS-style pose.
- // Use the acknowledged animation path so a first-ever move seeds and
- // displays the session cursor just as reliably as a coordinate click.
- reveal_pointer_action_for(&self.state, &cursor_id, x, y, false).await;
- ToolResult::text(format!(
- "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}); the user pointer was unchanged."
- ))
+ Err(err) => return Some(err.with_structured(mouse_hold_json(cursor_id, Some(hold)))),
+ _ => {}
}
-}
-// ── set_agent_cursor_enabled ──────────────────────────────────────────────────
-
-pub struct SetAgentCursorEnabledV2Tool {
- state: Arc<ToolState>,
+ match args.opt_u64("window_id") {
+ Some(xid) if xid != hold.xid => Some(
+ ToolResult::error(format!(
+ "Cursor '{cursor_id}' is holding a button for window_id {}, not {xid}.",
+ hold.xid
+ ))
+ .with_structured(mouse_hold_json(cursor_id, Some(hold))),
+ ),
+ _ => None,
+ }
}
-static CURSOR_ENABLED_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetAgentCursorEnabledV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_ENABLED_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_enabled"))
- }
+pub(crate) fn clear_mouse_hold_after_release(state: &ToolState, cursor_id: &str) {
+ state.mouse_hold.lock().unwrap().remove(cursor_id);
+}
- async fn invoke(&self, args: Value) -> ToolResult {
- let enabled = match args.get("enabled").and_then(Value::as_bool) {
- Some(value) => value,
- None => return ToolResult::error("Missing required boolean field `enabled`."),
- };
- let session = resolve_cursor_key(&args);
- self.state.cursor_registry.set_enabled(&session, enabled);
- crate::overlay::send_command_for(
- session.clone(),
- cursor_overlay::OverlayCommand::SetEnabled(enabled),
- );
- ToolResult::text(format!(
- "Agent cursor for session '{session}' {}.",
- if enabled { "enabled" } else { "disabled" }
- ))
- .with_structured(json!({"session":session,"enabled":enabled}))
+pub(crate) fn persistent_pointer_coordinates(
+ is_wayland: bool,
+ window_id: u64,
+ x: i32,
+ y: i32,
+ window_local_to_output: impl FnOnce(u64, i32, i32) -> anyhow::Result<AimedPoint>,
+) -> anyhow::Result<AimedPoint> {
+ if is_wayland {
+ window_local_to_output(window_id, x, y)
+ } else {
+ Ok((x, y).into())
}
}
-pub struct SetAgentCursorMotionV2Tool;
-
-static CURSOR_MOTION_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+fn release_mouse_hold_for_session<Release>(
+ state: &ToolState,
+ session_id: &str,
+ mut release: Release,
+) -> Result<(), String>
+where
+ Release: FnMut(&str, &MouseHoldState) -> anyhow::Result<()>,
+{
+ let hold = state.mouse_hold.lock().unwrap().get(session_id).cloned();
+ let Some(hold) = hold else {
+ return Ok(());
+ };
-fn cursor_number(value: Option<&Value>) -> Option<f64> {
- value.and_then(|value| {
- value
- .as_f64()
- .or_else(|| value.as_i64().map(|integer| integer as f64))
- })
+ release(session_id, &hold).map_err(|error| error.to_string())?;
+ state.mouse_hold.lock().unwrap().remove(session_id);
+ Ok(())
}
-#[async_trait]
-impl Tool for SetAgentCursorMotionV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_MOTION_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_motion"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let session = resolve_cursor_key(&args);
- let current = crate::overlay::current_motion_for(&session);
- let motion = current.with_overrides(
- cursor_number(args.get("start_handle")),
- cursor_number(args.get("end_handle")),
- cursor_number(args.get("arc_size")),
- cursor_number(args.get("arc_flow")),
- cursor_number(args.get("spring")),
- cursor_number(args.get("glide_duration_ms")),
- cursor_number(args.get("dwell_after_click_ms")),
- cursor_number(args.get("idle_hide_ms")),
- None,
- cursor_number(args.get("turn_radius")),
- );
- let motion = match motion.with_style_args(&args) {
- Ok(motion) => motion,
- Err(message) => return ToolResult::error(message),
- };
- crate::overlay::send_command_for(
- session.clone(),
- cursor_overlay::OverlayCommand::SetMotion(motion.clone()),
- );
- ToolResult::text(format!(
- "Agent cursor motion updated for session '{session}'."
- ))
- .with_structured(json!({"session":session,"motion":motion.output_json()}))
- }
+pub(crate) fn overlay_snap_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::SnapTo {
+ x: sx,
+ y: sy,
+ heading_radians: heading,
+ },
+ );
}
-pub struct SetAgentCursorThemeTool {
- state: Arc<ToolState>,
+/// [`overlay_move_to_for`] with the targeted element's screen rect
+/// `[x, y, width, height]` (same space as `sx`/`sy`).
+pub(crate) fn overlay_move_to_target_for(
+ cursor_id: &str,
+ sx: f64,
+ sy: f64,
+ heading: Option<f64>,
+ target: Option<[f64; 4]>,
+) {
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::MoveTo {
+ x: sx,
+ y: sy,
+ end_heading_radians: heading.unwrap_or(std::f64::consts::FRAC_PI_4),
+ target,
+ },
+ );
}
-static CURSOR_THEME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-fn cursor_reduced_motion(value: Option<&str>) -> cursor_overlay::ReducedMotion {
- match value {
- Some("on") => cursor_overlay::ReducedMotion::On,
- Some("off") => cursor_overlay::ReducedMotion::Off,
- _ => cursor_overlay::ReducedMotion::Auto,
- }
+pub(crate) async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {
+ overlay_glide_to_target_for(cursor_id, sx, sy, None).await;
}
-#[async_trait]
-impl Tool for SetAgentCursorThemeTool {
- fn def(&self) -> &ToolDef {
- CURSOR_THEME_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_theme"))
+/// [`overlay_glide_to_for`] with the targeted element's screen rect
+/// `[x, y, width, height]` (same space as `sx`/`sy`); `None` for pixel actions.
+async fn overlay_glide_to_target_for(cursor_id: &str, sx: f64, sy: f64, target: Option<[f64; 4]>) {
+ // Input always revives its agent cursor. Hiding it is useful while idle,
+ // but a hidden cursor must not make pointer or keyboard control invisible.
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::SetEnabled(true),
+ );
+ // Every Wayland backend owns its animation loop. Send one destination and
+ // let the Linux overlay layer select the semantic helper, layer-shell, or
+ // older helper fallback without an interpolated stream fighting it.
+ if crate::wayland::is_wayland() {
+ overlay_move_to_target_for(cursor_id, sx, sy, None, target);
+ return;
}
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let Some(theme_id) = args.get("theme_id").and_then(Value::as_str) else {
- return ToolResult::error("Missing required string field `theme_id`.");
- };
- let session = resolve_cursor_key(&args);
- let resolved_theme = match cursor_overlay::resolve_theme_selection(theme_id) {
- Ok(theme) => theme,
- Err(error) => {
- return ToolResult::error(format!(
- "Cursor theme '{theme_id}' cannot be selected: {error}"
- ));
- }
- };
- let (version, profile) = resolved_theme
- .as_deref()
- .map(|theme| (theme.version.as_str(), theme.profile.as_str()))
- .unwrap_or((
- cursor_overlay::DEFAULT_THEME_VERSION,
- cursor_overlay::THEME_PROFILE,
- ));
- let reduced_motion =
- cursor_reduced_motion(args.get("reduced_motion").and_then(Value::as_str));
- self.state
- .cursor_registry
- .update_config(&session, |config| {
- config.theme_id = theme_id.to_owned();
- config.reduced_motion = reduced_motion;
- });
+ let pos = crate::overlay::current_position_for(cursor_id);
+ if pos.is_none() {
crate::overlay::send_command_for(
- session.clone(),
- cursor_overlay::OverlayCommand::SetTheme {
- theme_id: theme_id.to_owned(),
- reduced_motion,
- },
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
);
- ToolResult::text(format!(
- "Agent cursor theme for session '{session}' set to '{theme_id}'."
- ))
- .with_structured(json!({"session":session,"theme":{
- "id":theme_id,
- "version":version,
- "profile":profile,
- "reduced_motion":reduced_motion,
- "fallback":null
- }}))
+ return;
}
+ crate::overlay::animate_cursor_to_target_for(cursor_id.to_owned(), sx, sy, target).await;
}
-pub struct GetAgentCursorStateV2Tool {
- state: Arc<ToolState>,
-}
-
-static CURSOR_STATE_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetAgentCursorStateV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_STATE_V2_DEF.get_or_init(|| canonical_cursor_def("get_agent_cursor_state"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let session = resolve_cursor_key(&args);
- let enabled = crate::overlay::is_enabled_for(&session);
- let motion = crate::overlay::current_motion_for(&session);
- let (theme_id, version, profile, fallback, visual) =
- crate::overlay::current_theme_state_for(&session).unwrap_or_else(|| {
- (
- cursor_overlay::DEFAULT_THEME_ID.into(),
- cursor_overlay::DEFAULT_THEME_VERSION.into(),
- cursor_overlay::THEME_PROFILE.into(),
- None,
- cursor_overlay::CursorVisualState::default(),
- )
- });
- let modifiers: Vec<&str> = [
- visual.delivery.map(|value| value.as_str()),
- visual.target.map(|value| value.as_str()),
- ]
- .into_iter()
- .flatten()
- .collect();
- // The last point this session's cursor was placed at, `null` until
- // it first moves (same registry source as macOS).
- let position = self
- .state
- .cursor_registry
- .get(&session)
- .and_then(|cursor| cursor.x.zip(cursor.y))
- .map(|(x, y)| json!({"x": x, "y": y}));
- ToolResult::text(format!("Agent cursor state for session '{session}'.")).with_structured(
- json!({
- "session":session,
- "enabled":enabled,
- "position":position,
- "theme":{
- "id":theme_id,
- "version":version,
- "profile":profile,
- "reduced_motion":visual.reduced_motion,
- "fallback":fallback
- },
- "visual_state":{
- "requested_action":visual.requested_action,
- "resolved_action":visual.resolved_action,
- "modifiers":modifiers,
- "phase":visual.phase(),
- "frame":visual.frame(),
- "preempted_count":visual.preempted_count
- },
- "motion":motion.output_json()
- }),
- )
- }
+/// The compositor snapshot a desktop-scope action is addressed in. One
+/// snapshot converts the action's points, places the overlay and sizes the
+/// virtual pointer, so they cannot disagree about the monitor layout.
+pub(crate) async fn desktop_input_space() -> anyhow::Result<crate::wayland::DesktopInputSpace> {
+ tokio::task::spawn_blocking(|| {
+ crate::interaction::admit_lock(
+ crate::interaction::policy::Scope::DesktopInput,
+ crate::interaction::facts::Fact::Unknown("desktop input addresses no window".into()),
+ )?;
+ if !crate::wayland::wayland_input_enabled() {
+ return Ok(crate::wayland::DesktopInputSpace::default());
+ }
+ crate::wayland::DesktopInputSpace::current()
+ })
+ .await
+ .map_err(|error| anyhow::anyhow!("task error: {error}"))?
}
-fn canonical_cursor_def(name: &str) -> ToolDef {
- let contract = cua_driver_contract::tool_contract(name)
- .unwrap_or_else(|| panic!("missing canonical cursor contract for {name}"));
- ToolDef::from_contract(&contract)
+/// Keep the logical cursor position in sync with every visibly targeted
+/// pointer action. Overlay delivery is intentionally best-effort: registry
+/// state is still updated when no renderer is running or its queue is closed.
+pub(crate) async fn reveal_pointer_action_for(
+ state: &ToolState,
+ cursor_id: &str,
+ sx: f64,
+ sy: f64,
+ click_pulse: bool,
+) {
+ reveal_pointer_action_target_for(state, cursor_id, sx, sy, click_pulse, None).await;
}
-// ── check_permissions ─────────────────────────────────────────────────────────
-
-pub struct CheckPermissionsTool;
-static PERMS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for CheckPermissionsTool {
- fn def(&self) -> &ToolDef {
- PERMS_DEF.get_or_init(|| ToolDef {
- name: "check_permissions".into(),
- description: "Check required permissions for cua-driver-rs on Linux.".into(),
- input_schema: json!({"type":"object","properties":{"prompt":{"type":"boolean","description":"Ignored: Linux has no permission prompts."}},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
+/// [`reveal_pointer_action_for`] with the targeted element's screen rect
+/// `[x, y, width, height]` (same space as `sx`/`sy`).
+pub(crate) async fn reveal_pointer_action_target_for(
+ state: &ToolState,
+ cursor_id: &str,
+ sx: f64,
+ sy: f64,
+ click_pulse: bool,
+ target: Option<[f64; 4]>,
+) {
+ if !sx.is_finite() || !sy.is_finite() {
+ return;
}
- async fn invoke(&self, _args: Value) -> ToolResult {
- // Check X11 connectivity (required for window enumeration and input injection).
- let x11_ok = tokio::task::spawn_blocking(|| {
- x11rb::rust_connection::RustConnection::connect(None).is_ok()
- })
- .await
- .unwrap_or(false);
-
- // Check AT-SPI: not merely "is there a session bus?" but "does
- // org.a11y.Bus actually answer on it?" — the previous env-var-or-
- // /run/user heuristic false-passed exactly the headless/container case
- // (/run/user exists, but no a11y bus → empty trees). Probe for real.
- let dbus_address = std::env::var("DBUS_SESSION_BUS_ADDRESS").ok();
- let atspi_ok = tokio::task::spawn_blocking(crate::health_report::probe_a11y_bus)
- .await
- .unwrap_or(false);
-
- let wayland_display = std::env::var("WAYLAND_DISPLAY").ok();
- let atspi_status = if atspi_ok {
- match &dbus_address {
- Some(a) => format!("✅ org.a11y.Bus reachable (DBUS_SESSION_BUS_ADDRESS={a})"),
- None => "✅ org.a11y.Bus reachable".to_string(),
- }
- } else if dbus_address.is_none() {
- "❌ no session bus (DBUS_SESSION_BUS_ADDRESS unset and none auto-discovered) — \
- AT-SPI trees will be empty; start the daemon inside the desktop session"
- .to_string()
- } else {
- "❌ session bus present but org.a11y.Bus has no owner — enable accessibility \
- (gsettings set org.gnome.desktop.interface toolkit-accessibility true) / \
- start at-spi-bus-launcher"
- .to_string()
- };
- let status_text = format!(
- "X11 display: {}\nWayland: {}\nAT-SPI (D-Bus): {}\nXSendEvent injection: {}",
- if x11_ok { "✅ connected" } else { "❌ DISPLAY not set or X11 unavailable" },
- match &wayland_display {
- Some(s) if crate::wayland::wayland_enabled() =>
- format!("✅ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend ENABLED"),
- Some(s) => format!(
- "⚠️ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend OFF; \
- set {}=1 to enable it",
- crate::wayland::ENABLE_WAYLAND_ENV
- ),
- None => "❌ not a Wayland session".to_string(),
- },
- atspi_status,
- if x11_ok { "✅ available" } else { "❌ requires X11" }
+ state.cursor_registry.set_enabled(cursor_id, true);
+ state.cursor_registry.update_position(cursor_id, sx, sy);
+ emit_cursor_hook(cursor_id, sx, sy, false);
+ overlay_glide_to_target_for(cursor_id, sx, sy, target).await;
+ if click_pulse {
+ emit_cursor_hook(cursor_id, sx, sy, true);
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
);
- ToolResult::text(status_text)
- .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "wayland_enabled": crate::wayland::wayland_enabled(), "atspi": atspi_ok, "dbus_session_bus_address": dbus_address, "xsend_event": x11_ok }))
}
}
-#[cfg(test)]
-#[tokio::test]
-async fn check_permissions_accepts_the_status_prompt_argument() {
- let result = build_registry_with_provider(false, None)
- .invoke("check_permissions", json!({ "prompt": false }))
- .await;
- let wire = serde_json::to_value(&result).unwrap();
- assert!(wire["structuredContent"].get("x11").is_some(), "{wire}");
-}
-
-// ── get_config ────────────────────────────────────────────────────────────────
-
-pub struct GetConfigTool {
- state: Arc<ToolState>,
-}
-static GCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetConfigTool {
- fn def(&self) -> &ToolDef {
- GCFG_DEF.get_or_init(|| ToolDef {
- name: "get_config".into(),
- description: "Return current cua-driver-rs configuration.".into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
+/// Report a commanded cursor move (or a press) to the embedder's cursor
+/// hook, with the same guards as macOS's `CursorRegistry::emit_cursor_event`:
+/// nothing is built unless a hook is registered, and an ended session never
+/// reaches it. Emitted from the logical cursor write path, so it works with
+/// no overlay renderer (a headless Xvfb session).
+pub(crate) fn emit_cursor_hook(cursor_id: &str, x: f64, y: f64, pressed: bool) {
+ if !cua_driver_core::cursor_hook::cursor_hook_enabled() {
+ return;
}
- async fn invoke(&self, _args: Value) -> ToolResult {
- let cfg = self.state.config.read().unwrap();
- let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
- ToolResult::text("cua-driver-rs configuration").with_structured(json!({
- "version": env!("CARGO_PKG_VERSION"),
- "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
- "platform": "linux",
- "capture_mode": cfg.capture_mode,
- "max_image_dimension": cfg.max_image_dimension,
- "cursor": cursor_overlay::cursor_options::saved_config_json(),
- "experimental_pip": pip_enabled,
- "experimental_pip_geometry": pip_geometry
- }))
+ if cursor_id.is_empty() || cua_driver_core::session::is_session_ended(cursor_id) {
+ return;
}
+ cua_driver_core::cursor_hook::push_cursor_event(
+ cua_driver_core::cursor_hook::CursorHookEvent {
+ cursor_id: cursor_id.to_owned(),
+ x,
+ y,
+ pressed,
+ },
+ );
}
-// ── set_config ────────────────────────────────────────────────────────────────
-
-fn with_cursor_motion_config_properties(mut schema: Value) -> Value {
- if let Some(properties) = schema.get_mut("properties").and_then(Value::as_object_mut) {
- properties.extend(cursor_overlay::motion_defaults::config_schema_properties());
- properties.extend(cursor_overlay::badge_mode::config_schema_properties());
- properties.extend(cursor_overlay::cursor_options::config_schema_properties());
+pub(crate) fn keyboard_window_center(xid: u64) -> Option<(f64, f64)> {
+ if xid == 0 {
+ return None;
+ }
+ if crate::wayland::is_wayland() {
+ return crate::wayland::window_geometry(xid).and_then(|(x, y, width, height)| {
+ (width > 0 && height > 0).then_some((
+ f64::from(x) + f64::from(width) / 2.0,
+ f64::from(y) + f64::from(height) / 2.0,
+ ))
+ });
}
- schema
+ window_screen_center(xid)
+ .ok()
+ .map(|(x, y)| (f64::from(x), f64::from(y)))
}
-pub struct SetConfigTool {
- state: Arc<ToolState>,
-}
-static SCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetConfigTool {
- fn def(&self) -> &ToolDef {
- SCFG_DEF.get_or_init(|| ToolDef {
- name: "set_config".into(),
- description: "Update cua-driver-rs configuration. capture_mode / \
- max_image_dimension take effect immediately.\n\n\
- Pass each setting as its own field: `{\"capture_mode\": \"som\", \"max_image_dimension\": 0}`.\n\n\
- The experimental_pip keys persist to ~/.cua-driver/config.json and apply on next \
- daemon restart (the PiP backend is initialised once at startup; \
- Linux ships only the trait stub today — see issue #1729).\n\n\
- `cursor.motion.style`, `cursor.motion.timing` and `cursor.motion.effects.<name>` \
- (`on`, `off`, or `default`) save the default cursor motion for sessions started afterwards. \
- `cursor.motion: \"default\"` clears every saved cursor motion default.\n\n\
- `cursor.badges` (`minimal` default, `verbose`, or `off`) sets the agent cursor badge \
- for every cursor at once.\n\n\
- `cursor.float` (resting float) is `on` (default), `off` or `default`; \
- `cursor.marks` (action marks) and `cursor.rotate` (turn toward travel) are `on`, \
- `off` (default) or `default`. All apply to every cursor at once.".into(),
- input_schema: with_cursor_motion_config_properties(json!({"type":"object","properties":{
- "capture_mode":{"type":"string","enum":["ax","vision"],"description":"Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
- "max_image_dimension":{"type":"integer","description":"Max dimension for screenshot resizing (0 = no limit)."},
- "experimental_pip":{"type":"boolean","description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed)."},
- "experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form."}
- },"additionalProperties":false})),
- read_only: false, destructive: false, idempotent: true, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if args.get("capture_scope").is_some() {
- return ToolResult::error(
- "config key 'capture_scope' is retired; select a window or desktop target on each action",
- )
- .with_structured(json!({
- "code": "config_key_retired",
- "key": "capture_scope",
- "replacement": "action.target",
- }));
- }
- let motion_keys = match cursor_overlay::motion_defaults::apply_config_args(&args) {
- Ok(keys) => keys,
- Err(message) => return ToolResult::error(message),
- };
- let badge_key = match cursor_overlay::badge_mode::apply_config_args(&args) {
- Ok(key) => key,
- Err(message) => return ToolResult::error(message),
- };
- let option_keys = match cursor_overlay::cursor_options::apply_config_args(&args) {
- Ok(keys) => keys,
- Err(message) => return ToolResult::error(message),
- };
- let mut cfg = self.state.config.write().unwrap();
- let mut parts: Vec<String> = motion_keys
- .iter()
- .map(|key| format!("{key} (applies to sessions started from now on)"))
- .collect();
- parts.extend(
- badge_key
- .into_iter()
- .chain(option_keys)
- .map(|key| format!("{key} (applies now)")),
- );
- if let Some(mode) = args.opt_str("capture_mode") {
- if let Err(e) =
- pip_preview::write_config_key("capture_mode", Value::String(mode.clone()))
- {
- tracing::warn!("set_config: failed to persist capture_mode: {e}");
- }
- parts.push(format!("capture_mode={mode}"));
- cfg.capture_mode = mode;
- }
- if let Some(dim) = args.opt_u64("max_image_dimension") {
- cfg.max_image_dimension = dim as u32;
- if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
- tracing::warn!("set_config: failed to persist max_image_dimension: {e}");
- }
- parts.push(format!("max_image_dimension={dim}"));
- }
- if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
- if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
- {
- return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
- }
- parts.push(format!("experimental_pip={enabled} (next restart)"));
- }
- if let Some(geom) = args.opt_str("experimental_pip_geometry") {
- if pip_preview::PipGeometry::parse(&geom).is_none() {
- return ToolResult::error(format!(
- "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
- ));
- }
- if let Err(e) = pip_preview::write_config_key(
- "experimental_pip_geometry",
- Value::String(geom.clone()),
- ) {
- return ToolResult::error(format!(
- "failed to persist experimental_pip_geometry: {e}"
- ));
- }
- parts.push(format!("experimental_pip_geometry={geom} (next restart)"));
- }
- let msg = if parts.is_empty() {
- "Config unchanged (no known parameters).".to_owned()
- } else {
- format!("Config updated: {}", parts.join(", "))
- };
- let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
- ToolResult::text(msg).with_structured(json!({
- "capture_mode": cfg.capture_mode,
- "max_image_dimension": cfg.max_image_dimension,
- "experimental_pip": pip_enabled,
- "experimental_pip_geometry": pip_geometry
- }))
- }
-}
-
-// ── get_accessibility_tree ────────────────────────────────────────────────────
-
-pub struct GetAccessibilityTreeTool;
-
-static GAX_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetAccessibilityTreeTool {
- fn def(&self) -> &ToolDef {
- GAX_DEF.get_or_init(|| ToolDef {
- name: "get_accessibility_tree".into(),
- description: "Return a lightweight snapshot of the desktop: running processes and \
- on-screen visible windows with their bounds and owner pid.\n\n\
- For the full AT-SPI subtree of a single window (with interactive element indices \
- you can click by), use get_window_state instead — this is a fast discovery read."
- .into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
+fn current_pointer_position() -> Option<(f64, f64)> {
+ if crate::wayland::is_wayland() {
+ return crate::wayland::last_synth_cursor_pos().map(|(x, y)| (f64::from(x), f64::from(y)));
}
- async fn invoke(&self, _args: Value) -> ToolResult {
- let (procs, windows) = tokio::task::spawn_blocking(|| {
- (
- crate::proc_fs::list_processes(),
- visible_live_desktop_windows(crate::wayland::list_windows_dispatch(None)),
- )
- })
- .await
- .unwrap_or_default();
-
- if crate::wayland::is_wayland() {
- crate::wayland::remember_observed_window_origins(&windows);
- }
- let mut lines = vec![format!(
- "{} running process(es), {} visible window(s)",
- procs.len(),
- windows.len()
- )];
- for p in &procs {
- let cmd = if p.cmdline.is_empty() {
- p.name.clone()
- } else {
- p.cmdline.clone()
- };
- lines.push(format!("- {} (pid {})", cmd, p.pid));
- }
- if !windows.is_empty() {
- lines.push(String::new());
- lines.push("Windows:".to_owned());
- for w in &windows {
- let title = if w.title.is_empty() {
- "(no title)".to_owned()
- } else {
- format!("\"{}\"", w.title)
- };
- lines.push(format!(
- "- pid={} window_id={} {} {}x{}+{}+{}",
- w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
- w.xid,
- title,
- w.width,
- w.height,
- w.x,
- w.y
- ));
- }
- lines.push(
- "→ Call get_window_state(pid, window_id) to inspect a window's UI.".to_owned(),
- );
- }
+ use x11rb::connection::Connection;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
- let structured = json!({
- "processes": procs.iter().map(|p| json!({"pid":p.pid,"name":p.name})).collect::<Vec<_>>(),
- "windows": windows.iter().map(window_record_json).collect::<Vec<_>>()
- });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
+ let (connection, screen_num) = RustConnection::connect(None).ok()?;
+ let root = connection.setup().roots[screen_num].root;
+ let reply = connection.query_pointer(root).ok()?.reply().ok()?;
+ Some((f64::from(reply.root_x), f64::from(reply.root_y)))
}
-// ── zoom ──────────────────────────────────────────────────────────────────────
+pub(crate) fn explicit_keyboard_cursor_target(
+ pid: u32,
+ xid: u64,
+ element: Option<&crate::atspi::TokenElement>,
+ pixel_target: Option<(f64, f64)>,
+) -> Option<(f64, f64)> {
+ if let Some(element) = element {
+ let (sx, sy) = element_screen_center(pid, element, (xid != 0).then_some(xid)).ok()?;
+ return Some((sx, sy));
+ }
-pub struct ZoomTool {
- state: Arc<ToolState>,
+ let (x, y) = pixel_target?;
+ if crate::wayland::wayland_input_enabled() {
+ return crate::wayland::window_geometry(xid)
+ .map(|(wx, wy, _, _)| (f64::from(wx) + x.round(), f64::from(wy) + y.round()));
+ }
+ window_local_to_screen(xid, x, y).ok()
}
-static ZOOM_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ZoomTool {
- fn def(&self) -> &ToolDef {
- ZOOM_DEF.get_or_init(|| ToolDef {
- name: "zoom".into(),
- description: "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in \
- screenshot pixels, with 20% padding. Output is at most 500 px wide. \
- Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\n\
- After a zoom, pass from_zoom=true to click/type_text to auto-translate \
- coordinates back to full-window space. Coordinate actions return \
- `screenshot_context_missing` when no current snapshot contains a \
- screenshot owned by this session. `from_zoom` actions return \
- `zoom_context_missing` when the zoom was never created or was replaced; call \
- `get_window_state`, then `zoom`, again on the same connection.".into(),
- input_schema: json!({
- "type":"object","required":["window_id","x1","y1","x2","y2"],"properties":{
- "window_id":{"type":"integer","description":"Window id of the window captured by get_window_state."},
- "pid":{"type":"integer","description":"Optional target pid. When omitted, the driver resolves the unique current snapshot for this session and window."},
- "x1":{"type":"number","description":"Left edge of the region in window-local screenshot pixels."},
- "y1":{"type":"number","description":"Top edge of the region in window-local screenshot pixels."},
- "x2":{"type":"number","description":"Right edge of the region in window-local screenshot pixels."},
- "y2":{"type":"number","description":"Bottom edge of the region in window-local screenshot pixels."}
- },"additionalProperties":false
- }),
- read_only: true, destructive: false, idempotent: true, open_world: false,
- })
+
+/// Position and reveal a named session's cursor before keyboard/value input.
+/// `preserve_legacy_element_visual` retains the existing element-only feedback
+/// for type_text/set_value anonymous calls without adding session-style fallback
+/// placement to them. Geometry and overlay failures are observational and never
+/// affect the tool's actual input result.
+/// Tell the cursor hook where an anonymous keyboard action types, before the
+/// keystrokes: the element or pixel target, else the window's center. The
+/// overlay is left alone (anonymous calls keep their legacy visuals); only an
+/// embedder publishing presence learns the position, so a viewer sees the
+/// agent's cursor go to the field it is about to type into.
+pub(crate) async fn announce_keyboard_target(
+ args: &Value,
+ pid: u32,
+ xid: u64,
+ element: Option<crate::atspi::TokenElement>,
+ pixel_target: Option<(f64, f64)>,
+) {
+ if !cua_driver_core::cursor_hook::cursor_hook_enabled() {
+ return;
+ }
+ let cursor_id = resolve_cursor_key(args);
+ let target = tokio::task::spawn_blocking(move || {
+ explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
+ .or_else(|| (xid != 0).then(|| keyboard_window_center(xid)).flatten())
+ })
+ .await
+ .ok()
+ .flatten();
+ if let Some((x, y)) = target {
+ emit_cursor_hook(&cursor_id, x, y, false);
}
+}
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let xid = match args.require_u64("window_id") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let requested_pid = match args.get("pid") {
- None => None,
- Some(value) => match value.as_u64().and_then(|pid| i32::try_from(pid).ok()) {
- Some(pid) => Some(pid),
- None => return ToolResult::error("pid must be a positive 32-bit integer"),
- },
- };
- let session_id = args.opt_str("_session_id");
- let (pid, screenshot) = match self.state.snapshots.screenshot_context_for_zoom(
- requested_pid,
- xid,
- session_id.as_deref(),
- ) {
- Ok(context) => context,
- Err(refusal) => return refusal,
- };
- let x1 = match args.opt_f64("x1") {
- Some(v) => v,
- None => return ToolResult::error("Missing x1"),
- };
- let y1 = match args.opt_f64("y1") {
- Some(v) => v,
- None => return ToolResult::error("Missing y1"),
- };
- let x2 = match args.opt_f64("x2") {
- Some(v) => v,
- None => return ToolResult::error("Missing x2"),
- };
- let y2 = match args.opt_f64("y2") {
- Some(v) => v,
- None => return ToolResult::error("Missing y2"),
- };
- if x2 <= x1 || y2 <= y1 {
- return ToolResult::error("x2 must be > x1 and y2 must be > y1");
+pub(crate) async fn position_named_session_keyboard_cursor(
+ state: &ToolState,
+ args: &Value,
+ pid: u32,
+ xid: u64,
+ element: Option<crate::atspi::TokenElement>,
+ pixel_target: Option<(f64, f64)>,
+ preserve_legacy_element_visual: bool,
+) {
+ let named_cursor_id = named_session_cursor_key(args);
+ let cursor_id = match named_cursor_id {
+ Some(ref cursor_id) => cursor_id.clone(),
+ None if preserve_legacy_element_visual && element.is_some() => resolve_cursor_key(args),
+ None => {
+ announce_keyboard_target(args, pid, xid, element, pixel_target).await;
+ return;
}
+ };
- let (x1, y1, x2, y2) = (
- x1 * screenshot.scale,
- y1 * screenshot.scale,
- x2 * screenshot.scale,
- y2 * screenshot.scale,
- );
- let state = self.state.clone();
- let result = tokio::task::spawn_blocking(move || {
- // Route through the Wayland-aware window capture dispatcher so
- // pure-Wayland sessions surface a typed "per-window capture not
- // supported yet" error instead of accidentally calling the
- // X11-only path with a foreign-toplevel id.
- let png = crate::wayland::screenshot_window_dispatch(xid)?;
- cursor_overlay::capture_utils::crop_png_to_jpeg(&png, x1, y1, x2, y2, 500)
+ let remembered = named_cursor_id.as_ref().and_then(|_| {
+ state
+ .cursor_registry
+ .get(&cursor_id)
+ .and_then(|cursor| cursor.x.zip(cursor.y))
+ });
+ let explicit = tokio::task::spawn_blocking(move || {
+ explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
+ })
+ .await
+ .ok()
+ .flatten();
+
+ let fallback = if named_cursor_id.is_some()
+ && explicit.is_none()
+ && cursor_overlay::keyboard_cursor_target(None, remembered, None, None).is_none()
+ {
+ tokio::task::spawn_blocking(move || {
+ let center = keyboard_window_center(xid);
+ let pointer = center.is_none().then(current_pointer_position).flatten();
+ (center, pointer)
})
- .await;
+ .await
+ .unwrap_or((None, None))
+ } else {
+ (None, None)
+ };
- match result {
- Ok(Ok(crop)) => {
- if let Err(refusal) = state.snapshots.set_zoom(
- pid,
- session_id.as_deref(),
- ZoomContext {
- screenshot,
- origin_x: crop.origin_x,
- origin_y: crop.origin_y,
- scale_inv: crop.scale_inv,
- },
- ) {
- return refusal;
- }
- use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
- let b64 = B64.encode(&crop.jpeg_bytes);
- let (w, h) = (crop.out_w, crop.out_h);
- use cua_driver_core::protocol::Content;
- ToolResult {
- content: vec![
- Content::image_jpeg(b64),
- Content::text(format!(
- "Zoom ({x1:.0},{y1:.0})–({x2:.0},{y2:.0}) → {w}×{h} px JPEG."
- )),
- ],
- is_error: None,
- // Surface 7: `mime_type` mirrors the MCP image part's `mimeType`
- // onto the structured payload (additive — `format` stays).
- structured_content: Some(json!({
- "width": w, "height": h, "format": "jpeg",
- "mime_type": "image/jpeg"
- })),
- action_record: None,
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("Zoom failed: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
+ let Some((sx, sy)) =
+ cursor_overlay::keyboard_cursor_target(explicit, remembered, fallback.0, fallback.1)
+ else {
+ return;
+ };
+ if xid != 0 {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
}
+ reveal_pointer_action_for(state, &cursor_id, sx, sy, false).await;
}
-// ── type_text_chars ───────────────────────────────────────────────────────────
-
-pub struct TypeTextCharsTool;
-static TYPE_CHARS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for TypeTextCharsTool {
- fn def(&self) -> &ToolDef {
- TYPE_CHARS_DEF.get_or_init(|| ToolDef {
- name: "type_text_chars".into(),
- description: "Type text character-by-character with a configurable inter-character \
- delay (default 30 ms). Useful for apps that miss keystrokes. \
- Otherwise identical to type_text (XSendEvent, no focus steal).".into(),
- input_schema: json!({
- "type":"object","required":["pid","text"],"properties":{
- "pid":{"type":"integer","description":"Target process ID."},
- "window_id":{"type":"integer","description":"Window id from list_windows. Required with element_index."},
- "text":{"type":"string","description":"Text to type, one character at a time."},
- "delay_ms":{"type":"integer","description":"Milliseconds between chars (default 30)."},
- "type_chars_only":{"type":"boolean","description":"Skip element focus, type directly. Default false."}
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- #[allow(unused_assignments)]
- let pid = args.u64_or("pid", 0) as u32;
- let text_raw = match args.require_str("text") {
- Ok(v) => v,
- Err(e) => return e,
- };
- // Same trailing-protocol-tag scrub as TypeTextTool — see
- // cua_driver_core::text_sanitize for rationale.
- let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
- .into_owned();
- let delay_ms = args.u64_or("delay_ms", 30);
- let xid_opt = args.opt_u64("window_id");
- let xid = match xid_opt {
- Some(x) => x,
- None => {
- let windows = tokio::task::spawn_blocking(move || {
- crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
- })
- .await
- .unwrap_or_default();
- // pid omitted: the keys go to the active window, like a
- // physical keyboard would, and the action adopts its pid.
- let chosen = if pid == 0 {
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active && w.pid.is_some())
- .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
- } else {
- // The pid's active window, else its largest on-screen
- // toplevel: LibreOffice and GIMP own hidden/utility
- // toplevels that a plain `first()` could pick, and an
- // unmapped window cannot take the virtual keyboard focus.
- let active = crate::x11::active_window();
- windows
- .iter()
- .find(|w| Some(w.xid) == active)
- .or_else(|| {
- windows
- .iter()
- .filter(|w| w.is_on_screen)
- .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
- })
- .or_else(|| windows.first())
- };
- match chosen {
- Some(w) => w.xid,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- let text_len = text.chars().count();
- let result = spawn_blocking_bounded(
- "type_text_chars",
- paced_typing_budget(text_len, delay_ms),
- move || {
- if crate::wayland::wayland_input_enabled() {
- // Per-char `wtype` loop with the requested delay — mirrors the
- // X11 XSendEvent per-char path. Sleeping here is fine because
- // we're inside spawn_blocking.
- let mut buf = [0u8; 4];
- for ch in text.chars() {
- let s = ch.encode_utf8(&mut buf);
- crate::wayland::type_text(xid, s)?;
- if delay_ms > 0 {
- std::thread::sleep(std::time::Duration::from_millis(delay_ms));
- }
- }
- return Ok(());
- }
- crate::input::send_type_text_with_delay(xid, &text, delay_ms)
- },
- )
- .await;
- match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) with {delay_ms}ms delay."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
+fn process_name(pid: u32) -> Option<String> {
+ let cmdline = fs::read(format!("/proc/{pid}/cmdline")).ok()?;
+ let first = String::from_utf8_lossy(&cmdline)
+ .split('\0')
+ .next()
+ .unwrap_or("")
+ .trim()
+ .to_owned();
+ if !first.is_empty() {
+ return std::path::Path::new(&first)
+ .file_name()
+ .map(|s| s.to_string_lossy().into_owned())
+ .or(Some(first));
}
+
+ let status = fs::read_to_string(format!("/proc/{pid}/status")).ok()?;
+ status
+ .lines()
+ .find(|l| l.starts_with("Name:"))
+ .map(|l| l[5..].trim().to_owned())
}
-// ── kill_app ──────────────────────────────────────────────────────────────────
-
-pub struct KillAppTool;
-static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for KillAppTool {
- fn def(&self) -> &ToolDef {
- KILL_DEF.get_or_init(|| ToolDef {
- name: "kill_app".into(),
- description: "Force-terminate a process by pid (kill -9 equivalent on Linux). \
- Use as escalation when the cooperative close path failed to make the process \
- exit. Unsaved state is lost — prefer the cooperative path first."
- .into(),
- input_schema: json!({"type":"object","required":["pid"],"properties":{
- "pid":{"type":"integer","description":"PID of the process to terminate."}
- },"additionalProperties":false}),
- read_only: false,
- destructive: true,
- idempotent: true,
- open_world: false,
- })
+pub(crate) fn is_terminal_process(pid: u32) -> bool {
+ // Canonical list lives in `crate::terminal::TERMINAL_PROCESS_NAMES`
+ // — keeps the additive contract centralised. Adding a new terminal
+ // here means appending one string in `crate::terminal`.
+ match process_name(pid).as_deref() {
+ Some(name) => crate::terminal::is_terminal_process_name(name),
+ None => false,
}
+}
- async fn protected_resource_scope(
- &self,
- adapter_id: &str,
- args: &Value,
- ) -> Result<Option<Value>, String> {
- if adapter_id != "process_control" {
- return Ok(None);
- }
- use cua_driver_core::browser::platform::BrowserPlatform;
- let pid = args
- .get("pid")
- .and_then(Value::as_i64)
- .filter(|pid| *pid > 0)
- .ok_or_else(|| "kill_app requires a positive integer pid".to_owned())?;
- let fingerprint = crate::browser_platform::LinuxBrowserPlatform::default()
- .process_fingerprint(pid)
- .await
- .map_err(|error| error.message)?;
- Ok(Some(json!({
- "kind": "process_instance",
- "fingerprint": fingerprint,
- })))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Some(expected) = args.get("_protected_process_fingerprint") {
- let current = match self
- .protected_resource_scope("process_control", &args)
- .await
- {
- Ok(Some(scope)) => scope["fingerprint"].clone(),
- Ok(None) => Value::Null,
- Err(message) => return kill_app_stale_process_refusal(message),
- };
- if current != *expected {
- return kill_app_stale_process_refusal(
- "the process identity changed at the termination boundary".to_owned(),
- );
- }
- }
- let pid_i = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(p) if p > 0 && p <= i32::MAX as i64 => p as i32,
- Some(_) => {
- return ToolResult::error("kill_app: `pid` must be a positive integer".to_string())
- }
- None => {
- return ToolResult::error(
- "kill_app: missing required integer field `pid`".to_string(),
- )
- }
+fn terminal_descendant_ttys(pid: u32) -> Vec<PathBuf> {
+ let mut parent_to_children: std::collections::HashMap<u32, Vec<u32>> =
+ std::collections::HashMap::new();
+ let proc_dir = std::path::Path::new("/proc");
+ let entries = match fs::read_dir(proc_dir) {
+ Ok(entries) => entries,
+ Err(_) => return Vec::new(),
+ };
+
+ for entry in entries.flatten() {
+ let pid_str = entry.file_name();
+ let pid_str = pid_str.to_string_lossy();
+ let child_pid: u32 = match pid_str.parse() {
+ Ok(pid) => pid,
+ Err(_) => continue,
};
- // Read the instance before signaling so the confirmation below can
- // tell its exit from a later process that reused the pid.
- let before = match crate::proc_fs::read_process_stat(pid_i as u32) {
- Ok(Some(before)) => before,
- Ok(None) => {
- return kill_app_failure(
- pid_i,
- "process_not_found",
- format!("kill_app: pid {pid_i} does not exist; no signal was sent."),
- )
- }
- Err(error) => {
- return kill_app_failure(
- pid_i,
- "process_identity_unavailable",
- format!(
- "kill_app: cannot read /proc/{pid_i}/stat to verify termination: \
- {error}; no signal was sent."
- ),
- )
- }
+ let status = match fs::read_to_string(proc_dir.join(&*pid_str).join("status")) {
+ Ok(status) => status,
+ Err(_) => continue,
};
- // SAFETY: libc::kill is a thin syscall wrapper, no thread-safety concerns.
- let rc = unsafe { libc::kill(pid_i, libc::SIGKILL) };
- if rc != 0 {
- let err = std::io::Error::last_os_error();
- return kill_app_failure(
- pid_i,
- "signal_failed",
- format!(
- "kill_app: kill(pid={pid_i}, SIGKILL) failed: {err}. \
- The process may not exist, or the daemon lacks permission to signal it."
- ),
- );
+ let parent_pid = status
+ .lines()
+ .find(|l| l.starts_with("PPid:"))
+ .and_then(|l| l[5..].trim().parse::<u32>().ok());
+ if let Some(parent_pid) = parent_pid {
+ parent_to_children
+ .entry(parent_pid)
+ .or_default()
+ .push(child_pid);
}
- // An accepted signal is not an exit: a sandbox (gVisor) can accept
- // SIGKILL for a process that keeps running. Report success only once
- // the same instance is gone.
- let deadline = tokio::time::Instant::now() + KILL_CONFIRM_TIMEOUT;
- loop {
- let now = crate::proc_fs::read_process_stat(pid_i as u32);
- if let Some(observation) = now
- .as_ref()
- .ok()
- .and_then(|now| crate::proc_fs::process_exit_observation(before, *now))
- {
- return ToolResult::text(format!(
- "✅ Terminated pid {pid_i} with SIGKILL (confirmed: {observation})."
- ))
- .with_structured(json!({
- "status": "terminated",
- "effect": "confirmed",
- "pid": pid_i,
- "terminated": true,
- "observation": observation,
- }));
- }
- if tokio::time::Instant::now() >= deadline {
- let detail = match now {
- Err(error) => format!("/proc/{pid_i}/stat became unreadable: {error}"),
- Ok(_) => format!(
- "the same process was still alive {} ms later",
- KILL_CONFIRM_TIMEOUT.as_millis()
- ),
- };
- let mut result = kill_app_failure(
- pid_i,
- "termination_unconfirmed",
- format!("kill_app: SIGKILL was accepted for pid {pid_i}, but {detail}."),
- );
- if let Some(structured) = result.structured_content.as_mut() {
- structured["effect"] = json!("suspected_noop");
- }
- return result;
+ }
+
+ let mut descendants = Vec::new();
+ let mut queue = std::collections::VecDeque::from([pid]);
+ while let Some(current) = queue.pop_front() {
+ if let Some(children) = parent_to_children.get(&current) {
+ for &child in children {
+ descendants.push(child);
+ queue.push_back(child);
}
- tokio::time::sleep(KILL_CONFIRM_INTERVAL).await;
}
}
-}
-
-/// How long `kill_app` waits for the signaled process to exit, and how often
-/// it re-reads `/proc/<pid>/stat` meanwhile.
-const KILL_CONFIRM_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
-const KILL_CONFIRM_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
-
-fn kill_app_failure(pid: i32, code: &str, message: String) -> ToolResult {
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "failed",
- "pid": pid,
- "terminated": false,
- "code": code,
- "message": message,
- }))
-}
+ descendants.sort_unstable();
-fn kill_app_stale_process_refusal(message: String) -> ToolResult {
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "refusal": {
- "code": "protected_resource_scope_stale",
- "message": message,
+ let mut ttys = Vec::new();
+ for child in descendants {
+ let tty = match fs::read_link(format!("/proc/{child}/fd/0")) {
+ Ok(path) => path,
+ Err(_) => continue,
+ };
+ if tty.starts_with("/dev/pts/") {
+ ttys.push(tty);
}
- }))
+ }
+ ttys
}
-// ── invoke_menu (Linux) ──────────────────────────────────────────────────────
-
-pub struct InvokeMenuTool;
-
-static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
- if path.is_empty() || path.len() > 16 {
- return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+fn terminal_tty_for_window(pid: u32, xid: u64) -> Option<PathBuf> {
+ if !is_terminal_process(pid) {
+ return None;
}
- path.into_iter()
- .enumerate()
- .map(|(index, segment)| {
- let segment = segment.trim();
- if segment.is_empty() {
- Err(format!("invoke_menu: path segment {index} is empty"))
- } else {
- Ok(segment.to_owned())
- }
- })
- .collect()
+ let mut windows = crate::x11::list_windows(Some(pid));
+ windows.sort_by_key(|w| w.xid);
+ let window_index = windows.iter().position(|w| w.xid == xid)?;
+ let ttys = terminal_descendant_ttys(pid);
+ ttys.get(window_index).cloned()
}
-fn menu_refusal(message: String) -> ToolResult {
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "refusal": { "code": "menu_path_unavailable", "message": message }
- }))
+/// Type into a terminal window without touching X focus. Resolves the window's
+/// pty, then borrows the emulator's master fd and writes to it (see
+/// `crate::tty`). Returns `Ok(false)` when the target isn't a terminal we can
+/// reach this way so the caller falls back to the generic XSendEvent path.
+pub(crate) fn inject_terminal_input(pid: u32, xid: u64, text: &str) -> anyhow::Result<bool> {
+ let Some(tty) = terminal_tty_for_window(pid, xid) else {
+ return Ok(false);
+ };
+ // tty is `/dev/pts/<N>`; the emulator (pid) holds the master for the same N.
+ let Some(ptn) = tty
+ .file_name()
+ .and_then(|s| s.to_str())
+ .and_then(|s| s.parse::<u32>().ok())
+ else {
+ return Ok(false);
+ };
+ crate::tty::inject_via_master(pid, ptn, text)
}
-#[async_trait]
-impl Tool for InvokeMenuTool {
- fn def(&self) -> &ToolDef {
- INVOKE_MENU_DEF.get_or_init(|| {
- let contract =
- cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
- ToolDef {
- name: contract.name,
- description: contract.description,
- input_schema: contract.input_schema,
- read_only: contract.annotations.read_only,
- destructive: contract.annotations.destructive,
- idempotent: contract.annotations.idempotent,
- open_world: contract.annotations.open_world,
- }
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::action_record::{
- ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
- EvidenceKind, RequestedDelivery,
- };
+pub(crate) fn capture_admission_error(error: anyhow::Error) -> ToolResult {
+ let code = cua_driver_core::capture_runtime::admission_error_code(&error);
+ ToolResult::error(format!("capture-bound click refused: {error}"))
+ .with_structured(json!({ "code": code, "effect": "refused" }))
+}
- // Menu paths are resolved and fired through AT-SPI, which needs no
- // window activation on X11: the default is background (focus-free,
- // under the focus guard). `delivery_mode:"foreground"` is the explicit
- // escalation that activates the window first, as every other tool.
- let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
- let mut args = args;
- if let Some(object) = args.as_object_mut() {
- // Not part of the closed InvokeMenuInput contract; consumed above.
- object.remove("delivery_mode");
- }
- let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let path = match normalized_menu_path(input.path) {
- Ok(path) => path,
- Err(error) => return menu_refusal(error),
- };
- let pid = input.pid;
- let window_id = input.window_id;
- if !crate::wayland::list_windows_dispatch(Some(pid))
- .iter()
- .any(|window| window.xid == window_id && window.pid == Some(pid))
- {
- return menu_refusal("invoke_menu: window_id does not belong to pid".into());
- }
+// ── px-focus helper (keyboard family) ───────────────────────────────────────
- if !crate::wayland::is_wayland() && !delivery.is_foreground() {
- let outcome = tokio::task::spawn_blocking(move || {
- crate::input::focus_guard::guarded(Some(pid), || {
- crate::atspi::native::invoke_menu_path_in(pid, window_id, &path)
- })
- })
- .await;
- return match outcome {
- Ok(Ok(((), guard))) => attach_focus_guard(
- ToolResult::text(
- "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=background, focus untouched); verify the command's semantic effect from fresh state.",
- )
- .with_structured(json!({
- "path": "ax",
- "verified": false,
- "effect": "unverifiable",
- "delivery_mode": "background",
- }))
- .with_action_record(
- ActionExecutionRecord::builder(
- ActionEffect::Unverifiable,
- ActionTransport::LinuxAtSpiAction,
- RequestedDelivery::Background,
- )
- .actual_delivery(ActualDelivery::Background)
- .evidence(ActionEvidence {
- kind: EvidenceKind::NativeApiResult,
- detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"
- .into(),
- })
- .build()
- .expect("invoke_menu record is valid"),
- ),
- guard.as_ref(),
- ),
- Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),
- Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
- };
+/// px-focus for the keyboard family (type_text / press_key / hotkey): pixel-click
+/// at (x,y) to establish real renderer focus before a keystroke — the *element px
+/// action* form of a keyboard tool. Reuses ClickTool's exact coordinate
+/// translation + delivery_mode so it lands on the same pixel a px-click would.
+/// `Ok(())` on success; `Err(ToolResult)` short-circuits the caller.
+///
+/// Retains the parent's authenticated session admission for the nested click.
+/// `point` is the window-local pixel to click for focus.
+pub(crate) async fn focus_by_pixel(
+ state: &Arc<ToolState>,
+ pid: u32,
+ window_id: Option<u64>,
+ (x, y): (f64, f64),
+ foreground: bool,
+ parent_args: &Value,
+ from_zoom: bool,
+) -> Result<(), ToolResult> {
+ let mut click_args = json!({
+ "pid": pid, "x": x, "y": y,
+ "delivery_mode": if foreground { "foreground" } else { "background" },
+ });
+ if let Some(wid) = window_id {
+ click_args["window_id"] = json!(wid);
+ }
+ for field in [
+ "session",
+ "_session_id",
+ "_transport_session_id",
+ "cursor_id",
+ "scope",
+ "coordinate_frame",
+ ] {
+ if let Some(value) = parent_args.get(field) {
+ click_args[field] = value.clone();
}
+ }
+ if from_zoom {
+ click_args["from_zoom"] = json!(true);
+ }
+ let focus = ClickTool {
+ state: state.clone(),
+ }
+ .invoke(click_args)
+ .await;
+ if focus.is_error == Some(true) {
+ return Err(focus);
+ }
+ // Brief settle so the renderer registers focus before the keystrokes.
+ tokio::time::sleep(std::time::Duration::from_millis(120)).await;
+ Ok(())
+}
- let activation = if crate::wayland::is_wayland() {
- let result = tokio::task::spawn_blocking(move || {
- crate::wayland::activate_window_for_input_target(window_id, Some(pid))
- })
- .await;
- match result {
- Ok(Ok(())) => None,
- Ok(Err(error)) => {
- return menu_refusal(format!(
- "invoke_menu: target window could not be activated: {error}"
- ))
- }
- Err(error) => {
- return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
- }
- }
- } else {
- match tokio::task::spawn_blocking(move || {
- crate::input::x11_activate_window_persistent(window_id)
- })
+/// Establish widget-local focus inside a nested-compositor target without
+/// changing the compositor's focused toplevel. AX uses Component.GrabFocus;
+/// PX sends a private per-surface left click at the requested local point.
+pub(crate) async fn focus_nested_inject_target(
+ pid: u32,
+ window_id: u64,
+ element: Option<crate::atspi::TokenElement>,
+ pixel: Option<(f64, f64)>,
+) -> Result<(), ToolResult> {
+ if let Some(element) = element {
+ let index = element.index;
+ return match tokio::task::spawn_blocking(move || crate::atspi::focus_token(pid, &element))
.await
- {
- Ok(Ok(prior)) => Some(prior),
- Ok(Err(error)) => {
- return menu_refusal(format!(
- "invoke_menu: target window could not be activated: {error}"
- ))
- }
- Err(error) => {
- return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
- }
- }
+ {
+ Ok(Ok(true)) => Ok(()),
+ Ok(Ok(false)) => Err(ToolResult::error(format!(
+ "AT-SPI Component.GrabFocus returned false for element {index}"
+ ))),
+ Ok(Err(error)) => Err(stale_observed_element("focus", index, &error)
+ .unwrap_or_else(|| ToolResult::error(error.to_string()))),
+ Err(error) => Err(ToolResult::error(format!("Task error: {error}"))),
};
-
- let outcome = tokio::task::spawn_blocking(move || {
- let result = crate::atspi::native::invoke_menu_path_in(pid, window_id, &path);
- if let Some(Some(prior_window)) = activation {
- let _ = crate::input::x11_activate_window_persistent(prior_window);
- }
- result
+ }
+ if let Some((x, y)) = pixel {
+ return match tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_click(pid, window_id, x, y, 1, 1)
})
- .await;
-
- match outcome {
- Ok(Ok(())) => ToolResult::text(
- "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=foreground); verify the command's semantic effect from fresh state.",
- )
- .with_structured(json!({
- "path": "ax",
- "verified": false,
- "effect": "unverifiable",
- "delivery_mode": "foreground",
- }))
- .with_action_record(
- ActionExecutionRecord::builder(
- ActionEffect::Unverifiable,
- ActionTransport::LinuxAtSpiAction,
- RequestedDelivery::Foreground,
- )
- .actual_delivery(ActualDelivery::Foreground)
- .evidence(ActionEvidence {
- kind: EvidenceKind::NativeApiResult,
- detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"
- .into(),
- })
- .build()
- .expect("invoke_menu record is valid"),
- ),
- Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),
- Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
- }
+ .await
+ {
+ Ok(Ok(())) => Ok(()),
+ Ok(Err(error)) => Err(ToolResult::error(error.to_string())),
+ Err(error) => Err(ToolResult::error(format!("Task error: {error}"))),
+ };
}
+ Ok(())
}
-// ── set_window_frame (Linux) ─────────────────────────────────────────────────
+pub(crate) fn is_modifier(k: &str) -> bool {
+ matches!(
+ k.to_lowercase().as_str(),
+ "ctrl"
+ | "control"
+ | "shift"
+ | "alt"
+ | "super"
+ | "meta"
+ | "cmd"
+ | "command"
+ | "win"
+ | "windows"
+ )
+}
-pub struct SetWindowFrameTool;
+/// Numeric-aware equality for the `set_value` read-back ("40" == "40.0").
+pub(crate) fn values_agree(wanted: &str, observed: &str) -> bool {
+ let wanted = wanted.trim();
+ let observed = observed.trim();
+ if wanted == observed {
+ return true;
+ }
+ match (wanted.parse::<f64>(), observed.parse::<f64>()) {
+ (Ok(a), Ok(b)) => (a - b).abs() <= 1e-6 * a.abs().max(1.0),
+ _ => false,
+ }
+}
-static SET_WINDOW_FRAME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+/// Read the true X11 root-window size in pixels: (width, height).
+/// Shared by `get_screen_size` and `get_desktop_state`.
+pub(crate) fn x11_screen_size() -> anyhow::Result<(u32, u32)> {
+ use x11rb::connection::Connection;
+ use x11rb::rust_connection::RustConnection;
+ let (conn, screen_num) = RustConnection::connect(None)
+ .map_err(|e| anyhow::anyhow!("{e}{}", crate::no_display_hint()))?;
+ let setup = conn.setup();
+ let screen = &setup.roots[screen_num];
+ let w = screen.width_in_pixels as u32;
+ let h = screen.height_in_pixels as u32;
+ // WSLg / headless XWayland quirk: the X server connects but the
+ // root screen advertises a 0-px geometry until a real output is
+ // attached. Returning {width:0,height:0} here would propagate a
+ // success with zero dimensions to the client, which then either
+ // divides by zero when scaling or feeds the value into `int(...)`
+ // after the missing key collapses to None. Fail loudly with an
+ // actionable, typed error instead (never emit a 0/null where the
+ // client expects a usable int). See issue #2005.
+ if w == 0 || h == 0 {
+ anyhow::bail!(
+ "X11 connected but reports a 0x0 root screen — no usable \
+ display geometry.{}",
+ crate::no_display_hint()
+ );
+ }
+ Ok((w, h))
+}
-#[async_trait]
-impl Tool for SetWindowFrameTool {
- fn def(&self) -> &ToolDef {
- SET_WINDOW_FRAME_DEF.get_or_init(|| {
- let contract = cua_driver_contract::tool_contract("set_window_frame")
- .expect("set_window_frame contract");
- ToolDef {
- name: contract.name,
- description: contract.description,
- input_schema: contract.input_schema,
- read_only: contract.annotations.read_only,
- destructive: contract.annotations.destructive,
- idempotent: contract.annotations.idempotent,
- open_world: contract.annotations.open_world,
- }
- })
+/// Put the desktop image in the exact coordinate frame consumed by desktop
+/// actions. Native Wayland capture buffers may use backing pixels while the
+/// compositor's pointer protocol and reported screen geometry use logical
+/// pixels. Returning the backing image alongside logical dimensions violates
+/// the screenshot-to-action contract and makes every vision-grounded action
+/// miss by the output scale.
+pub(crate) fn normalize_desktop_capture_for_action_frame(
+ png: Vec<u8>,
+ action_width: u32,
+ action_height: u32,
+) -> anyhow::Result<(Vec<u8>, u32, u32, f64)> {
+ if action_width == 0 || action_height == 0 {
+ anyhow::bail!("desktop action frame is empty: {action_width}x{action_height}");
}
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_contract::SetWindowFrameInput;
- use cua_driver_core::action_record::{
- effect_from_value_readback, ActionEvidence, ActionExecutionRecord, ActionTransport,
- ActualDelivery, EvidenceKind, RequestedDelivery,
- };
+ let (capture_width, capture_height) = crate::capture::png_dimensions_pub(&png)?;
+ let scale_x = f64::from(capture_width) / f64::from(action_width);
+ let scale_y = f64::from(capture_height) / f64::from(action_height);
+ if (scale_x - scale_y).abs() > 0.01 {
+ anyhow::bail!(
+ "desktop capture {capture_width}x{capture_height} cannot be mapped uniformly to \
+ action frame {action_width}x{action_height} (scale {scale_x:.4}x{scale_y:.4})"
+ );
+ }
- let input: SetWindowFrameInput =
- match cua_driver_core::tool_args::parse_typed_input("set_window_frame", args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- if crate::wayland::is_wayland() {
- return ToolResult::error(
- "set_window_frame: this Wayland compositor exposes no protocol that permits a client to set another top-level window's geometry",
- );
- }
- let values = [input.x, input.y, input.width, input.height];
- if values.iter().any(|value| !value.is_finite())
- || input.width <= 0.0
- || input.height <= 0.0
- {
- return ToolResult::error(
- "set_window_frame: x/y must be finite and width/height must be finite positive numbers",
- );
- }
- let to_i32 = |name: &str, value: f64| -> Result<i32, String> {
- let rounded = value.round();
- if rounded < i32::MIN as f64 || rounded > i32::MAX as f64 {
- Err(format!("{name} is out of range for an X11 coordinate"))
- } else {
- Ok(rounded as i32)
- }
- };
- let to_u32 = |name: &str, value: f64| -> Result<u32, String> {
- let rounded = value.round();
- if rounded < 1.0 || rounded > u32::MAX as f64 {
- Err(format!("{name} is out of range for an X11 size"))
- } else {
- Ok(rounded as u32)
- }
- };
- let x = match to_i32("x", input.x) {
- Ok(value) => value,
- Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
- };
- let y = match to_i32("y", input.y) {
- Ok(value) => value,
- Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
- };
- let width = match to_u32("width", input.width) {
- Ok(value) => value,
- Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
- };
- let height = match to_u32("height", input.height) {
- Ok(value) => value,
- Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
- };
- let window_id = input.window_id;
- let pid = input.pid;
- let outcome = tokio::task::spawn_blocking(move || {
- let before = crate::x11::list_windows(Some(pid))
- .into_iter()
- .find(|window| window.xid == window_id)
- .map(|window| (window.x, window.y, window.width, window.height));
- let (observed, confirmed, mutation_error) =
- crate::x11::set_window_frame(window_id, pid, x, y, width, height)
- .map_err(|error| error.to_string())?;
- let observed_frame =
- observed.map(|window| (window.x, window.y, window.width, window.height));
- let changed = observed_frame.is_some_and(|observed| before != Some(observed));
- Ok::<_, String>((observed_frame, confirmed, changed, mutation_error))
- })
- .await;
- let (observed, confirmed, changed, mutation_error) = match outcome {
- Ok(Ok(outcome)) => outcome,
- Ok(Err(error)) => return ToolResult::error(format!("set_window_frame: {error}")),
- Err(error) => {
- return ToolResult::error(format!(
- "set_window_frame: blocking task failed: {error}"
- ));
- }
- };
- let effect = effect_from_value_readback(confirmed, changed, observed.is_some());
- let requested = (x, y, width, height);
- let mut record = ActionExecutionRecord::builder(
- effect,
- ActionTransport::LinuxX11ConfigureWindow,
- RequestedDelivery::NotApplicable,
- )
- .actual_delivery(ActualDelivery::NotApplicable)
- .detail(format!(
- "requested={requested:?} observed={observed:?} mutation_error={mutation_error:?}"
- ));
- if observed.is_some() {
- record = record.evidence(ActionEvidence {
- kind: EvidenceKind::ValueReadback,
- detail: if confirmed {
- "X11 geometry readback matched the requested frame".into()
- } else {
- "X11 geometry readback did not match the requested frame".into()
- },
- });
- }
- ToolResult::text(if confirmed {
- "Set and verified the requested window frame."
- } else if observed.is_none() {
- "The native window mutation was attempted, but its resulting frame could not be read back."
- } else {
- "The window frame did not settle at the requested geometry."
- })
- .with_action_record(record.build().expect("set_window_frame record is valid"))
- }
-}
-
-// ── bring_to_front (Linux) ───────────────────────────────────────────────────
-
-pub struct BringToFrontTool;
-
-static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for BringToFrontTool {
- fn def(&self) -> &ToolDef {
- BTF_DEF.get_or_init(|| ToolDef {
- name: "bring_to_front".into(),
- description:
- "Persistently activate a window so subsequent input lands on it. \
- This deliberately breaks the no-foreground contract and is not part \
- of the normal input ladder. For an ordinary `background_unavailable` \
- response, retry only the refused action with \
- `delivery_mode:\"foreground\"`; the input tool performs its own \
- activate, act, and restore sequence. Use `bring_to_front` only for a \
- focus-proxy surface that must remain foreground across multiple calls, \
- such as a remote desktop session, or when repeated action-scoped \
- activation prevents the remote surface from accepting input. \
- X11: EWMH _NET_ACTIVE_WINDOW activation (the `wmctrl -a` equivalent, \
- with proper timestamp handling to beat focus-stealing prevention). \
- Wayland: activates through a target-addressable compositor adapter \
- (wlroots foreign-toplevel or the GNOME Shell helper) and refuses \
- when the compositor offers no safe adapter. Matches the macOS / \
- Windows bring_to_front rung."
- .into(),
- input_schema: serde_json::json!({
- "type":"object","required":["pid"],"properties":{
- "pid":{"type":"integer","description":"Process ID of the app to activate."},
- "window_id":{"type":"integer","description":"X11 window id (xid) to activate. If omitted, the first window of `pid` is used."}
- },"additionalProperties":false
- }),
- read_only: false, destructive: false, idempotent: true, open_world: false,
- })
+ if capture_width == action_width && capture_height == action_height {
+ return Ok((png, action_width, action_height, 1.0));
}
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let pid = args.u64_or("pid", 0) as u32;
- if crate::wayland::is_wayland() {
- let window_id = match args.opt_u64("window_id") {
- Some(window_id) => window_id,
- None => match crate::wayland::list_windows_dispatch(Some(pid)).first() {
- Some(window) => window.xid,
- None => {
- return ToolResult::error(format!(
- "bring_to_front: no window_id given and no Wayland windows found for pid {pid}."
- ))
- }
- },
- };
- if crate::wayland::wayland_input_enabled() && crate::wayland::hyprland::is_session() {
- if args.opt_u64("window_id").is_none() {
- return foreground_hyprland_refusal("an exact window_id is required");
- }
- return foreground_hyprland_action(
- &args,
- pid,
- window_id,
- crate::wayland::hyprland_input::Action::Activate,
- )
- .await;
- }
- let result = tokio::task::spawn_blocking(move || {
- crate::wayland::activate_window_for_input_target(window_id, Some(pid))
- })
- .await;
- return match result {
- Ok(Ok(())) => {
- ToolResult::text(format!("Brought Wayland window {window_id} to front."))
- .with_structured(serde_json::json!({
- "window_id": window_id,
- "platform": "linux",
- "session": "wayland",
- }))
- }
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
+ let image = image::load_from_memory_with_format(&png, image::ImageFormat::Png)?;
+ let resized = image.resize_exact(
+ action_width,
+ action_height,
+ image::imageops::FilterType::Lanczos3,
+ );
+ let mut encoded = std::io::Cursor::new(Vec::new());
+ resized.write_to(&mut encoded, image::ImageFormat::Png)?;
+ Ok((encoded.into_inner(), action_width, action_height, scale_x))
+}
- // X11: resolve the target xid (window_id, else first window for pid).
- let xid = match args.opt_u64("window_id") {
- Some(x) => x,
- None => {
- let windows =
- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.xid,
- None => {
- return ToolResult::error(format!(
- "bring_to_front: no window_id given and no windows found for pid {pid}."
- ))
- }
- }
- }
- };
- let r =
- tokio::task::spawn_blocking(move || crate::input::x11_activate_window_persistent(xid))
- .await;
- match r {
- Ok(Ok(prior)) => ToolResult::text(format!(
- "✅ Brought window {xid} to front (X11 _NET_ACTIVE_WINDOW)."
- ))
- .with_structured(serde_json::json!({
- "window_id": xid,
- "prior_active": prior,
- "platform": "linux",
- })),
- Ok(Err(e)) => ToolResult::error(format!("bring_to_front failed: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
+#[cfg(test)]
+#[tokio::test]
+async fn check_permissions_accepts_the_status_prompt_argument() {
+ let result = build_registry_with_provider(false, None)
+ .invoke("check_permissions", json!({ "prompt": false }))
+ .await;
+ let wire = serde_json::to_value(&result).unwrap();
+ assert!(wire["structuredContent"].get("x11").is_some(), "{wire}");
}
// ── registry ─────────────────────────────────────────────────────────────────
@@ -15611,30 +4718,293 @@ pub fn build_registry_with_provider(
}
#[cfg(test)]
-mod click_button_schema_tests;
+mod pid_window_target_tests {
+ use super::*;
+ use cua_driver_core::window_target::{resolve_pid_window_target, PidWindowTargetResolution};
-#[cfg(test)]
-mod pid_window_target_tests;
+ fn window(xid: u64, pid: u32) -> crate::x11::WindowInfo {
+ crate::x11::WindowInfo {
+ xid,
+ pid: Some(pid),
+ app_name: "editor".into(),
+ title: format!("Document {xid}"),
+ is_on_screen: true,
+ z_index: Some(1),
+ x: 0,
+ y: 0,
+ width: 640,
+ height: 480,
+ }
+ }
-#[cfg(test)]
-mod session_cursor_target_tests;
+ #[test]
+ fn same_pid_sibling_windows_are_ambiguous() {
+ let candidates =
+ window_target_candidates_for_pid([window(7, 42), window(8, 42), window(9, 99)], 42);
+ assert!(matches!(
+ resolve_pid_window_target(candidates),
+ PidWindowTargetResolution::Ambiguous(windows)
+ if windows.iter().map(|window| window.window_id).collect::<Vec<_>>() == [7, 8]
+ ));
+ }
+}
#[cfg(test)]
-mod display_only_tests;
-#[cfg(test)]
-mod window_capture_dimension_tests;
+mod desktop_capture_frame_tests {
+ use super::{capture_admission_error, normalize_desktop_capture_for_action_frame};
-#[cfg(test)]
-mod desktop_capture_frame_tests;
+ fn png(width: u32, height: u32) -> Vec<u8> {
+ let rgba = vec![0x7f; (width * height * 4) as usize];
+ cua_driver_core::image_utils::encode_rgba_to_png(&rgba, width, height)
+ .expect("encode fixture")
+ }
-#[cfg(test)]
-mod background_budget_tests;
+ #[test]
+ fn native_wayland_capture_is_resized_to_the_reported_action_frame() {
+ let (normalized, width, height, scale) =
+ normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1000)
+ .expect("normalize 2x capture");
+
+ assert_eq!((width, height), (1600, 1000));
+ assert_eq!(
+ cua_driver_core::image_utils::png_dimensions(&normalized).unwrap(),
+ (1600, 1000)
+ );
+ assert_eq!(scale, 2.0);
+ }
+
+ #[test]
+ fn nonuniform_capture_mapping_fails_instead_of_distorting_coordinates() {
+ let error = normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1200)
+ .expect_err("nonuniform mapping must fail closed");
+ assert!(error.to_string().contains("cannot be mapped uniformly"));
+ }
+
+ #[test]
+ fn capture_refusal_exposes_specific_code_and_refused_effect() {
+ let refusal = capture_admission_error(anyhow::Error::new(
+ cua_driver_core::capture_runtime::CaptureActionError::NativeActionFrameMismatch,
+ ));
+ let structured = refusal.structured_content.unwrap();
+ assert_eq!(structured["code"], "capture_frame_mismatch");
+ assert_eq!(structured["effect"], "refused");
+ }
+}
#[cfg(test)]
-mod background_keyboard_route_tests;
+mod background_keyboard_route_tests {
+ use super::{element_needs_real_click, split_key_combo, wm_chord_kind, WmChord};
+
+ #[test]
+ fn key_combos_split_into_modifiers_and_key() {
+ assert_eq!(
+ split_key_combo("alt+F4"),
+ (vec!["alt".to_owned()], "F4".to_owned())
+ );
+ assert_eq!(
+ split_key_combo("ctrl+shift+t"),
+ (vec!["ctrl".to_owned(), "shift".to_owned()], "t".to_owned())
+ );
+ assert_eq!(split_key_combo("+"), (vec![], "+".to_owned()));
+ assert_eq!(
+ split_key_combo("ctrl++"),
+ (vec!["ctrl".to_owned()], "+".to_owned())
+ );
+ assert_eq!(split_key_combo("Return"), (vec![], "Return".to_owned()));
+ // Not a modifier prefix: left untouched for the keysym resolver.
+ assert_eq!(split_key_combo("a+b"), (vec![], "a+b".to_owned()));
+ }
+
+ #[test]
+ fn wm_chords_are_recognised() {
+ let m = |xs: &[&str]| xs.iter().map(|x| x.to_string()).collect::<Vec<_>>();
+ assert_eq!(
+ wm_chord_kind("F4", &m(&["alt"])),
+ Some(WmChord::CloseWindow)
+ );
+ assert_eq!(
+ wm_chord_kind("f4", &m(&["Alt_L"])),
+ Some(WmChord::CloseWindow)
+ );
+ assert_eq!(
+ wm_chord_kind("Tab", &m(&["alt"])),
+ Some(WmChord::Unavailable)
+ );
+ assert_eq!(
+ wm_chord_kind("Tab", &m(&["alt", "shift"])),
+ Some(WmChord::Unavailable)
+ );
+ assert_eq!(
+ wm_chord_kind("a", &m(&["super"])),
+ Some(WmChord::Unavailable)
+ );
+ assert_eq!(
+ wm_chord_kind("t", &m(&["ctrl", "alt"])),
+ Some(WmChord::Unavailable)
+ );
+ assert_eq!(
+ wm_chord_kind("F4", &m(&["ctrl", "alt"])),
+ Some(WmChord::Unavailable)
+ );
+ // Application chords stay with the application.
+ assert_eq!(wm_chord_kind("c", &m(&["ctrl"])), None);
+ assert_eq!(wm_chord_kind("F4", &m(&["ctrl"])), None);
+ assert_eq!(wm_chord_kind("f", &m(&["alt"])), None);
+ assert_eq!(wm_chord_kind("F4", &m(&[])), None);
+ assert_eq!(wm_chord_kind("s", &m(&["ctrl", "shift"])), None);
+ }
+
+ #[test]
+ fn focus_taking_roles_need_a_real_click() {
+ for role in ["spin button", "Text", "entry", "slider", "combo box"] {
+ assert!(element_needs_real_click(role), "{role}");
+ }
+ for role in ["menu item", "push button", "menu", "check box", "page tab"] {
+ assert!(!element_needs_real_click(role), "{role}");
+ }
+ }
+}
#[cfg(test)]
-mod visibility_tests;
+mod visibility_tests {
+ use super::*;
+
+ #[test]
+ fn framed_elements_lead_and_keep_their_relative_order() {
+ let elements = vec![
+ json!({"element_index": 0, "role": "menu"}),
+ json!({"element_index": 1, "role": "push button", "frame": {"x": 1, "y": 1, "w": 2, "h": 2}}),
+ json!({"element_index": 2, "role": "menu"}),
+ json!({"element_index": 3, "role": "paragraph", "frame": {"x": 1, "y": 9, "w": 2, "h": 2}}),
+ ];
+ let ordered: Vec<u64> = framed_elements_first(elements)
+ .iter()
+ .map(|e| e["element_index"].as_u64().unwrap())
+ .collect();
+ assert_eq!(ordered, vec![1, 3, 0, 2]);
+ }
+
+ #[test]
+ fn a_frameless_popup_returns_only_the_menu_entries_drawn_inside_it() {
+ let elements = vec![
+ json!({"element_index": 1, "role": "push button", "frame": {"x": 5, "y": 5, "w": 20, "h": 20}}),
+ json!({"element_index": 2, "role": "menu item", "label": "Paragraph...", "frame": {"x": 2, "y": 30, "w": 180, "h": 22}}),
+ json!({"element_index": 3, "role": "menu item", "label": "elsewhere", "frame": {"x": 400, "y": 30, "w": 180, "h": 22}}),
+ json!({"element_index": 4, "role": "menu", "label": "Format"}),
+ ];
+ let kept: Vec<u64> = popup_menu_elements(elements.clone(), (0, 0), 200, 400)
+ .iter()
+ .map(|e| e["element_index"].as_u64().unwrap())
+ .collect();
+ assert_eq!(kept, vec![2]);
+ assert_eq!(
+ popup_menu_elements(elements[..1].to_vec(), (0, 0), 200, 400).len(),
+ 1
+ );
+ }
+
+ #[test]
+ fn a_combo_popup_returns_the_rows_of_the_list_that_fills_it() {
+ // Qt: the combo list lives under the combo box inside the dialog; the
+ // main window's menubar entries are elsewhere on the screen.
+ let elements = vec![
+ json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": -600, "y": -300, "w": 50, "h": 20}}),
+ json!({"element_index": 7, "role": "list", "frame": {"x": 1, "y": 1, "w": 198, "h": 118}}),
+ json!({"element_index": 8, "role": "list item", "label": "Desktop", "parent_index": 7, "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),
+ json!({"element_index": 9, "role": "list item", "label": "user", "parent_index": 7, "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),
+ json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),
+ ];
+ let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)
+ .iter()
+ .map(|e| e["element_index"].as_u64().unwrap())
+ .collect();
+ assert_eq!(kept, vec![8, 9]);
+ }
+
+ #[test]
+ fn a_list_popup_without_an_indexed_container_keeps_its_item_rows_only() {
+ let elements = vec![
+ json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": 1, "y": 1, "w": 50, "h": 20}}),
+ json!({"element_index": 8, "role": "list item", "label": "Desktop", "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),
+ json!({"element_index": 9, "role": "tree item", "label": "user", "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),
+ json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),
+ ];
+ let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)
+ .iter()
+ .map(|e| e["element_index"].as_u64().unwrap())
+ .collect();
+ assert_eq!(kept, vec![8, 9]);
+ }
+
+ #[test]
+ fn menu_roles_take_the_real_press() {
+ for role in ["menu", "Menu Item", "check menu item", "radio menu item"] {
+ assert!(element_is_menu_role(role), "{role}");
+ }
+ for role in ["menu bar", "push button", "text"] {
+ assert!(!element_is_menu_role(role), "{role}");
+ }
+ }
+
+ #[test]
+ fn overlays_name_the_follow_up_call_per_dialog_and_popup() {
+ let overlays = WindowOverlays {
+ dialogs: vec![json!({
+ "window_id": 71, "title": "Position and Size", "transient_for": 3, "modal": true,
+ "bounds": {"x": 100, "y": 100, "width": 400, "height": 300}
+ })],
+ popups: vec![json!({
+ "window_id": 72, "title": "", "pid": 5,
+ "bounds": {"x": 10, "y": 30, "width": 200, "height": 400}
+ })],
+ covers_window: true,
+ window_rect: Some((0, 0, 800, 600)),
+ };
+ let note = overlays.follow_up(5).unwrap();
+ assert!(
+ note.contains(
+ "dialog \"Position and Size\" (window_id 71, transient of window 3, modal)"
+ ),
+ "{note}"
+ );
+ assert!(
+ note.contains("get_window_state(pid=5, window_id=71)"),
+ "{note}"
+ );
+ assert!(
+ note.contains("popup (window_id 72, bounds x=10 y=30 200x400)"),
+ "{note}"
+ );
+ assert!(rects_intersect((0, 0, 800, 600), (100, 100, 400, 300)));
+ assert!(!rects_intersect((0, 0, 800, 600), (800, 0, 10, 10)));
+ }
+
+ #[test]
+ fn a_cross_process_dialog_names_its_real_owning_pid_in_the_follow_up() {
+ // GIMP's "Export Image as JPEG" dialog spawned as a separate process
+ // (owning_pid 999) but is transient-for the target pid (5)'s window.
+ let overlays = WindowOverlays {
+ dialogs: vec![json!({
+ "window_id": 90, "title": "Export Image as JPEG", "transient_for": 10,
+ "modal": false,
+ "bounds": {"x": 0, "y": 0, "width": 300, "height": 200},
+ "owning_pid": 999,
+ })],
+ popups: vec![],
+ covers_window: false,
+ window_rect: Some((0, 0, 800, 600)),
+ };
+ let note = overlays.follow_up(5).unwrap();
+ assert!(
+ note.contains("owned by a different process pid 999, not pid 5"),
+ "{note}"
+ );
+ assert!(
+ note.contains("get_window_state(pid=5, window_id=90)"),
+ "{note}"
+ );
+ }
+}
#[cfg(test)]
mod cursor_hook_emission_tests {
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_budget_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_budget_tests.rs
deleted file mode 100644
index edcb78ed06..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_budget_tests.rs
+++ /dev/null
@@ -1,51 +0,0 @@
-use super::*;
-
-#[test]
-fn synthetic_pointer_drop_detection_covers_gtk_vcl_and_qt() {
- assert!(maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libgtk-3.so.0\n"
- ));
- assert!(maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libgtk-4.so.1\n"
- ));
- assert!(maps_indicate_synthetic_pointer_dropped(
- "7f /opt/libreoffice/program/libmergedlo.so\n"
- ));
- assert!(maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libvcllo.so\n"
- ));
- assert!(maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libQt5Gui.so.5\n"
- ));
- assert!(!maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libgtk-x11-2.0.so.0\n"
- ));
- assert!(!maps_indicate_synthetic_pointer_dropped(
- "7f /usr/lib/libX11.so.6\n"
- ));
-}
-
-#[test]
-fn desktop_window_lines_expose_pid_and_window_id() {
- let windows = vec![crate::x11::WindowInfo {
- xid: 0x2e00003,
- pid: Some(4321),
- app_name: "libreoffice".into(),
- title: "Untitled 1 - LibreOffice Calc".into(),
- is_on_screen: true,
- z_index: None,
- x: 0,
- y: 27,
- width: 1920,
- height: 1053,
- }];
- let text = desktop_window_lines(&windows, 1.0);
- assert!(text.contains("pid=4321 window_id=48234499"));
- assert!(text.contains("LibreOffice Calc"));
- assert!(text.contains("app=libreoffice"));
- assert!(text.contains("get_window_state(pid, window_id)"));
- assert!(desktop_window_lines(&[], 1.0).contains("none"));
- // A downsized desktop screenshot reports window rects in ITS pixels.
- let scaled = desktop_window_lines(&windows, 1.5);
- assert!(scaled.contains("1280x702 at (0,18)"), "{scaled}");
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_keyboard_route_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_keyboard_route_tests.rs
deleted file mode 100644
index 3e11a11fc2..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/background_keyboard_route_tests.rs
+++ /dev/null
@@ -1,70 +0,0 @@
-use super::{element_needs_real_click, split_key_combo, wm_chord_kind, WmChord};
-
-#[test]
-fn key_combos_split_into_modifiers_and_key() {
- assert_eq!(
- split_key_combo("alt+F4"),
- (vec!["alt".to_owned()], "F4".to_owned())
- );
- assert_eq!(
- split_key_combo("ctrl+shift+t"),
- (vec!["ctrl".to_owned(), "shift".to_owned()], "t".to_owned())
- );
- assert_eq!(split_key_combo("+"), (vec![], "+".to_owned()));
- assert_eq!(
- split_key_combo("ctrl++"),
- (vec!["ctrl".to_owned()], "+".to_owned())
- );
- assert_eq!(split_key_combo("Return"), (vec![], "Return".to_owned()));
- // Not a modifier prefix: left untouched for the keysym resolver.
- assert_eq!(split_key_combo("a+b"), (vec![], "a+b".to_owned()));
-}
-
-#[test]
-fn wm_chords_are_recognised() {
- let m = |xs: &[&str]| xs.iter().map(|x| x.to_string()).collect::<Vec<_>>();
- assert_eq!(
- wm_chord_kind("F4", &m(&["alt"])),
- Some(WmChord::CloseWindow)
- );
- assert_eq!(
- wm_chord_kind("f4", &m(&["Alt_L"])),
- Some(WmChord::CloseWindow)
- );
- assert_eq!(
- wm_chord_kind("Tab", &m(&["alt"])),
- Some(WmChord::Unavailable)
- );
- assert_eq!(
- wm_chord_kind("Tab", &m(&["alt", "shift"])),
- Some(WmChord::Unavailable)
- );
- assert_eq!(
- wm_chord_kind("a", &m(&["super"])),
- Some(WmChord::Unavailable)
- );
- assert_eq!(
- wm_chord_kind("t", &m(&["ctrl", "alt"])),
- Some(WmChord::Unavailable)
- );
- assert_eq!(
- wm_chord_kind("F4", &m(&["ctrl", "alt"])),
- Some(WmChord::Unavailable)
- );
- // Application chords stay with the application.
- assert_eq!(wm_chord_kind("c", &m(&["ctrl"])), None);
- assert_eq!(wm_chord_kind("F4", &m(&["ctrl"])), None);
- assert_eq!(wm_chord_kind("f", &m(&["alt"])), None);
- assert_eq!(wm_chord_kind("F4", &m(&[])), None);
- assert_eq!(wm_chord_kind("s", &m(&["ctrl", "shift"])), None);
-}
-
-#[test]
-fn focus_taking_roles_need_a_real_click() {
- for role in ["spin button", "Text", "entry", "slider", "combo box"] {
- assert!(element_needs_real_click(role), "{role}");
- }
- for role in ["menu item", "push button", "menu", "check box", "page tab"] {
- assert!(!element_needs_real_click(role), "{role}");
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/click_button_schema_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/click_button_schema_tests.rs
deleted file mode 100644
index 3d13735e85..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/click_button_schema_tests.rs
+++ /dev/null
@@ -1,57 +0,0 @@
-use super::{chromium_background_must_refuse, maps_indicate_gtk, ClickTool};
-use cua_driver_core::tool::Tool;
-
-/// Surface 5: schema must advertise the three canonical button values and
-/// describe the back-compat default. Linux already routed button=middle/right
-/// pre-Surface-5; this freezes the schema shape so the contract can't drift.
-#[test]
-fn schema_advertises_button_enum_and_description() {
- let tool = ClickTool {
- state: super::ToolState::new(),
- };
- let d = tool.def();
- let props = d.input_schema.get("properties").expect("properties");
- let button = props.get("button").expect("button field present");
- assert_eq!(button.get("type").and_then(|v| v.as_str()), Some("string"));
- let enum_vals: Vec<&str> = button
- .get("enum")
- .and_then(|v| v.as_array())
- .expect("button.enum present")
- .iter()
- .filter_map(|v| v.as_str())
- .collect();
- for need in ["left", "right", "middle"] {
- assert!(enum_vals.contains(&need), "missing {need} in button.enum");
- }
- let desc = button
- .get("description")
- .and_then(|v| v.as_str())
- .expect("button.description present");
- let lc = desc.to_ascii_lowercase();
- assert!(lc.contains("left"), "description should mention default");
- assert!(
- lc.contains("wayland"),
- "description should call out wayland fallback"
- );
-}
-
-#[test]
-fn chromium_background_requires_focus_free_inject_mode() {
- assert!(chromium_background_must_refuse(false, false, true));
- assert!(!chromium_background_must_refuse(false, true, true));
- assert!(!chromium_background_must_refuse(true, false, true));
- assert!(!chromium_background_must_refuse(false, false, false));
-}
-
-#[test]
-fn gtk_process_maps_are_detected_without_matching_unrelated_libraries() {
- assert!(maps_indicate_gtk(
- "7f00-7f01 r-xp /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.32"
- ));
- assert!(maps_indicate_gtk(
- "7f00-7f01 r-xp /nix/store/hash-gtk4/lib/libgtk-4.so.1"
- ));
- assert!(!maps_indicate_gtk(
- "7f00-7f01 r-xp /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so"
- ));
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/desktop_capture_frame_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/desktop_capture_frame_tests.rs
deleted file mode 100644
index 82baf3fc4d..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/desktop_capture_frame_tests.rs
+++ /dev/null
@@ -1,37 +0,0 @@
-use super::{capture_admission_error, normalize_desktop_capture_for_action_frame};
-
-fn png(width: u32, height: u32) -> Vec<u8> {
- let rgba = vec![0x7f; (width * height * 4) as usize];
- cua_driver_core::image_utils::encode_rgba_to_png(&rgba, width, height).expect("encode fixture")
-}
-
-#[test]
-fn native_wayland_capture_is_resized_to_the_reported_action_frame() {
- let (normalized, width, height, scale) =
- normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1000)
- .expect("normalize 2x capture");
-
- assert_eq!((width, height), (1600, 1000));
- assert_eq!(
- cua_driver_core::image_utils::png_dimensions(&normalized).unwrap(),
- (1600, 1000)
- );
- assert_eq!(scale, 2.0);
-}
-
-#[test]
-fn nonuniform_capture_mapping_fails_instead_of_distorting_coordinates() {
- let error = normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1200)
- .expect_err("nonuniform mapping must fail closed");
- assert!(error.to_string().contains("cannot be mapped uniformly"));
-}
-
-#[test]
-fn capture_refusal_exposes_specific_code_and_refused_effect() {
- let refusal = capture_admission_error(anyhow::Error::new(
- cua_driver_core::capture_runtime::CaptureActionError::NativeActionFrameMismatch,
- ));
- let structured = refusal.structured_content.unwrap();
- assert_eq!(structured["code"], "capture_frame_mismatch");
- assert_eq!(structured["effect"], "refused");
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs
deleted file mode 100644
index 4876c86c8a..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/display_only_tests.rs
+++ /dev/null
@@ -1,35 +0,0 @@
-use super::{GetWindowStateTool, ToolState};
-use cua_driver_core::tool::Tool;
-
-// T3 Code feature-detects the preview mode from this schema property.
-#[test]
-fn display_only_is_advertised() {
- let tool = GetWindowStateTool {
- state: ToolState::new(),
- };
- assert_eq!(
- tool.def().input_schema["properties"]["display_only"]["type"],
- "boolean"
- );
-}
-
-#[tokio::test]
-async fn display_only_refuses_accessibility_snapshot_requests() {
- let tool = GetWindowStateTool {
- state: ToolState::new(),
- };
- for tree in [None, Some(true)] {
- let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
- if let Some(tree) = tree {
- args["include_accessibility_tree"] = serde_json::json!(tree);
- }
- let result = tool.invoke(args).await;
- assert_eq!(
- serde_json::to_value(result).unwrap(),
- serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
- "display_only requires include_accessibility_tree:false"
- ))
- .unwrap()
- );
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/get_window_state_actions_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/get_window_state_actions_tests.rs
deleted file mode 100644
index fddb546a42..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/get_window_state_actions_tests.rs
+++ /dev/null
@@ -1,41 +0,0 @@
-use super::*;
-use crate::atspi::AtspiNode;
-
-fn node(actions: Vec<String>) -> AtspiNode {
- AtspiNode {
- element_index: Some(1),
- role: "button".to_owned(),
- name: Some("ok".to_owned()),
- value: None,
- checked: None,
- enabled: Some(true),
- selected: None,
- description: None,
- actions,
- element_key: 1,
- identity: None,
- depth: 0,
- parent_element_index: None,
- in_web_content: false,
- object_ref: None,
- }
-}
-
-#[test]
-fn element_entry_omits_actions_when_empty() {
- let n = node(Vec::new());
- let entry = build_element_entry(&n, None, None).unwrap();
- assert!(entry.get("actions").is_none());
-}
-
-#[test]
-fn element_entry_filters_blank_action_names() {
- let n = node(vec![
- "Press".to_owned(),
- "".to_owned(),
- " ".to_owned(),
- "Open".to_owned(),
- ]);
- let entry = build_element_entry(&n, None, None).unwrap();
- assert_eq!(entry["actions"], json!(["Press", "Open"]));
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_app_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_app_tests.rs
deleted file mode 100644
index 9adec8731e..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_app_tests.rs
+++ /dev/null
@@ -1,116 +0,0 @@
-use super::*;
-use crate::installed_apps::InstalledApp;
-
-fn app(name: &str, bundle_id: &str, launch_path: &str) -> InstalledApp {
- InstalledApp {
- name: name.to_owned(),
- bundle_id: bundle_id.to_owned(),
- launch_path: launch_path.to_owned(),
- last_used: None,
- startup_wm_class: None,
- }
-}
-
-fn fixture() -> Vec<InstalledApp> {
- vec![
- app("Galculator", "galculator", "galculator"),
- app(
- "Google Chrome",
- "google-chrome",
- "/usr/bin/google-chrome-stable",
- ),
- app("File Manager", "thunar", "thunar"),
- app(
- "File Manager Settings",
- "thunar-settings",
- "thunar-settings",
- ),
- ]
-}
-
-/// Process state letter from `/proc/<pid>/stat`, or `None` once the entry
-/// is gone. `comm` may itself contain spaces and parentheses, so the state
-/// is read after the final `)` rather than by splitting from the left.
-fn proc_state(pid: u32) -> Option<char> {
- let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
- let after_comm = stat.rsplit_once(')')?.1;
- after_comm.split_whitespace().next()?.chars().next()
-}
-
-#[test]
-fn launched_children_are_reaped_instead_of_lingering_as_zombies() {
- // A launched app that exits must not stay in the process table. The
- // driver never waits on what it launches, so without an explicit
- // reaper every launch leaked a pid slot for the daemon's lifetime and
- // left "does this pid exist" liveness checks reporting a terminated
- // app as still running.
- // `/bin/sh` rather than a richer coreutils binary: it is the one
- // executable POSIX and the Nix build sandbox both guarantee, and the
- // sandbox has no `/bin/true`.
- let pid = spawn_launch_command("/bin/sh -c exit", &[]).expect("/bin/sh should spawn");
- let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
- loop {
- match proc_state(pid) {
- None => break, // reaped, entry gone
- Some(state) if state != 'Z' => {} // still running or exiting
- Some(_) => {
- assert!(
- std::time::Instant::now() < deadline,
- "pid {pid} was still a zombie after the reaper deadline"
- );
- }
- }
- assert!(
- std::time::Instant::now() < deadline,
- "pid {pid} was never reaped"
- );
- std::thread::sleep(std::time::Duration::from_millis(25));
- }
-}
-
-#[test]
-fn matches_exact_display_name_case_insensitively() {
- let apps = fixture();
- let hit = match_installed_app(&apps, "google chrome").expect("should match");
- assert_eq!(hit.bundle_id, "google-chrome");
-}
-
-#[test]
-fn matches_desktop_file_id_and_exec_basename() {
- let apps = fixture();
- assert_eq!(
- match_installed_app(&apps, "galculator").unwrap().name,
- "Galculator"
- );
- assert_eq!(
- match_installed_app(&apps, "google-chrome-stable")
- .unwrap()
- .name,
- "Google Chrome"
- );
-}
-
-#[test]
-fn matches_unambiguous_display_name_substring() {
- let apps = fixture();
- assert_eq!(
- match_installed_app(&apps, "chrome").unwrap().name,
- "Google Chrome"
- );
-}
-
-#[test]
-fn refuses_ambiguous_substring_and_unknown_names() {
- let apps = fixture();
- // "file manager" is a substring of two entries — refuse to guess.
- // ("File Manager" itself still resolves via the exact-name rung.)
- assert!(match_installed_app(&apps, "file man").is_none());
- assert_eq!(
- match_installed_app(&apps, "file manager")
- .unwrap()
- .bundle_id,
- "thunar"
- );
- assert!(match_installed_app(&apps, "gnome-calculator").is_none());
- assert!(match_installed_app(&apps, "").is_none());
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_resolution_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_resolution_tests.rs
deleted file mode 100644
index 614c4f1028..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/launch_resolution_tests.rs
+++ /dev/null
@@ -1,102 +0,0 @@
-use super::*;
-
-fn window(class: &str) -> crate::x11::WindowInfo {
- crate::x11::WindowInfo {
- xid: 1,
- pid: Some(7),
- app_name: class.into(),
- title: "t".into(),
- is_on_screen: true,
- z_index: None,
- x: 0,
- y: 0,
- width: 1,
- height: 1,
- }
-}
-
-#[test]
-fn launch_queries_match_window_classes() {
- assert!(window_matches_launch(
- &window("org.gnome.Nautilus"),
- "nautilus"
- ));
- assert!(window_matches_launch(
- &window("org.gnome.Nautilus"),
- "org.gnome.Nautilus"
- ));
- assert!(window_matches_launch(
- &window("Gnome-terminal"),
- "gnome-terminal"
- ));
- assert!(window_matches_launch(
- &window("Gnome-control-center"),
- "gnome-control-center"
- ));
- assert!(window_matches_launch(
- &window("Org.gnome.Nautilus"),
- "org.gnome.Nautilus.desktop"
- ));
- assert!(window_matches_launch(
- &window("Org.gnome.Nautilus"),
- "Files\u{0}org.gnome.Nautilus\u{0}nautilus --new-window"
- ));
- assert!(window_matches_launch(
- &window("Gnome-control-center"),
- "Settings\u{0}Settings\u{0}gnome-control-center"
- ));
- assert!(!window_matches_launch(&window("Gedit"), "nautilus"));
- assert!(!window_matches_launch(&window(""), "nautilus"));
-}
-
-fn owned(xid: u64, pid: u32, class: &str) -> crate::x11::WindowInfo {
- crate::x11::WindowInfo {
- xid,
- pid: Some(pid),
- ..window(class)
- }
-}
-
-#[test]
-fn a_launcher_splash_waits_for_the_app_window() {
- let splash = owned(10, 377, "libreoffice");
- let calc = owned(20, 398, "libreoffice-calc");
- let none = std::collections::HashSet::new();
- let is_splash = |w: &crate::x11::WindowInfo| w.xid == 10;
-
- assert!(settle_launch(&[splash.clone()], 377, "libreoffice", &none, is_splash).is_none());
-
- let launched = settle_launch(&[splash, calc], 377, "libreoffice", &none, is_splash).unwrap();
- assert_eq!(launched.pid, Some(398));
- assert_eq!(
- launched.windows.iter().map(|w| w.xid).collect::<Vec<_>>(),
- [20]
- );
-
- let gimp = owned(30, 377, "gimp");
- let launched = settle_launch(&[gimp], 377, "gimp", &none, is_splash).unwrap();
- assert_eq!((launched.pid, launched.handed_off), (Some(377), false));
-}
-
-#[test]
-fn a_missing_process_counts_as_exited() {
- assert!(process_exited(u32::MAX - 1));
- assert!(!process_exited(std::process::id()));
-}
-
-#[test]
-fn a_handoff_timeout_is_a_refusal_not_a_null_pid_success() {
- let result = launch_handoff_timeout_result("Nautilus", 4242, 8);
- assert_eq!(
- result.is_error,
- Some(true),
- "a hand-off timeout must be reported as an error, not success"
- );
- let s = result.structured_content.unwrap();
- assert_eq!(s["code"], "launch_handoff_timeout");
- assert_eq!(s["effect"], "refused");
- assert_eq!(s["pid"], Value::Null);
- assert_eq!(s["running"], Value::Null);
- assert_eq!(s["launcher_pid"], 4242);
- assert_eq!(s["waited_secs"], 8);
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/list_windows_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/list_windows_tests.rs
deleted file mode 100644
index e94134275a..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/list_windows_tests.rs
+++ /dev/null
@@ -1,102 +0,0 @@
-use super::*;
-
-#[test]
-fn record_has_bounds_and_flat_legacy_fields() {
- let w = crate::x11::WindowInfo {
- xid: 42,
- pid: Some(1234),
- app_name: "example-app".to_owned(),
- title: "Example".to_owned(),
- is_on_screen: true,
- z_index: Some(3),
- x: 10,
- y: 20,
- width: 300,
- height: 400,
- };
- let rec = window_record_json(&w);
-
- // Canonical cross-platform shape: nested `bounds` object.
- let bounds = rec
- .get("bounds")
- .expect("record must carry a `bounds` object");
- assert_eq!(bounds["x"], json!(10));
- assert_eq!(bounds["y"], json!(20));
- assert_eq!(bounds["width"], json!(300));
- assert_eq!(bounds["height"], json!(400));
-
- // Legacy alias: flat fields must still be present.
- assert_eq!(rec["x"], json!(10));
- assert_eq!(rec["y"], json!(20));
- assert_eq!(rec["width"], json!(300));
- assert_eq!(rec["height"], json!(400));
-
- // Cross-platform companions.
- assert_eq!(rec["app_name"], json!("example-app"));
- assert_eq!(rec["is_on_screen"], json!(true));
- assert_eq!(rec["z_index"], json!(3));
- assert_eq!(rec["window_id"], json!(42));
- assert_eq!(rec["title"], json!("Example"));
-}
-
-#[test]
-fn unavailable_wayland_order_serializes_as_null() {
- let w = crate::x11::WindowInfo {
- xid: 43,
- pid: Some(1234),
- app_name: "native-wayland-app".to_owned(),
- title: "Example".to_owned(),
- is_on_screen: true,
- z_index: None,
- x: 0,
- y: 0,
- width: 300,
- height: 400,
- };
-
- assert_eq!(window_record_json(&w)["z_index"], Value::Null);
-}
-
-#[test]
-fn desktop_discovery_retains_only_visible_windows_with_live_or_unknown_owners() {
- let window = |xid, pid, is_on_screen| crate::x11::WindowInfo {
- xid,
- pid,
- app_name: "native-wayland-app".to_owned(),
- title: "Example".to_owned(),
- is_on_screen,
- z_index: None,
- x: 10,
- y: 20,
- width: 300,
- height: 400,
- };
- let windows = visible_live_desktop_windows(vec![
- window(1, Some(std::process::id()), true),
- window(2, Some(std::process::id()), false),
- window(3, Some(u32::MAX), true),
- window(4, None, true),
- window(5, None, false),
- ]);
-
- assert_eq!(
- windows.iter().map(|w| w.xid).collect::<Vec<_>>(),
- vec![1, 4]
- );
- let record = window_record_json(&windows[0]);
- assert_eq!(
- record["bounds"],
- json!({"x": 10, "y": 20, "width": 300, "height": 400})
- );
- assert_eq!(record["app_name"], json!("native-wayland-app"));
- assert_eq!(record["is_on_screen"], json!(true));
- assert_eq!(record["z_index"], Value::Null);
-}
-
-#[test]
-fn chromium_launch_detection_uses_executable_basename() {
- assert!(chromium_family_program("/usr/bin/google-chrome-stable"));
- assert!(chromium_family_program("CuaTestHarness.Electron"));
- assert!(chromium_family_program("chromium-browser"));
- assert!(!chromium_family_program("/usr/bin/gnome-text-editor"));
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/megapixel_cap_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/megapixel_cap_tests.rs
deleted file mode 100644
index 51fc69679c..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/megapixel_cap_tests.rs
+++ /dev/null
@@ -1,25 +0,0 @@
-use super::{megapixel_long_edge_cap, WINDOW_SCREENSHOT_MAX_PIXELS};
-
-#[test]
-fn a_window_shot_above_the_api_threshold_is_capped_below_it() {
- let edge = megapixel_long_edge_cap(1568, 861, WINDOW_SCREENSHOT_MAX_PIXELS).expect("capped");
- let scale = edge as f64 / 1568.0;
- let (w, h) = (
- (1568.0 * scale).round() as u64,
- (861.0 * scale).round() as u64,
- );
- assert!(w * h <= WINDOW_SCREENSHOT_MAX_PIXELS, "{w}x{h}");
- assert!((1440..1568).contains(&edge), "{edge}");
-}
-
-#[test]
-fn a_shot_within_the_threshold_is_left_alone() {
- assert_eq!(
- megapixel_long_edge_cap(1280, 800, WINDOW_SCREENSHOT_MAX_PIXELS),
- None
- );
- assert_eq!(
- megapixel_long_edge_cap(0, 800, WINDOW_SCREENSHOT_MAX_PIXELS),
- None
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_resolver_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_resolver_tests.rs
deleted file mode 100644
index 687cb50eb5..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_resolver_tests.rs
+++ /dev/null
@@ -1,95 +0,0 @@
-use super::*;
-
-fn win(xid: u64, z: Option<usize>, on_screen: bool, w: u32, h: u32) -> crate::x11::WindowInfo {
- crate::x11::WindowInfo {
- xid,
- pid: Some(7),
- app_name: "gimp".into(),
- title: format!("w{xid}"),
- is_on_screen: on_screen,
- z_index: z,
- x: 0,
- y: 0,
- width: w,
- height: h,
- }
-}
-
-#[test]
-fn point_resolver_prefers_the_topmost_window_by_stacking_position() {
- // Main window listed last (as `_NET_CLIENT_LIST_STACKING` bottom-up
- // enumeration does when a dialog sits above it); the dialog wins.
- let dialog = crate::x11::WindowInfo {
- z_index: Some(5),
- ..win(2, None, true, 400, 300)
- };
- let main = crate::x11::WindowInfo {
- z_index: Some(3),
- ..win(1, None, true, 1920, 1080)
- };
- assert_eq!(
- topmost_window_at(&[main.clone(), dialog.clone()], 7, 100.0, 100.0),
- Some(2)
- );
- assert_eq!(topmost_window_at(&[dialog, main], 7, 100.0, 100.0), Some(2));
-}
-
-#[test]
-fn point_resolver_ignores_windows_without_a_stacking_position_when_one_has_it() {
- let ranked = win(2, Some(0), true, 400, 300);
- let unranked = win(1, None, true, 1920, 1080);
- assert_eq!(
- topmost_window_at(&[ranked, unranked], 7, 10.0, 10.0),
- Some(2)
- );
-}
-
-#[test]
-fn pid_only_resolver_orders_focus_then_dialog_then_active_then_largest() {
- let main = win(1, Some(1), true, 1920, 1080);
- let dialog = win(2, Some(2), true, 400, 300);
- let dock = win(3, Some(0), true, 200, 900);
- let hidden = win(4, None, false, 3000, 3000);
- let windows = vec![main, dialog, dock, hidden];
- let transient = |xid: u64| (xid == 2).then_some(1u64);
- // 1. core focus inside the pid wins outright (even over a dialog).
- assert_eq!(
- crate::x11::pick_pid_window(&windows, Some(3), transient, Some(1)),
- Some(3)
- );
- // 2. no focus in the pid: its topmost transient dialog.
- assert_eq!(
- crate::x11::pick_pid_window(&windows, None, transient, Some(1)),
- Some(2)
- );
- // 3. no dialog: the WM's active window when it is the pid's.
- assert_eq!(
- crate::x11::pick_pid_window(&windows, None, |_| None, Some(3)),
- Some(3)
- );
- // 4. otherwise the largest mapped toplevel (never the unmapped one).
- assert_eq!(
- crate::x11::pick_pid_window(&windows, None, |_| None, None),
- Some(1)
- );
- assert_eq!(
- crate::x11::pick_pid_window(&windows, None, |_| None, Some(99)),
- Some(1)
- );
- // A focused window that is not the pid's is ignored.
- assert_eq!(
- crate::x11::pick_pid_window(&windows, Some(42), |_| None, None),
- Some(1)
- );
-}
-
-#[test]
-fn pid_only_resolver_skips_unmapped_dialogs() {
- let main = win(1, Some(1), true, 800, 600);
- let unmapped_dialog = win(2, Some(2), false, 400, 300);
- let windows = vec![main, unmapped_dialog];
- assert_eq!(
- crate::x11::pick_pid_window(&windows, None, |xid| (xid == 2).then_some(1u64), None),
- Some(1)
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_target_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_target_tests.rs
deleted file mode 100644
index dcf3a31f27..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pid_window_target_tests.rs
+++ /dev/null
@@ -1,28 +0,0 @@
-use super::*;
-use cua_driver_core::window_target::{resolve_pid_window_target, PidWindowTargetResolution};
-
-fn window(xid: u64, pid: u32) -> crate::x11::WindowInfo {
- crate::x11::WindowInfo {
- xid,
- pid: Some(pid),
- app_name: "editor".into(),
- title: format!("Document {xid}"),
- is_on_screen: true,
- z_index: Some(1),
- x: 0,
- y: 0,
- width: 640,
- height: 480,
- }
-}
-
-#[test]
-fn same_pid_sibling_windows_are_ambiguous() {
- let candidates =
- window_target_candidates_for_pid([window(7, 42), window(8, 42), window(9, 99)], 42);
- assert!(matches!(
- resolve_pid_window_target(candidates),
- PidWindowTargetResolution::Ambiguous(windows)
- if windows.iter().map(|window| window.window_id).collect::<Vec<_>>() == [7, 8]
- ));
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pointer_argument_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pointer_argument_tests.rs
deleted file mode 100644
index e9d95c76a5..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/pointer_argument_tests.rs
+++ /dev/null
@@ -1,101 +0,0 @@
-use super::*;
-
-#[test]
-fn pid_only_pointer_call_is_refused_not_clicked_at_origin() {
- let refusal =
- require_point_args("click", &json!({"target_id": "s1:147", "pid": 1867})).expect("refused");
- assert_eq!(refusal.is_error, Some(true));
- let structured = refusal.structured_content.expect("structured");
- assert_eq!(structured["code"], "invalid_arguments");
- assert_eq!(structured["effect"], "refused");
- assert!(structured["accepted_forms"].as_array().unwrap().len() == 3);
-}
-
-#[test]
-fn complete_point_passes() {
- assert!(require_point_args(
- "click",
- &json!({"pid": 1, "window_id": 2, "x": 3, "y": 4.5})
- )
- .is_none());
- assert!(require_point_args("click", &json!({"x": 1, "y": null})).is_some());
- assert!(require_point_args("click", &json!({"y": 1})).is_some());
-}
-
-#[test]
-fn foreground_report_becomes_evidence() {
- let report = crate::input::ForegroundReport {
- already_active: false,
- retried_activation: false,
- confirm_ms: 12,
- focus_after: crate::input::FocusAfter::SamePid,
- window_change: Some("appeared: window 9 \"Brightness-Contrast\"".into()),
- window_change_owned: true,
- };
- let v = foreground_structured("x11_xtest_fg", report, serde_json::Map::new());
- assert_eq!(v["effect"], "confirmed");
- assert_eq!(v["focus_after"], "same_pid");
- assert_eq!(v["evidence"][0]["kind"], "window_change");
- let quiet = crate::input::ForegroundReport {
- already_active: true,
- retried_activation: false,
- confirm_ms: 0,
- focus_after: crate::input::FocusAfter::Target,
- window_change: None,
- window_change_owned: false,
- };
- let v = foreground_structured("x11_xtest_fg", quiet, serde_json::Map::new());
- assert_eq!(v["effect"], "unverifiable");
- assert_eq!(v["evidence"][0]["kind"], "native_api_result");
- let closed = crate::input::ForegroundReport {
- already_active: true,
- retried_activation: false,
- confirm_ms: 0,
- focus_after: crate::input::FocusAfter::Unknown,
- window_change: Some("closed: window 3 \"Format Cells\"".into()),
- window_change_owned: true,
- };
- let v = foreground_structured("x11_xtest_fg", closed, serde_json::Map::new());
- assert_eq!(v["effect"], "confirmed");
- assert_eq!(v["evidence"][0]["kind"], "window_change");
- // A popup that names no pid is listed in the target's window set (a VCL
- // menu carries none), but it may be another client's: once the focus
- // left the target process its change is no confirmation.
- let unattributed = |focus_after| crate::input::ForegroundReport {
- already_active: true,
- retried_activation: false,
- confirm_ms: 0,
- focus_after,
- window_change: Some("appeared: popup window 7 (200x300 at 1,2)".into()),
- window_change_owned: false,
- };
- let v = foreground_structured(
- "x11_xtest_fg",
- unattributed(crate::input::FocusAfter::Elsewhere),
- serde_json::Map::new(),
- );
- assert_eq!(v["effect"], "suspected_noop");
- let v = foreground_structured(
- "x11_xtest_fg",
- unattributed(crate::input::FocusAfter::Unknown),
- serde_json::Map::new(),
- );
- assert_eq!(v["effect"], "unverifiable");
- assert_eq!(v["evidence"][0]["kind"], "native_api_result");
- let v = foreground_structured(
- "x11_xtest_fg",
- unattributed(crate::input::FocusAfter::SamePid),
- serde_json::Map::new(),
- );
- assert_eq!(v["effect"], "confirmed");
- let lost = crate::input::ForegroundReport {
- already_active: true,
- retried_activation: false,
- confirm_ms: 0,
- focus_after: crate::input::FocusAfter::Elsewhere,
- window_change: None,
- window_change_owned: false,
- };
- let v = foreground_structured("x11_xtest_fg", lost, serde_json::Map::new());
- assert_eq!(v["effect"], "suspected_noop");
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/press_key_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/press_key_tests.rs
deleted file mode 100644
index f4911628a7..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/press_key_tests.rs
+++ /dev/null
@@ -1,45 +0,0 @@
-use super::is_enter_key;
-use super::press_key_chord;
-
-#[test]
-fn unmodified_press_stays_on_the_single_key_route() {
- assert_eq!(press_key_chord(&[], "return"), None);
-}
-
-#[test]
-fn enter_key_gate_accepts_the_documented_enter_spelling() {
- assert!(is_enter_key("enter"));
- assert!(is_enter_key("Enter"));
- assert!(is_enter_key("ENTER"));
-}
-
-#[test]
-fn enter_key_gate_accepts_the_documented_return_spelling() {
- // `key_name_to_keysym` resolves "return" and "enter" to the same
- // physical key (XK_Return, 0xFF0D); the pty short-circuit must not
- // drop a documented spelling on terminals (they discard synthetic
- // XSendEvent keys, so a dropped short-circuit loses the keypress).
- assert!(is_enter_key("return"));
- assert!(is_enter_key("Return"));
- assert!(is_enter_key("RETURN"));
-}
-
-#[test]
-fn enter_key_gate_rejects_non_enter_keys() {
- assert!(!is_enter_key("tab"));
- assert!(!is_enter_key("escape"));
- assert!(!is_enter_key("space"));
- assert!(!is_enter_key(""));
-}
-
-#[test]
-fn modifiers_are_promoted_to_a_chord_in_order() {
- assert_eq!(
- press_key_chord(&["ctrl".to_owned()], "s"),
- Some(vec!["ctrl".to_owned(), "s".to_owned()])
- );
- assert_eq!(
- press_key_chord(&["ctrl".to_owned(), "shift".to_owned()], "t"),
- Some(vec!["ctrl".to_owned(), "shift".to_owned(), "t".to_owned()])
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/session_cursor_target_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/session_cursor_target_tests.rs
deleted file mode 100644
index d5ee4b44a9..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/session_cursor_target_tests.rs
+++ /dev/null
@@ -1,28 +0,0 @@
-use super::{cursor_control_scope, reveal_pointer_action_for, CursorControlScope, ToolState};
-use serde_json::json;
-
-#[test]
-fn real_pointer_control_requires_explicit_desktop_scope() {
- assert_eq!(cursor_control_scope(&json!({})), CursorControlScope::Agent);
- assert_eq!(
- cursor_control_scope(&json!({"scope": "window"})),
- CursorControlScope::Agent
- );
- assert_eq!(
- cursor_control_scope(&json!({"scope": "desktop"})),
- CursorControlScope::Desktop
- );
-}
-
-#[tokio::test]
-async fn pointer_position_survives_an_unavailable_overlay() {
- let state = ToolState::new();
- reveal_pointer_action_for(&state, "no-renderer", 123.0, 456.0, true).await;
-
- let cursor = state
- .cursor_registry
- .get("no-renderer")
- .expect("pointer action records its position independently of rendering");
- assert!(cursor.config.enabled, "input must revive its agent cursor");
- assert_eq!(cursor.x.zip(cursor.y), Some((123.0, 456.0)));
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/set_value_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/set_value_tests.rs
deleted file mode 100644
index c4a909c92e..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/set_value_tests.rs
+++ /dev/null
@@ -1,48 +0,0 @@
-use super::*;
-
-#[test]
-fn a_file_chooser_keeping_the_basename_counts_as_committed() {
- assert!(path_normalised_to_basename(
- "/home/user/Desktop/tone.wav",
- "tone.wav"
- ));
- assert!(!path_normalised_to_basename(
- "/home/user/Desktop/tone.wav",
- "other.wav"
- ));
- assert!(!path_normalised_to_basename("tone.wav", "tone.wav"));
- let ok = set_value_result(
- 3,
- "/home/user/Desktop/tone.wav",
- "ax",
- Some("tone.wav".into()),
- None,
- );
- let s = ok.structured_content.unwrap();
- assert_eq!(s["verified"], true);
- assert_eq!(s["effect"], "confirmed");
-}
-
-#[test]
-fn readback_agreement_is_numeric_aware() {
- assert!(values_agree("40", "40.0"));
- assert!(values_agree("40", " 40 "));
- assert!(values_agree("abc", "abc"));
- assert!(!values_agree("40", "41"));
- assert!(!values_agree("40", "forty"));
-}
-
-#[test]
-fn set_value_result_is_verified_only_on_matching_readback() {
- let ok = set_value_result(3, "40", "click_type_mpx", Some("40.0".into()), None);
- let s = ok.structured_content.unwrap();
- assert_eq!(s["verified"], true);
- assert_eq!(s["effect"], "confirmed");
- let noop = set_value_result(3, "40", "ax", Some("10.0".into()), None);
- let s = noop.structured_content.unwrap();
- assert_eq!(s["verified"], false);
- assert_eq!(s["effect"], "suspected_noop");
- assert_eq!(s["readback"], "10.0");
- let blind = set_value_result(3, "40", "ax", None, None);
- assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/token_window_scope_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/token_window_scope_tests.rs
deleted file mode 100644
index 8650af2d1a..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/token_window_scope_tests.rs
+++ /dev/null
@@ -1,172 +0,0 @@
-use super::{
- element_click_placement, prove_token_element, stale_observed_element, token_element_for,
-};
-use crate::atspi::native::{window_element_membership, ObjectRef};
-use crate::atspi::snapshot::{cached_element, forget_window, update_snapshot, CachedElement};
-use crate::atspi::{AtspiIdentity, AtspiNode, Snapshots, TokenElement, WindowProof};
-use cua_driver_core::element_token::{token_for, ResolvedElement};
-use cua_driver_core::protocol::ToolResult;
-
-// One process owns windows A and B; each has a text field at the same index.
-const PID: u32 = 515_151;
-const WINDOW_A: u64 = 1;
-const WINDOW_B: u64 = 2;
-const INDEX: usize = 3;
-
-fn identity(path: &str) -> AtspiIdentity {
- AtspiIdentity {
- bus_name: ":1.42".into(),
- path: path.into(),
- frame_bus_name: ":1.42".into(),
- frame_path: "/frame".into(),
- }
-}
-
-fn object_ref(path: &str) -> Option<ObjectRef> {
- Some(ObjectRef {
- bus: ":1.42".into(),
- path: path.into(),
- })
-}
-
-fn observed(path: &str) -> CachedElement {
- CachedElement {
- key: 0,
- identity: identity(path),
- object_ref: object_ref(path),
- role: "text".into(),
- in_web_content: false,
- bounds: None,
- }
-}
-
-fn text_field(path: &str) -> AtspiNode {
- AtspiNode {
- element_index: Some(INDEX),
- role: "text".into(),
- name: None,
- value: None,
- checked: None,
- enabled: Some(true),
- selected: None,
- description: None,
- actions: Vec::new(),
- element_key: 0,
- identity: Some(identity(path)),
- depth: 0,
- parent_element_index: None,
- in_web_content: false,
- object_ref: object_ref(path),
- }
-}
-
-fn refusal_code(refusal: &ToolResult) -> &str {
- assert_eq!(refusal.is_error, Some(true));
- let structured = refusal.structured_content.as_ref().expect("structured");
- assert_eq!(structured["status"], "refused");
- structured["refusal"]["code"]
- .as_str()
- .expect("refusal code")
-}
-
-#[test]
-fn an_unproven_window_element_refuses_before_dispatch() {
- // The membership walk no longer finds the element in window 2's scope.
- let proven = |membership: anyhow::Result<()>| membership.map(|()| WindowProof::Proven);
- let unproven = proven(window_element_membership(Some(0), None, || false));
- let refusal = token_element_for("click", 2, 3, observed("/a"), unproven).expect_err("refused");
- assert_eq!(refusal_code(&refusal), "window_scope_unproven");
- let gone = proven(window_element_membership(None, Some(0), || true));
- let refusal = token_element_for("type_text", 2, 3, observed("/a"), gone).expect_err("refused");
- assert_eq!(refusal_code(&refusal), "window_scope_unproven");
- assert!(token_element_for("click", 2, 3, observed("/a"), Ok(WindowProof::Proven)).is_ok());
-}
-
-#[test]
-fn a_membership_failure_in_the_click_glide_refuses_instead_of_acting() {
- let unproven = window_element_membership(Some(0), None, || false).map(|()| (2, 0.0, 0.0));
- let refusal = element_click_placement(unproven, Some(2)).expect_err("refused");
- assert_eq!(refusal_code(&refusal), "window_scope_unproven");
- // Any other lookup failure only skips the cursor glide.
- let other = element_click_placement(
- Err(anyhow::anyhow!("element exposes no Component interface")),
- Some(2),
- );
- assert!(matches!(other, Ok(None)));
- assert!(matches!(
- element_click_placement(Ok((2, 10.0, 20.0)), Some(2)),
- Ok(Some((2, x, y))) if x == 10.0 && y == 20.0
- ));
-}
-
-#[tokio::test]
-async fn a_target_without_an_element_token_needs_no_membership_walk() {
- let none = ResolvedElement::<CachedElement>::None;
- assert!(matches!(
- prove_token_element(1, &none, "click").await,
- Ok(None)
- ));
-}
-
-#[test]
-fn a_proven_token_dispatches_to_its_own_element_while_another_window_is_retained() {
- // Both windows' snapshots are retained: in the runtime store and in the
- // (pid, window) side index.
- let snapshots = Snapshots::new();
- let tokens =
- [(WINDOW_A, "/window_a/text"), (WINDOW_B, "/window_b/text")].map(|(window, path)| {
- let payload = update_snapshot(PID, window, &[text_field(path)], &[]);
- (snapshots.publish(PID as i32, window, payload), window, path)
- });
- let mut pid_only = Vec::new();
- for (snapshot, window, path) in tokens {
- let args = serde_json::json!({ "element_token": token_for(snapshot, INDEX) });
- let Ok(ResolvedElement::Element {
- window_id,
- element_index,
- element,
- }) = snapshots.resolve(PID as i32, &args)
- else {
- panic!("the token for window {window} resolves to its element");
- };
- assert_eq!(window_id, window);
- // The fresh walk scoped to the window finds the observed identity.
- let proof =
- window_element_membership(Some(0), Some(0), || true).map(|()| WindowProof::Proven);
- let token = token_element_for("type_text", window_id, element_index, element, proof)
- .expect("membership proven");
- // Text, focus, value and scroll dispatch address this object.
- assert_eq!(token.object().expect("addressable").path, path);
- assert!(!token.may_reresolve, "a proven token never re-resolves");
- pid_only.push(cached_element(PID, None, INDEX).and_then(|element| element.object_ref));
- }
- // The PID-only lookup the earlier dispatch used gives both tokens the same
- // window's element, so it acted on the wrong window for one of them.
- assert_eq!(pid_only[0], pid_only[1]);
- forget_window(PID, WINDOW_A);
- forget_window(PID, WINDOW_B);
-}
-
-#[test]
-fn only_an_unproven_session_may_re_resolve_a_token_index() {
- let proven = TokenElement::new(INDEX, observed("/window_b/text"), WindowProof::Proven);
- assert!(!proven.may_reresolve);
- let unproven = TokenElement::new(INDEX, observed("/window_b/text"), WindowProof::NotRequired);
- assert!(unproven.may_reresolve);
-}
-
-#[test]
-fn a_gone_observed_object_refuses_as_a_stale_token() {
- let mut gone = observed("/window_b/text");
- gone.object_ref = None;
- let token = TokenElement::new(INDEX, gone, WindowProof::Proven);
- let error = token.object().expect_err("no D-Bus address");
- let refusal = stale_observed_element("type_text", INDEX, &error).expect("stale");
- assert_eq!(
- refusal.structured_content.expect("structured")["code"],
- "stale_element_token"
- );
- // A write the toolkit rejected is not a stale token.
- let rejected = anyhow::anyhow!("EditableText write was rejected");
- assert!(stale_observed_element("type_text", INDEX, &rejected).is_none());
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/type_text_readback_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/type_text_readback_tests.rs
deleted file mode 100644
index edff46a7aa..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/type_text_readback_tests.rs
+++ /dev/null
@@ -1,19 +0,0 @@
-use super::*;
-
-#[test]
-fn typed_text_is_confirmed_only_when_read_back() {
- let confirmed =
- type_text_readback_result(1, 3, "-40", "via targeted AT-SPI", Some("-40.0".into()));
- let s = confirmed.structured_content.clone().unwrap();
- assert_eq!(s["effect"], "confirmed");
- assert_eq!(s["verified"], true);
- assert_eq!(s["evidence"][0]["kind"], "value_readback");
- let contained = type_text_readback_result(1, 3, "ab", "r", Some("xxabyy".into()));
- assert_eq!(contained.structured_content.unwrap()["effect"], "confirmed");
- let mismatch = type_text_readback_result(1, 3, "ab", "r", Some("zz".into()));
- let s = mismatch.structured_content.unwrap();
- assert_eq!(s["effect"], "unverifiable");
- assert_eq!(s["readback"], "zz");
- let blind = type_text_readback_result(1, 3, "ab", "r", None);
- assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/visibility_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/visibility_tests.rs
deleted file mode 100644
index f961be5caf..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/visibility_tests.rs
+++ /dev/null
@@ -1,135 +0,0 @@
-use super::*;
-
-#[test]
-fn framed_elements_lead_and_keep_their_relative_order() {
- let elements = vec![
- json!({"element_index": 0, "role": "menu"}),
- json!({"element_index": 1, "role": "push button", "frame": {"x": 1, "y": 1, "w": 2, "h": 2}}),
- json!({"element_index": 2, "role": "menu"}),
- json!({"element_index": 3, "role": "paragraph", "frame": {"x": 1, "y": 9, "w": 2, "h": 2}}),
- ];
- let ordered: Vec<u64> = framed_elements_first(elements)
- .iter()
- .map(|e| e["element_index"].as_u64().unwrap())
- .collect();
- assert_eq!(ordered, vec![1, 3, 0, 2]);
-}
-
-#[test]
-fn a_frameless_popup_returns_only_the_menu_entries_drawn_inside_it() {
- let elements = vec![
- json!({"element_index": 1, "role": "push button", "frame": {"x": 5, "y": 5, "w": 20, "h": 20}}),
- json!({"element_index": 2, "role": "menu item", "label": "Paragraph...", "frame": {"x": 2, "y": 30, "w": 180, "h": 22}}),
- json!({"element_index": 3, "role": "menu item", "label": "elsewhere", "frame": {"x": 400, "y": 30, "w": 180, "h": 22}}),
- json!({"element_index": 4, "role": "menu", "label": "Format"}),
- ];
- let kept: Vec<u64> = popup_menu_elements(elements.clone(), (0, 0), 200, 400)
- .iter()
- .map(|e| e["element_index"].as_u64().unwrap())
- .collect();
- assert_eq!(kept, vec![2]);
- assert_eq!(
- popup_menu_elements(elements[..1].to_vec(), (0, 0), 200, 400).len(),
- 1
- );
-}
-
-#[test]
-fn a_combo_popup_returns_the_rows_of_the_list_that_fills_it() {
- // Qt: the combo list lives under the combo box inside the dialog; the
- // main window's menubar entries are elsewhere on the screen.
- let elements = vec![
- json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": -600, "y": -300, "w": 50, "h": 20}}),
- json!({"element_index": 7, "role": "list", "frame": {"x": 1, "y": 1, "w": 198, "h": 118}}),
- json!({"element_index": 8, "role": "list item", "label": "Desktop", "parent_index": 7, "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),
- json!({"element_index": 9, "role": "list item", "label": "user", "parent_index": 7, "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),
- json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),
- ];
- let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)
- .iter()
- .map(|e| e["element_index"].as_u64().unwrap())
- .collect();
- assert_eq!(kept, vec![8, 9]);
-}
-
-#[test]
-fn a_list_popup_without_an_indexed_container_keeps_its_item_rows_only() {
- let elements = vec![
- json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": 1, "y": 1, "w": 50, "h": 20}}),
- json!({"element_index": 8, "role": "list item", "label": "Desktop", "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),
- json!({"element_index": 9, "role": "tree item", "label": "user", "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),
- json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),
- ];
- let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)
- .iter()
- .map(|e| e["element_index"].as_u64().unwrap())
- .collect();
- assert_eq!(kept, vec![8, 9]);
-}
-
-#[test]
-fn menu_roles_take_the_real_press() {
- for role in ["menu", "Menu Item", "check menu item", "radio menu item"] {
- assert!(element_is_menu_role(role), "{role}");
- }
- for role in ["menu bar", "push button", "text"] {
- assert!(!element_is_menu_role(role), "{role}");
- }
-}
-
-#[test]
-fn overlays_name_the_follow_up_call_per_dialog_and_popup() {
- let overlays = WindowOverlays {
- dialogs: vec![json!({
- "window_id": 71, "title": "Position and Size", "transient_for": 3, "modal": true,
- "bounds": {"x": 100, "y": 100, "width": 400, "height": 300}
- })],
- popups: vec![json!({
- "window_id": 72, "title": "", "pid": 5,
- "bounds": {"x": 10, "y": 30, "width": 200, "height": 400}
- })],
- covers_window: true,
- window_rect: Some((0, 0, 800, 600)),
- };
- let note = overlays.follow_up(5).unwrap();
- assert!(
- note.contains("dialog \"Position and Size\" (window_id 71, transient of window 3, modal)"),
- "{note}"
- );
- assert!(
- note.contains("get_window_state(pid=5, window_id=71)"),
- "{note}"
- );
- assert!(
- note.contains("popup (window_id 72, bounds x=10 y=30 200x400)"),
- "{note}"
- );
- assert!(rects_intersect((0, 0, 800, 600), (100, 100, 400, 300)));
- assert!(!rects_intersect((0, 0, 800, 600), (800, 0, 10, 10)));
-}
-
-#[test]
-fn a_cross_process_dialog_names_its_real_owning_pid_in_the_follow_up() {
- // GIMP's "Export Image as JPEG" dialog spawned as a separate process
- // (owning_pid 999) but is transient-for the target pid (5)'s window.
- let overlays = WindowOverlays {
- dialogs: vec![json!({
- "window_id": 90, "title": "Export Image as JPEG", "transient_for": 10,
- "modal": false,
- "bounds": {"x": 0, "y": 0, "width": 300, "height": 200},
- "owning_pid": 999,
- })],
- popups: vec![],
- covers_window: false,
- window_rect: Some((0, 0, 800, 600)),
- };
- let note = overlays.follow_up(5).unwrap();
- assert!(
- note.contains("owned by a different process pid 999, not pid 5"),
- "{note}"
- );
- assert!(
- note.contains("get_window_state(pid=5, window_id=90)"),
- "{note}"
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/window_capture_dimension_tests.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/window_capture_dimension_tests.rs
deleted file mode 100644
index f9ad1be0ca..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_/window_capture_dimension_tests.rs
+++ /dev/null
@@ -1,13 +0,0 @@
-use super::{GetWindowStateTool, ToolState};
-use cua_driver_core::tool::Tool;
-
-#[test]
-fn schema_exposes_zero_as_native_resolution() {
- let tool = GetWindowStateTool {
- state: ToolState::new(),
- };
- assert_eq!(
- tool.def().input_schema["properties"]["max_image_dimension"]["minimum"],
- 0
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/invoke_menu.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/invoke_menu.rs
new file mode 100644
index 0000000000..1717664225
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/invoke_menu.rs
@@ -0,0 +1,205 @@
+use async_trait::async_trait;
+use cua_driver_contract::InvokeMenuInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+use super::impl_::attach_focus_guard;
+
+// ── invoke_menu (Linux) ──────────────────────────────────────────────────────
+
+pub struct InvokeMenuTool;
+
+static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
+ if path.is_empty() || path.len() > 16 {
+ return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ }
+ path.into_iter()
+ .enumerate()
+ .map(|(index, segment)| {
+ let segment = segment.trim();
+ if segment.is_empty() {
+ Err(format!("invoke_menu: path segment {index} is empty"))
+ } else {
+ Ok(segment.to_owned())
+ }
+ })
+ .collect()
+}
+
+fn menu_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": { "code": "menu_path_unavailable", "message": message }
+ }))
+}
+
+#[async_trait]
+impl Tool for InvokeMenuTool {
+ fn def(&self) -> &ToolDef {
+ INVOKE_MENU_DEF.get_or_init(|| {
+ let contract =
+ cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
+ EvidenceKind, RequestedDelivery,
+ };
+
+ // Menu paths are resolved and fired through AT-SPI, which needs no
+ // window activation on X11: the default is background (focus-free,
+ // under the focus guard). `delivery_mode:"foreground"` is the explicit
+ // escalation that activates the window first, as every other tool.
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ let mut args = args;
+ if let Some(object) = args.as_object_mut() {
+ // Not part of the closed InvokeMenuInput contract; consumed above.
+ object.remove("delivery_mode");
+ }
+ let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let path = match normalized_menu_path(input.path) {
+ Ok(path) => path,
+ Err(error) => return menu_refusal(error),
+ };
+ let pid = input.pid;
+ let window_id = input.window_id;
+ if !crate::wayland::list_windows_dispatch(Some(pid))
+ .iter()
+ .any(|window| window.xid == window_id && window.pid == Some(pid))
+ {
+ return menu_refusal("invoke_menu: window_id does not belong to pid".into());
+ }
+
+ if !crate::wayland::is_wayland() && !delivery.is_foreground() {
+ let outcome = tokio::task::spawn_blocking(move || {
+ crate::input::focus_guard::guarded(Some(pid), || {
+ crate::atspi::native::invoke_menu_path_in(pid, window_id, &path)
+ })
+ })
+ .await;
+ return match outcome {
+ Ok(Ok(((), guard))) => attach_focus_guard(
+ ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=background, focus untouched); verify the command's semantic effect from fresh state.",
+ )
+ .with_structured(json!({
+ "path": "ax",
+ "verified": false,
+ "effect": "unverifiable",
+ "delivery_mode": "background",
+ }))
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::LinuxAtSpiAction,
+ RequestedDelivery::Background,
+ )
+ .actual_delivery(ActualDelivery::Background)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"
+ .into(),
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ guard.as_ref(),
+ ),
+ Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),
+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
+ };
+ }
+
+ let activation = if crate::wayland::is_wayland() {
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::activate_window_for_input_target(window_id, Some(pid))
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => None,
+ Ok(Err(error)) => {
+ return menu_refusal(format!(
+ "invoke_menu: target window could not be activated: {error}"
+ ))
+ }
+ Err(error) => {
+ return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
+ }
+ }
+ } else {
+ match tokio::task::spawn_blocking(move || {
+ crate::input::x11_activate_window_persistent(window_id)
+ })
+ .await
+ {
+ Ok(Ok(prior)) => Some(prior),
+ Ok(Err(error)) => {
+ return menu_refusal(format!(
+ "invoke_menu: target window could not be activated: {error}"
+ ))
+ }
+ Err(error) => {
+ return menu_refusal(format!("invoke_menu: activation task failed: {error}"))
+ }
+ }
+ };
+
+ let outcome = tokio::task::spawn_blocking(move || {
+ let result = crate::atspi::native::invoke_menu_path_in(pid, window_id, &path);
+ if let Some(Some(prior_window)) = activation {
+ let _ = crate::input::x11_activate_window_persistent(prior_window);
+ }
+ result
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(())) => ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=foreground); verify the command's semantic effect from fresh state.",
+ )
+ .with_structured(json!({
+ "path": "ax",
+ "verified": false,
+ "effect": "unverifiable",
+ "delivery_mode": "foreground",
+ }))
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::LinuxAtSpiAction,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Foreground)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"
+ .into(),
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),
+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/kill_app.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/kill_app.rs
new file mode 100644
index 0000000000..d06ed5d16b
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/kill_app.rs
@@ -0,0 +1,186 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── kill_app ──────────────────────────────────────────────────────────────────
+
+pub struct KillAppTool;
+static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for KillAppTool {
+ fn def(&self) -> &ToolDef {
+ KILL_DEF.get_or_init(|| ToolDef {
+ name: "kill_app".into(),
+ description: "Force-terminate a process by pid (kill -9 equivalent on Linux). \
+ Use as escalation when the cooperative close path failed to make the process \
+ exit. Unsaved state is lost — prefer the cooperative path first."
+ .into(),
+ input_schema: json!({"type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"PID of the process to terminate."}
+ },"additionalProperties":false}),
+ read_only: false,
+ destructive: true,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn protected_resource_scope(
+ &self,
+ adapter_id: &str,
+ args: &Value,
+ ) -> Result<Option<Value>, String> {
+ if adapter_id != "process_control" {
+ return Ok(None);
+ }
+ use cua_driver_core::browser::platform::BrowserPlatform;
+ let pid = args
+ .get("pid")
+ .and_then(Value::as_i64)
+ .filter(|pid| *pid > 0)
+ .ok_or_else(|| "kill_app requires a positive integer pid".to_owned())?;
+ let fingerprint = crate::browser_platform::LinuxBrowserPlatform::default()
+ .process_fingerprint(pid)
+ .await
+ .map_err(|error| error.message)?;
+ Ok(Some(json!({
+ "kind": "process_instance",
+ "fingerprint": fingerprint,
+ })))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Some(expected) = args.get("_protected_process_fingerprint") {
+ let current = match self
+ .protected_resource_scope("process_control", &args)
+ .await
+ {
+ Ok(Some(scope)) => scope["fingerprint"].clone(),
+ Ok(None) => Value::Null,
+ Err(message) => return kill_app_stale_process_refusal(message),
+ };
+ if current != *expected {
+ return kill_app_stale_process_refusal(
+ "the process identity changed at the termination boundary".to_owned(),
+ );
+ }
+ }
+ let pid_i = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(p) if p > 0 && p <= i32::MAX as i64 => p as i32,
+ Some(_) => {
+ return ToolResult::error("kill_app: `pid` must be a positive integer".to_string())
+ }
+ None => {
+ return ToolResult::error(
+ "kill_app: missing required integer field `pid`".to_string(),
+ )
+ }
+ };
+ // Read the instance before signaling so the confirmation below can
+ // tell its exit from a later process that reused the pid.
+ let before = match crate::proc_fs::read_process_stat(pid_i as u32) {
+ Ok(Some(before)) => before,
+ Ok(None) => {
+ return kill_app_failure(
+ pid_i,
+ "process_not_found",
+ format!("kill_app: pid {pid_i} does not exist; no signal was sent."),
+ )
+ }
+ Err(error) => {
+ return kill_app_failure(
+ pid_i,
+ "process_identity_unavailable",
+ format!(
+ "kill_app: cannot read /proc/{pid_i}/stat to verify termination: \
+ {error}; no signal was sent."
+ ),
+ )
+ }
+ };
+ // SAFETY: libc::kill is a thin syscall wrapper, no thread-safety concerns.
+ let rc = unsafe { libc::kill(pid_i, libc::SIGKILL) };
+ if rc != 0 {
+ let err = std::io::Error::last_os_error();
+ return kill_app_failure(
+ pid_i,
+ "signal_failed",
+ format!(
+ "kill_app: kill(pid={pid_i}, SIGKILL) failed: {err}. \
+ The process may not exist, or the daemon lacks permission to signal it."
+ ),
+ );
+ }
+ // An accepted signal is not an exit: a sandbox (gVisor) can accept
+ // SIGKILL for a process that keeps running. Report success only once
+ // the same instance is gone.
+ let deadline = tokio::time::Instant::now() + KILL_CONFIRM_TIMEOUT;
+ loop {
+ let now = crate::proc_fs::read_process_stat(pid_i as u32);
+ if let Some(observation) = now
+ .as_ref()
+ .ok()
+ .and_then(|now| crate::proc_fs::process_exit_observation(before, *now))
+ {
+ return ToolResult::text(format!(
+ "✅ Terminated pid {pid_i} with SIGKILL (confirmed: {observation})."
+ ))
+ .with_structured(json!({
+ "status": "terminated",
+ "effect": "confirmed",
+ "pid": pid_i,
+ "terminated": true,
+ "observation": observation,
+ }));
+ }
+ if tokio::time::Instant::now() >= deadline {
+ let detail = match now {
+ Err(error) => format!("/proc/{pid_i}/stat became unreadable: {error}"),
+ Ok(_) => format!(
+ "the same process was still alive {} ms later",
+ KILL_CONFIRM_TIMEOUT.as_millis()
+ ),
+ };
+ let mut result = kill_app_failure(
+ pid_i,
+ "termination_unconfirmed",
+ format!("kill_app: SIGKILL was accepted for pid {pid_i}, but {detail}."),
+ );
+ if let Some(structured) = result.structured_content.as_mut() {
+ structured["effect"] = json!("suspected_noop");
+ }
+ return result;
+ }
+ tokio::time::sleep(KILL_CONFIRM_INTERVAL).await;
+ }
+ }
+}
+
+/// How long `kill_app` waits for the signaled process to exit, and how often
+/// it re-reads `/proc/<pid>/stat` meanwhile.
+const KILL_CONFIRM_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
+const KILL_CONFIRM_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
+
+fn kill_app_failure(pid: i32, code: &str, message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "failed",
+ "pid": pid,
+ "terminated": false,
+ "code": code,
+ "message": message,
+ }))
+}
+
+fn kill_app_stale_process_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": {
+ "code": "protected_resource_scope_stale",
+ "message": message,
+ }
+ }))
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/launch_app.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/launch_app.rs
new file mode 100644
index 0000000000..2ca7848d1c
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/launch_app.rs
@@ -0,0 +1,733 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+use super::impl_::{chromium_family_program, exec_basename, window_record_json};
+
+// ── launch_app ───────────────────────────────────────────────────────────────
+
+pub struct LaunchAppTool;
+static LAUNCH_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+/// Spawn a launcher command line (an executable plus arguments, e.g. an XDG
+/// `Exec=` value with field codes stripped) in the background and return the
+/// child pid.
+fn spawn_launch_command(cmd: &str, additional_arguments: &[String]) -> std::io::Result<u32> {
+ let mut parts = cmd.split_whitespace();
+ let prog = parts.next().unwrap_or(cmd);
+ let mut rest: Vec<String> = parts.map(str::to_owned).collect();
+ rest.extend(additional_arguments.iter().cloned());
+ let mut launch = std::process::Command::new(prog);
+ launch
+ .args(&rest)
+ // The child must not inherit the driver's stdio: in private-worker mode
+ // stdout is the JSON-RPC stream to the SDK, and a chatty app (Chromium's
+ // zygote logs, GTK warnings) writing there corrupts a response and shuts
+ // the worker down.
+ .stdin(std::process::Stdio::null())
+ .stdout(std::process::Stdio::null())
+ .stderr(std::process::Stdio::null())
+ // Enable accessibility for this child without toggling GNOME's global
+ // ScreenReaderEnabled setting (which can launch Orca). Native
+ // toolkits ignore these when they do not need them.
+ .env("ACCESSIBILITY_ENABLED", "1")
+ .env("NO_AT_BRIDGE", "0");
+ if chromium_family_program(prog)
+ && !rest
+ .iter()
+ .any(|arg| arg == "--force-renderer-accessibility")
+ {
+ launch.arg("--force-renderer-accessibility");
+ }
+ let child = launch.spawn()?;
+ let pid = child.id();
+ reap_in_background(child);
+ Ok(pid)
+}
+
+/// Collect a launched child's exit status in the background.
+///
+/// `std::process::Child` does not reap on drop and the driver never waits on
+/// an app it launches, so every launched app used to linger as a zombie for
+/// the daemon's lifetime: it held a pid slot, and — because the pid stays
+/// present — made "does this pid exist" liveness checks report a terminated
+/// app as still running.
+///
+/// Reaping is scoped to one thread per child rather than setting
+/// `SIGCHLD` to `SIG_IGN`, which is process-global and would break every
+/// caller that needs an exit status, including the `xdg-open` check below
+/// and any `Command::status()`/`output()` elsewhere in the daemon. The
+/// thread blocks in `wait` until the app exits, so it costs nothing while
+/// the app runs and never delays the launch itself.
+fn reap_in_background(mut child: std::process::Child) {
+ let pid = child.id();
+ let reaper = std::thread::Builder::new()
+ .name(format!("cua-reap-{pid}"))
+ .stack_size(64 * 1024)
+ .spawn(move || {
+ if let Err(e) = child.wait() {
+ tracing::debug!(pid, "launched app could not be reaped: {e}");
+ }
+ });
+ if let Err(e) = reaper {
+ tracing::debug!(pid, "could not start reaper thread for launched app: {e}");
+ }
+}
+
+/// Match a launch_app `name` that failed direct exec against installed XDG
+/// .desktop applications: exact display name, desktop-file id, or `Exec=`
+/// basename first, then a display-name substring when it is unambiguous.
+fn match_installed_app<'a>(
+ apps: &'a [crate::installed_apps::InstalledApp],
+ query: &str,
+) -> Option<&'a crate::installed_apps::InstalledApp> {
+ let q = query.to_ascii_lowercase();
+ if q.is_empty() {
+ return None;
+ }
+ apps.iter()
+ .find(|a| {
+ a.name.to_ascii_lowercase() == q
+ || a.bundle_id.to_ascii_lowercase() == q
+ || exec_basename(&a.launch_path) == q
+ })
+ .or_else(|| {
+ let mut matches = apps
+ .iter()
+ .filter(|a| a.name.to_ascii_lowercase().contains(&q));
+ match (matches.next(), matches.next()) {
+ (Some(only), None) => Some(only),
+ _ => None,
+ }
+ })
+}
+
+/// Watch a just-spawned `xdg-open` long enough to catch a fast failure.
+/// xdg-open's generic fallback can `exec` the target app and stay alive for
+/// its whole lifetime, so a child still running after the grace period counts
+/// as success; a quick non-zero exit (2 = file not found, 3 = no handler
+/// tool, 4 = action failed) is the only reliable failure signal.
+fn xdg_open_failure(mut child: std::process::Child) -> Option<String> {
+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(1500);
+ loop {
+ // A branch that observes an exit status has already reaped the child;
+ // one that gives up on it still owes that, so hand it to the reaper
+ // instead of dropping it and leaving a zombie behind.
+ match child.try_wait() {
+ Ok(Some(status)) if status.success() => return None,
+ Ok(Some(status)) => return Some(format!("xdg-open failed ({status})")),
+ Ok(None) if std::time::Instant::now() >= deadline => {
+ reap_in_background(child);
+ return None;
+ }
+ Ok(None) => std::thread::sleep(std::time::Duration::from_millis(50)),
+ Err(e) => {
+ reap_in_background(child);
+ return Some(format!("could not observe xdg-open: {e}"));
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod launch_app_tests {
+ use super::*;
+ use crate::installed_apps::InstalledApp;
+
+ fn app(name: &str, bundle_id: &str, launch_path: &str) -> InstalledApp {
+ InstalledApp {
+ name: name.to_owned(),
+ bundle_id: bundle_id.to_owned(),
+ launch_path: launch_path.to_owned(),
+ last_used: None,
+ startup_wm_class: None,
+ }
+ }
+
+ fn fixture() -> Vec<InstalledApp> {
+ vec![
+ app("Galculator", "galculator", "galculator"),
+ app(
+ "Google Chrome",
+ "google-chrome",
+ "/usr/bin/google-chrome-stable",
+ ),
+ app("File Manager", "thunar", "thunar"),
+ app(
+ "File Manager Settings",
+ "thunar-settings",
+ "thunar-settings",
+ ),
+ ]
+ }
+
+ /// Process state letter from `/proc/<pid>/stat`, or `None` once the entry
+ /// is gone. `comm` may itself contain spaces and parentheses, so the state
+ /// is read after the final `)` rather than by splitting from the left.
+ fn proc_state(pid: u32) -> Option<char> {
+ let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
+ let after_comm = stat.rsplit_once(')')?.1;
+ after_comm.split_whitespace().next()?.chars().next()
+ }
+
+ #[test]
+ fn launched_children_are_reaped_instead_of_lingering_as_zombies() {
+ // A launched app that exits must not stay in the process table. The
+ // driver never waits on what it launches, so without an explicit
+ // reaper every launch leaked a pid slot for the daemon's lifetime and
+ // left "does this pid exist" liveness checks reporting a terminated
+ // app as still running.
+ // `/bin/sh` rather than a richer coreutils binary: it is the one
+ // executable POSIX and the Nix build sandbox both guarantee, and the
+ // sandbox has no `/bin/true`.
+ let pid = spawn_launch_command("/bin/sh -c exit", &[]).expect("/bin/sh should spawn");
+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
+ loop {
+ match proc_state(pid) {
+ None => break, // reaped, entry gone
+ Some(state) if state != 'Z' => {} // still running or exiting
+ Some(_) => {
+ assert!(
+ std::time::Instant::now() < deadline,
+ "pid {pid} was still a zombie after the reaper deadline"
+ );
+ }
+ }
+ assert!(
+ std::time::Instant::now() < deadline,
+ "pid {pid} was never reaped"
+ );
+ std::thread::sleep(std::time::Duration::from_millis(25));
+ }
+ }
+
+ #[test]
+ fn matches_exact_display_name_case_insensitively() {
+ let apps = fixture();
+ let hit = match_installed_app(&apps, "google chrome").expect("should match");
+ assert_eq!(hit.bundle_id, "google-chrome");
+ }
+
+ #[test]
+ fn matches_desktop_file_id_and_exec_basename() {
+ let apps = fixture();
+ assert_eq!(
+ match_installed_app(&apps, "galculator").unwrap().name,
+ "Galculator"
+ );
+ assert_eq!(
+ match_installed_app(&apps, "google-chrome-stable")
+ .unwrap()
+ .name,
+ "Google Chrome"
+ );
+ }
+
+ #[test]
+ fn matches_unambiguous_display_name_substring() {
+ let apps = fixture();
+ assert_eq!(
+ match_installed_app(&apps, "chrome").unwrap().name,
+ "Google Chrome"
+ );
+ }
+
+ #[test]
+ fn refuses_ambiguous_substring_and_unknown_names() {
+ let apps = fixture();
+ // "file manager" is a substring of two entries — refuse to guess.
+ // ("File Manager" itself still resolves via the exact-name rung.)
+ assert!(match_installed_app(&apps, "file man").is_none());
+ assert_eq!(
+ match_installed_app(&apps, "file manager")
+ .unwrap()
+ .bundle_id,
+ "thunar"
+ );
+ assert!(match_installed_app(&apps, "gnome-calculator").is_none());
+ assert!(match_installed_app(&apps, "").is_none());
+ }
+}
+
+#[async_trait]
+impl Tool for LaunchAppTool {
+ fn def(&self) -> &ToolDef {
+ LAUNCH_DEF.get_or_init(|| ToolDef {
+ name: "launch_app".into(),
+ description: "Launch a Linux app in the background. Provide launch_path (preferred — \
+ round-trip the value from list_apps), name (tried as a direct command, then \
+ matched against installed .desktop applications, then handed to xdg-open if it \
+ is a URL or existing file path), bundle_id (ignored on Linux), or urls (list of \
+ URLs to open). Resolution precedence: launch_path > name > bundle_id. Errors \
+ when the name resolves to nothing launchable.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "launch_path":{"type":"string","description":"Round-trip the `launch_path` returned by `list_apps` — the Exec= command from the .desktop file with XDG field codes already stripped. Highest precedence on Linux; spawned directly via the system shell."},
+ "name":{"type":"string","description":"App name or command to launch. Tried as a direct command first, then matched against installed .desktop applications (exact display name, desktop-file id, or Exec basename; else an unambiguous display-name substring)."},
+ "bundle_id":{"type":"string","description":"Ignored on Linux (macOS/Windows concept)."},
+ "urls":{"type":"array","items":{"type":"string"},"description":"URLs to open via xdg-open."},
+ "additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: false, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let launch_path_opt = args.opt_str("launch_path");
+ let name_opt = args.opt_str("name");
+ let urls: Vec<String> = args.str_array("urls");
+ let additional_arguments: Vec<String> = args.str_array("additional_arguments");
+ if args.get("cdp_debugging_port").is_some() {
+ return ToolResult::error(
+ "cdp_debugging_port moved to browser_prepare so DevTools is never enabled on an unproven user profile",
+ );
+ }
+ if launch_path_opt
+ .as_deref()
+ .into_iter()
+ .chain(name_opt.as_deref())
+ .chain(additional_arguments.iter().map(String::as_str))
+ .any(cua_driver_core::launch_guard::contains_remote_debugging_flag)
+ {
+ return ToolResult::error(
+ cua_driver_core::launch_guard::REMOTE_DEBUGGING_LAUNCH_REFUSAL,
+ );
+ }
+
+ if launch_path_opt.is_none() && name_opt.is_none() && urls.is_empty() {
+ return ToolResult::error("Provide at least one of: launch_path, name, or urls.");
+ }
+
+ // Snapshot the top-levels before spawning so a window created by an
+ // already-running service (D-Bus activation) can be told apart.
+ let windows_before: std::collections::HashSet<u64> =
+ tokio::task::spawn_blocking(|| crate::wayland::list_windows_dispatch(None))
+ .await
+ .unwrap_or_default()
+ .into_iter()
+ .map(|w| w.xid)
+ .collect();
+ let result = tokio::task::spawn_blocking(
+ move || -> anyhow::Result<(String, Option<u32>, String)> {
+ // Open URLs via xdg-open.
+ if !urls.is_empty() {
+ let mut children = Vec::new();
+ for url in &urls {
+ children.push((
+ url.clone(),
+ std::process::Command::new("xdg-open").arg(url).spawn()?,
+ ));
+ }
+ for (url, child) in children {
+ if let Some(reason) = xdg_open_failure(child) {
+ anyhow::bail!("could not open '{url}': {reason}");
+ }
+ }
+ return Ok((
+ format!("Opened {} URL(s) via xdg-open.", urls.len()),
+ None,
+ "xdg-open".to_owned(),
+ ));
+ }
+ // launch_path > name. Both go through the same direct-exec path
+ // (so XDG `Exec=` commands round-trip), but launch_path is the
+ // canonical form preferred by list_apps callers.
+ let command = launch_path_opt.as_deref().or(name_opt.as_deref());
+ if let Some(cmd) = command {
+ match spawn_launch_command(cmd, &additional_arguments) {
+ Ok(pid) => {
+ return Ok((
+ format!("✅ Launched {cmd} (pid {pid}) in background."),
+ Some(pid),
+ cmd.to_owned(),
+ ));
+ }
+ Err(_) => {
+ // Not an executable on PATH. Resolve the name against
+ // installed XDG .desktop applications — the same
+ // source list_apps reads — and run the match's Exec=.
+ let installed = crate::installed_apps::list_installed_apps();
+ if let Some(app) = match_installed_app(&installed, cmd) {
+ let pid =
+ spawn_launch_command(&app.launch_path, &additional_arguments)
+ .map_err(|e| {
+ anyhow::anyhow!(
+ "'{cmd}' matched installed app '{}' but its launcher \
+ `{}` failed to start: {e}",
+ app.name,
+ app.launch_path
+ )
+ })?;
+ return Ok((
+ format!(
+ "✅ Launched {} (`{}`, pid {pid}) in background — \
+ resolved '{cmd}' via its desktop entry.",
+ app.name, app.launch_path
+ ),
+ Some(pid),
+ format!("{}\u{0}{cmd}\u{0}{}", app.name, app.launch_path),
+ ));
+ }
+ // xdg-open handles URLs and file paths, not app
+ // names — only fall through for something it can
+ // plausibly open, and surface its fast non-zero
+ // exit instead of reporting a launch that never
+ // happened.
+ if !cmd.contains("://") && !std::path::Path::new(cmd).exists() {
+ anyhow::bail!(
+ "'{cmd}' is not an executable on PATH and matches no \
+ installed .desktop application. Call list_apps and \
+ round-trip its launch_path."
+ );
+ }
+ let child = std::process::Command::new("xdg-open").arg(cmd).spawn()?;
+ if let Some(reason) = xdg_open_failure(child) {
+ anyhow::bail!("could not open '{cmd}': {reason}");
+ }
+ // xdg-open may spawn a helper and exit, so do not
+ // claim its pid is the app pid.
+ return Ok((
+ format!("Opened '{cmd}' via xdg-open."),
+ None,
+ cmd.to_owned(),
+ ));
+ }
+ }
+ }
+ unreachable!()
+ },
+ )
+ .await;
+
+ match result {
+ Ok(Ok((message, pid_opt, name))) => {
+ if let Some(launcher_pid) = pid_opt {
+ // `name` may carry NUL-separated match keys (display name,
+ // query, Exec=) for a desktop-entry launch.
+ let query = name.clone();
+ let name = name.split('\u{0}').next().unwrap_or("").to_owned();
+ let resolved = tokio::task::spawn_blocking(move || {
+ resolve_launched_windows(launcher_pid, &query, &windows_before)
+ })
+ .await
+ .unwrap_or_else(|_| LaunchedWindows::default());
+ let windows: Vec<Value> =
+ resolved.windows.iter().map(window_record_json).collect();
+ let mut text = message;
+ let mut structured = json!({
+ "pid": resolved.pid,
+ "bundle_id": Value::Null,
+ "name": name,
+ "running": resolved.pid.is_some(),
+ "active": false,
+ "windows": windows,
+ "launcher_pid": launcher_pid,
+ });
+ if resolved.handed_off {
+ match resolved.pid {
+ Some(pid) => {
+ structured["handoff"] = json!("dbus_activation");
+ text.push_str(&format!(
+ " The launcher handed the request to another process \
+ (a wrapper or D-Bus activation); the window belongs to pid {pid}."
+ ));
+ }
+ None => {
+ // The launcher exited (D-Bus activation or a failed
+ // start) and the hand-off budget in
+ // `resolve_launched_windows` passed with no new
+ // window ever appearing. This is not a success: we
+ // have no pid, no window, and no way to know the app
+ // actually started. Report a typed refusal instead
+ // of a `pid: null` shape that reads as success.
+ return launch_handoff_timeout_result(
+ &name,
+ launcher_pid,
+ LAUNCH_HANDOFF_BUDGET.as_secs(),
+ );
+ }
+ }
+ }
+ ToolResult::text(text).with_structured(structured)
+ } else {
+ ToolResult::text(message).with_structured(json!({
+ "pid": Value::Null,
+ "bundle_id": Value::Null,
+ "name": name,
+ "running": Value::Null,
+ "active": false,
+ "windows": [],
+ }))
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("Failed to launch: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+/// The `launch_app` D-Bus-activation hand-off timed out: the launcher exited
+/// (handing the request to a running service) but no new window of the app
+/// ever appeared within the deadline. This has no pid and no confirmed
+/// window, so it is a refusal (`code: "launch_handoff_timeout"`), never a
+/// `pid: null` shape that a caller could mistake for success.
+fn launch_handoff_timeout_result(name: &str, launcher_pid: u32, waited_secs: u64) -> ToolResult {
+ ToolResult::error(format!(
+ "'{name}' was handed off for D-Bus activation but no new window appeared within \
+ {waited_secs}s; the launch could not be confirmed. Call list_windows to check \
+ whether it started anyway, or retry."
+ ))
+ .with_structured(json!({
+ "code": "launch_handoff_timeout",
+ "effect": "refused",
+ "handoff": "dbus_activation",
+ "name": name,
+ "launcher_pid": launcher_pid,
+ "pid": Value::Null,
+ "running": Value::Null,
+ "waited_secs": waited_secs,
+ }))
+}
+
+/// What `launch_app` could attribute to the launch after the spawn.
+#[derive(Default)]
+struct LaunchedWindows {
+ /// The process that owns the app's window: the launcher itself, or the
+ /// wrapper target / running service it handed off to.
+ pid: Option<u32>,
+ windows: Vec<crate::x11::WindowInfo>,
+ /// The window belongs to another process than the launcher, or the
+ /// launcher exited without one.
+ handed_off: bool,
+}
+
+/// Whether the launched process is gone or a zombie (already reaped or being
+/// reaped by the launch thread).
+fn process_exited(pid: u32) -> bool {
+ match std::fs::read_to_string(format!("/proc/{pid}/stat")) {
+ Ok(stat) => stat
+ .rsplit_once(')')
+ .and_then(|(_, rest)| rest.split_whitespace().next())
+ .is_some_and(|state| state == "Z" || state == "X"),
+ Err(_) => true,
+ }
+}
+
+/// Case-insensitive identity match between a launch query
+/// (`nautilus`, `org.gnome.Nautilus`, `gnome-control-center`, a display
+/// name) and a window's WM_CLASS / app id.
+fn window_matches_launch(window: &crate::x11::WindowInfo, query: &str) -> bool {
+ query
+ .split('\u{0}')
+ .filter(|key| !key.trim().is_empty())
+ .any(|key| window_matches_launch_key(window, key))
+}
+
+fn window_matches_launch_key(window: &crate::x11::WindowInfo, query: &str) -> bool {
+ let class = window.app_name.to_ascii_lowercase();
+ if class.is_empty() {
+ return false;
+ }
+ let query = query.to_ascii_lowercase();
+ let stem = query
+ .rsplit('/')
+ .next()
+ .unwrap_or(&query)
+ .trim_end_matches(".desktop")
+ .to_owned();
+ let last = stem.rsplit('.').next().unwrap_or(&stem).to_owned();
+ let class_last = class.rsplit('.').next().unwrap_or(&class).to_owned();
+ class == stem
+ || class_last == last
+ || class.contains(&last)
+ || last.contains(&class_last)
+ || stem
+ .split_whitespace()
+ .next()
+ .is_some_and(|word| class.contains(word))
+}
+
+/// How long a launch may take to show a window while the launcher process is
+/// still alive (the app is starting; LibreOffice on a small VM needs well over
+/// ten seconds), and once it has exited (a wrapper or D-Bus activation has
+/// taken over, so the window is either imminent or never coming).
+const LAUNCH_STARTING_BUDGET: std::time::Duration = std::time::Duration::from_secs(20);
+const LAUNCH_HANDOFF_BUDGET: std::time::Duration = std::time::Duration::from_secs(8);
+
+/// Resolve the window(s) of a launch. Prefers the launcher's own window;
+/// otherwise adopts a newly mapped top-level whose WM_CLASS matches the
+/// launch, whoever owns it: a wrapper script that exec'd the real binary
+/// (`libreoffice` → `soffice.bin`), or the running D-Bus service GNOME apps
+/// hand their request to.
+fn resolve_launched_windows(
+ launcher_pid: u32,
+ query: &str,
+ before: &std::collections::HashSet<u64>,
+) -> LaunchedWindows {
+ let started = std::time::Instant::now();
+ loop {
+ let all = crate::wayland::list_windows_dispatch(None);
+ if let Some(launched) = settle_launch(&all, launcher_pid, query, before, |w| {
+ crate::x11::window_is_splash(w.xid)
+ }) {
+ return launched;
+ }
+ let launcher_exited = process_exited(launcher_pid);
+ let budget = if launcher_exited {
+ LAUNCH_HANDOFF_BUDGET
+ } else {
+ LAUNCH_STARTING_BUDGET
+ };
+ if started.elapsed() >= budget {
+ return LaunchedWindows {
+ pid: (!launcher_exited).then_some(launcher_pid),
+ windows: Vec::new(),
+ handed_off: launcher_exited,
+ };
+ }
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ }
+}
+
+fn settle_launch(
+ all: &[crate::x11::WindowInfo],
+ launcher_pid: u32,
+ query: &str,
+ before: &std::collections::HashSet<u64>,
+ is_splash: impl Fn(&crate::x11::WindowInfo) -> bool,
+) -> Option<LaunchedWindows> {
+ let own: Vec<_> = all
+ .iter()
+ .filter(|w| w.pid == Some(launcher_pid) && !is_splash(w))
+ .cloned()
+ .collect();
+ if !own.is_empty() {
+ return Some(LaunchedWindows {
+ pid: Some(launcher_pid),
+ windows: own,
+ handed_off: false,
+ });
+ }
+ let fresh: Vec<_> = all
+ .iter()
+ .filter(|w| !before.contains(&w.xid) && w.pid.is_some())
+ .filter(|w| window_matches_launch(w, query) && !is_splash(w))
+ .cloned()
+ .collect();
+ let pid = fresh.first()?.pid?;
+ Some(LaunchedWindows {
+ pid: Some(pid),
+ windows: fresh.into_iter().filter(|w| w.pid == Some(pid)).collect(),
+ handed_off: true,
+ })
+}
+
+#[cfg(test)]
+mod launch_resolution_tests {
+ use super::*;
+
+ fn window(class: &str) -> crate::x11::WindowInfo {
+ crate::x11::WindowInfo {
+ xid: 1,
+ pid: Some(7),
+ app_name: class.into(),
+ title: "t".into(),
+ is_on_screen: true,
+ z_index: None,
+ x: 0,
+ y: 0,
+ width: 1,
+ height: 1,
+ }
+ }
+
+ #[test]
+ fn launch_queries_match_window_classes() {
+ assert!(window_matches_launch(
+ &window("org.gnome.Nautilus"),
+ "nautilus"
+ ));
+ assert!(window_matches_launch(
+ &window("org.gnome.Nautilus"),
+ "org.gnome.Nautilus"
+ ));
+ assert!(window_matches_launch(
+ &window("Gnome-terminal"),
+ "gnome-terminal"
+ ));
+ assert!(window_matches_launch(
+ &window("Gnome-control-center"),
+ "gnome-control-center"
+ ));
+ assert!(window_matches_launch(
+ &window("Org.gnome.Nautilus"),
+ "org.gnome.Nautilus.desktop"
+ ));
+ assert!(window_matches_launch(
+ &window("Org.gnome.Nautilus"),
+ "Files\u{0}org.gnome.Nautilus\u{0}nautilus --new-window"
+ ));
+ assert!(window_matches_launch(
+ &window("Gnome-control-center"),
+ "Settings\u{0}Settings\u{0}gnome-control-center"
+ ));
+ assert!(!window_matches_launch(&window("Gedit"), "nautilus"));
+ assert!(!window_matches_launch(&window(""), "nautilus"));
+ }
+
+ fn owned(xid: u64, pid: u32, class: &str) -> crate::x11::WindowInfo {
+ crate::x11::WindowInfo {
+ xid,
+ pid: Some(pid),
+ ..window(class)
+ }
+ }
+
+ #[test]
+ fn a_launcher_splash_waits_for_the_app_window() {
+ let splash = owned(10, 377, "libreoffice");
+ let calc = owned(20, 398, "libreoffice-calc");
+ let none = std::collections::HashSet::new();
+ let is_splash = |w: &crate::x11::WindowInfo| w.xid == 10;
+
+ assert!(settle_launch(&[splash.clone()], 377, "libreoffice", &none, is_splash).is_none());
+
+ let launched =
+ settle_launch(&[splash, calc], 377, "libreoffice", &none, is_splash).unwrap();
+ assert_eq!(launched.pid, Some(398));
+ assert_eq!(
+ launched.windows.iter().map(|w| w.xid).collect::<Vec<_>>(),
+ [20]
+ );
+
+ let gimp = owned(30, 377, "gimp");
+ let launched = settle_launch(&[gimp], 377, "gimp", &none, is_splash).unwrap();
+ assert_eq!((launched.pid, launched.handed_off), (Some(377), false));
+ }
+
+ #[test]
+ fn a_missing_process_counts_as_exited() {
+ assert!(process_exited(u32::MAX - 1));
+ assert!(!process_exited(std::process::id()));
+ }
+
+ #[test]
+ fn a_handoff_timeout_is_a_refusal_not_a_null_pid_success() {
+ let result = launch_handoff_timeout_result("Nautilus", 4242, 8);
+ assert_eq!(
+ result.is_error,
+ Some(true),
+ "a hand-off timeout must be reported as an error, not success"
+ );
+ let s = result.structured_content.unwrap();
+ assert_eq!(s["code"], "launch_handoff_timeout");
+ assert_eq!(s["effect"], "refused");
+ assert_eq!(s["pid"], Value::Null);
+ assert_eq!(s["running"], Value::Null);
+ assert_eq!(s["launcher_pid"], 4242);
+ assert_eq!(s["waited_secs"], 8);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/list_apps.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/list_apps.rs
new file mode 100644
index 0000000000..e7e66490ab
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/list_apps.rs
@@ -0,0 +1,261 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+use super::impl_::{exec_basename, window_record_json};
+
+// ── list_apps ────────────────────────────────────────────────────────────────
+
+pub struct ListAppsTool;
+static LIST_APPS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ListAppsTool {
+ fn def(&self) -> &ToolDef {
+ LIST_APPS_DEF.get_or_init(|| ToolDef {
+ name: "list_apps".into(),
+ description:
+ "List Linux apps — both currently running and installed-but-not-running — \
+ with per-app state flags:\n\n\
+ - running: is a process for this app live? (pid is 0 when false)\n\
+ - active: reserved (Linux X11/Wayland focus model differs from frontmost-app); \
+ always false.\n\
+ - kind: `\"desktop\"` for XDG `.desktop` launcher entries.\n\
+ - launch_path: the launcher command from `Exec=` (field codes stripped). \
+ Pass to `launch_app(launch_path=...)`.\n\
+ - bundle_id: the XDG \"desktop file id\" — the `.desktop` file's path \
+ relative to its XDG `applications/` root with the `.desktop` suffix \
+ stripped and path separators replaced with `-` \
+ (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).\n\
+ - last_used: RFC3339 mtime of the `.desktop` file, when readable.\n\
+ - windows: the app's current top-level windows (same records as list_windows); \
+ empty for a process without one.\n\n\
+ Running apps come from `/proc`. Installed apps come from XDG Desktop Entry \
+ files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's \
+ applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are \
+ filtered. A `.desktop` file whose launcher matches a running process \
+ (by basename), or whose WM_CLASS a top-level window carries, is merged \
+ into a single entry with `running: true`.\n\n\
+ Use this for \"is X installed?\" as well as \"is X running?\". For per-window \
+ state — visibility, geometry, titles — call list_windows instead."
+ .into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ let apps = tokio::task::spawn_blocking(|| -> Vec<serde_json::Value> {
+ let procs = crate::proc_fs::list_processes();
+ let installed = crate::installed_apps::list_installed_apps();
+ let windows = crate::wayland::list_windows_dispatch(None);
+
+ // An app whose launcher is not its process (`libreoffice --calc`
+ // runs `soffice.bin`, `google-chrome` runs `chrome`, `gnome-terminal`
+ // hands off to `gnome-terminal-server`) never matches by
+ // basename. Its top-level windows still carry its WM_CLASS, so
+ // owning one is what makes it running.
+ let by_window: std::collections::HashMap<usize, u32> = installed
+ .iter()
+ .enumerate()
+ .filter_map(|(i, app)| {
+ windows
+ .iter()
+ .find(|w| w.pid.is_some() && app.owns_window_class(&w.app_name))
+ .and_then(|w| w.pid)
+ .map(|pid| (i, pid))
+ })
+ .collect();
+
+ // Match running processes to installed apps by executable
+ // basename (Exec=firefox %u → "firefox"; cmdline /usr/bin/firefox
+ // → "firefox"). Many distros register multiple .desktop entries
+ // sharing the same basename (e.g. several `firefox` profiles, a
+ // `code` and a `code-insiders` both shelling `code`), so the
+ // bucket is `Vec<usize>` — we pick a winner per running process
+ // via `disambiguate_installed_match` instead of overwriting.
+ let mut by_exe: std::collections::HashMap<String, Vec<usize>> =
+ std::collections::HashMap::new();
+ for (i, app) in installed.iter().enumerate() {
+ let basename = exec_basename(&app.launch_path);
+ if !basename.is_empty() {
+ by_exe.entry(basename).or_default().push(i);
+ }
+ }
+
+ let mut consumed: std::collections::HashSet<usize> = std::collections::HashSet::new();
+ let mut out = Vec::new();
+ for p in &procs {
+ let key_source = if !p.cmdline.is_empty() {
+ &p.cmdline
+ } else {
+ &p.name
+ };
+ let basename = exec_basename(key_source);
+ if basename.is_empty() {
+ continue;
+ }
+ let candidates = by_exe.get(&basename).map(|v| v.as_slice()).unwrap_or(&[]);
+ let merged = disambiguate_installed_match(candidates, &installed, key_source)
+ .or_else(|| {
+ by_window
+ .iter()
+ .filter(|(_, pid)| **pid == p.pid)
+ .map(|(idx, _)| *idx)
+ .min()
+ });
+ if let Some(idx) = merged {
+ consumed.insert(idx);
+ }
+ let (name, bundle_id, launch_path, kind, last_used) = match merged {
+ Some(idx) => {
+ let a = &installed[idx];
+ (
+ a.name.clone(),
+ Some(a.bundle_id.clone()),
+ Some(a.launch_path.clone()),
+ Some("desktop".to_owned()),
+ a.last_used.clone(),
+ )
+ }
+ None => (
+ if !p.name.is_empty() {
+ p.name.clone()
+ } else {
+ basename.clone()
+ },
+ None,
+ None,
+ None,
+ None,
+ ),
+ };
+ let owned: Vec<Value> = windows
+ .iter()
+ .filter(|w| w.pid == Some(p.pid))
+ .filter(|w| {
+ merged.is_none_or(|idx| installed[idx].owns_window_class(&w.app_name))
+ })
+ .map(window_record_json)
+ .collect();
+ out.push(json!({
+ "pid": p.pid,
+ "bundle_id": bundle_id,
+ "name": name,
+ "running": true,
+ "active": false,
+ "kind": kind,
+ "launch_path": launch_path,
+ "last_used": last_used,
+ "windows": owned,
+ }));
+ }
+ for (i, app) in installed.iter().enumerate() {
+ if consumed.contains(&i) {
+ continue;
+ }
+ // One process can own windows of several entries (a single
+ // `soffice.bin` showing Calc and Writer): each entry with a
+ // window is running under its own id.
+ let window_pid = by_window.get(&i).copied();
+ let owned: Vec<Value> = windows
+ .iter()
+ .filter(|w| w.pid.is_some() && w.pid == window_pid)
+ .filter(|w| app.owns_window_class(&w.app_name))
+ .map(window_record_json)
+ .collect();
+ out.push(json!({
+ "pid": window_pid.unwrap_or(0),
+ "bundle_id": app.bundle_id.clone(),
+ "name": app.name.clone(),
+ "running": window_pid.is_some(),
+ "active": false,
+ "kind": "desktop",
+ "launch_path": app.launch_path.clone(),
+ "last_used": app.last_used.clone(),
+ "windows": owned,
+ }));
+ }
+ out
+ })
+ .await
+ .unwrap_or_default();
+
+ let running_count = apps
+ .iter()
+ .filter(|a| a["running"].as_bool().unwrap_or(false))
+ .count();
+ let total = apps.len();
+ let installed_only = total - running_count;
+ let mut lines = vec![format!(
+ "✅ Found {total} app(s): {running_count} running, {installed_only} installed-not-running."
+ )];
+ for app in apps
+ .iter()
+ .filter(|a| a["running"].as_bool().unwrap_or(false))
+ {
+ let name = app["name"].as_str().unwrap_or("?");
+ let pid = app["pid"].as_u64().unwrap_or(0);
+ lines.push(format!("- {name} (pid {pid})"));
+ }
+ // Unified `apps` array + legacy `processes` alias for older callers.
+ let structured = json!({
+ "apps": apps,
+ "processes": apps.iter().filter(|a| a["running"].as_bool().unwrap_or(false))
+ .map(|a| json!({
+ "pid": a["pid"], "name": a["name"]
+ })).collect::<Vec<_>>(),
+ });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
+
+/// Pick which of several `.desktop`-derived installed apps best matches a
+/// running process whose basename collided. Precedence:
+/// 1. exact full-launch-path equality with the process's cmdline token
+/// (so `/usr/bin/firefox` beats `/snap/firefox/current/firefox`)
+/// 2. most recently modified launcher (`last_used` desc, RFC3339 string
+/// ordering is lexicographic-safe for our format)
+/// 3. first candidate (deterministic by source order)
+fn disambiguate_installed_match(
+ candidates: &[usize],
+ installed: &[crate::installed_apps::InstalledApp],
+ proc_key_source: &str,
+) -> Option<usize> {
+ if candidates.is_empty() {
+ return None;
+ }
+ if candidates.len() == 1 {
+ return Some(candidates[0]);
+ }
+
+ // Look for an exact path match against the cmdline's first token.
+ let proc_path = proc_key_source.split_whitespace().next().unwrap_or("");
+ if !proc_path.is_empty() {
+ if let Some(&idx) = candidates
+ .iter()
+ .find(|&&i| installed[i].launch_path == proc_path)
+ {
+ return Some(idx);
+ }
+ }
+
+ // Fall back to the candidate with the most recent `last_used`.
+ candidates
+ .iter()
+ .copied()
+ .max_by(|&a, &b| {
+ installed[a]
+ .last_used
+ .as_deref()
+ .unwrap_or("")
+ .cmp(installed[b].last_used.as_deref().unwrap_or(""))
+ })
+ .or_else(|| candidates.first().copied())
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/list_windows.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/list_windows.rs
new file mode 100644
index 0000000000..30d17beef9
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/list_windows.rs
@@ -0,0 +1,79 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+use super::impl_::window_record_json;
+
+// ── list_windows ─────────────────────────────────────────────────────────────
+
+pub struct ListWindowsTool;
+static LIST_WINDOWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ListWindowsTool {
+ fn def(&self) -> &ToolDef {
+ LIST_WINDOWS_DEF.get_or_init(|| ToolDef {
+ name: "list_windows".into(),
+ description: "List top-level windows. Each record includes z_index (integer or null; \
+ higher values are closer to the front; null means stacking order is unavailable \
+ and callers must not infer one). To select a frontmost candidate, take the maximum \
+ integer z_index; if every value is null, use an explicit fallback instead of \
+ relying on array order. Filter with `app` or `pid`: an unfiltered call lists \
+ every window on the host.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "pid":{"type":"integer","description":"Only list windows owned by this process ID."},
+ "app":{"type":"string","description":cua_driver_core::tool_args::LIST_WINDOWS_APP_FILTER_DESCRIPTION},
+ "on_screen_only":{"type":"boolean","description":"When true, filter to visible windows only. Default false."}
+ },"additionalProperties":false}),
+ read_only: true, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let filter_pid = args.opt_u64("pid").map(|v| v as u32);
+ let on_screen_only = args.bool_or("on_screen_only", false);
+ let mut windows =
+ tokio::task::spawn_blocking(move || crate::wayland::list_windows_dispatch(filter_pid))
+ .await
+ .unwrap_or_default();
+ // Exited applications can remain visible in AT-SPI/X11 as zombies;
+ // never return those stale targets to callers.
+ windows.retain(|window| window.pid.is_none_or(crate::proc_fs::is_process_live));
+ if on_screen_only {
+ windows.retain(|window| window.is_on_screen);
+ }
+ if let Some(filter) = args.opt_str("app") {
+ windows.retain(|window| {
+ cua_driver_core::tool_args::app_filter_matches(&filter, &window.app_name, None)
+ });
+ }
+ if crate::wayland::is_wayland() {
+ crate::wayland::remember_observed_window_origins(&windows);
+ }
+ let mut lines = vec![format!("Found {} windows:", windows.len())];
+ for w in &windows {
+ lines.push(format!(
+ " window_id={} pid={} \"{}\" {}x{}+{}+{}{}",
+ w.xid,
+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),
+ w.title,
+ w.width,
+ w.height,
+ w.x,
+ w.y,
+ if w.app_name.is_empty() {
+ String::new()
+ } else {
+ format!(" app={}", w.app_name)
+ }
+ ));
+ }
+ let structured =
+ json!({ "windows": windows.iter().map(window_record_json).collect::<Vec<_>>() });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/mod.rs
index 38b9c42d14..03360d3c05 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/mod.rs
@@ -5,10 +5,74 @@
use cua_driver_core::tool::ToolRegistry;
+#[cfg(target_os = "linux")]
+mod bring_to_front;
+#[cfg(target_os = "linux")]
+mod check_permissions;
+#[cfg(target_os = "linux")]
+mod click;
+#[cfg(target_os = "linux")]
+mod cursor_tools;
+#[cfg(target_os = "linux")]
+mod double_click;
+#[cfg(target_os = "linux")]
+mod drag;
+#[cfg(target_os = "linux")]
+mod get_accessibility_tree;
+#[cfg(target_os = "linux")]
+mod get_config;
+#[cfg(target_os = "linux")]
+mod get_cursor_position;
+#[cfg(target_os = "linux")]
+mod get_desktop_state;
+#[cfg(target_os = "linux")]
+mod get_screen_size;
+#[cfg(target_os = "linux")]
+mod get_window_state;
+#[cfg(target_os = "linux")]
+mod hotkey;
#[cfg(target_os = "linux")]
mod impl_;
#[cfg(target_os = "linux")]
+mod invoke_menu;
+#[cfg(target_os = "linux")]
+mod kill_app;
+#[cfg(target_os = "linux")]
+mod launch_app;
+#[cfg(target_os = "linux")]
+mod list_apps;
+#[cfg(target_os = "linux")]
+mod list_windows;
+#[cfg(target_os = "linux")]
+mod mouse_button_down;
+#[cfg(target_os = "linux")]
+mod mouse_button_up;
+#[cfg(target_os = "linux")]
+mod mouse_drag;
+#[cfg(target_os = "linux")]
+mod move_cursor;
+#[cfg(target_os = "linux")]
pub(crate) mod page;
+#[cfg(target_os = "linux")]
+mod parallel_mouse_drag;
+#[cfg(target_os = "linux")]
+mod press_key;
+#[cfg(target_os = "linux")]
+mod right_click;
+#[cfg(target_os = "linux")]
+mod scroll;
+#[cfg(target_os = "linux")]
+mod set_config;
+#[cfg(target_os = "linux")]
+mod set_value;
+#[cfg(target_os = "linux")]
+mod set_window_frame;
+#[cfg(target_os = "linux")]
+mod type_text;
+#[cfg(target_os = "linux")]
+mod type_text_chars;
+#[cfg(target_os = "linux")]
+mod zoom;
#[cfg(not(target_os = "linux"))]
mod stubs;
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_down.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_down.rs
new file mode 100644
index 0000000000..0266de2674
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_down.rs
@@ -0,0 +1,207 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::ArgsExt as _,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ coordinate_frame_schema, desktop_frame_requested, desktop_frame_unmapped,
+ desktop_to_window_local, foreground_budget, input_error_result, mouse_button_name,
+ mouse_hold_json, overlay_glide_to_for, overlay_snap_to_for, parse_mouse_button,
+ persistent_pointer_coordinates, press_and_record_x11_hold, resolve_cursor_key,
+ screenshot_scale, spawn_blocking_bounded, window_local_to_screen, MouseHoldState, ToolState,
+};
+
+// ── mouse_button_down / mouse_drag / mouse_button_up ────────────────────────
+
+pub struct MouseButtonDownTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static MDOWN_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for MouseButtonDownTool {
+ fn def(&self) -> &ToolDef {
+ MDOWN_DEF.get_or_init(|| ToolDef {
+ name: "mouse_button_down".into(),
+ description: "Press and hold a mouse button at (x,y) via background X11 delivery. \
+ Does not release the button; pair with mouse_drag / mouse_button_up. \
+ Returns the current held-button state.".into(),
+ input_schema: json!({"type":"object","required":["pid","window_id","x","y"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema_with(
+ "When both are present, session takes precedence over cursor_id."
+ ),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"Window id from list_windows."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "button": cua_driver_core::tool_schema::button_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ if let Some(held) = self
+ .state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .get(&cursor_id)
+ .cloned()
+ {
+ return ToolResult::error(format!(
+ "Cursor '{cursor_id}' already has a held mouse button. Call mouse_button_up first."
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&held)));
+ }
+
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let xid = match args.opt_u64("window_id") {
+ Some(v) => v,
+ None => return ToolResult::error("window_id is required on Linux."),
+ };
+ let button_name = args.str_or("button", "left");
+ let button = parse_mouse_button(button_name.as_str());
+ let mut x = args.f64_or("x", 0.0);
+ let mut y = args.f64_or("y", 0.0);
+ let desktop_frame = desktop_frame_requested(&args);
+ if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated once below.
+ } else if args.bool_or("from_zoom", false) {
+ match self.state.zoom_context(&args, pid, Some(xid)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(x, y);
+ x = wx;
+ y = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ x *= ratio;
+ y *= ratio;
+ }
+ if desktop_frame {
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
+ Ok(Ok((lx, ly))) => {
+ x = lx;
+ y = ly;
+ }
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
+ {
+ overlay_glide_to_for(&cursor_id, sx, sy).await;
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ }
+
+ let xi = x as i32;
+ let yi = y as i32;
+ let is_wl = crate::wayland::is_wayland();
+ // Native Wayland: route through the persistent virtual-pointer module
+ // so the held button survives across tool calls; the X11 path keeps
+ // the existing input::send_button_down behaviour.
+ let result = if is_wl {
+ let cid = cursor_id.clone();
+ let aimed = match persistent_pointer_coordinates(
+ is_wl,
+ xid,
+ xi,
+ yi,
+ crate::wayland::window_local_to_output,
+ ) {
+ Ok(point) => point,
+ Err(error) => return input_error_result(error),
+ };
+ let state = self.state.clone();
+ let hold = MouseHoldState {
+ pid,
+ xid,
+ button,
+ x,
+ y,
+ };
+ spawn_blocking_bounded("mouse_button_down", foreground_budget(0), move || {
+ crate::wayland::persistent_vptr::press(&cid, xid, aimed, button, || {
+ state.mouse_hold.lock().unwrap().insert(cid.clone(), hold);
+ })
+ })
+ .await
+ } else {
+ let hold = MouseHoldState {
+ pid,
+ xid,
+ button,
+ x,
+ y,
+ };
+ press_and_record_x11_hold(self.state.clone(), cursor_id.clone(), hold, move || {
+ crate::input::send_button_down(xid, xi, yi, button)
+ })
+ .await
+ };
+ match result {
+ Ok(Ok(())) => {
+ let hold = MouseHoldState {
+ pid,
+ xid,
+ button,
+ x,
+ y,
+ };
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
+ {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ overlay_snap_to_for(&cursor_id, sx, sy, None);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ }
+ ToolResult::text(format!(
+ "✅ Cursor '{cursor_id}' held {} button down at ({x:.1}, {y:.1}).",
+ mouse_button_name(button),
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()).with_structured(mouse_hold_json(
+ &cursor_id,
+ self.state.mouse_hold.lock().unwrap().get(&cursor_id),
+ )),
+ Err(e) => {
+ ToolResult::error(format!("Task error: {e}")).with_structured(mouse_hold_json(
+ &cursor_id,
+ self.state.mouse_hold.lock().unwrap().get(&cursor_id),
+ ))
+ }
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_up.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_up.rs
new file mode 100644
index 0000000000..545ef90b21
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_button_up.rs
@@ -0,0 +1,263 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::ArgsExt as _,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ clear_mouse_hold_after_release, coordinate_frame_schema, desktop_frame_requested,
+ desktop_to_window_local, foreground_budget, held_target_mismatch, mouse_button_name,
+ mouse_hold_json, overlay_glide_to_for, overlay_snap_to_for, resolve_cursor_key,
+ screenshot_scale, spawn_blocking_bounded, window_local_to_screen, MouseHoldState, ToolState,
+};
+
+#[cfg(test)]
+#[test]
+fn coordinate_less_mouse_button_up_survives_snapshot_replacement() {
+ let state = ToolState::new();
+ let cursor_id = "held-after-replacement";
+ let hold = MouseHoldState {
+ pid: 42,
+ xid: 7,
+ button: 1,
+ x: 120.0,
+ y: 80.0,
+ };
+ state.snapshots.publish_for_session(
+ hold.pid as i32,
+ hold.xid,
+ crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
+ Some("press-owner"),
+ Some(2.0),
+ );
+ state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .insert(cursor_id.to_owned(), hold.clone());
+
+ state.snapshots.publish_for_session(
+ hold.pid as i32,
+ hold.xid,
+ crate::atspi::snapshot::AtspiSnapshot::from_nodes(&[]),
+ Some("replacement-owner"),
+ Some(1.0),
+ );
+
+ let release_args = json!({"_session_id": "press-owner"});
+ assert_eq!(
+ mouse_button_up_coordinates(&state, &release_args, &hold).unwrap(),
+ (hold.x, hold.y),
+ "coordinate-less release must use the stored native hold point"
+ );
+
+ let stale_coordinate_args = json!({"_session_id": "press-owner", "x": 60.0, "y": 40.0});
+ let refusal = mouse_button_up_coordinates(&state, &stale_coordinate_args, &hold)
+ .expect_err("coordinate-bearing release must still reject a replaced snapshot");
+ assert_eq!(
+ refusal.structured_content.as_ref().unwrap()["code"],
+ "screenshot_context_missing"
+ );
+
+ clear_mouse_hold_after_release(&state, cursor_id);
+ assert!(state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .insert(cursor_id.to_owned(), hold)
+ .is_none());
+}
+
+fn mouse_button_up_coordinates(
+ state: &ToolState,
+ args: &Value,
+ hold: &MouseHoldState,
+) -> Result<(f64, f64), ToolResult> {
+ if args.get("x").is_none() && args.get("y").is_none() {
+ return Ok((hold.x, hold.y));
+ }
+
+ let mut x = args.opt_f64("x").unwrap_or(hold.x);
+ let mut y = args.opt_f64("y").unwrap_or(hold.y);
+ if args.bool_or("from_zoom", false) {
+ let context = state.zoom_context(args, hold.pid, Some(hold.xid))?;
+ return Ok(context.zoom_to_window(x, y));
+ }
+
+ let ratio = screenshot_scale(state, args, hold.pid, Some(hold.xid))?;
+ x *= ratio;
+ y *= ratio;
+ Ok((x, y))
+}
+
+pub struct MouseButtonUpTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static MUP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for MouseButtonUpTool {
+ fn def(&self) -> &ToolDef {
+ MUP_DEF.get_or_init(|| ToolDef {
+ name: "mouse_button_up".into(),
+ description: "Release a previously-held mouse button via background X11 delivery. \
+ If x/y are omitted, releases at the last held position. Returns the current held-button state.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema_with(
+ "When both are present, session takes precedence over cursor_id."
+ ),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Optional process ID; must match the held button's target."},
+ "window_id":{"type":"integer","description":"Optional window id; must match the held button's target."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let Some(hold) = self
+ .state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .get(&cursor_id)
+ .cloned()
+ else {
+ // A press whose recording failed and whose release niri did not
+ // confirm stays with the persistent pointer: release it there.
+ if crate::wayland::is_wayland() {
+ let cid = cursor_id.clone();
+ let released =
+ spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
+ crate::wayland::persistent_vptr::release_unrecorded(&cid)
+ })
+ .await;
+ let unheld = mouse_hold_json(&cursor_id, None);
+ match released {
+ Ok(Ok(true)) => {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ return ToolResult::text(format!(
+ "✅ Cursor '{cursor_id}' released a button an earlier call pressed but could not record."
+ ))
+ .with_structured(unheld);
+ }
+ Ok(Ok(false)) => {}
+ Ok(Err(e)) => return ToolResult::error(e.to_string()).with_structured(unheld),
+ Err(e) => {
+ return ToolResult::error(format!("Task error: {e}"))
+ .with_structured(unheld)
+ }
+ }
+ }
+ return ToolResult::error(format!(
+ "No mouse button is currently held for cursor '{cursor_id}'."
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, None));
+ };
+ if let Some(err) = held_target_mismatch(&args, &cursor_id, &hold) {
+ return err;
+ }
+
+ let xid = hold.xid;
+
+ let desktop_frame = args.opt_f64("x").is_some()
+ && args.opt_f64("y").is_some()
+ && desktop_frame_requested(&args);
+ let (mut x, mut y) = if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated below.
+ (args.f64_or("x", hold.x), args.f64_or("y", hold.y))
+ } else {
+ match mouse_button_up_coordinates(&self.state, &args, &hold) {
+ Ok(point) => point,
+ Err(refusal) => return refusal,
+ }
+ };
+ if desktop_frame {
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
+ Ok(Ok((lx, ly))) => {
+ x = lx;
+ y = ly;
+ }
+ Ok(Err(e)) => {
+ return ToolResult::error(format!(
+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
+ }
+ Err(e) => {
+ return ToolResult::error(format!("Task error: {e}"))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
+ }
+ }
+ }
+
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
+ {
+ overlay_glide_to_for(&cursor_id, sx, sy).await;
+ }
+
+ let button = hold.button;
+ let xi = x as i32;
+ let yi = y as i32;
+ // Native Wayland: release through the persistent virtual-pointer so
+ // the same vptr device that emitted the press also emits the release
+ // (single logical drag rather than a click pair).
+ let result = if crate::wayland::is_wayland() {
+ let cid = cursor_id.clone();
+ spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
+ crate::wayland::persistent_vptr::release(&cid, button)
+ })
+ .await
+ } else {
+ // XSendEvent does not read the call's action, so the release
+ // still goes out if the call ends first.
+ spawn_blocking_bounded("mouse_button_up", foreground_budget(0), move || {
+ crate::input::send_button_up(xid, xi, yi, button)
+ })
+ .await
+ };
+ match result {
+ Ok(Ok(())) => {
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y)).await
+ {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ overlay_snap_to_for(&cursor_id, sx, sy, None);
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ clear_mouse_hold_after_release(&self.state, &cursor_id);
+ let cleared = mouse_hold_json(&cursor_id, None);
+ ToolResult::text(format!(
+ "✅ Cursor '{cursor_id}' released held {} button at ({x:.1}, {y:.1}).",
+ mouse_button_name(button),
+ ))
+ .with_structured(cleared)
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string())
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
+ Err(e) => ToolResult::error(format!("Task error: {e}"))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_drag.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_drag.rs
new file mode 100644
index 0000000000..ed26f07a04
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/mouse_drag.rs
@@ -0,0 +1,254 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::ArgsExt as _,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ drag_steps, foreground_budget, held_target_mismatch, mouse_button_name, mouse_hold_json,
+ overlay_glide_to_for, overlay_move_to_target_for, overlay_snap_to_for,
+ persistent_pointer_coordinates, require_point_args, resolve_cursor_key, screenshot_scale,
+ spawn_blocking_bounded, window_local_to_screen, AimedPoint, ToolState,
+};
+
+fn overlay_move_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {
+ overlay_move_to_target_for(cursor_id, sx, sy, heading, None);
+}
+
+pub struct MouseDragTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static MDRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for MouseDragTool {
+ fn def(&self) -> &ToolDef {
+ MDRAG_DEF.get_or_init(|| ToolDef {
+ name: "mouse_drag".into(),
+ description: "Move a previously-held mouse button to a new point via background X11 delivery. \
+ Requires an active mouse_button_down state; does not release the button. \
+ Returns the updated held-button state.".into(),
+ input_schema: json!({"type":"object","required":["x","y"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema_with(
+ "When both are present, session takes precedence over cursor_id."
+ ),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Optional process ID; must match the held button's target."},
+ "window_id":{"type":"integer","description":"Optional window id; must match the held button's target."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},
+ "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let Some(mut hold) = self
+ .state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .get(&cursor_id)
+ .cloned()
+ else {
+ return ToolResult::error(format!(
+ "No mouse button is currently held for cursor '{cursor_id}'. Call mouse_button_down first."
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, None));
+ };
+ if let Some(err) = held_target_mismatch(&args, &cursor_id, &hold) {
+ return err;
+ }
+
+ if let Some(refusal) = require_point_args("mouse_drag", &args) {
+ return refusal;
+ }
+ let mut to_x = args.f64_or("x", 0.0);
+ let mut to_y = args.f64_or("y", 0.0);
+ if args.bool_or("from_zoom", false) {
+ match self.state.zoom_context(&args, hold.pid, Some(hold.xid)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(to_x, to_y);
+ to_x = wx;
+ to_y = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, hold.pid, Some(hold.xid)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ to_x *= ratio;
+ to_y *= ratio;
+ }
+
+ let xid = hold.xid;
+
+ let from_x = hold.x;
+ let from_y = hold.y;
+ let duration_ms = args.u64_or("duration_ms", 500);
+ let steps = drag_steps(&args);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, from_x, from_y)).await
+ {
+ overlay_glide_to_for(&cursor_id, sx, sy).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ overlay_snap_to_for(&cursor_id, sx, sy, None);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ }
+
+ let button = hold.button;
+ let step_delay_ms = if steps > 1 {
+ duration_ms / steps as u64
+ } else {
+ duration_ms
+ };
+ let mut result: anyhow::Result<()> = Ok(());
+ let mut moved = 0;
+ let mut prev_x = from_x;
+ let mut prev_y = from_y;
+ let is_wl = crate::wayland::is_wayland();
+ for i in 1..=steps {
+ let t = i as f64 / steps as f64;
+ let ix = from_x + (to_x - from_x) * t;
+ let iy = from_y + (to_y - from_y) * t;
+ // Native Wayland: route motion through the persistent virtual-
+ // pointer so the held button stays held across the entire drag;
+ // X11 keeps the existing input::send_motion path.
+ let cid_inner = cursor_id.clone();
+ let move_result = if is_wl {
+ match persistent_pointer_coordinates(
+ is_wl,
+ xid,
+ ix.round() as i32,
+ iy.round() as i32,
+ crate::wayland::window_local_to_output,
+ ) {
+ Ok(AimedPoint {
+ point: (output_x, output_y),
+ ..
+ }) => {
+ spawn_blocking_bounded("mouse_drag", foreground_budget(0), move || {
+ crate::wayland::persistent_vptr::move_to(&cid_inner, output_x, output_y)
+ })
+ .await
+ }
+ Err(error) => Ok(Err(error)),
+ }
+ } else {
+ spawn_blocking_bounded("mouse_drag", foreground_budget(0), move || {
+ crate::input::send_motion(
+ xid,
+ ix.round() as i32,
+ iy.round() as i32,
+ Some(button),
+ )
+ })
+ .await
+ };
+ match move_result {
+ Ok(Ok(())) => {
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy))
+ .await
+ {
+ let heading = if (ix - prev_x).abs() > f64::EPSILON
+ || (iy - prev_y).abs() > f64::EPSILON
+ {
+ Some((iy - prev_y).atan2(ix - prev_x))
+ } else {
+ None
+ };
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ overlay_move_to_for(&cursor_id, sx, sy, heading);
+ }
+ prev_x = ix;
+ prev_y = iy;
+ moved = i;
+ if step_delay_ms > 0 {
+ tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms)).await;
+ }
+ }
+ Ok(Err(e)) => {
+ result = Err(e);
+ break;
+ }
+ Err(e) => {
+ result = Err(anyhow::anyhow!("Task error: {e}"));
+ break;
+ }
+ }
+ }
+
+ match result {
+ Ok(()) => {
+ hold.x = to_x;
+ hold.y = to_y;
+ self.state
+ .mouse_hold
+ .lock()
+ .unwrap()
+ .insert(cursor_id.clone(), hold.clone());
+ if let Ok(Ok((sx, sy))) =
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y))
+ .await
+ {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx, sy);
+ overlay_snap_to_for(
+ &cursor_id,
+ sx,
+ sy,
+ Some((to_y - from_y).atan2(to_x - from_x)),
+ );
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ }
+ ToolResult::text(format!(
+ "✅ Cursor '{cursor_id}' dragged held {} button to ({to_x:.1}, {to_y:.1}).",
+ mouse_button_name(hold.button),
+ ))
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))
+ }
+ Err(e) if e.is::<crate::wayland::persistent_vptr::HeldPointerEnded>() => {
+ // The persistent pointer released the button and discarded
+ // the cursor: it holds nothing any more.
+ self.state.mouse_hold.lock().unwrap().remove(&cursor_id);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ let mut structured = mouse_hold_json(&cursor_id, None);
+ structured["effect"] = json!("partial");
+ ToolResult::error(format!(
+ "mouse_drag stopped after {moved} of {steps} steps: {e}"
+ ))
+ .with_structured(structured)
+ }
+ Err(e) => ToolResult::error(e.to_string())
+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold))),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/move_cursor.rs
new file mode 100644
index 0000000000..2e19c12af5
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/move_cursor.rs
@@ -0,0 +1,140 @@
+use async_trait::async_trait;
+use cua_driver_contract::MoveCursorInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ foreground_budget, input_error_result, resolve_cursor_key, reveal_pointer_action_for,
+ spawn_blocking_bounded, ToolState,
+};
+
+// ── move_cursor ───────────────────────────────────────────────────────────────
+
+pub struct MoveCursorTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static MCURSOR_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum CursorControlScope {
+ Agent,
+ Desktop,
+}
+
+fn cursor_control_scope(args: &Value) -> CursorControlScope {
+ match args.get("scope").and_then(Value::as_str) {
+ Some("desktop") => CursorControlScope::Desktop,
+ _ => CursorControlScope::Agent,
+ }
+}
+
+#[async_trait]
+impl Tool for MoveCursorTool {
+ fn def(&self) -> &ToolDef {
+ MCURSOR_DEF.get_or_init(|| ToolDef {
+ name: "move_cursor".into(),
+ description: "Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates.".into(),
+ input_schema: json!({"type":"object","required":["x","y"],"properties":{
+ "x":{"type":"number","description":"Destination X. Window scope: screen coordinates of the agent cursor overlay. Desktop scope: get_desktop_state screenshot pixels."},
+ "y":{"type":"number","description":"Destination Y, in the same space as x."},
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Cursor instance to move. Default: 'default'."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"\"window\" (default) moves only the agent cursor overlay; \"desktop\" moves the real OS pointer."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if cursor_control_scope(&args) == CursorControlScope::Desktop {
+ let input = match parse_typed_projection::<MoveCursorInput>("move_cursor", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let (x, y) = (input.x, input.y);
+ let xi = x.round() as i32;
+ let yi = y.round() as i32;
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland {
+ "wayland_desktop"
+ } else {
+ "xtest_desktop"
+ };
+ let result = if wayland {
+ spawn_blocking_bounded("move_cursor", foreground_budget(0), move || {
+ let space = crate::wayland::DesktopInputSpace::current()?;
+ crate::wayland::move_cursor_desktop(&space, xi, yi)
+ })
+ .await
+ } else {
+ spawn_blocking_bounded("move_cursor", foreground_budget(0), move || {
+ crate::input::send_move_xtest_desktop(xi, yi)
+ })
+ .await
+ };
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Moved the real desktop pointer to ({xi}, {yi})."
+ ))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"x":xi,"y":yi,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
+ input_error_result(error)
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let x = args.f64_or("x", 0.0);
+ let y = args.f64_or("y", 0.0);
+ let cursor_id = resolve_cursor_key(&args);
+ // End pointing upper-left (45°) — matches Swift's
+ // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so the
+ // overlay arrow settles to the natural macOS-style pose.
+ // Use the acknowledged animation path so a first-ever move seeds and
+ // displays the session cursor just as reliably as a coordinate click.
+ reveal_pointer_action_for(&self.state, &cursor_id, x, y, false).await;
+ ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}); the user pointer was unchanged."
+ ))
+ }
+}
+
+#[cfg(test)]
+mod session_cursor_target_tests {
+ use super::{cursor_control_scope, reveal_pointer_action_for, CursorControlScope, ToolState};
+ use serde_json::json;
+
+ #[test]
+ fn real_pointer_control_requires_explicit_desktop_scope() {
+ assert_eq!(cursor_control_scope(&json!({})), CursorControlScope::Agent);
+ assert_eq!(
+ cursor_control_scope(&json!({"scope": "window"})),
+ CursorControlScope::Agent
+ );
+ assert_eq!(
+ cursor_control_scope(&json!({"scope": "desktop"})),
+ CursorControlScope::Desktop
+ );
+ }
+
+ #[tokio::test]
+ async fn pointer_position_survives_an_unavailable_overlay() {
+ let state = ToolState::new();
+ reveal_pointer_action_for(&state, "no-renderer", 123.0, 456.0, true).await;
+
+ let cursor = state
+ .cursor_registry
+ .get("no-renderer")
+ .expect("pointer action records its position independently of rendering");
+ assert!(cursor.config.enabled, "input must revive its agent cursor");
+ assert_eq!(cursor.x.zip(cursor.y), Some((123.0, 456.0)));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/parallel_mouse_drag.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/parallel_mouse_drag.rs
new file mode 100644
index 0000000000..873dffe55b
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/parallel_mouse_drag.rs
@@ -0,0 +1,357 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ foreground_budget, linux_input_error, parse_mouse_button, spawn_blocking_bounded,
+ window_local_to_screen, ToolState,
+};
+
+pub struct ParallelMouseDragTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static PMDRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+/// cua-compositor path for parallel_mouse_drag: build window-local drag paths
+/// and run them as concurrent multi-cursor injections over the control socket.
+/// Coordinates stay window-local (the compositor maps them per app_id), so no
+/// X11 geometry/MPX is needed — the X11 path's hard blocker on Wayland.
+async fn parallel_drag_inject(args: &Value) -> ToolResult {
+ let Some(items) = args.get("drags").and_then(|v| v.as_array()) else {
+ return ToolResult::error("drags[] is required.");
+ };
+ if items.len() < 2 {
+ return ToolResult::error("parallel_mouse_drag requires at least two drag items.");
+ }
+ let mut drags = Vec::with_capacity(items.len());
+ for (i, item) in items.iter().enumerate() {
+ let Some(xid) = item.get("window_id").and_then(|v| v.as_u64()) else {
+ return ToolResult::error("each drag item requires window_id.");
+ };
+ let local: Vec<(f64, f64)> = if let Some(pts) = item.get("path").and_then(|v| v.as_array())
+ {
+ pts.iter()
+ .filter_map(|p| {
+ let a = p.as_array()?;
+ Some((a.first()?.as_f64()?, a.get(1)?.as_f64()?))
+ })
+ .collect()
+ } else {
+ let g = |k: &str| item.get(k).and_then(|v| v.as_f64());
+ match (g("from_x"), g("from_y"), g("to_x"), g("to_y")) {
+ (Some(fx), Some(fy), Some(tx), Some(ty)) => vec![(fx, fy), (tx, ty)],
+ _ => {
+ return ToolResult::error(
+ "each drag item requires path[] or from_x/from_y/to_x/to_y.",
+ )
+ }
+ }
+ };
+ if local.len() < 2 {
+ return ToolResult::error("drag path needs at least 2 points.");
+ }
+ let steps = item
+ .get("steps")
+ .and_then(|v| v.as_u64())
+ .unwrap_or(60)
+ .clamp(1, 300) as usize;
+ let x_button = parse_mouse_button(
+ item.get("button")
+ .and_then(|v| v.as_str())
+ .unwrap_or("left"),
+ ) as u32;
+ let app = match tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_target_for_window(xid)
+ })
+ .await
+ {
+ Ok(Ok(target)) => target,
+ Ok(Err(error)) => return ToolResult::error(error.to_string()),
+ Err(error) => return ToolResult::error(format!("Task error: {error}")),
+ };
+ drags.push(crate::wayland::InjectDrag {
+ app_id: app,
+ idx: i,
+ x_button,
+ path: local,
+ steps,
+ });
+ }
+ let n = drags.len();
+ match tokio::task::spawn_blocking(move || crate::wayland::inject_parallel_drags(&drags)).await {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Ran {n} concurrent drags (multi-cursor via cua-compositor)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+}
+
+#[async_trait]
+impl Tool for ParallelMouseDragTool {
+ fn def(&self) -> &ToolDef {
+ PMDRAG_DEF.get_or_init(|| ToolDef {
+ name: "parallel_mouse_drag".into(),
+ description: "Run multiple mouse drag gestures concurrently via Linux MPX/XI2 virtual master pointers. \
+ Each drag item runs on its own session-scoped master pointer (true same-window concurrent draws on X11). \
+ Each item presses once, glides continuously through its whole path, and releases once — one smooth held \
+ drag, not a chain of clicks. A path is given either as a straight segment (from_x/from_y → to_x/to_y) or \
+ as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels (e.g. fn:\"x\" is a diagonal, \
+ fn:\"300+120*sin(x/40)\" a sine wave). Functions support + - * / ^, sin/cos/tan, sqrt, abs, exp, ln, pi, e.".into(),
+ input_schema: json!({"type":"object","required":["drags"],"properties":{
+ "drags":{"type":"array","minItems":2,"description":"Two or more drag gestures to run concurrently, each on its own virtual master pointer.","items":{"type":"object","required":["session","window_id"],"properties":{
+ "session":{"type":"string","description":"Session/cursor id; also keys the virtual master pointer."},
+ "window_id":{"type":"integer","description":"Window id from list_windows."},
+ "path":{"type":"array","items":{"type":"array","items":{"type":"number"}},"description":"Explicit window-local waypoints [[x,y],...] (>=2); pressed once, glided through, released once. Takes precedence over fn/from-to."},
+ "fn":{"type":"string","description":"Expression y(x) in window-local pixels; sampled over [x_from,x_to]. Mutually exclusive with from_x/to_x."},
+ "x_from":{"type":"number","description":"Domain start (window-local x) when `fn` is used."},
+ "x_to":{"type":"number","description":"Domain end (window-local x) when `fn` is used."},
+ "samples":{"type":"integer","minimum":2,"maximum":400,"description":"Waypoints sampled along `fn`. Default: 80."},
+ "from_x":{"type":"number","description":"Straight-segment start X (window-local pixels)."},
+ "from_y":{"type":"number","description":"Straight-segment start Y (window-local pixels)."},
+ "to_x":{"type":"number","description":"Straight-segment end X (window-local pixels)."},
+ "to_y":{"type":"number","description":"Straight-segment end Y (window-local pixels)."},
+ "button": cua_driver_core::tool_schema::button_schema(),
+ "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Default: 1500 for fn paths, 500 for straight."},
+ "steps":{"type":"integer","minimum":1,"maximum":300,"description":"Motion sub-steps along the whole path. Default: scaled to path length."}
+ },"additionalProperties":false}}
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ // Nested cua-compositor: run the drags as concurrent multi-cursor
+ // injections (window-local, no X11 MPX/geometry needed).
+ if crate::wayland::is_inject_mode() {
+ return parallel_drag_inject(&args).await;
+ }
+ // Native Wayland without the inject socket: MPX/XI2 + uinput master
+ // pointers don't exist on Wayland. Surface a typed error instead of
+ // silently calling the X11 path that's guaranteed to fail.
+ if crate::wayland::is_wayland() {
+ return ToolResult::error(
+ "parallel_mouse_drag requires the cua-compositor inject socket on Wayland \
+ (set CUA_INJECT_SOCKET to the cua-compositor control socket), \
+ or run the target under X11.",
+ );
+ }
+ match tokio::task::spawn_blocking(crate::input::check_parallel_pointer_support).await {
+ Ok(Ok(())) => {}
+ Ok(Err(e)) => return ToolResult::error(e.to_string()),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+
+ let Some(items) = args.get("drags").and_then(|v| v.as_array()) else {
+ return ToolResult::error("drags[] is required.");
+ };
+ if items.len() < 2 {
+ return ToolResult::error("parallel_mouse_drag requires at least two drag items.");
+ }
+
+ let mut drags = Vec::with_capacity(items.len());
+ for item in items {
+ let Some(session) = item.get("session").and_then(|v| v.as_str()) else {
+ return ToolResult::error("each drag item requires session.");
+ };
+ let Some(xid) = item.get("window_id").and_then(|v| v.as_u64()) else {
+ return ToolResult::error("each drag item requires window_id.");
+ };
+
+ // Build the window-local waypoint path from one of: an explicit
+ // `path` of [x,y] points, a function `fn` (y = f(x) sampled over
+ // [x_from, x_to]), or a straight from→to segment.
+ let is_fn = item.get("fn").and_then(|v| v.as_str()).is_some();
+ let local: Vec<(f64, f64)> = if let Some(pts) =
+ item.get("path").and_then(|v| v.as_array())
+ {
+ let mut out = Vec::with_capacity(pts.len());
+ for p in pts {
+ let a = p.as_array();
+ let (Some(px), Some(py)) = (
+ a.and_then(|a| a.first()).and_then(|v| v.as_f64()),
+ a.and_then(|a| a.get(1)).and_then(|v| v.as_f64()),
+ ) else {
+ return ToolResult::error("each `path` entry must be [x, y].");
+ };
+ out.push((px, py));
+ }
+ if out.len() < 2 {
+ return ToolResult::error("`path` needs at least 2 points.");
+ }
+ out
+ } else if let Some(expr_str) = item.get("fn").and_then(|v| v.as_str()) {
+ let Some(x_from) = item.get("x_from").and_then(|v| v.as_f64()) else {
+ return ToolResult::error("`fn` requires x_from.");
+ };
+ let Some(x_to) = item.get("x_to").and_then(|v| v.as_f64()) else {
+ return ToolResult::error("`fn` requires x_to.");
+ };
+ let samples = item
+ .get("samples")
+ .and_then(|v| v.as_u64())
+ .unwrap_or(80)
+ .clamp(2, 400);
+ match crate::input::sample_function(expr_str, x_from, x_to, samples) {
+ Ok(pts) => pts,
+ Err(e) => return ToolResult::error(e.to_string()),
+ }
+ } else {
+ let coerce = |k: &str| item.get(k).and_then(|v| v.as_f64());
+ match (coerce("from_x"), coerce("from_y"), coerce("to_x"), coerce("to_y")) {
+ (Some(fx), Some(fy), Some(tx), Some(ty)) => vec![(fx, fy), (tx, ty)],
+ _ => return ToolResult::error("each drag item requires either `fn`+x_from+x_to, or from_x/from_y/to_x/to_y."),
+ }
+ };
+
+ let button = parse_mouse_button(
+ item.get("button")
+ .and_then(|v| v.as_str())
+ .unwrap_or("left"),
+ );
+ let duration_ms = item
+ .get("duration_ms")
+ .and_then(|v| v.as_u64())
+ .unwrap_or(if is_fn { 1500 } else { 500 });
+
+ // One translate gives the window origin; the path is a pure offset.
+ let origin =
+ match tokio::task::spawn_blocking(move || window_local_to_screen(xid, 0.0, 0.0))
+ .await
+ {
+ Ok(Ok(o)) => o,
+ Ok(Err(e)) => return ToolResult::error(e.to_string()),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ };
+ let path: Vec<(i32, i32)> = local
+ .iter()
+ .map(|(lx, ly)| {
+ (
+ (origin.0 + lx).round() as i32,
+ (origin.1 + ly).round() as i32,
+ )
+ })
+ .collect();
+
+ // Default sub-step count scaled to path length (smooth glide),
+ // overridable via `steps`.
+ let total_len: f64 = path
+ .windows(2)
+ .map(|w| {
+ (((w[1].0 - w[0].0) as f64).powi(2) + ((w[1].1 - w[0].1) as f64).powi(2)).sqrt()
+ })
+ .sum();
+ let steps = item
+ .get("steps")
+ .and_then(|v| v.as_u64())
+ .map(|s| (s as usize).clamp(1, 300))
+ .unwrap_or_else(|| ((total_len / 3.0).round() as usize).clamp(24, 300));
+
+ let start = path[0];
+ self.state
+ .cursor_registry
+ .update_position(session, start.0 as f64, start.1 as f64);
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SnapTo {
+ x: start.0 as f64,
+ y: start.1 as f64,
+ heading_radians: None,
+ },
+ );
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+
+ drags.push((
+ session.to_owned(),
+ crate::input::VirtualPointerDrag {
+ target_window: xid,
+ button,
+ path,
+ duration_ms,
+ steps,
+ },
+ ));
+ }
+
+ let drags_for_task = drags.clone();
+ // The drags run side by side, so the longest one bounds the call.
+ let budget = drags
+ .iter()
+ .map(|(_, drag)| {
+ foreground_budget(drag.steps) + std::time::Duration::from_millis(drag.duration_ms)
+ })
+ .max()
+ .unwrap_or_default();
+ let result = spawn_blocking_bounded("parallel_mouse_drag", budget, move || {
+ crate::input::send_parallel_virtual_pointer_drags(&drags_for_task)
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ for (session, drag) in &drags {
+ let n = drag.path.len();
+ let end = drag.path[n - 1];
+ let prev = drag.path[n.saturating_sub(2)];
+ self.state
+ .cursor_registry
+ .update_position(session, end.0 as f64, end.1 as f64);
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SnapTo {
+ x: end.0 as f64,
+ y: end.1 as f64,
+ heading_radians: Some(
+ ((end.1 - prev.1) as f64).atan2((end.0 - prev.0) as f64),
+ ),
+ },
+ );
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: end.0 as f64,
+ y: end.1 as f64,
+ },
+ );
+ }
+ ToolResult::text(format!(
+ "✅ Ran {} MPX drag gesture(s) concurrently.",
+ drags.len()
+ ))
+ .with_structured(json!({"count": drags.len()}))
+ }
+ Ok(Err(e)) => {
+ for (session, _) in &drags {
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ }
+ linux_input_error(e)
+ }
+ Err(e) => {
+ for (session, _) in &drags {
+ crate::overlay::send_command_for(
+ session.to_owned(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ }
+ ToolResult::error(format!("Task error: {e}"))
+ }
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/press_key.rs
new file mode 100644
index 0000000000..c567d83e82
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/press_key.rs
@@ -0,0 +1,514 @@
+use async_trait::async_trait;
+use cua_driver_contract::PressKeyInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ background_key_route, coordinate_frame_is_desktop, coordinate_frame_schema, focus_by_pixel,
+ focus_hyprland_foreground, focus_nested_inject_target, foreground_budget,
+ foreground_hyprland_action, foreground_hyprland_key, foreground_hyprland_refusal,
+ hyprland_foreground, inject_terminal_input, input_error_result, invalid_pointer_arguments,
+ isolated_hyprland_action, isolated_hyprland_background, isolated_hyprland_refusal,
+ key_route_result, position_named_session_keyboard_cursor, prove_token_element,
+ resolve_cursor_key, spawn_blocking_bounded, split_key_combo, stale_observed_element,
+ unavailable_chromium_background, unavailable_gtk_keyboard_background,
+ unavailable_wayland_focused_input_background, unavailable_webkit_keyboard_background,
+ wm_chord_background, KeyRoute, ToolState,
+};
+
+/// True when `key` names the Enter key in the shared X keysym vocabulary
+/// (`key_name_to_keysym`). The terminal pty short-circuit below applies to
+/// every spelling of that physical key — `enter`, `return`, any case — so an
+/// agent following the documented key names cannot silently lose the keypress
+/// on a terminal window (terminals discard synthetic XSendEvent keys).
+fn is_enter_key(key: &str) -> bool {
+ crate::input::key_name_to_keysym(key).ok() == Some(0xFF0D)
+}
+
+// ── press_key ─────────────────────────────────────────────────────────────────
+
+pub struct PressKeyTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static PRESS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+/// Turn a modified `press_key` request into the chord an underlying keyboard
+/// path expects when it takes a single key and no separate modifier list.
+///
+/// Returns `None` when there is nothing to promote, so the caller keeps the
+/// plain single-key route. Dropping the modifiers instead is not a silent
+/// no-op: the bare key still reaches the app and is typed as a character.
+fn press_key_chord(mods: &[String], key: &str) -> Option<Vec<String>> {
+ if mods.is_empty() {
+ return None;
+ }
+ let mut chord = mods.to_vec();
+ chord.push(key.to_owned());
+ Some(chord)
+}
+
+#[cfg(test)]
+mod press_key_tests {
+ use super::is_enter_key;
+ use super::press_key_chord;
+
+ #[test]
+ fn unmodified_press_stays_on_the_single_key_route() {
+ assert_eq!(press_key_chord(&[], "return"), None);
+ }
+
+ #[test]
+ fn enter_key_gate_accepts_the_documented_enter_spelling() {
+ assert!(is_enter_key("enter"));
+ assert!(is_enter_key("Enter"));
+ assert!(is_enter_key("ENTER"));
+ }
+
+ #[test]
+ fn enter_key_gate_accepts_the_documented_return_spelling() {
+ // `key_name_to_keysym` resolves "return" and "enter" to the same
+ // physical key (XK_Return, 0xFF0D); the pty short-circuit must not
+ // drop a documented spelling on terminals (they discard synthetic
+ // XSendEvent keys, so a dropped short-circuit loses the keypress).
+ assert!(is_enter_key("return"));
+ assert!(is_enter_key("Return"));
+ assert!(is_enter_key("RETURN"));
+ }
+
+ #[test]
+ fn enter_key_gate_rejects_non_enter_keys() {
+ assert!(!is_enter_key("tab"));
+ assert!(!is_enter_key("escape"));
+ assert!(!is_enter_key("space"));
+ assert!(!is_enter_key(""));
+ }
+
+ #[test]
+ fn modifiers_are_promoted_to_a_chord_in_order() {
+ assert_eq!(
+ press_key_chord(&["ctrl".to_owned()], "s"),
+ Some(vec!["ctrl".to_owned(), "s".to_owned()])
+ );
+ assert_eq!(
+ press_key_chord(&["ctrl".to_owned(), "shift".to_owned()], "t"),
+ Some(vec!["ctrl".to_owned(), "shift".to_owned(), "t".to_owned()])
+ );
+ }
+}
+
+#[async_trait]
+impl Tool for PressKeyTool {
+ fn def(&self) -> &ToolDef {
+ PRESS_DEF.get_or_init(|| ToolDef {
+ name: "press_key".into(),
+ description: "Press a key via XSendEvent to a window. No focus steal.".into(),
+ input_schema: json!({
+ "type":"object","required":["key"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to send the key to the focused application."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
+ "key":{"type":"string","description":"Key name: enter/return, tab, escape, space, backspace, delete, insert, home, end, pageup, pagedown, up, down, left, right, f1-f12, or any single ASCII character."},
+ "modifiers":{"type":"array","items":{"type":"string"},"description":"Modifier keys held while the key is pressed: ctrl, shift, alt, super."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Pixel Y (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to send the key to the focused application. Default \"window\"."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if (args.opt_str("scope").as_deref() == Some("desktop")
+ || coordinate_frame_is_desktop(&args))
+ && args.get("pid").is_none()
+ {
+ let input = match parse_typed_projection::<PressKeyInput>("press_key", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let key = input.key;
+ let display = key.clone();
+ let modifiers = input.modifiers.unwrap_or_default();
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result =
+ spawn_blocking_bounded("desktop press_key", foreground_budget(1), move || {
+ if wayland && modifiers.is_empty() {
+ crate::wayland::press_key_focused(&key)
+ } else if wayland {
+ let mut keys = modifiers;
+ keys.push(key);
+ crate::wayland::hotkey_focused(&keys)
+ } else {
+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_key_xtest(&key, &refs)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop key '{display}'."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let mut pid = args.u64_or("pid", 0) as u32;
+ let key = match args.require_str("key") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let mut mods: Vec<String> = args.str_array("modifiers");
+ let (combo_mods, key) = split_key_combo(&key);
+ mods.extend(combo_mods);
+
+ // Resolve the element token into both its owning window and exact
+ // child. Foreground delivery establishes child focus inside the
+ // verified top-level activation transaction; background XSendEvent
+ // retains the historical direct-window path.
+ let window_id_arg = args.opt_u64("window_id");
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "press_key").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let resolved_element_index = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ let xid_opt = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
+ };
+ let xid = match xid_opt {
+ Some(x) => x,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || {
+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
+ })
+ .await
+ .unwrap_or_default();
+ // pid omitted: the keys go to the active window, like a
+ // physical keyboard would, and the action adopts its pid.
+ let chosen = if pid == 0 {
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active && w.pid.is_some())
+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
+ } else {
+ // The pid's active window, else its largest on-screen
+ // toplevel: LibreOffice and GIMP own hidden/utility
+ // toplevels that a plain `first()` could pick, and an
+ // unmapped window cannot take the virtual keyboard focus.
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active)
+ .or_else(|| {
+ windows
+ .iter()
+ .filter(|w| w.is_on_screen)
+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
+ })
+ .or_else(|| windows.first())
+ };
+ match chosen {
+ Some(w) => {
+ if pid == 0 {
+ pid = w.pid.unwrap_or(0);
+ }
+ w.xid
+ }
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // ── px form: pixel-click to focus, then the key goes to the focused element ──
+ // Reuses click's translation + delivery_mode; after it, deliver via the plain
+ // background path (the focus-click already handled fronting when fg). Pass x,y
+ // (no element_token) for Chromium/Electron surfaces the AX path can't focus.
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ if isolated_hyprland_background(delivery) {
+ if xid_opt.is_none() {
+ return isolated_hyprland_refusal(
+ "an exact window_id is required for isolated keys",
+ );
+ }
+ if resolved_element_index.is_some()
+ || args.get("x").is_some()
+ || args.get("y").is_some()
+ {
+ return isolated_hyprland_refusal(
+ "isolated keys address the exact top-level; first click the child explicitly",
+ );
+ }
+ position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, false)
+ .await;
+ return isolated_hyprland_action(
+ &args,
+ pid,
+ xid,
+ crate::wayland::hyprland_input::Action::Key {
+ key,
+ modifiers: mods,
+ },
+ )
+ .await;
+ }
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+
+ if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
+ return refusal;
+ }
+ if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {
+ return result;
+ }
+
+ let px = args.get("x").and_then(|value| value.as_f64());
+ let py = args.get("y").and_then(|value| value.as_f64());
+ if px.is_some() != py.is_some() {
+ return invalid_pointer_arguments(
+ "press_key",
+ "x and y must be passed together (a pixel focus point), or neither",
+ );
+ }
+ if px.is_some() && resolved_element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to press_key, not both.",
+ );
+ }
+
+ if hyprland_foreground(delivery) {
+ if xid_opt.is_none() {
+ return foreground_hyprland_refusal("an exact window_id is required");
+ }
+ let action = match foreground_hyprland_key(key, mods) {
+ Ok(action) => action,
+ Err(error) => return error,
+ };
+ if let Err(error) =
+ focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
+ .await
+ {
+ return error;
+ }
+ if px.is_none() {
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ None,
+ false,
+ )
+ .await;
+ }
+ return foreground_hyprland_action(&args, pid, xid, action).await;
+ }
+
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ px.zip(py),
+ false,
+ )
+ .await;
+
+ // Nested cua-compositor addresses the owning Wayland client directly.
+ // Preserve legacy modifiers by promoting the request to a chord.
+ if crate::wayland::is_inject_mode() {
+ if let Err(error) =
+ focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
+ {
+ return error;
+ }
+ let result = match press_key_chord(&mods, &key) {
+ None => {
+ let key_w = key.clone();
+ tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_press_key(pid, xid, &key_w)
+ })
+ .await
+ }
+ Some(chord) => {
+ tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_hotkey(pid, xid, &chord)
+ })
+ .await
+ }
+ };
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Pressed key '{key}' (focus-free via cua-compositor)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ let px_target = {
+ if let (Some(cx), Some(cy)) = (px, py) {
+ let from_zoom = args.bool_or("from_zoom", false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ Some(xid),
+ (cx, cy),
+ delivery.is_foreground(),
+ &args,
+ from_zoom,
+ )
+ .await
+ {
+ return e;
+ }
+ Some((cx.round() as i32, cy.round() as i32))
+ } else {
+ None
+ }
+ };
+
+ // Native Wayland: send the key to the focused surface via virtual-keyboard.
+ // `wayland::press_key` carries no modifier list, so a modified request has
+ // to become a chord here — otherwise the modifiers are dropped and the
+ // bare key is typed as a character (Ctrl+S inserts a literal "s").
+ if crate::wayland::wayland_input_enabled() {
+ let key_w = key.clone();
+ let chord = press_key_chord(&mods, &key);
+ let element = token.clone();
+ let result = spawn_blocking_bounded("press_key", foreground_budget(1), move || {
+ crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::RestoreMode::RestorePrevious,
+ || {
+ if let Some(element) = &element {
+ if !crate::atspi::focus_token(pid, element)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {}",
+ element.index
+ );
+ }
+ }
+ match chord {
+ Some(keys) => crate::wayland::hotkey_focused(&keys),
+ None => crate::wayland::press_key_focused(&key_w),
+ }
+ },
+ )
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Pressed key '{key}' (via Wayland virtual-keyboard)."
+ )),
+ Ok(Err(e)) => resolved_element_index
+ .and_then(|idx| stale_observed_element("press_key", idx, &e))
+ .unwrap_or_else(|| ToolResult::error(e.to_string())),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ let key_for_task = key.clone();
+ let cursor_id = resolve_cursor_key(&args);
+ // Foreground delivery is one atomic activate-and-XTest transaction.
+ // A preceding PX click establishes internal widget focus, but that
+ // click restores the prior top-level before returning.
+ let deliver_fg = delivery.is_foreground();
+ let result = spawn_blocking_bounded(
+ "press_key",
+ foreground_budget(1),
+ move || -> anyhow::Result<KeyRoute> {
+ if resolved_element_index.is_none()
+ && mods.is_empty()
+ && is_enter_key(&key_for_task)
+ && inject_terminal_input(pid, xid, "\n")?
+ {
+ return Ok(KeyRoute::Terminal);
+ }
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ // foreground: activate the window first, then inject a REAL key via
+ // XTest. Synthetic XSendEvent keys (`send_key`) are dropped by
+ // GTK/Qt/Chromium/Firefox, so the foreground rung must use XTest —
+ // it delivers to the now-focused window. background = real key
+ // events through the XI2 virtual master keyboard (no focus steal),
+ // else direct XSendEvent for apps that accept it.
+ if deliver_fg {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::keyboard(),
+ || {
+ if let Some(element) = &token {
+ let element_index = element.index;
+ if !crate::atspi::focus_token(pid, element)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {element_index}"
+ );
+ }
+ }
+ crate::input::send_key_xtest(&key_for_task, &m)
+ },
+ )
+ .map(|((), report)| KeyRoute::Foreground(report));
+ }
+ background_key_route(
+ &cursor_id,
+ pid,
+ xid,
+ px_target,
+ &key_for_task,
+ &m,
+ token.as_ref(),
+ )
+ })
+ .await;
+ let mode_label = if deliver_fg {
+ "foreground"
+ } else {
+ "background"
+ };
+ match result {
+ Ok(Ok(route)) => key_route_result(&format!("Pressed key '{key}'"), route, mode_label),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/right_click.rs
new file mode 100644
index 0000000000..3775f5247a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/right_click.rs
@@ -0,0 +1,329 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::ArgsExt as _,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::{
+ click::ClickTool,
+ impl_::{
+ checked_element_local_coords, coordinate_frame_schema, desktop_frame_requested,
+ desktop_frame_unmapped, desktop_to_window_local, element_screen_center_and_rect,
+ foreground_budget, hyprland_foreground, input_error_result, prove_token_element,
+ require_point_args, resolve_cursor_key, reveal_pointer_action_for,
+ reveal_pointer_action_target_for, screenshot_scale, spawn_blocking_bounded,
+ unavailable_chromium_background, unavailable_gtk_pointer_background,
+ unavailable_wayland_focused_input_background, unavailable_webkit_background,
+ window_local_to_screen, with_pixel_points, x11_pixel_click_no_focus_steal, AimedPoint,
+ PointerRoute, ToolState,
+ },
+};
+
+// ── right_click ───────────────────────────────────────────────────────────────
+
+pub struct RightClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static RCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for RightClickTool {
+ fn def(&self) -> &ToolDef {
+ RCLICK_DEF.get_or_init(|| ToolDef {
+ name: "right_click".into(),
+ description: "Right-click at (x,y) or an element_token (AT-SPI bounds). Background \
+ delivery on X11 sends real button events from the session's virtual master pointer \
+ (path mpx_pointer); a point covered by another application's window is refused \
+ (target_occluded), one covered by this app's own dialog is retargeted to it \
+ (retargeted_to), one outside the window is refused (point_outside_window; x/y are \
+ window-local pixels, the result reports window_point / screen_point). No focus \
+ steal. When a context menu opens, the result names it (`popup: {window_id, bounds, \
+ title}`): call get_window_state(pid, window_id=<that id>) to index its menu items \
+ and click them by element_token. Provide either (window_id + x/y) or (pid + element_token). \
+ After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),
+ input_schema: json!({"type":"object","required":[],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it)."},
+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},
+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
+ return refusal;
+ }
+ if hyprland_foreground(delivery) {
+ // Share exact-target validation and the admitted native click lifecycle.
+ let mut args = args;
+ args["button"] = json!("right");
+ args["count"] = json!(1);
+ return ClickTool {
+ state: self.state.clone(),
+ }
+ .invoke(args)
+ .await;
+ }
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "right_click").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let window_id_resolved: Option<u64> = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
+ };
+ if let Some(token) = token {
+ let idx = token.index;
+ let xid_hint = window_id_resolved;
+ let target = token.clone();
+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {
+ checked_element_local_coords(pid, &target, xid_hint)
+ })
+ .await;
+ return match result {
+ Ok(Ok((xid, lx, ly))) => {
+ if let Ok(Ok(((sx, sy), rect))) = tokio::task::spawn_blocking(move || {
+ element_screen_center_and_rect(pid, &token, Some(xid))
+ })
+ .await
+ {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ reveal_pointer_action_target_for(
+ &self.state,
+ &cursor_id,
+ sx,
+ sy,
+ true,
+ Some(rect),
+ )
+ .await;
+ }
+ let lxi = lx as i32;
+ let lyi = ly as i32;
+ let wayland_point = match crate::wayland::wayland_input_enabled()
+ .then(|| crate::wayland::window_local_to_output(xid, lxi, lyi))
+ .transpose()
+ {
+ Ok(point) => point,
+ Err(error) => return input_error_result(error),
+ };
+ let cursor_id_for_task = cursor_id.clone();
+ let click_result = spawn_blocking_bounded(
+ "right_click",
+ foreground_budget(0),
+ move || -> anyhow::Result<PointerRoute> {
+ if crate::wayland::is_inject_mode() {
+ return crate::wayland::inject_click(pid, xid, lx, ly, 1, 3)
+ .map(|()| PointerRoute::Wayland);
+ }
+ if crate::wayland::wayland_input_enabled() {
+ let aimed = wayland_point.unwrap_or_else(|| (lxi, lyi).into());
+ return crate::wayland::click(xid, aimed, 1, 3)
+ .map(|()| PointerRoute::Wayland);
+ }
+ if delivery.is_foreground() {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ let (sx, sy) =
+ window_local_to_screen(xid, lxi as f64, lyi as f64)?;
+ crate::input::send_click_xtest_desktop(
+ sx.round() as i32,
+ sy.round() as i32,
+ 3,
+ 1,
+ )
+ },
+ )
+ .map(|(value, _report)| value)
+ .map(|()| PointerRoute::Foreground);
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 3, 1)
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ match click_result {
+ Ok(Ok(route)) => ToolResult::text(format!(
+ "Right-clicked element [{idx}] {}",
+ route.text_suffix(mode_label)
+ ))
+ .with_structured(route.structured(mode_label)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ if let Some(refusal) = require_point_args("right_click", &args) {
+ return refusal;
+ }
+ let xid = match window_id_resolved {
+ Some(v) => v,
+ None => return ToolResult::error("Provide either element_token or window_id + x/y."),
+ };
+ let desktop_frame = desktop_frame_requested(&args);
+ let from_zoom = args.bool_or("from_zoom", false);
+ let mut x = args.f64_or("x", 0.0);
+ let mut y = args.f64_or("y", 0.0);
+ if desktop_frame {
+ // Desktop pixels: no zoom/screenshot scaling; translated once below.
+ } else if from_zoom {
+ match self.state.zoom_context(&args, pid, Some(xid)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(x, y);
+ x = wx;
+ y = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(xid)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ x *= ratio;
+ y *= ratio;
+ }
+ if desktop_frame {
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {
+ Ok(Ok((lx, ly))) => {
+ x = lx;
+ y = ly;
+ }
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ let wayland_output_point = if crate::wayland::wayland_input_enabled() {
+ match crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32) {
+ Ok(point) => Some(point),
+ Err(error) => return input_error_result(error),
+ }
+ } else {
+ None
+ };
+ let glide_target = if let Some(AimedPoint {
+ point: (sx, sy), ..
+ }) = wayland_output_point
+ {
+ Some((sx as f64, sy as f64))
+ } else {
+ tokio::task::spawn_blocking(move || window_local_to_screen(xid, x, y))
+ .await
+ .ok()
+ .and_then(|r| r.ok())
+ };
+ if let Some((sx, sy)) = glide_target {
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
+ }
+ let (xi, yi) = (x as i32, y as i32);
+ let cursor_id_for_task = cursor_id.clone();
+ let result = spawn_blocking_bounded(
+ "right_click",
+ foreground_budget(0),
+ move || -> anyhow::Result<PointerRoute> {
+ if crate::wayland::is_inject_mode() {
+ return crate::wayland::inject_click(pid, xid, x, y, 1, 3)
+ .map(|()| PointerRoute::Wayland);
+ }
+ if crate::wayland::wayland_input_enabled() {
+ let aimed = wayland_output_point.unwrap_or_else(|| (xi, yi).into());
+ return crate::wayland::click(xid, aimed, 1, 3).map(|()| PointerRoute::Wayland);
+ }
+ if delivery.is_foreground() {
+ return crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ // Real XTest right-click at the screen point (synthetic
+ // XSendEvent is dropped by GTK/Qt; MPX needs /dev/uinput).
+ // Mirrors the single-click foreground path.
+ if let Ok((sx, sy)) = window_local_to_screen(xid, xi as f64, yi as f64)
+ {
+ crate::input::send_click_xtest_desktop(
+ sx.round() as i32,
+ sy.round() as i32,
+ 3,
+ 1,
+ )?;
+ return Ok(());
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)
+ .map(|_| ())
+ },
+ )
+ .map(|(value, _report)| value)
+ .map(|()| PointerRoute::Foreground);
+ }
+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ let outcome = match result {
+ Ok(Ok(route)) => ToolResult::text(format!(
+ "Right-clicked at ({x:.1}, {y:.1}) {}",
+ route.text_suffix(mode_label)
+ ))
+ .with_structured(route.structured(mode_label)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ with_pixel_points(
+ outcome,
+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),
+ (xi, yi),
+ )
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/scroll.rs
new file mode 100644
index 0000000000..3d533f6ec5
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/scroll.rs
@@ -0,0 +1,835 @@
+use async_trait::async_trait;
+use cua_driver_contract::ScrollInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::{parse_typed_projection, ArgsExt as _},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ coordinate_frame_schema, desktop_frame_requested, desktop_frame_unmapped, desktop_input_space,
+ desktop_to_window_local, explicit_keyboard_cursor_target, foreground_budget,
+ foreground_hyprland_refusal, hyprland_foreground, hyprland_input_result, input_error_result,
+ invalid_pointer_arguments, is_webkitgtk_embedder, isolated_hyprland_background,
+ isolated_hyprland_refusal, isolated_hyprland_result, isolated_hyprland_task_error,
+ keyboard_window_center, named_session_cursor_key, prove_token_element, resolve_cursor_key,
+ resolve_element_local_coords, reveal_pointer_action_for, screenshot_scale,
+ spawn_blocking_bounded, spawn_isolated_hyprland, stale_observed_element,
+ unavailable_chromium_background, unavailable_gtk_pointer_background,
+ unavailable_wayland_focused_input_background, unavailable_webkit_background,
+ window_local_to_screen, window_screen_center, ToolState,
+};
+
+/// Wheel detents for a compositor scroll: one per line. On niri a page is
+/// the detent approximation the Hyprland route documents; other compositors
+/// keep one detent per page, which no run has measured against.
+fn wheel_ticks(amount: usize, by: cua_driver_contract::ScrollBy, niri: bool) -> u32 {
+ let detents = match by {
+ cua_driver_contract::ScrollBy::Page if niri => {
+ amount * crate::wayland::hyprland_input::PAGE_WHEEL_DETENTS
+ }
+ _ => amount,
+ };
+ detents as u32
+}
+
+#[cfg(test)]
+#[test]
+fn a_niri_page_scroll_sends_more_than_one_detent_per_page() {
+ // `by: "page"` used to send `amount` detents, the same as lines: two
+ // pages moved a WebKitGTK scroller 80 px under niri, less than its
+ // 128 px viewport.
+ use cua_driver_contract::ScrollBy;
+ assert_eq!(wheel_ticks(2, ScrollBy::Line, true), 2);
+ assert_eq!(
+ wheel_ticks(2, ScrollBy::Page, true),
+ 2 * crate::wayland::hyprland_input::PAGE_WHEEL_DETENTS as u32
+ );
+ // Sway, KWin and GNOME keep their earlier page scroll.
+ assert_eq!(wheel_ticks(2, ScrollBy::Page, false), 2);
+}
+
+/// Pixel scroll shares click and drag's refusal order: an unsupported native
+/// background delivery must report `background_unavailable` before the
+/// screenshot frame is consulted, because native Wayland may legitimately
+/// withhold the screenshot (`surface_identity_unproven`) that the frame needs.
+fn coordinate_scroll_scale(
+ native_refusal: Option<ToolResult>,
+ resolve_scale: impl FnOnce() -> Result<f64, ToolResult>,
+) -> Result<f64, ToolResult> {
+ if let Some(refusal) = native_refusal {
+ return Err(refusal);
+ }
+ resolve_scale()
+}
+
+async fn foreground_hyprland_scroll(
+ args: &Value,
+ pid: u32,
+ xid: u64,
+ actions: Vec<crate::wayland::hyprland_input::Action>,
+) -> ToolResult {
+ if !crate::wayland::hyprland_input::enabled() {
+ return foreground_hyprland_refusal("production Hyprland input plugin is unavailable");
+ }
+ let owner = named_session_cursor_key(args);
+ let (_cancellation, dispatch) = match spawn_isolated_hyprland(args, move |cancellation| {
+ crate::wayland::hyprland_input::execute_foreground_scroll(
+ owner,
+ pid,
+ xid,
+ actions,
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(_) => return foreground_hyprland_refusal("authenticated admitted lifecycle required"),
+ };
+ hyprland_input_result(
+ match dispatch.await {
+ Ok(result) => result,
+ Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
+ error.into(),
+ 0,
+ )),
+ },
+ true,
+ )
+}
+
+#[cfg(test)]
+#[test]
+fn coordinate_scroll_keeps_wayland_background_refusal_ahead_of_screenshot_context() {
+ use std::sync::atomic::{AtomicBool, Ordering};
+
+ let resolved = AtomicBool::new(false);
+ let native_refusal = crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ let result = coordinate_scroll_scale(Some(native_refusal), || {
+ resolved.store(true, Ordering::SeqCst);
+ Err(ToolResult::error(
+ "screenshot context must not be consulted",
+ ))
+ })
+ .expect_err("unsupported Wayland background delivery must refuse pixel scroll");
+
+ assert_eq!(
+ result.structured_content.as_ref().unwrap()["code"],
+ "background_unavailable"
+ );
+ assert!(
+ !resolved.load(Ordering::SeqCst),
+ "scroll snapshot context must not be consulted before the native refusal"
+ );
+ assert_eq!(coordinate_scroll_scale(None, || Ok(1.5)).unwrap(), 1.5);
+
+ let source = include_str!("scroll.rs");
+ let invoke = source
+ .rsplit_once("impl Tool for ScrollTool {")
+ .unwrap()
+ .1
+ .split_once("async fn invoke")
+ .unwrap()
+ .1;
+ let (before_scale, _) = invoke
+ .split_once("coordinate_scroll_scale(native_refusal")
+ .expect("pixel scroll must resolve its frame through the refusal-first helper");
+ assert!(
+ before_scale.contains("unavailable_wayland_focused_input_background(delivery, true)"),
+ "pixel scroll must compute the native Wayland refusal before its frame"
+ );
+ assert!(
+ !before_scale.contains("screenshot_scale("),
+ "pixel scroll must not consult the screenshot frame before the native refusal"
+ );
+}
+
+// ── scroll ────────────────────────────────────────────────────────────────────
+
+pub struct ScrollTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+fn atspi_scroll_result(progress: crate::atspi::ScrollProgress, foreground: bool) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
+ };
+ let effect = if !progress.complete && progress.acknowledged > 0 {
+ ActionEffect::Partial
+ } else {
+ ActionEffect::Unverifiable
+ };
+ let mut record = ActionExecutionRecord::builder(
+ effect,
+ ActionTransport::LinuxAtSpiAction,
+ if foreground {
+ RequestedDelivery::Foreground
+ } else {
+ RequestedDelivery::Background
+ },
+ )
+ .actual_delivery(if !progress.complete {
+ ActualDelivery::Unknown
+ } else if foreground {
+ ActualDelivery::Foreground
+ } else {
+ ActualDelivery::Background
+ });
+ if progress.acknowledged > 0 {
+ record = record.delivered_count(progress.acknowledged);
+ }
+ let text = if progress.complete {
+ format!(
+ "AT-SPI acknowledged {} scroll action(s); effect remains unverified.",
+ progress.acknowledged
+ )
+ } else {
+ format!("AT-SPI scroll stopped after {} acknowledged action(s): {}. Refresh state before another action.", progress.acknowledged, progress.detail.as_deref().unwrap_or("unknown outcome"))
+ };
+ let record = record
+ .detail(text.clone())
+ .build()
+ .expect("AT-SPI scroll preserves acknowledged progress");
+ let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
+ let result = if progress.complete {
+ ToolResult::text(text)
+ } else {
+ ToolResult::error(text)
+ };
+ result.with_structured(public).with_action_record(record)
+}
+
+#[test]
+fn atspi_scroll_outcomes_preserve_uncertainty_and_acknowledged_count() {
+ use cua_driver_core::action_record::ActionEffect;
+ for (complete, acknowledged, effect) in [
+ (false, 0, ActionEffect::Unverifiable),
+ (false, 1, ActionEffect::Partial),
+ (true, 2, ActionEffect::Unverifiable),
+ ] {
+ let result = atspi_scroll_result(
+ crate::atspi::ScrollProgress {
+ acknowledged,
+ complete,
+ detail: None,
+ },
+ false,
+ );
+ let record = result.action_record.unwrap();
+ assert_eq!(record.effect, effect);
+ let public = serde_json::to_value(record.public_result().unwrap()).unwrap();
+ assert_eq!(
+ public["delivery"]["mode"],
+ if complete { "background" } else { "unknown" }
+ );
+ assert_eq!(
+ public["delivery"]["delivered_count"],
+ if acknowledged == 0 {
+ Value::Null
+ } else {
+ json!(acknowledged)
+ }
+ );
+ }
+}
+static SCROLL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ScrollTool {
+ fn def(&self) -> &ToolDef {
+ SCROLL_DEF.get_or_init(|| ToolDef {
+ name: "scroll".into(),
+ description: "Scroll the target pid's focused region via XSendEvent Button4/5. \
+ direction required; by defaults to line, amount defaults to 3.".into(),
+ input_schema: json!({
+ // `pid` is conditionally required (validated in code), so only
+ // `direction` is pinned — matches the scroll→["direction"] canon
+ // in cua_driver_core::tool_schema.
+ "type":"object","required":["direction"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
+ "pid":{"type":"integer","description":"Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid)."},
+ "direction":{"type":"string","enum":["up","down","left","right"],"description":"Scroll direction."},
+ "by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},
+ "amount":{"type":"integer","minimum":1,"maximum":50,"description":"Number of scroll steps. Default: 3."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y; optional with element_token (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_token."},
+ "y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_token."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with x,y and no pid/window_id for get_desktop_state screen coordinates. Default \"window\"."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: false, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_id = resolve_cursor_key(&args);
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<ScrollInput>("scroll", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let direction = input.direction.as_str().to_owned();
+ let (x, y) = (input.x, input.y);
+ let x = x.round() as i32;
+ let y = y.round() as i32;
+ let amount = input.amount.unwrap_or(3).clamp(1, 50) as usize;
+ let ticks = wheel_ticks(
+ amount,
+ input.by.unwrap_or(cua_driver_contract::ScrollBy::Line),
+ crate::wayland::niri::is_session(),
+ );
+ let display = direction.clone();
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_desktop" } else { "xtest" };
+ let space = match desktop_input_space().await {
+ Ok(space) => space,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ if named_session_cursor_key(&args).is_some() {
+ let (overlay_x, overlay_y) = space.to_layout(x, y);
+ reveal_pointer_action_for(
+ &self.state,
+ &cursor_id,
+ f64::from(overlay_x),
+ f64::from(overlay_y),
+ false,
+ )
+ .await;
+ }
+ let result = spawn_blocking_bounded(
+ "desktop scroll",
+ foreground_budget(ticks as usize),
+ move || {
+ if wayland {
+ crate::wayland::scroll_desktop(&space, x, y, &direction, ticks)
+ } else {
+ crate::input::send_scroll_xtest_desktop(x, y, &direction, amount)
+ }
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Scrolled desktop {display} × {amount}."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) if crate::input::foreground::error_code(&error).is_some() => {
+ input_error_result(error)
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let mut pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let direction = match args.require_str("direction") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;
+ let by = match serde_json::from_value::<cua_driver_contract::ScrollBy>(
+ args.get("by").cloned().unwrap_or(json!("line")),
+ ) {
+ Ok(by) => by,
+ Err(error) => return ToolResult::error(format!("Invalid scroll by: {error}")),
+ };
+ // The Linux scroll implementation doesn't pre-focus the element (X11
+ // scroll buttons go to the window root), but the token still needs to
+ // be validated so a stale token surfaces an error instead of silently
+ // no-op'ing.
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "scroll").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let xid_opt: Option<u64> = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
+ Some(*window_id)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => args.opt_u64("window_id"),
+ };
+
+ // Resolve XID: use window_id if given, else first window for pid.
+ let xid = match xid_opt {
+ Some(x) => x,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || {
+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
+ })
+ .await
+ .unwrap_or_default();
+ // pid omitted: the keys go to the active window, like a
+ // physical keyboard would, and the action adopts its pid.
+ let chosen = if pid == 0 {
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active && w.pid.is_some())
+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
+ } else {
+ // The pid's active window, else its largest on-screen
+ // toplevel: LibreOffice and GIMP own hidden/utility
+ // toplevels that a plain `first()` could pick, and an
+ // unmapped window cannot take the virtual keyboard focus.
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active)
+ .or_else(|| {
+ windows
+ .iter()
+ .filter(|w| w.is_on_screen)
+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
+ })
+ .or_else(|| windows.first())
+ };
+ match chosen {
+ Some(w) => {
+ if pid == 0 {
+ pid = w.pid.unwrap_or(0);
+ }
+ w.xid
+ }
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ let isolated_background = isolated_hyprland_background(delivery);
+ let pixel_target = match (
+ args.get("x").and_then(|value| value.as_f64()),
+ args.get("y").and_then(|value| value.as_f64()),
+ ) {
+ (Some(x), Some(y)) if desktop_frame_requested(&args) => {
+ // Desktop-frame pixels (from get_desktop_state) against a named
+ // window: map into the window-local frame the pipeline expects.
+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await
+ {
+ Ok(Ok(local)) => Some(local),
+ Ok(Err(e)) => return desktop_frame_unmapped(xid, e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+ (Some(x), Some(y)) => {
+ // Pixel targets use the latest screenshot's coordinate frame.
+ // Apply the same buffer-to-window ratio as click/drag before
+ // positioning either the agent cursor or the input device.
+ // An unsupported native background delivery refuses first.
+ let native_refusal = (!isolated_background)
+ .then(|| unavailable_wayland_focused_input_background(delivery, true))
+ .flatten();
+ let ratio = match coordinate_scroll_scale(native_refusal, || {
+ screenshot_scale(&self.state, &args, pid, Some(xid))
+ }) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ Some((x * ratio, y * ratio))
+ }
+ (None, None) => None,
+ (Some(_), None) => return invalid_pointer_arguments("scroll", "x was given without y"),
+ (None, Some(_)) => return invalid_pointer_arguments("scroll", "y was given without x"),
+ };
+ let resolved_element_index = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ if pixel_target.is_some() && resolved_element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to scroll, not both.",
+ );
+ }
+
+ if named_session_cursor_key(&args).is_some() {
+ let element = token.clone();
+ let visual_target = tokio::task::spawn_blocking(move || {
+ explicit_keyboard_cursor_target(pid, xid, element.as_ref(), pixel_target)
+ .or_else(|| keyboard_window_center(xid))
+ })
+ .await
+ .ok()
+ .flatten();
+ if let Some((sx, sy)) = visual_target {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, false).await;
+ }
+ }
+
+ if !isolated_background {
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ }
+ if let Some(element) = token.clone() {
+ let idx = element.index;
+ // WebKitGTK acknowledges the AT-SPI scroll action without moving
+ // the DOM scroller. On native Wayland, use a real compositor wheel
+ // event at the resolved element instead of reporting a silent
+ // success. Other AX targets keep their semantic route before any
+ // coordinate-based compositor fallback.
+ if !(crate::wayland::wayland_input_enabled() && is_webkitgtk_embedder(pid)) {
+ let direction_for_ax = direction.clone();
+ let ax_result = tokio::task::spawn_blocking(move || {
+ crate::atspi::scroll_token(pid, &element, &direction_for_ax, amount, by)
+ })
+ .await;
+ if let Ok(Err(error)) = &ax_result {
+ // The observed element is gone: nothing was scrolled.
+ if let Some(stale) = stale_observed_element("scroll", idx, error) {
+ return stale;
+ }
+ }
+ match ax_result {
+ Ok(Ok(progress)) => {
+ return atspi_scroll_result(progress, delivery.is_foreground())
+ }
+ Err(error) => {
+ return atspi_scroll_result(
+ crate::atspi::ScrollProgress {
+ acknowledged: 0,
+ complete: false,
+ detail: Some(error.to_string()),
+ },
+ delivery.is_foreground(),
+ )
+ }
+ Ok(Err(_)) => {}
+ }
+ }
+ }
+ if hyprland_foreground(delivery) {
+ if xid_opt.is_none() {
+ return foreground_hyprland_refusal(
+ "an exact window_id or window-bound element token is required",
+ );
+ }
+ let point = if let Some(element) = token.clone() {
+ match tokio::task::spawn_blocking(move || {
+ resolve_element_local_coords(pid, &element, Some(xid))
+ })
+ .await
+ {
+ Ok(Ok((_, x, y))) => Some((x, y)),
+ _ => {
+ return foreground_hyprland_refusal(
+ "could not resolve exact element coordinates",
+ )
+ }
+ }
+ } else {
+ pixel_target
+ };
+ // `by: "page"` is the documented 3-detent approximation, not an
+ // exact viewport; large requests become several wheel packets.
+ let actions =
+ match crate::wayland::hyprland_input::scroll_actions(point, &direction, amount, by)
+ {
+ Ok(actions) => actions,
+ Err(error) => return foreground_hyprland_refusal(error.to_string()),
+ };
+ return foreground_hyprland_scroll(&args, pid, xid, actions).await;
+ }
+ if isolated_background {
+ if xid_opt.is_none() {
+ return isolated_hyprland_refusal(
+ "an exact window_id or window-bound element token is required",
+ );
+ }
+ let owner = named_session_cursor_key(&args);
+ let element = token.clone();
+ let (_cancellation, dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ let point = if let Some(element) = &element {
+ let (_, x, y) = resolve_element_local_coords(pid, element, Some(xid))?;
+ Some((x, y))
+ } else {
+ pixel_target
+ };
+ let actions = crate::wayland::hyprland_input::scroll_actions(
+ point, &direction, amount, by,
+ )?;
+ crate::wayland::hyprland_input::execute_scroll(
+ owner,
+ pid,
+ xid,
+ actions,
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(refusal) => return refusal,
+ };
+ return match dispatch.await {
+ Ok(result) => isolated_hyprland_result(result),
+ Err(error) => isolated_hyprland_task_error(error, false),
+ };
+ }
+
+ if crate::wayland::is_inject_mode() {
+ let Some((x, y)) = pixel_target else {
+ return crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ };
+ let direction_for_inject = direction.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_scroll(pid, xid, x, y, &direction_for_inject, amount as u32)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Scrolled {direction} {amount} ticks (focus-free via cua-compositor)."
+ ))
+ .with_structured(json!({
+ "verified": false,
+ "delivery_mode": if delivery.is_foreground() { "foreground" } else { "background" },
+ "route": "cua_compositor_inject"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ if crate::wayland::wayland_input_enabled() {
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, false) {
+ return refusal;
+ }
+ let direction_for_wayland = direction.clone();
+ let local_point = match (pixel_target, token.clone()) {
+ (Some(point), _) => Some(point),
+ (None, Some(element)) => {
+ let idx = element.index;
+ match tokio::task::spawn_blocking(move || {
+ resolve_element_local_coords(pid, &element, Some(xid))
+ })
+ .await
+ {
+ Ok(Ok((_resolved_xid, x, y))) => Some((x, y)),
+ Ok(Err(error)) => {
+ return ToolResult::error(format!(
+ "Could not resolve scroll element [{idx}] coordinates: {error}"
+ ))
+ }
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Scroll element coordinate task failed: {error}"
+ ))
+ }
+ }
+ }
+ (None, None) => None,
+ };
+ let output_point = match local_point
+ .map(|(x, y)| {
+ crate::wayland::window_local_to_output(xid, x.round() as i32, y.round() as i32)
+ })
+ .transpose()
+ {
+ Ok(point) => point,
+ Err(error) => return input_error_result(error),
+ };
+ let ticks = wheel_ticks(amount, by, crate::wayland::niri::is_session());
+ let result =
+ spawn_blocking_bounded("scroll", foreground_budget(ticks as usize), move || {
+ crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::pointer_gesture_restore(),
+ || {
+ crate::wayland::scroll_at(
+ xid,
+ output_point,
+ &direction_for_wayland,
+ ticks,
+ )
+ },
+ )
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Scrolled {direction} {ticks} ticks (delivery_mode=foreground)."
+ ))
+ .with_structured(json!({ "verified": false, "delivery_mode": "foreground" })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ if let Some(refusal) = unavailable_webkit_background(pid, delivery) {
+ return refusal;
+ }
+ if let Some(refusal) = unavailable_gtk_pointer_background(pid, delivery) {
+ return refusal;
+ }
+
+ // An element-addressed scroll must land over the element. The old
+ // fallback used (0, 0) in the window, which can report success while
+ // Chromium/GTK routes the wheel to the toplevel instead of the
+ // requested scroll region.
+ let element_point = match token {
+ Some(element) => {
+ let idx = element.index;
+ match tokio::task::spawn_blocking(move || {
+ resolve_element_local_coords(pid, &element, Some(xid))
+ })
+ .await
+ {
+ Ok(Ok((_resolved_xid, local_x, local_y))) => {
+ let screen = window_local_to_screen(xid, local_x, local_y).ok();
+ Some(((local_x, local_y), screen))
+ }
+ Ok(Err(e)) => {
+ return ToolResult::error(format!(
+ "Could not resolve scroll element [{idx}] coordinates: {e}"
+ ))
+ }
+ Err(e) => {
+ return ToolResult::error(format!(
+ "Scroll element coordinate task failed: {e}"
+ ))
+ }
+ }
+ }
+ None => pixel_target.map(|local| {
+ let screen = window_local_to_screen(xid, local.0, local.1).ok();
+ (local, screen)
+ }),
+ };
+
+ // X11 scroll buttons: 4=up, 5=down, 6=left, 7=right
+ // Note: "page" scroll is still per-click on X11; send more ticks for page.
+ let button: u8 = match direction.as_str() {
+ "up" => 4,
+ "left" => 6,
+ "right" => 7,
+ _ => 5,
+ };
+ let cursor_id_for_task = cursor_id.clone();
+ let direction_for_wayland = direction.clone();
+ let amount_u32 = amount as u32;
+ let result = spawn_blocking_bounded(
+ "scroll",
+ foreground_budget(amount_u32 as usize),
+ move || -> anyhow::Result<()> {
+ if crate::wayland::wayland_input_enabled() {
+ return crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::pointer_gesture_restore(),
+ || crate::wayland::scroll(xid, &direction_for_wayland, amount_u32),
+ );
+ }
+ // foreground: activate the window, then scroll, then restore — for
+ // surfaces that only route wheel events to the active window.
+ let x11_scroll = || -> anyhow::Result<()> {
+ // X11: synthetic Button4-7 XSendEvents are dropped by XInput2
+ // toolkits (GTK never scrolls). On a real Xorg host, drive a real
+ // wheel detent through the MPX uinput pointer over the window's
+ // center — libinput turns it into the XI2 smooth-scroll GTK reads —
+ // without stealing focus. Falls back to the legacy XSendEvent
+ // Button4-7 path on Xvfb / unsupported servers.
+ // Button → axis/sign: 4=up(+v) 5=down(-v) 6=left(-h) 7=right(+h).
+ if crate::input::real_pointer_input_available() {
+ let pointer_point = element_point
+ .and_then(|(_, screen)| screen)
+ .map(|(x, y)| (x as i32, y as i32))
+ .or_else(|| window_screen_center(xid).ok());
+ if let Some((cx, cy)) = pointer_point {
+ let horizontal = matches!(button, 6 | 7);
+ let ticks = match button {
+ 4 | 7 => amount as i32,
+ _ => -(amount as i32), // 5 (down) and 6 (left)
+ };
+ match crate::input::send_virtual_pointer_scroll(
+ &cursor_id_for_task,
+ &crate::input::VirtualPointerScroll {
+ target_window: xid,
+ x: cx,
+ y: cy,
+ horizontal,
+ ticks,
+ },
+ ) {
+ Ok(()) => return Ok(()),
+ Err(error) if crate::input::is_uinput_unavailable(&error) => {
+ return Err(error)
+ }
+ Err(e) => tracing::warn!("MPX scroll fell back to XSendEvent: {e}"),
+ }
+ }
+ }
+ let (local_x, local_y) =
+ element_point.map(|(local, _)| local).unwrap_or((0.0, 0.0));
+ crate::input::send_click(xid, local_x as i32, local_y as i32, amount, button)
+ };
+ if delivery.is_foreground() {
+ let point = element_point
+ .and_then(|(_, screen)| screen)
+ .or_else(|| {
+ window_screen_center(xid)
+ .ok()
+ .map(|(x, y)| (x as f64, y as f64))
+ })
+ .ok_or_else(|| {
+ anyhow::anyhow!("could not resolve foreground scroll point")
+ })?;
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ crate::input::send_click_xtest_desktop(
+ point.0 as i32,
+ point.1 as i32,
+ button,
+ amount,
+ )
+ },
+ )
+ .map(|(value, _report)| value)
+ } else {
+ x11_scroll()
+ }
+ },
+ )
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Scrolled {direction} {amount} ticks (delivery_mode={mode_label})."
+ ))
+ .with_structured(json!({ "verified": false, "delivery_mode": mode_label })),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/set_config.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_config.rs
new file mode 100644
index 0000000000..27103b56dd
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_config.rs
@@ -0,0 +1,142 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+// ── set_config ────────────────────────────────────────────────────────────────
+
+fn with_cursor_motion_config_properties(mut schema: Value) -> Value {
+ if let Some(properties) = schema.get_mut("properties").and_then(Value::as_object_mut) {
+ properties.extend(cursor_overlay::motion_defaults::config_schema_properties());
+ properties.extend(cursor_overlay::badge_mode::config_schema_properties());
+ properties.extend(cursor_overlay::cursor_options::config_schema_properties());
+ }
+ schema
+}
+
+pub struct SetConfigTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static SCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetConfigTool {
+ fn def(&self) -> &ToolDef {
+ SCFG_DEF.get_or_init(|| ToolDef {
+ name: "set_config".into(),
+ description: "Update cua-driver-rs configuration. capture_mode / \
+ max_image_dimension take effect immediately.\n\n\
+ Pass each setting as its own field: `{\"capture_mode\": \"som\", \"max_image_dimension\": 0}`.\n\n\
+ The experimental_pip keys persist to ~/.cua-driver/config.json and apply on next \
+ daemon restart (the PiP backend is initialised once at startup; \
+ Linux ships only the trait stub today — see issue #1729).\n\n\
+ `cursor.motion.style`, `cursor.motion.timing` and `cursor.motion.effects.<name>` \
+ (`on`, `off`, or `default`) save the default cursor motion for sessions started afterwards. \
+ `cursor.motion: \"default\"` clears every saved cursor motion default.\n\n\
+ `cursor.badges` (`minimal` default, `verbose`, or `off`) sets the agent cursor badge \
+ for every cursor at once.\n\n\
+ `cursor.float` (resting float) is `on` (default), `off` or `default`; \
+ `cursor.marks` (action marks) and `cursor.rotate` (turn toward travel) are `on`, \
+ `off` (default) or `default`. All apply to every cursor at once.".into(),
+ input_schema: with_cursor_motion_config_properties(json!({"type":"object","properties":{
+ "capture_mode":{"type":"string","enum":["ax","vision"],"description":"Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
+ "max_image_dimension":{"type":"integer","description":"Max dimension for screenshot resizing (0 = no limit)."},
+ "experimental_pip":{"type":"boolean","description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed)."},
+ "experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form."}
+ },"additionalProperties":false})),
+ read_only: false, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.get("capture_scope").is_some() {
+ return ToolResult::error(
+ "config key 'capture_scope' is retired; select a window or desktop target on each action",
+ )
+ .with_structured(json!({
+ "code": "config_key_retired",
+ "key": "capture_scope",
+ "replacement": "action.target",
+ }));
+ }
+ let motion_keys = match cursor_overlay::motion_defaults::apply_config_args(&args) {
+ Ok(keys) => keys,
+ Err(message) => return ToolResult::error(message),
+ };
+ let badge_key = match cursor_overlay::badge_mode::apply_config_args(&args) {
+ Ok(key) => key,
+ Err(message) => return ToolResult::error(message),
+ };
+ let option_keys = match cursor_overlay::cursor_options::apply_config_args(&args) {
+ Ok(keys) => keys,
+ Err(message) => return ToolResult::error(message),
+ };
+ let mut cfg = self.state.config.write().unwrap();
+ let mut parts: Vec<String> = motion_keys
+ .iter()
+ .map(|key| format!("{key} (applies to sessions started from now on)"))
+ .collect();
+ parts.extend(
+ badge_key
+ .into_iter()
+ .chain(option_keys)
+ .map(|key| format!("{key} (applies now)")),
+ );
+ if let Some(mode) = args.opt_str("capture_mode") {
+ if let Err(e) =
+ pip_preview::write_config_key("capture_mode", Value::String(mode.clone()))
+ {
+ tracing::warn!("set_config: failed to persist capture_mode: {e}");
+ }
+ parts.push(format!("capture_mode={mode}"));
+ cfg.capture_mode = mode;
+ }
+ if let Some(dim) = args.opt_u64("max_image_dimension") {
+ cfg.max_image_dimension = dim as u32;
+ if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
+ tracing::warn!("set_config: failed to persist max_image_dimension: {e}");
+ }
+ parts.push(format!("max_image_dimension={dim}"));
+ }
+ if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
+ {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
+ }
+ parts.push(format!("experimental_pip={enabled} (next restart)"));
+ }
+ if let Some(geom) = args.opt_str("experimental_pip_geometry") {
+ if pip_preview::PipGeometry::parse(&geom).is_none() {
+ return ToolResult::error(format!(
+ "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
+ ));
+ }
+ if let Err(e) = pip_preview::write_config_key(
+ "experimental_pip_geometry",
+ Value::String(geom.clone()),
+ ) {
+ return ToolResult::error(format!(
+ "failed to persist experimental_pip_geometry: {e}"
+ ));
+ }
+ parts.push(format!("experimental_pip_geometry={geom} (next restart)"));
+ }
+ let msg = if parts.is_empty() {
+ "Config unchanged (no known parameters).".to_owned()
+ } else {
+ format!("Config updated: {}", parts.join(", "))
+ };
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
+ ToolResult::text(msg).with_structured(json!({
+ "capture_mode": cfg.capture_mode,
+ "max_image_dimension": cfg.max_image_dimension,
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry
+ }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_value.rs
new file mode 100644
index 0000000000..38961bbf32
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_value.rs
@@ -0,0 +1,430 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ attach_focus_guard, background_virtual_key, background_virtual_text, foreground_budget,
+ input_error_result, non_ax_escalation, position_named_session_keyboard_cursor,
+ prove_token_element, resolve_cursor_key, resolve_element_local_coords, spawn_blocking_bounded,
+ stale_observed_element, values_agree, window_local_to_screen, x11_pixel_click_no_focus_steal,
+ ToolState,
+};
+
+// ── set_value ─────────────────────────────────────────────────────────────────
+
+pub struct SetValueTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static SV_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetValueTool {
+ fn def(&self) -> &ToolDef {
+ SV_DEF.get_or_init(|| ToolDef {
+ name: "set_value".into(),
+ description: "Set value of an AT-SPI element via SetValue action.".into(),
+ input_schema: json!({
+ "type":"object","required":["value"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"Omit when element_token is supplied (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "value":{"type":"string","description":"New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let value = match args.get("value") {
+ Some(Value::String(text)) => text.clone(),
+ Some(Value::Number(number)) => number.to_string(),
+ _ => {
+ return ToolResult::error("set_value: 'value' must be a string or a number.")
+ .with_structured(json!({ "code": "invalid_arguments" }))
+ }
+ };
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "set_value").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let (token, xid) = match (token, &resolved) {
+ (
+ Some(token),
+ cua_driver_core::element_token::ResolvedElement::Element { window_id, .. },
+ ) => (token, *window_id),
+ _ => {
+ return ToolResult::error(
+ "set_value requires element_token to address the target element.",
+ )
+ }
+ };
+ let idx = token.index;
+ let value_for_task = value.clone();
+ let xid_opt = Some(xid);
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ let cursor_id = resolve_cursor_key(&args);
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ Some(token.clone()),
+ None,
+ true,
+ )
+ .await;
+ // 1. Focus-free accessibility write on the exact snapshot object.
+ let target = token.clone();
+ let ax = spawn_blocking_bounded("set_value", foreground_budget(0), move || {
+ crate::atspi::set_value_token(pid, &target, &value_for_task)
+ })
+ .await;
+ let ax_error = match ax {
+ Ok(Ok(())) => {
+ let target = token.clone();
+ let readback =
+ tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&target))
+ .await
+ .ok()
+ .flatten();
+ return set_value_result(idx, &value, "ax", readback, None);
+ }
+ Ok(Err(error)) if crate::atspi::native::is_no_value_route(&error) => error,
+ Ok(Err(error)) if crate::atspi::native::is_no_such_option(&error) => {
+ return no_such_option_result(idx, &error)
+ }
+ Ok(Err(error)) => {
+ return stale_observed_element("set_value", idx, &error)
+ .unwrap_or_else(|| input_error_result(error))
+ }
+ Err(error) => return ToolResult::error(format!("Task error: {error}")),
+ };
+ // 2. No accessibility write route. A combo box is opened with a click
+ // and the option picked from its popup (whose items only exist and
+ // act while it is open), then the popup is closed. Anything else is
+ // typed like a user would: click the field (real pointer), select
+ // all, type, commit with Tab.
+ let is_combo = token.observed.role.eq_ignore_ascii_case("combo box");
+ let wayland = crate::wayland::wayland_input_enabled();
+ let background_route = !delivery.is_foreground()
+ && !wayland
+ && crate::input::real_pointer_input_available()
+ && crate::input::real_keyboard_input_available();
+ let foreground_route = delivery.is_foreground() && !wayland;
+ if !background_route && !foreground_route {
+ let detail = format!(
+ "{ax_error}; the keyboard fallback needs {}",
+ if wayland {
+ "an X11 session"
+ } else {
+ "the focus-free real-input route (a writable /dev/uinput), or delivery_mode:\"foreground\""
+ }
+ );
+ let mut structured = json!({
+ "code": "set_value_unavailable",
+ "detail": detail,
+ "effect": "none",
+ });
+ // Only X11 can run the foreground keyboard fallback.
+ if !wayland {
+ structured["escalation"] = json!({ "recommended": "foreground", "reason": "click the field and type the value with delivery_mode:\"foreground\"" });
+ }
+ return ToolResult::error(format!(
+ "set_value: element [{idx}] has no accessibility write route and no keyboard \
+ fallback is available: {detail}."
+ ))
+ .with_structured(structured);
+ }
+ let value_for_keys = value.clone();
+ let target = token.clone();
+ let typed = spawn_blocking_bounded(
+ "set_value keyboard fallback",
+ foreground_budget(value.len()),
+ move || -> anyhow::Result<Option<crate::input::FocusGuardReport>> {
+ let (win, lx, ly) = resolve_element_local_coords(pid, &target, xid_opt)?;
+ if is_combo {
+ return pick_from_popup(
+ &cursor_id,
+ pid,
+ &target,
+ (win, lx, ly),
+ &value_for_keys,
+ foreground_route,
+ );
+ }
+ if foreground_route {
+ let (sx, sy) = window_local_to_screen(win, lx, ly)?;
+ crate::interaction::with_x11_foreground(
+ win,
+ crate::input::ForegroundOptions::pointer(),
+ || {
+ crate::input::send_click_xtest_desktop(
+ sx.round() as i32,
+ sy.round() as i32,
+ 1,
+ 1,
+ )?;
+ std::thread::sleep(std::time::Duration::from_millis(80));
+ crate::input::send_key_xtest("a", &["ctrl"])?;
+ crate::input::send_type_text_xtest(&value_for_keys)?;
+ crate::input::send_key_xtest("Tab", &[])
+ },
+ )?;
+ return Ok(None);
+ }
+ let (_, guard) = crate::input::focus_guard::guarded(Some(pid), || {
+ x11_pixel_click_no_focus_steal(&cursor_id, win, lx as i32, ly as i32, 1, 1)?;
+ std::thread::sleep(std::time::Duration::from_millis(80));
+ background_virtual_key(&cursor_id, pid, win, "a", &["ctrl"])?;
+ background_virtual_text(&cursor_id, pid, win, &value_for_keys)?;
+ background_virtual_key(&cursor_id, pid, win, "Tab", &[])?;
+ Ok(())
+ })?;
+ Ok(guard)
+ },
+ )
+ .await;
+ match typed {
+ Ok(Ok(guard)) => {
+ // Let the toolkit commit on focus-out before reading back.
+ tokio::time::sleep(std::time::Duration::from_millis(150)).await;
+ let readback =
+ tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&token))
+ .await
+ .ok()
+ .flatten();
+ set_value_result(
+ idx,
+ &value,
+ match (is_combo, foreground_route) {
+ (true, true) => "click_select_fg",
+ (true, false) => "click_select_mpx",
+ (false, true) => "click_type_fg",
+ (false, false) => "click_type_mpx",
+ },
+ readback,
+ guard.as_ref(),
+ )
+ }
+ Ok(Err(error)) if crate::atspi::native::is_no_such_option(&error) => {
+ no_such_option_result(idx, &error)
+ }
+ Ok(Err(error)) => input_error_result(error),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ }
+ }
+}
+
+/// `set_value` named an option the combo box or list does not have.
+fn no_such_option_result(idx: usize, error: &anyhow::Error) -> ToolResult {
+ let detail = error.to_string();
+ let detail = detail
+ .strip_prefix(crate::atspi::native::NO_SUCH_OPTION)
+ .map(|rest| rest.trim_start_matches(':').trim())
+ .unwrap_or(&detail)
+ .to_owned();
+ ToolResult::error(format!(
+ "set_value: element [{idx}] has {detail}; nothing was changed."
+ ))
+ .with_structured(json!({ "code": "option_not_found", "detail": detail, "effect": "none" }))
+}
+
+/// How long `set_value` waits for an opened combo box popup to list its
+/// options.
+const POPUP_OPTIONS_WAIT: std::time::Duration = std::time::Duration::from_millis(1500);
+
+/// Keyboard-fallback `set_value` on a combo box whose options cannot be
+/// selected while it is closed: click it open (the real pointer: XTest in
+/// the foreground, the session's MPX pointer in the background), activate
+/// the option `value` names from the popup, and press Escape if the popup
+/// is still open afterwards (an unknown option, or a toolkit that keeps it
+/// open), so no popup is left behind.
+fn pick_from_popup(
+ cursor_id: &str,
+ pid: u32,
+ token: &crate::atspi::TokenElement,
+ (win, lx, ly): (u64, f64, f64),
+ value: &str,
+ foreground: bool,
+) -> anyhow::Result<Option<crate::input::FocusGuardReport>> {
+ let close_popup = || -> anyhow::Result<()> {
+ if !crate::atspi::options_showing_token(token) {
+ return Ok(());
+ }
+ if foreground {
+ crate::interaction::with_x11_foreground(
+ win,
+ crate::input::ForegroundOptions::keyboard(),
+ || crate::input::send_key_xtest("Escape", &[]),
+ )
+ .map(|_| ())
+ } else {
+ background_virtual_key(cursor_id, pid, win, "Escape", &[]).map(|_| ())
+ }
+ };
+ let pick = || -> anyhow::Result<String> {
+ let deadline = std::time::Instant::now() + POPUP_OPTIONS_WAIT;
+ loop {
+ match crate::atspi::activate_option_token(token, value) {
+ Ok(Some(name)) => return Ok(name),
+ Ok(None) if std::time::Instant::now() < deadline => {
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ }
+ Ok(None) => anyhow::bail!("the combo box popup listed no options"),
+ Err(error) => return Err(error),
+ }
+ }
+ };
+ let open_and_pick = || -> anyhow::Result<()> {
+ let picked = pick();
+ // Let the toolkit close the popup on activation before checking.
+ std::thread::sleep(std::time::Duration::from_millis(150));
+ let closed = close_popup();
+ picked?;
+ closed
+ };
+ if foreground {
+ let (sx, sy) = window_local_to_screen(win, lx, ly)?;
+ crate::interaction::with_x11_foreground(
+ win,
+ crate::input::ForegroundOptions::pointer(),
+ || crate::input::send_click_xtest_desktop(sx.round() as i32, sy.round() as i32, 1, 1),
+ )?;
+ open_and_pick()?;
+ return Ok(None);
+ }
+ let (_, guard) = crate::input::focus_guard::guarded(Some(pid), || {
+ x11_pixel_click_no_focus_steal(cursor_id, win, lx as i32, ly as i32, 1, 1)?;
+ open_and_pick()
+ })?;
+ Ok(guard)
+}
+
+/// `set_value` result with an honest verification: `verified` only when the
+/// element read back the requested value.
+fn set_value_result(
+ idx: usize,
+ value: &str,
+ path: &str,
+ readback: Option<String>,
+ guard: Option<&crate::input::FocusGuardReport>,
+) -> ToolResult {
+ let normalised_path = readback
+ .as_deref()
+ .is_some_and(|seen| path_normalised_to_basename(value, seen));
+ let verified = readback
+ .as_deref()
+ .is_some_and(|seen| values_agree(value, seen))
+ || normalised_path;
+ let mut structured = json!({
+ "path": path,
+ "verified": verified,
+ "effect": if verified { "confirmed" } else if readback.is_some() { "suspected_noop" } else { "unverifiable" },
+ });
+ let mut text = format!("Set value of element [{idx}] to '{value}' (path={path})");
+ match readback {
+ Some(seen) if normalised_path => {
+ structured["readback"] = json!(seen);
+ text.push_str(&format!(
+ "; the field reads back '{seen}': a file chooser navigated to the path's \
+ directory and kept the file name (the write committed)."
+ ));
+ }
+ Some(seen) if verified => text.push_str(&format!("; read back '{seen}'.")),
+ Some(seen) => {
+ structured["readback"] = json!(seen);
+ structured["escalation"] = non_ax_escalation();
+ text.push_str(&format!(
+ "; but the element reads back '{seen}' — the write may not have committed. \
+ Confirm with get_window_state or a screenshot."
+ ));
+ }
+ None => {
+ structured["readback"] = Value::Null;
+ text.push_str("; not verified — confirm with get_window_state.");
+ }
+ }
+ attach_focus_guard(ToolResult::text(text).with_structured(structured), guard)
+}
+
+/// A Qt/GTK file chooser given an absolute path in its "File name" field
+/// moves to that directory and shows only the final component: the readback
+/// is the basename, and the write committed.
+fn path_normalised_to_basename(value: &str, seen: &str) -> bool {
+ let value = value.trim();
+ let seen = seen.trim();
+ value.contains('/')
+ && !seen.is_empty()
+ && value
+ .rsplit('/')
+ .next()
+ .is_some_and(|base| !base.is_empty() && base == seen)
+}
+
+#[cfg(test)]
+mod set_value_tests {
+ use super::*;
+
+ #[test]
+ fn a_file_chooser_keeping_the_basename_counts_as_committed() {
+ assert!(path_normalised_to_basename(
+ "/home/user/Desktop/tone.wav",
+ "tone.wav"
+ ));
+ assert!(!path_normalised_to_basename(
+ "/home/user/Desktop/tone.wav",
+ "other.wav"
+ ));
+ assert!(!path_normalised_to_basename("tone.wav", "tone.wav"));
+ let ok = set_value_result(
+ 3,
+ "/home/user/Desktop/tone.wav",
+ "ax",
+ Some("tone.wav".into()),
+ None,
+ );
+ let s = ok.structured_content.unwrap();
+ assert_eq!(s["verified"], true);
+ assert_eq!(s["effect"], "confirmed");
+ }
+
+ #[test]
+ fn readback_agreement_is_numeric_aware() {
+ assert!(values_agree("40", "40.0"));
+ assert!(values_agree("40", " 40 "));
+ assert!(values_agree("abc", "abc"));
+ assert!(!values_agree("40", "41"));
+ assert!(!values_agree("40", "forty"));
+ }
+
+ #[test]
+ fn set_value_result_is_verified_only_on_matching_readback() {
+ let ok = set_value_result(3, "40", "click_type_mpx", Some("40.0".into()), None);
+ let s = ok.structured_content.unwrap();
+ assert_eq!(s["verified"], true);
+ assert_eq!(s["effect"], "confirmed");
+ let noop = set_value_result(3, "40", "ax", Some("10.0".into()), None);
+ let s = noop.structured_content.unwrap();
+ assert_eq!(s["verified"], false);
+ assert_eq!(s["effect"], "suspected_noop");
+ assert_eq!(s["readback"], "10.0");
+ let blind = set_value_result(3, "40", "ax", None, None);
+ assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/set_window_frame.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_window_frame.rs
new file mode 100644
index 0000000000..aa4b309e06
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/set_window_frame.rs
@@ -0,0 +1,145 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::Value;
+
+// ── set_window_frame (Linux) ─────────────────────────────────────────────────
+
+pub struct SetWindowFrameTool;
+
+static SET_WINDOW_FRAME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetWindowFrameTool {
+ fn def(&self) -> &ToolDef {
+ SET_WINDOW_FRAME_DEF.get_or_init(|| {
+ let contract = cua_driver_contract::tool_contract("set_window_frame")
+ .expect("set_window_frame contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_contract::SetWindowFrameInput;
+ use cua_driver_core::action_record::{
+ effect_from_value_readback, ActionEvidence, ActionExecutionRecord, ActionTransport,
+ ActualDelivery, EvidenceKind, RequestedDelivery,
+ };
+
+ let input: SetWindowFrameInput =
+ match cua_driver_core::tool_args::parse_typed_input("set_window_frame", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ if crate::wayland::is_wayland() {
+ return ToolResult::error(
+ "set_window_frame: this Wayland compositor exposes no protocol that permits a client to set another top-level window's geometry",
+ );
+ }
+ let values = [input.x, input.y, input.width, input.height];
+ if values.iter().any(|value| !value.is_finite())
+ || input.width <= 0.0
+ || input.height <= 0.0
+ {
+ return ToolResult::error(
+ "set_window_frame: x/y must be finite and width/height must be finite positive numbers",
+ );
+ }
+ let to_i32 = |name: &str, value: f64| -> Result<i32, String> {
+ let rounded = value.round();
+ if rounded < i32::MIN as f64 || rounded > i32::MAX as f64 {
+ Err(format!("{name} is out of range for an X11 coordinate"))
+ } else {
+ Ok(rounded as i32)
+ }
+ };
+ let to_u32 = |name: &str, value: f64| -> Result<u32, String> {
+ let rounded = value.round();
+ if rounded < 1.0 || rounded > u32::MAX as f64 {
+ Err(format!("{name} is out of range for an X11 size"))
+ } else {
+ Ok(rounded as u32)
+ }
+ };
+ let x = match to_i32("x", input.x) {
+ Ok(value) => value,
+ Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
+ };
+ let y = match to_i32("y", input.y) {
+ Ok(value) => value,
+ Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
+ };
+ let width = match to_u32("width", input.width) {
+ Ok(value) => value,
+ Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
+ };
+ let height = match to_u32("height", input.height) {
+ Ok(value) => value,
+ Err(error) => return ToolResult::error(format!("set_window_frame: {error}")),
+ };
+ let window_id = input.window_id;
+ let pid = input.pid;
+ let outcome = tokio::task::spawn_blocking(move || {
+ let before = crate::x11::list_windows(Some(pid))
+ .into_iter()
+ .find(|window| window.xid == window_id)
+ .map(|window| (window.x, window.y, window.width, window.height));
+ let (observed, confirmed, mutation_error) =
+ crate::x11::set_window_frame(window_id, pid, x, y, width, height)
+ .map_err(|error| error.to_string())?;
+ let observed_frame =
+ observed.map(|window| (window.x, window.y, window.width, window.height));
+ let changed = observed_frame.is_some_and(|observed| before != Some(observed));
+ Ok::<_, String>((observed_frame, confirmed, changed, mutation_error))
+ })
+ .await;
+ let (observed, confirmed, changed, mutation_error) = match outcome {
+ Ok(Ok(outcome)) => outcome,
+ Ok(Err(error)) => return ToolResult::error(format!("set_window_frame: {error}")),
+ Err(error) => {
+ return ToolResult::error(format!(
+ "set_window_frame: blocking task failed: {error}"
+ ));
+ }
+ };
+ let effect = effect_from_value_readback(confirmed, changed, observed.is_some());
+ let requested = (x, y, width, height);
+ let mut record = ActionExecutionRecord::builder(
+ effect,
+ ActionTransport::LinuxX11ConfigureWindow,
+ RequestedDelivery::NotApplicable,
+ )
+ .actual_delivery(ActualDelivery::NotApplicable)
+ .detail(format!(
+ "requested={requested:?} observed={observed:?} mutation_error={mutation_error:?}"
+ ));
+ if observed.is_some() {
+ record = record.evidence(ActionEvidence {
+ kind: EvidenceKind::ValueReadback,
+ detail: if confirmed {
+ "X11 geometry readback matched the requested frame".into()
+ } else {
+ "X11 geometry readback did not match the requested frame".into()
+ },
+ });
+ }
+ ToolResult::text(if confirmed {
+ "Set and verified the requested window frame."
+ } else if observed.is_none() {
+ "The native window mutation was attempted, but its resulting frame could not be read back."
+ } else {
+ "The window frame did not settle at the requested geometry."
+ })
+ .with_action_record(record.build().expect("set_window_frame record is valid"))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text.rs
new file mode 100644
index 0000000000..0d1074d5f9
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text.rs
@@ -0,0 +1,1170 @@
+use async_trait::async_trait;
+use cua_driver_contract::TypeTextInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::{fs, sync::Arc};
+
+use super::impl_::{
+ announce_keyboard_target, atspi_up, background_keyboard_refusal, background_virtual_text,
+ coordinate_frame_is_desktop, coordinate_frame_schema, focus_by_pixel,
+ focus_hyprland_foreground, focus_nested_inject_target, foreground_hyprland_refusal,
+ foreground_structured, hyprland_foreground, hyprland_input_result, inject_terminal_input,
+ input_error_result, invalid_pointer_arguments, is_chromium_embedder, is_popup_keyboard_grab,
+ is_terminal_process, is_webkitgtk_embedder, isolated_hyprland_background,
+ isolated_hyprland_refusal, isolated_hyprland_result, isolated_hyprland_task_error,
+ named_session_cursor_key, position_named_session_keyboard_cursor, prove_token_element,
+ resolve_cursor_key, spawn_blocking_bounded, spawn_isolated_hyprland, stale_observed_element,
+ typing_budget, unavailable_chromium_background, unavailable_wayland_focused_input_background,
+ unavailable_webkit_keyboard_background, values_agree, KeyRoute, ToolState,
+};
+
+/// Structured payload for a `type_text` response. Keeps the legacy
+/// `path`/`characters`/`verified` fields for back-compat and adds the cross-tool
+/// `effect` tri-state. Linux's AT-SPI `insertText` return value acknowledges
+/// the method call but does not read the widget value back, so it and every
+/// keystroke / XSendEvent / XTest / Wayland rung are `"unverifiable"` (the
+/// caller confirms through a separate observation). No escalation is
+/// attached to a successful delivery: the public projection renders one as
+/// `delivery_failed`, which is wrong for text that landed but was not read
+/// back. Mirrors the macOS `type_text` contract.
+fn type_text_structured(path: &str, characters: usize, verified: bool) -> Value {
+ let s = json!({
+ "path": path,
+ "characters": characters,
+ "verified": verified,
+ "effect": if verified { "confirmed" } else { "unverifiable" },
+ });
+ // No escalation on a successful-but-unverified delivery: the public
+ // projection renders a foreground escalation as `delivery_failed`, which
+ // made models re-type text that had already landed. Refusals carry their own.
+ let _ = verified;
+ s
+}
+
+/// Structured payload for the Electron/Chromium AX-echo case: the AT-SPI
+/// `insertText` rung returned success on a Chromium embedder, but on those
+/// surfaces the a11y bridge can accept and echo the write while the renderer
+/// never observes it — so a path=="ax" "confirm" is a shim echo, not ground
+/// truth. Mirrors the macOS `type_text` `ax_echo_surface` branch: report
+/// effect:"unverifiable" + escalation:{recommended:"px"} (a renderer-focus
+/// problem — pixel-focus the field, not foreground-activate it).
+fn type_text_structured_electron(text_len: usize) -> Value {
+ json!({
+ "path": "ax",
+ "characters": text_len,
+ "verified": false,
+ "effect": "unverifiable",
+ "escalation": {
+ "recommended": "px",
+ "reason": "Electron/web surface — the AX write was echoed but the \
+ renderer may not have observed it. Confirm via the screenshot; \
+ if it didn't land, re-type with the element px action (pass x,y \
+ to pixel-focus the field, then type)."
+ }
+ })
+}
+
+/// [`type_text_ax_result`] for an element-addressed insert: the snapshot
+/// object is read back (as `set_value` does) and, when the typed text is now
+/// part of its value, the result is `effect: confirmed` with a
+/// `value_readback` evidence item instead of `unverifiable`.
+async fn type_text_ax_result_verified(
+ pid: u32,
+ token: crate::atspi::TokenElement,
+ text: &str,
+ route: &str,
+) -> ToolResult {
+ let text_len = text.chars().count();
+ if is_chromium_embedder(pid) {
+ return type_text_ax_result(pid, text_len, route);
+ }
+ let idx = token.index;
+ let readback = tokio::task::spawn_blocking(move || crate::atspi::read_value_token(&token))
+ .await
+ .ok()
+ .flatten();
+ type_text_readback_result(pid, idx, text, route, readback)
+}
+
+/// The typed text is confirmed when the element's fresh value contains it
+/// (an insert at the caret) or agrees with it numerically (a spin button
+/// that reformats "-40" as "-40.0").
+fn type_text_readback_result(
+ pid: u32,
+ idx: usize,
+ text: &str,
+ route: &str,
+ readback: Option<String>,
+) -> ToolResult {
+ let text_len = text.chars().count();
+ match readback {
+ Some(seen)
+ if !text.trim().is_empty() && (seen.contains(text) || values_agree(text, &seen)) =>
+ {
+ let shown: String = seen.chars().take(200).collect();
+ ToolResult::text(format!(
+ "Typed {text_len} character(s) into element [{idx}] ({route}); it reads back '{shown}'."
+ ))
+ .with_structured(json!({
+ "path": "ax",
+ "characters": text_len,
+ "verified": true,
+ "effect": "confirmed",
+ "readback": shown,
+ "evidence": [{
+ "kind": "value_readback",
+ "detail": format!("element [{idx}] reads back the typed text"),
+ }],
+ }))
+ }
+ Some(seen) => {
+ let mut result = type_text_ax_result(pid, text_len, route);
+ let shown: String = seen.chars().take(200).collect();
+ if let Some(structured) = result.structured_content.as_mut() {
+ structured["readback"] = json!(shown);
+ }
+ result
+ .content
+ .push(cua_driver_core::protocol::Content::text(format!(
+ "The element reads back '{shown}', which does not contain the typed text; \
+ confirm with get_window_state."
+ )));
+ result
+ }
+ None => type_text_ax_result(pid, text_len, route),
+ }
+}
+
+#[cfg(test)]
+mod type_text_readback_tests {
+ use super::*;
+
+ #[test]
+ fn typed_text_is_confirmed_only_when_read_back() {
+ let confirmed =
+ type_text_readback_result(1, 3, "-40", "via targeted AT-SPI", Some("-40.0".into()));
+ let s = confirmed.structured_content.clone().unwrap();
+ assert_eq!(s["effect"], "confirmed");
+ assert_eq!(s["verified"], true);
+ assert_eq!(s["evidence"][0]["kind"], "value_readback");
+ let contained = type_text_readback_result(1, 3, "ab", "r", Some("xxabyy".into()));
+ assert_eq!(contained.structured_content.unwrap()["effect"], "confirmed");
+ let mismatch = type_text_readback_result(1, 3, "ab", "r", Some("zz".into()));
+ let s = mismatch.structured_content.unwrap();
+ assert_eq!(s["effect"], "unverifiable");
+ assert_eq!(s["readback"], "zz");
+ let blind = type_text_readback_result(1, 3, "ab", "r", None);
+ assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");
+ }
+}
+
+/// Build the success `ToolResult` for an AT-SPI insert. The EditableText
+/// method's boolean is a delivery acknowledgement, not a fresh value readback,
+/// so native widgets remain `unverifiable`. Chromium embedders additionally
+/// recommend the pixel rung because their accessibility bridge can acknowledge
+/// a write the renderer never observes.
+fn type_text_ax_result(pid: u32, text_len: usize, route: &str) -> ToolResult {
+ if is_chromium_embedder(pid) {
+ return ToolResult::text(format!(
+ "📨 Sent (unverified) {text_len} character(s) ({route}). — Electron/web \
+ surface: the AX layer accepts and echoes the write but the renderer may \
+ not have observed it, so the driver cannot confirm via AX. Verify via the \
+ screenshot; if it didn't land, re-type with the px form (pass x,y to \
+ pixel-focus the field)."
+ ))
+ .with_structured(type_text_structured_electron(text_len));
+ }
+ ToolResult::text(format!("Typed {text_len} character(s) ({route})."))
+ .with_structured(type_text_structured("ax", text_len, false))
+}
+
+/// Qt5 maps its own GUI library; only then is the synthetic-FocusIn bridge
+/// workaround in `type_text` worth two extra AT-SPI walks.
+fn is_qt5_process(pid: u32) -> bool {
+ fs::read_to_string(format!("/proc/{pid}/maps"))
+ .map(|maps| maps.contains("libQt5Gui") || maps.contains("libQt5Widgets"))
+ .unwrap_or(false)
+}
+
+#[cfg(test)]
+#[test]
+fn type_text_routes_isolated_background_before_generic_wayland_refusal() {
+ let source = include_str!("type_text.rs");
+ let invoke = source
+ .rsplit_once("impl Tool for TypeTextTool {")
+ .unwrap()
+ .1
+ .split_once("async fn invoke")
+ .unwrap()
+ .1;
+ let isolated = invoke
+ .find("isolated_hyprland_background(delivery)")
+ .expect("type_text must select isolated Hyprland background delivery");
+ let generic = invoke
+ .find("unavailable_wayland_focused_input_background")
+ .expect("type_text must retain the generic Wayland refusal");
+ assert!(
+ isolated < generic,
+ "the plugin route must run before the generic focused-input refusal"
+ );
+ assert!(invoke.contains("execute_background_text"));
+}
+
+// ── type_text ─────────────────────────────────────────────────────────────────
+
+pub struct TypeTextTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for TypeTextTool {
+ fn def(&self) -> &ToolDef {
+ TYPE_DEF.get_or_init(|| ToolDef {
+ name: "type_text".into(),
+ description: "Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.".into(),
+ input_schema: json!({
+ "type":"object","required":["text"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Omit with scope \"desktop\" to type into the focused application."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with x/y; optional with element_token (the token carries it)."},
+ "text":{"type":"string","description":"Text to type."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\") — a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget."},
+ "y":{"type":"number","description":"Pixel Y of the field (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with no pid to type into the focused application. Default \"window\"."},
+ "coordinate_frame": coordinate_frame_schema(),
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if (args.opt_str("scope").as_deref() == Some("desktop")
+ || coordinate_frame_is_desktop(&args))
+ && args.get("pid").is_none()
+ {
+ let input = match parse_typed_projection::<TypeTextInput>("type_text", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let text =
+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&input.text)
+ .into_owned();
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result = spawn_blocking_bounded(
+ "desktop type_text",
+ typing_budget(text.chars().count(), crate::wayland::niri::is_session()),
+ move || {
+ if wayland {
+ crate::wayland::type_text_focused(&text)
+ } else {
+ crate::input::send_type_text_xtest(&text)
+ }
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text("Typed text into the focused desktop application.")
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let mut pid = args.u64_or("pid", 0) as u32;
+ let text_raw = match args.require_str("text") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ // Strip trailing agent-protocol closing tags — see
+ // cua_driver_core::text_sanitize docs for rationale.
+ let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
+ .into_owned();
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let token = match prove_token_element(pid, &resolved, "type_text").await {
+ Ok(token) => token,
+ Err(refusal) => return refusal,
+ };
+ let (resolved_elem_idx, resolved_window_id) = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element {
+ element_index,
+ window_id,
+ ..
+ } => (Some(*element_index), Some(*window_id)),
+ cua_driver_core::element_token::ResolvedElement::None => (None, None),
+ };
+ let xid_opt = resolved_window_id.or(args.opt_u64("window_id"));
+
+ // Resolve XID: use window_id if given, else first window for pid.
+ let xid = match xid_opt {
+ Some(x) => x,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || {
+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
+ })
+ .await
+ .unwrap_or_default();
+ // pid omitted: the keys go to the active window, like a
+ // physical keyboard would, and the action adopts its pid.
+ let chosen = if pid == 0 {
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active && w.pid.is_some())
+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
+ } else {
+ // The pid's active window, else its largest on-screen
+ // toplevel: LibreOffice and GIMP own hidden/utility
+ // toplevels that a plain `first()` could pick, and an
+ // unmapped window cannot take the virtual keyboard focus.
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active)
+ .or_else(|| {
+ windows
+ .iter()
+ .filter(|w| w.is_on_screen)
+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
+ })
+ .or_else(|| windows.first())
+ };
+ match chosen {
+ Some(w) => {
+ if pid == 0 {
+ pid = w.pid.unwrap_or(0);
+ }
+ w.xid
+ }
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
+ if isolated_hyprland_background(delivery)
+ && resolved_elem_idx.is_none()
+ && args.get("x").is_none()
+ && args.get("y").is_none()
+ {
+ if xid_opt.is_none() {
+ return isolated_hyprland_refusal(
+ "an exact window_id is required for isolated text",
+ );
+ }
+ match crate::wayland::hyprland_input::text_actions(&text) {
+ Ok(actions) if !actions.is_empty() => {}
+ Ok(_) => return isolated_hyprland_refusal("background text must not be empty"),
+ Err(error) => return isolated_hyprland_refusal(error.to_string()),
+ }
+ position_named_session_keyboard_cursor(&self.state, &args, pid, xid, None, None, true)
+ .await;
+ let owner = named_session_cursor_key(&args);
+ let (_cancellation, dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ crate::wayland::hyprland_input::execute_background_text(
+ owner,
+ pid,
+ xid,
+ &text,
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(refusal) => return refusal,
+ };
+ return match dispatch.await {
+ Ok(result) => isolated_hyprland_result(result),
+ Err(error) => isolated_hyprland_task_error(error, false),
+ };
+ }
+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
+ return refusal;
+ }
+ if resolved_elem_idx.is_none() {
+ if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {
+ return refusal;
+ }
+ }
+
+ let px = args.get("x").and_then(|value| value.as_f64());
+ let py = args.get("y").and_then(|value| value.as_f64());
+ if px.is_some() != py.is_some() {
+ return invalid_pointer_arguments(
+ "type_text",
+ "x and y must be passed together (a pixel focus point), or neither",
+ );
+ }
+ if px.is_some() && resolved_elem_idx.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to type_text, not both.",
+ );
+ }
+
+ if hyprland_foreground(delivery) {
+ if xid_opt.is_none() {
+ return foreground_hyprland_refusal("an exact window_id is required");
+ }
+ // Native editables retain their verifiable, addressed AT-SPI route.
+ if !is_chromium_embedder(pid) && !is_webkitgtk_embedder(pid) {
+ if let Some(element) = token.clone() {
+ let text_ax = text.clone();
+ if matches!(
+ tokio::task::spawn_blocking(move || crate::atspi::type_into_token(
+ pid, &element, &text_ax
+ ))
+ .await,
+ Ok(Ok(()))
+ ) {
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ None,
+ true,
+ )
+ .await;
+ return type_text_ax_result(
+ pid,
+ text.chars().count(),
+ "via targeted AT-SPI",
+ );
+ }
+ }
+ }
+ match crate::wayland::hyprland_input::text_actions(&text) {
+ Ok(actions) if !actions.is_empty() => {}
+ Ok(_) => return foreground_hyprland_refusal("foreground text must not be empty"),
+ Err(error) => return foreground_hyprland_refusal(error.to_string()),
+ }
+ if let Err(error) =
+ focus_hyprland_foreground(&self.state, &args, pid, xid, token.clone(), px.zip(py))
+ .await
+ {
+ return error;
+ }
+ if named_session_cursor_key(&args).is_some() {
+ if px.is_none() {
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ None,
+ true,
+ )
+ .await;
+ }
+ } else {
+ announce_keyboard_target(&args, pid, xid, token.clone(), px.zip(py)).await;
+ }
+ let owner = named_session_cursor_key(&args);
+ let (_cancellation, dispatch) =
+ match spawn_isolated_hyprland(&args, move |cancellation| {
+ crate::wayland::hyprland_input::execute_foreground_text(
+ owner,
+ pid,
+ xid,
+ &text,
+ cancellation,
+ )
+ }) {
+ Ok(dispatch) => dispatch,
+ Err(_) => {
+ return foreground_hyprland_refusal(
+ "authenticated admitted lifecycle required",
+ )
+ }
+ };
+ return hyprland_input_result(
+ match dispatch.await {
+ Ok(result) => result,
+ Err(error) => Err(crate::wayland::hyprland_input::unknown_dispatch(
+ error.into(),
+ 0,
+ )),
+ },
+ true,
+ );
+ }
+
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ token.clone(),
+ px.zip(py),
+ true,
+ )
+ .await;
+
+ let text_len = text.chars().count();
+ // Native toolkit editables have a stronger focus-free route than raw
+ // compositor keyboard injection. Keep Chromium/WebKit on real key events
+ // because their accessibility bridges may echo a write that never reaches
+ // renderer-owned state.
+ if let Some(element) = token.clone().filter(|_| {
+ crate::wayland::is_inject_mode()
+ && !is_chromium_embedder(pid)
+ && !is_webkitgtk_embedder(pid)
+ }) {
+ let text_at = text.clone();
+ let target = element.clone();
+ let targeted = tokio::task::spawn_blocking(move || {
+ crate::atspi::type_into_token(pid, &target, &text_at)
+ })
+ .await;
+ if let Ok(Ok(())) = targeted {
+ return type_text_ax_result_verified(pid, element, &text, "via targeted AT-SPI")
+ .await;
+ }
+ }
+ // The private nested compositor can target the owning Wayland client
+ // directly. Establish widget-local focus first, without changing the
+ // compositor's focused toplevel, so keys reach the addressed control.
+ if crate::wayland::is_inject_mode() {
+ if let Err(error) =
+ focus_nested_inject_target(pid, xid, token.clone(), px.zip(py)).await
+ {
+ return error;
+ }
+ let text_w = text.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ crate::wayland::inject_type_text(pid, xid, &text_w)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) (focus-free via cua-compositor)."
+ ))
+ .with_structured(type_text_structured(
+ "key_events",
+ text_len,
+ false,
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // ── px form: focus by pixel-click, then type into the now-focused element ──
+ // Pass x,y (no element_token) for an *element px action*: pixel-click
+ // the field to give the renderer the real keyboard focus the AT-SPI path
+ // can't, then fall through to the focused-element type path below (it
+ // escalates AT-SPI → key events and lands once focused). Reuses ClickTool's
+ // exact coordinate translation + delivery_mode.
+ if let (Some(cx), Some(cy)) = (px, py) {
+ if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ let from_zoom = args.bool_or("from_zoom", false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ Some(xid),
+ (cx, cy),
+ delivery.is_foreground(),
+ &args,
+ from_zoom,
+ )
+ .await
+ {
+ return e;
+ }
+ // resolved_elem_idx stays None → the type path below writes to the now-
+ // focused element via the background key / AT-SPI rung.
+ }
+
+ if resolved_elem_idx.is_some() {
+ if let Some(refusal) = unavailable_webkit_keyboard_background(pid, delivery) {
+ return refusal;
+ }
+ }
+
+ // Renderer EditableText writes can update the accessible value without
+ // emitting the DOM input event. For an explicit foreground request on
+ // native Wayland, focus the named field and send real keyboard input so
+ // Chromium/WebKit observe the same event sequence as a user.
+ if delivery.is_foreground()
+ && crate::wayland::wayland_input_enabled()
+ && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid))
+ {
+ if let Some(element) = token.clone() {
+ let idx = element.index;
+ let text_w = text.clone();
+ let result = spawn_blocking_bounded(
+ "type_text",
+ typing_budget(text_w.chars().count(), crate::wayland::niri::is_session()),
+ move || {
+ crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::RestoreMode::RestorePrevious,
+ || {
+ if !crate::atspi::focus_token(pid, &element)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {idx}"
+ );
+ }
+ crate::wayland::type_text_focused(&text_w)
+ },
+ )
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) (via Wayland virtual-keyboard)."
+ ))
+ .with_structured(type_text_structured(
+ "key_events",
+ text_len,
+ false,
+ )),
+ Ok(Err(error)) => stale_observed_element("type_text", idx, &error)
+ .unwrap_or_else(|| ToolResult::error(error.to_string())),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ }
+
+ // AX addressing names one exact editable. Try this focus-free route
+ // before native Wayland keyboard injection, which can only target the
+ // compositor's globally focused surface.
+ if let Some(element) = token.clone() {
+ let text_at = text.clone();
+ let target = element.clone();
+ let targeted = tokio::task::spawn_blocking(move || {
+ crate::atspi::type_into_token(pid, &target, &text_at)
+ })
+ .await;
+ if let Ok(Err(error)) = &targeted {
+ // The observed object is gone: nothing was written to it.
+ if let Some(stale) = stale_observed_element("type_text", element.index, error) {
+ return stale;
+ }
+ }
+ match targeted {
+ Ok(Ok(())) => {
+ return type_text_ax_result_verified(
+ pid,
+ element,
+ &text,
+ "via targeted AT-SPI",
+ )
+ .await;
+ }
+ Ok(Err(_)) | Err(_)
+ if !delivery.is_foreground() && crate::wayland::wayland_input_enabled() =>
+ {
+ return crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ }
+ _ => {}
+ }
+ }
+
+ // Native Wayland: keys go to the *focused* surface (no pid/window
+ // targeting in the protocol). Type via the virtual-keyboard tool; pair
+ // with a prior `click`/`activate` to focus the intended window.
+ if crate::wayland::wayland_input_enabled() {
+ if !delivery.is_foreground() {
+ return crate::input::delivery::background_unavailable_error(
+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
+ );
+ }
+ let text_w = text.clone();
+ let element = token.clone();
+ let result = spawn_blocking_bounded(
+ "type_text",
+ typing_budget(text_w.chars().count(), crate::wayland::niri::is_session()),
+ move || {
+ crate::interaction::with_target_foreground(
+ pid,
+ xid,
+ crate::interaction::RestoreMode::RestorePrevious,
+ || {
+ if let Some(element) = &element {
+ if !crate::atspi::focus_token(pid, element)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {}",
+ element.index
+ );
+ }
+ }
+ crate::wayland::type_text_focused(&text_w)
+ },
+ )
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) (via Wayland virtual-keyboard)."
+ ))
+ .with_structured(type_text_structured(
+ "key_events",
+ text_len,
+ false,
+ )),
+ Ok(Err(e)) => resolved_elem_idx
+ .and_then(|idx| stale_observed_element("type_text", idx, &e))
+ .unwrap_or_else(|| ToolResult::error(e.to_string())),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Terminal short-circuit: when the target window's WM_CLASS marks it
+ // as a terminal emulator (Ghostty / Alacritty / kitty / …), skip the
+ // AT-SPI path entirely. Terminals expose an editable text area that
+ // AT-SPI `insertText` would aim at, but the write never reaches the
+ // pty, so the user sees "type acknowledged but nothing appeared".
+ // Try pty-master injection first (most reliable), then XTest key
+ // synthesis. Either way the structured response reports
+ // `path: "key_events"` so callers can verify the route taken.
+ let pid_is_terminal = is_terminal_process(pid);
+ let wm_class_is_terminal =
+ tokio::task::spawn_blocking(move || crate::terminal::is_terminal_window(xid))
+ .await
+ .unwrap_or(false);
+ if pid_is_terminal || wm_class_is_terminal {
+ let text_len = text.chars().count();
+ let text_t = text.clone();
+ let foreground = delivery.is_foreground();
+ let cursor_id = resolve_cursor_key(&args);
+ let result = spawn_blocking_bounded(
+ "type_text terminal",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {
+ // pty-master injection is preferred — it skips the X event
+ // queue entirely. Falls through to key events if the terminal
+ // isn't reachable that way (descendant pty unresolvable).
+ if inject_terminal_input(pid, xid, &text_t)? {
+ return Ok(("pty", None));
+ }
+ if foreground {
+ crate::interaction::with_x11_foreground(xid, crate::input::ForegroundOptions::pointer(), || {
+ crate::input::send_type_text_xtest(&text_t)
+ }).map(|(value, _report)| value)?;
+ return Ok(("key_events_fg", None));
+ }
+ // Background: real key events through the XI2 virtual master
+ // keyboard (xterm/gnome-terminal accept them like physical input).
+ match background_text_route(&cursor_id, pid, xid, &text_t)? {
+ Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),
+ _ => Ok(("background_unavailable", None)),
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(atspi_up()),
+ Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),
+ Ok(Ok((path, None))) => ToolResult::text(format!(
+ "Typed {text_len} character(s) (terminal emulator: pty/XTest key events)."
+ ))
+ .with_structured(type_text_structured(path, text_len, false)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // Foreground means the caller explicitly permits activation: behave like
+ // a user. Activate the window and confirm it holds the input focus,
+ // GrabFocus the indexed element when one was given, then type REAL key
+ // events via XTest. The AT-SPI EditableText probes below are skipped on
+ // purpose: on large trees (LibreOffice) they stall for the full bus
+ // timeout, Chromium/WebKitGTK echo them without the renderer seeing the
+ // input, and for a spreadsheet cell / canvas they would target the
+ // wrong widget anyway. The whole transaction is deadline-bounded.
+ if delivery.is_foreground() {
+ let text_f = text.clone();
+ let element = token.clone();
+ let result = spawn_blocking_bounded(
+ "foreground type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || {
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::keyboard(),
+ || {
+ if let Some(element) = &element {
+ if !crate::atspi::focus_token(pid, element)? {
+ anyhow::bail!(
+ "AT-SPI Component.GrabFocus returned false for element {}",
+ element.index
+ );
+ }
+ }
+ crate::input::send_type_text_xtest(&text_f)
+ },
+ )
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(((), report))) => {
+ let mut extra = serde_json::Map::new();
+ extra.insert("characters".into(), json!(text_len));
+ ToolResult::text(format!(
+ "Typed {text_len} character(s) as real key events \
+ (delivery_mode=foreground, focus_after={}); not verified — \
+ confirm with a screenshot.",
+ report.focus_after.as_str()
+ ))
+ .with_structured(foreground_structured(
+ "key_events_fg",
+ report,
+ extra,
+ ))
+ }
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // With a focus-free real keyboard available, type where the widget
+ // focus is — exactly what a physical keyboard does after the click that
+ // focused the field. The blind AT-SPI editable search below can pick a
+ // different editable of the app (it "succeeded" into GIMP's wrong entry
+ // and VS Code's search box); it stays as the fallback for hosts without
+ // the MPX route and for explicit element targets.
+ if !delivery.is_foreground()
+ && resolved_elem_idx.is_none()
+ && crate::input::real_keyboard_input_available()
+ {
+ let text_b = text.clone();
+ let cursor_id = resolve_cursor_key(&args);
+ let result = spawn_blocking_bounded(
+ "background type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || background_text_route(&cursor_id, pid, xid, &text_b),
+ )
+ .await;
+ match result {
+ Ok(Ok(Some(KeyRoute::Mpx(report)))) => {
+ return type_text_mpx_result(text_len, report)
+ }
+ Ok(Err(e)) => return input_error_result(e),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ Ok(Ok(_)) => {}
+ }
+ }
+
+ // Prefer the focused widget — the element the user just clicked. If a
+ // NON-editable input holds keyboard focus (a spreadsheet cell, a
+ // terminal, a canvas), the focus-free AT-SPI editable search below would
+ // grab the wrong field (e.g. gnumeric's name box instead of the selected
+ // cell, or skip a terminal entirely), so synth-type into the focused
+ // widget instead: terminals via pty injection, everything else via
+ // XSendEvent to the focused window. A focused *editable* (Some(true)) or
+ // nothing focused (None) falls through to the existing AT-SPI-first flow.
+ let focus_kind = tokio::task::spawn_blocking(move || {
+ crate::atspi::focused_is_editable(pid).ok().flatten()
+ })
+ .await
+ .ok()
+ .flatten();
+ if focus_kind == Some(false) {
+ if !delivery.is_foreground() {
+ // A spreadsheet cell / canvas / terminal holds the widget focus:
+ // real key events aimed at this window are the only thing that
+ // types there. The XI2 virtual master keyboard delivers them
+ // without touching the user's focus; terminals take the pty.
+ let text_b = text.clone();
+ let cursor_id = resolve_cursor_key(&args);
+ let result = spawn_blocking_bounded(
+ "background type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || -> anyhow::Result<Option<KeyRoute>> {
+ if inject_terminal_input(pid, xid, &text_b)? {
+ return Ok(Some(KeyRoute::Terminal));
+ }
+ background_text_route(&cursor_id, pid, xid, &text_b)
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(Some(KeyRoute::Terminal))) => ToolResult::text(format!(
+ "Typed {text_len} character(s) into the focused terminal (pty)."
+ ))
+ .with_structured(type_text_structured("key_events", text_len, false)),
+ Ok(Ok(Some(KeyRoute::Mpx(report)))) => type_text_mpx_result(text_len, report),
+ Ok(Ok(_)) => background_keyboard_refusal(atspi_up()),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ let text_f = text.clone();
+ let result = spawn_blocking_bounded(
+ "foreground type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || {
+ if inject_terminal_input(pid, xid, &text_f)? {
+ return Ok(());
+ }
+ // XTest (real input to the focused window), NOT XSendEvent: GTK/Qt
+ // drop synthetic key events, so a spreadsheet cell / canvas would
+ // stay empty. The click that gave this widget focus already put it
+ // under the X input focus, so XTest-to-focus lands correctly.
+ crate::interaction::with_x11_foreground(
+ xid,
+ crate::input::ForegroundOptions::pointer(),
+ || crate::input::send_type_text_xtest(&text_f),
+ )
+ .map(|(value, _report)| value)
+ },
+ )
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) into the focused widget."
+ ))
+ .with_structured(type_text_structured(
+ "key_events",
+ text_len,
+ false,
+ )),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Try AT-SPI EditableText first (focus-free, works for Qt6/GTK4).
+ let text_clone = text.clone();
+ let atspi_result =
+ tokio::task::spawn_blocking(move || crate::atspi::type_into_editable(pid, &text_clone))
+ .await;
+
+ match atspi_result {
+ Ok(Ok(())) => {
+ // AT-SPI succeeded — focus-free typing worked (Qt6, GTK4, etc.)!
+ // Electron/Chromium can echo this write without the renderer
+ // observing it, so the confirm is suppressed there (mirrors macOS).
+ return type_text_ax_result(pid, text_len, "via AT-SPI");
+ }
+ _ => {
+ // AT-SPI failed (no editable exposed). Qt5 doesn't expose widgets
+ // when unfocused, so try the synthetic-focus workaround.
+ }
+ }
+
+ // Qt5 workaround: send synthetic FocusIn to make Qt5's AT-SPI bridge
+ // expose the widget tree, type via AT-SPI, then send FocusOut.
+ // This doesn't change the X11 active window, so the test's focus check passes.
+ let text_clone2 = text.clone();
+ let qt5_result = spawn_blocking_bounded(
+ "Qt5 type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || {
+ if !is_qt5_process(pid) {
+ anyhow::bail!("not a Qt5 process; synthetic-FocusIn bridge workaround skipped");
+ }
+ // Send FocusIn to trigger Qt5's bridge
+ crate::input::send_focus_in(xid)?;
+ std::thread::sleep(std::time::Duration::from_millis(100));
+
+ // Try AT-SPI again now that widgets should be exposed
+ let result = crate::atspi::type_into_editable(pid, &text_clone2);
+
+ // Restore state with FocusOut
+ crate::input::send_focus_out(xid)?;
+
+ result
+ },
+ )
+ .await;
+
+ match qt5_result {
+ Ok(Ok(())) => {
+ return type_text_ax_result(pid, text_len, "via AT-SPI with focus workaround");
+ }
+ _ => {
+ // AT-SPI still didn't work. Fall back to X11 XSendEvent.
+ }
+ }
+
+ // Track which path the final fallback chain took, so the
+ // structured response stays honest. The closure can't borrow
+ // a local mutably across `spawn_blocking`, so funnel the
+ // decision through the success type instead.
+ // delivery_mode: background (default) = focus-free AT-SPI / XSendEvent;
+ // foreground = activate the window (EWMH), then synthesize REAL key
+ // events to it via XTest — the escalation when background didn't land
+ // (e.g. a GTK dialog whose widget ignores synthetic XSendEvent keys).
+ let cursor_id = resolve_cursor_key(&args);
+ let result = spawn_blocking_bounded(
+ "type_text",
+ typing_budget(text_len, crate::wayland::niri::is_session()),
+ move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {
+ // Terminals: write to the pty master (focus-free, below the toolkit).
+ if inject_terminal_input(pid, xid, &text)? {
+ return Ok(("key_events", None));
+ }
+ // GUI apps: X11 only routes keystrokes to the *focused* toplevel's
+ // focused widget, so background XSendEvent typing doesn't land. Fill
+ // the editable field via AT-SPI instead — focus-free and toolkit-
+ // agnostic. Fall back to Tk send or XSendEvent when no a11y field is exposed.
+ if crate::atspi::insert_text(pid, &text).unwrap_or(false) {
+ return Ok(("ax", None));
+ }
+ // Tk apps: use Tk's `send` command (no AT-SPI bridge, so AT-SPI above
+ // returned false). This is the Tk-specific override, like CDP for Chromium.
+ if crate::input::inject_tk_send(&text).unwrap_or(false) {
+ return Ok(("key_events", None));
+ }
+ if delivery.is_foreground() {
+ crate::interaction::with_x11_foreground(xid, crate::input::ForegroundOptions::pointer(), || {
+ crate::input::send_type_text_xtest(&text)
+ }).map(|(value, _report)| value)?;
+ return Ok(("key_events_fg", None));
+ }
+ // No a11y editable, no pty: real key events through the XI2 virtual
+ // master keyboard are the last focus-free route.
+ match background_text_route(&cursor_id, pid, xid, &text)? {
+ Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),
+ _ => Ok(("background_unavailable", None)),
+ }
+ })
+ .await;
+ let mode_label = if delivery.is_foreground() {
+ "foreground"
+ } else {
+ "background"
+ };
+ match result {
+ // AT-SPI's boolean acknowledges the EditableText call; it is not a
+ // fresh value readback. Keep the result unverifiable and apply the
+ // stricter Chromium escalation where appropriate.
+ Ok(Ok(("ax", _))) => type_text_ax_result(
+ pid,
+ text_len,
+ &format!("via X11, delivery_mode={mode_label}"),
+ ),
+ Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(atspi_up()),
+ Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),
+ Ok(Ok((path, None))) => ToolResult::text(format!(
+ "Typed {text_len} character(s) (via X11, delivery_mode={mode_label})."
+ ))
+ .with_structured(type_text_structured(path, text_len, false)),
+ Ok(Err(e)) => input_error_result(e),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+/// Background text delivery through the XI2 virtual master keyboard. `None`
+/// when the host has no such route (Xvfb, no `/dev/uinput`); a uinput
+/// permission error propagates so the tool can attach the operator hint.
+fn background_text_route(
+ cursor_id: &str,
+ pid: u32,
+ xid: u64,
+ text: &str,
+) -> anyhow::Result<Option<KeyRoute>> {
+ if !crate::input::real_keyboard_input_available() {
+ return Ok(None);
+ }
+ match crate::input::focus_guard::guarded(Some(pid), || {
+ background_virtual_text(cursor_id, pid, xid, text)
+ }) {
+ Ok((mut report, guard)) => {
+ report.focus_guard = guard;
+ Ok(Some(KeyRoute::Mpx(report)))
+ }
+ Err(error) if crate::input::is_uinput_unavailable(&error) => Err(error),
+ Err(error) if is_popup_keyboard_grab(&error) => Err(error),
+ Err(error) => {
+ tracing::warn!(pid, "MPX keyboard text delivery failed: {error}");
+ Err(error.context("virtual master keyboard delivery failed"))
+ }
+ }
+}
+
+/// `type_text` result for a delivery through the virtual master keyboard.
+fn type_text_mpx_result(
+ text_len: usize,
+ report: crate::input::KeyboardDeliveryReport,
+) -> ToolResult {
+ let mut structured = type_text_structured(report.path, text_len, false);
+ for (key, value) in report.to_json().as_object().into_iter().flatten() {
+ structured[key] = value.clone();
+ }
+ structured["delivery_mode"] = json!("background");
+ let outcome = if report.skipped_characters.is_empty() {
+ format!("Typed {text_len} character(s)")
+ } else {
+ "Partial typing result".to_owned()
+ };
+ let mut text = format!(
+ "{outcome} as real key events {} (delivery_mode=background); \
+ not verified — confirm with a screenshot.",
+ report.route_phrase()
+ );
+ if !report.skipped_characters.is_empty() {
+ text.push_str(&format!(
+ " {} character(s) had no keycode in the current keymap and were skipped: {:?}.",
+ report.skipped_characters.len(),
+ report.skipped_characters.iter().collect::<String>()
+ ));
+ }
+ if !report.virtual_focus_held || !report.delivery_confirmed {
+ structured["effect"] = json!("suspected_noop");
+ text.push_str(
+ " Warning: the target lost the virtual keyboard's focus or the server did \
+ not confirm the last key; the text may not have landed.",
+ );
+ }
+ text.push_str(&report.delivery_notes());
+ let result = if report.skipped_characters.is_empty() {
+ ToolResult::text(text)
+ } else {
+ ToolResult::error(text)
+ };
+ result.with_structured(structured)
+}
+
+#[cfg(test)]
+#[test]
+fn mpx_type_text_with_skipped_characters_is_an_error() {
+ let report = |skipped_characters: Vec<char>| crate::input::KeyboardDeliveryReport {
+ virtual_focus_held: true,
+ core_focus_unchanged: true,
+ delivery_confirmed: true,
+ key_events: 2,
+ skipped_characters,
+ focus_guard: None,
+ released_stuck: Vec::new(),
+ path: crate::input::MPX_UINPUT_PATH,
+ grab_popup: None,
+ };
+ assert_eq!(type_text_mpx_result(2, report(Vec::new())).is_error, None);
+ let skipped = type_text_mpx_result(2, report(vec!['約']));
+ assert_eq!(skipped.is_error, Some(true));
+ assert!(!format!("{:?}", skipped.content).contains("Typed 2 character(s)"));
+ assert_eq!(
+ skipped.structured_content.unwrap()["skipped_characters"],
+ "約"
+ );
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text_chars.rs
new file mode 100644
index 0000000000..20a2a01e53
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/type_text_chars.rs
@@ -0,0 +1,124 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+use super::impl_::{paced_typing_budget, spawn_blocking_bounded};
+
+// ── type_text_chars ───────────────────────────────────────────────────────────
+
+pub struct TypeTextCharsTool;
+static TYPE_CHARS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for TypeTextCharsTool {
+ fn def(&self) -> &ToolDef {
+ TYPE_CHARS_DEF.get_or_init(|| ToolDef {
+ name: "type_text_chars".into(),
+ description: "Type text character-by-character with a configurable inter-character \
+ delay (default 30 ms). Useful for apps that miss keystrokes. \
+ Otherwise identical to type_text (XSendEvent, no focus steal).".into(),
+ input_schema: json!({
+ "type":"object","required":["pid","text"],"properties":{
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"Window id from list_windows. Required with element_index."},
+ "text":{"type":"string","description":"Text to type, one character at a time."},
+ "delay_ms":{"type":"integer","description":"Milliseconds between chars (default 30)."},
+ "type_chars_only":{"type":"boolean","description":"Skip element focus, type directly. Default false."}
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ #[allow(unused_assignments)]
+ let pid = args.u64_or("pid", 0) as u32;
+ let text_raw = match args.require_str("text") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ // Same trailing-protocol-tag scrub as TypeTextTool — see
+ // cua_driver_core::text_sanitize for rationale.
+ let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
+ .into_owned();
+ let delay_ms = args.u64_or("delay_ms", 30);
+ let xid_opt = args.opt_u64("window_id");
+ let xid = match xid_opt {
+ Some(x) => x,
+ None => {
+ let windows = tokio::task::spawn_blocking(move || {
+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })
+ })
+ .await
+ .unwrap_or_default();
+ // pid omitted: the keys go to the active window, like a
+ // physical keyboard would, and the action adopts its pid.
+ let chosen = if pid == 0 {
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active && w.pid.is_some())
+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))
+ } else {
+ // The pid's active window, else its largest on-screen
+ // toplevel: LibreOffice and GIMP own hidden/utility
+ // toplevels that a plain `first()` could pick, and an
+ // unmapped window cannot take the virtual keyboard focus.
+ let active = crate::x11::active_window();
+ windows
+ .iter()
+ .find(|w| Some(w.xid) == active)
+ .or_else(|| {
+ windows
+ .iter()
+ .filter(|w| w.is_on_screen)
+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))
+ })
+ .or_else(|| windows.first())
+ };
+ match chosen {
+ Some(w) => w.xid,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ let text_len = text.chars().count();
+ let result = spawn_blocking_bounded(
+ "type_text_chars",
+ paced_typing_budget(text_len, delay_ms),
+ move || {
+ if crate::wayland::wayland_input_enabled() {
+ // Per-char `wtype` loop with the requested delay — mirrors the
+ // X11 XSendEvent per-char path. Sleeping here is fine because
+ // we're inside spawn_blocking.
+ let mut buf = [0u8; 4];
+ for ch in text.chars() {
+ let s = ch.encode_utf8(&mut buf);
+ crate::wayland::type_text(xid, s)?;
+ if delay_ms > 0 {
+ std::thread::sleep(std::time::Duration::from_millis(delay_ms));
+ }
+ }
+ return Ok(());
+ }
+ crate::input::send_type_text_with_delay(xid, &text, delay_ms)
+ },
+ )
+ .await;
+ match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) with {delay_ms}ms delay."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/zoom.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/zoom.rs
new file mode 100644
index 0000000000..f17f41f45a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/zoom.rs
@@ -0,0 +1,145 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ snapshot_store::ZoomContext,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+// ── zoom ──────────────────────────────────────────────────────────────────────
+
+pub struct ZoomTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static ZOOM_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ZoomTool {
+ fn def(&self) -> &ToolDef {
+ ZOOM_DEF.get_or_init(|| ToolDef {
+ name: "zoom".into(),
+ description: "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in \
+ screenshot pixels, with 20% padding. Output is at most 500 px wide. \
+ Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\n\
+ After a zoom, pass from_zoom=true to click/type_text to auto-translate \
+ coordinates back to full-window space. Coordinate actions return \
+ `screenshot_context_missing` when no current snapshot contains a \
+ screenshot owned by this session. `from_zoom` actions return \
+ `zoom_context_missing` when the zoom was never created or was replaced; call \
+ `get_window_state`, then `zoom`, again on the same connection.".into(),
+ input_schema: json!({
+ "type":"object","required":["window_id","x1","y1","x2","y2"],"properties":{
+ "window_id":{"type":"integer","description":"Window id of the window captured by get_window_state."},
+ "pid":{"type":"integer","description":"Optional target pid. When omitted, the driver resolves the unique current snapshot for this session and window."},
+ "x1":{"type":"number","description":"Left edge of the region in window-local screenshot pixels."},
+ "y1":{"type":"number","description":"Top edge of the region in window-local screenshot pixels."},
+ "x2":{"type":"number","description":"Right edge of the region in window-local screenshot pixels."},
+ "y2":{"type":"number","description":"Bottom edge of the region in window-local screenshot pixels."}
+ },"additionalProperties":false
+ }),
+ read_only: true, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let xid = match args.require_u64("window_id") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let requested_pid = match args.get("pid") {
+ None => None,
+ Some(value) => match value.as_u64().and_then(|pid| i32::try_from(pid).ok()) {
+ Some(pid) => Some(pid),
+ None => return ToolResult::error("pid must be a positive 32-bit integer"),
+ },
+ };
+ let session_id = args.opt_str("_session_id");
+ let (pid, screenshot) = match self.state.snapshots.screenshot_context_for_zoom(
+ requested_pid,
+ xid,
+ session_id.as_deref(),
+ ) {
+ Ok(context) => context,
+ Err(refusal) => return refusal,
+ };
+ let x1 = match args.opt_f64("x1") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing x1"),
+ };
+ let y1 = match args.opt_f64("y1") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing y1"),
+ };
+ let x2 = match args.opt_f64("x2") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing x2"),
+ };
+ let y2 = match args.opt_f64("y2") {
+ Some(v) => v,
+ None => return ToolResult::error("Missing y2"),
+ };
+ if x2 <= x1 || y2 <= y1 {
+ return ToolResult::error("x2 must be > x1 and y2 must be > y1");
+ }
+
+ let (x1, y1, x2, y2) = (
+ x1 * screenshot.scale,
+ y1 * screenshot.scale,
+ x2 * screenshot.scale,
+ y2 * screenshot.scale,
+ );
+ let state = self.state.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ // Route through the Wayland-aware window capture dispatcher so
+ // pure-Wayland sessions surface a typed "per-window capture not
+ // supported yet" error instead of accidentally calling the
+ // X11-only path with a foreign-toplevel id.
+ let png = crate::wayland::screenshot_window_dispatch(xid)?;
+ cursor_overlay::capture_utils::crop_png_to_jpeg(&png, x1, y1, x2, y2, 500)
+ })
+ .await;
+
+ match result {
+ Ok(Ok(crop)) => {
+ if let Err(refusal) = state.snapshots.set_zoom(
+ pid,
+ session_id.as_deref(),
+ ZoomContext {
+ screenshot,
+ origin_x: crop.origin_x,
+ origin_y: crop.origin_y,
+ scale_inv: crop.scale_inv,
+ },
+ ) {
+ return refusal;
+ }
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ let b64 = B64.encode(&crop.jpeg_bytes);
+ let (w, h) = (crop.out_w, crop.out_h);
+ use cua_driver_core::protocol::Content;
+ ToolResult {
+ content: vec![
+ Content::image_jpeg(b64),
+ Content::text(format!(
+ "Zoom ({x1:.0},{y1:.0})–({x2:.0},{y2:.0}) → {w}×{h} px JPEG."
+ )),
+ ],
+ is_error: None,
+ // Surface 7: `mime_type` mirrors the MCP image part's `mimeType`
+ // onto the structured payload (additive — `format` stays).
+ structured_content: Some(json!({
+ "width": w, "height": h, "format": "jpeg",
+ "mime_type": "image/jpeg"
+ })),
+ action_record: None,
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("Zoom failed: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
index e78c5a9394..4f870d064d 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/keyboard.rs
@@ -10,7 +10,7 @@
//! the system input queue, not posted messages. This module exposes
//! [`is_xaml_host_hwnd`] so callers (e.g. the `type_text` tool) can route
//! around the PostMessage path for those targets — see CUA-543 and
-//! `tools::impl_::TypeTextTool` for the routing logic. The actual UIA
+//! `tools::type_text::TypeTextTool` for the routing logic. The actual UIA
//! `ValuePattern.SetValue` injection lives in the tools layer alongside
//! the existing `set_value` tool; this module deliberately stays
//! Win32-only so the unit tests don't depend on UIA initialisation.
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/bring_to_front.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/bring_to_front.rs
new file mode 100644
index 0000000000..f9b6d321ca
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/bring_to_front.rs
@@ -0,0 +1,209 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── bring_to_front ────────────────────────────────────────────────────────
+
+pub struct BringToFrontTool;
+static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for BringToFrontTool {
+ fn def(&self) -> &ToolDef {
+ BTF_DEF.get_or_init(|| ToolDef {
+ name: "bring_to_front".into(),
+ description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \
+ the OS foreground. \n\n\
+ **This deliberately breaks the no-foreground contract.** It is not part of \
+ the normal input ladder. For an ordinary `background_unavailable` response, \
+ retry only the refused action with `delivery_mode:\"foreground\"`; the input \
+ tool performs its own activate, act, and restore sequence. Use \
+ `bring_to_front` only for a focus-proxy surface that must remain foreground \
+ across multiple calls, such as an RDP or Windows App session, or when repeated \
+ action-scoped activation prevents the remote surface from accepting input. \n\n\
+ Implementation uses the `AttachThreadInput` trick to bypass Windows' \
+ foreground-lock when the daemon is not at UIAccess integrity. Returns \
+ structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \
+ Windows only; macOS / Linux return an error pointing at platform-native \
+ alternatives.".into(),
+ input_schema: json!({"type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"Optional HWND. Defaults to pid's first visible top-level window."}
+ },"additionalProperties":false}),
+ read_only: false,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let hwnd_opt = args.opt_u64("window_id");
+
+ // Resolve hwnd: explicit, or auto from pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows =
+ tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "bring_to_front: no windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // Run the foreground swap on a blocking thread. The AttachThreadInput
+ // trick mirrors `send_key_synthesized` (input/keyboard.rs:313-345)
+ // and is validated by `flash-repro/16-edge-launch-fg.ps1` for the
+ // Edge launch focus-steal recovery case.
+ let outcome =
+ tokio::task::spawn_blocking(move || -> Result<(u64, u64, bool, bool), String> {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::Graphics::Dwm::DwmFlush;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetForegroundWindow, IsIconic, IsWindow, SetWindowPos, ShowWindowAsync,
+ HWND_NOTOPMOST, HWND_TOPMOST, SWP_NOACTIVATE, SWP_NOMOVE, SWP_NOSIZE,
+ SW_RESTORE,
+ };
+
+ let target = HWND(hwnd as *mut _);
+ if !unsafe { IsWindow(target) }.as_bool() {
+ return Err(format!("hwnd 0x{hwnd:x} is not a valid window"));
+ }
+
+ let prev_fg = unsafe { GetForegroundWindow() };
+ let prev_fg_addr = prev_fg.0 as u64;
+
+ // Iconic windows have no rendered pixels and live at the sentinel
+ // (-32000, -32000) position. Restore before changing z-order so
+ // bring_to_front is also the advertised recovery path for capture.
+ let was_minimized = unsafe { IsIconic(target) }.as_bool();
+ if was_minimized {
+ let _ = unsafe { ShowWindowAsync(target, SW_RESTORE) };
+ for _ in 0..20 {
+ if !unsafe { IsIconic(target) }.as_bool() {
+ break;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(25));
+ }
+ if unsafe { IsIconic(target) }.as_bool() {
+ return Err(format!(
+ "restore request did not complete for minimized hwnd 0x{hwnd:x}"
+ ));
+ }
+ }
+
+ // Lock-free z-order raise FIRST: bring the window to the top of the
+ // normal band (the HWND_TOPMOST→HWND_NOTOPMOST force-to-front trick)
+ // so it's brought to the VISIBLE front even when the foreground-lock
+ // denies focus. SWP_NOACTIVATE → no focus steal; SetWindowPos z-order
+ // is not gated by the foreground-lock / UIAccess. This is the same
+ // technique the delivery_mode:"foreground" pointer path uses.
+ let raised = unsafe {
+ let a = SetWindowPos(
+ target,
+ HWND_TOPMOST,
+ 0,
+ 0,
+ 0,
+ 0,
+ SWP_NOACTIVATE | SWP_NOMOVE | SWP_NOSIZE,
+ )
+ .is_ok();
+ let b = SetWindowPos(
+ target,
+ HWND_NOTOPMOST,
+ 0,
+ 0,
+ 0,
+ 0,
+ SWP_NOACTIVATE | SWP_NOMOVE | SWP_NOSIZE,
+ )
+ .is_ok();
+ a && b
+ };
+
+ // The caller explicitly requested a visible transition, so the
+ // shared foreground helper may claim the most-recent-input token.
+ let _ = unsafe { crate::input::force_foreground_assisted(target) };
+ let now_fg = unsafe { GetForegroundWindow() };
+
+ // A restored HWND can stop reporting iconic before its compositor
+ // surface is painted. Flush DWM before returning, but do not make
+ // the restore operation depend on any particular capture backend.
+ // Capture has its own WGC fallback for freshly restored surfaces.
+ if was_minimized {
+ let _ = unsafe { DwmFlush() };
+ }
+ Ok((prev_fg_addr, now_fg.0 as u64, raised, was_minimized))
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok((prev, now, raised, restored))) => {
+ let focused = now == hwnd;
+ let msg = if focused {
+ format!("✅ bring_to_front: pid {pid} hwnd 0x{hwnd:x} is now foreground (was 0x{prev:x}).")
+ } else if raised {
+ format!(
+ "bring_to_front: exact target hwnd 0x{hwnd:x} was raised in z-order, but \
+ Windows kept foreground on hwnd 0x{now:x}; foreground activation failed."
+ )
+ } else {
+ format!(
+ "bring_to_front could neither focus nor raise target 0x{hwnd:x} (current \
+ foreground 0x{now:x})."
+ )
+ };
+ let structured = if focused {
+ json!({
+ "previous_fg_hwnd": format!("0x{prev:x}"),
+ "now_fg_hwnd": format!("0x{now:x}"),
+ "target_hwnd": format!("0x{hwnd:x}"),
+ "landed_on_target": true,
+ "raised": raised,
+ "restored": restored,
+ })
+ } else {
+ json!({
+ "code": "foreground_unavailable",
+ "effect": "refused",
+ "pid": pid,
+ "window_id": hwnd,
+ "actual_foreground_window_id": now,
+ "previous_fg_hwnd": format!("0x{prev:x}"),
+ "now_fg_hwnd": format!("0x{now:x}"),
+ "target_hwnd": format!("0x{hwnd:x}"),
+ "landed_on_target": false,
+ "raised": raised,
+ "restored": restored,
+ })
+ };
+ if focused {
+ ToolResult::text(msg).with_structured(structured)
+ } else {
+ ToolResult::error(msg).with_structured(structured)
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("bring_to_front: {e}")),
+ Err(join) => {
+ ToolResult::error(format!("bring_to_front: blocking-task join error: {join}"))
+ }
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/check_permissions.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/check_permissions.rs
new file mode 100644
index 0000000000..36f2805b14
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/check_permissions.rs
@@ -0,0 +1,172 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── check_permissions ─────────────────────────────────────────────────────────
+
+pub struct CheckPermissionsTool;
+static PERMS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for CheckPermissionsTool {
+ fn def(&self) -> &ToolDef {
+ PERMS_DEF.get_or_init(|| ToolDef {
+ name: "check_permissions".into(),
+ description: "Check required permissions for cua-driver-rs on Windows.".into(),
+ input_schema: json!({"type":"object","properties":{"prompt":{"type":"boolean","description":"Ignored: Windows has no permission prompts."}},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ // `elevated` reports the daemon's actual token integrity level, NOT the
+ // legacy `TokenIsElevated` UAC-elevation flag. The two diverge in two
+ // important cases that matter to cua-driver:
+ //
+ // 1. RID-500 built-in Administrator — runs at High IL all the time
+ // because RID 500 has no UAC split token (FilterAdministratorToken=0
+ // default). `TokenIsElevated` returns false (never UAC-prompted)
+ // but the IL is High and admin-level access works fine.
+ //
+ // 2. Task-Scheduler-spawned processes registered at RunLevel=Highest
+ // and triggered by AtLogon — Task Scheduler hands out the user's
+ // full admin token without going through the UAC prompt path, so
+ // `TokenIsElevated` returns false while IL is High. This is the
+ // common case for cua-driver's autostart task on non-RID-500
+ // admin users — the daemon IS at High IL (cross-AppContainer UIA
+ // works, Calculator-class UWP apps drive cleanly) but the legacy
+ // flag reports it as a "standard user", which confused everyone
+ // tracking #1602 / #1601. See issue #1640.
+ //
+ // #1646 / dogfood-iteration-3: PR #1647's in-process windows-rs
+ // implementation of TokenIntegrityLevel STILL misreported the IL.
+ // Verified externally on the cuademo dogfood VM: a daemon at
+ // demonstrably High IL (RID 0x3000 by direct PowerShell-C# Win32
+ // read against the daemon's pid) was reported by the daemon's own
+ // check_permissions as Medium IL (RID 0x2000). Both code paths used
+ // identical Win32 APIs (OpenProcess + OpenProcessToken +
+ // GetTokenInformation(TokenIntegrityLevel=25)), yet they disagreed.
+ // Root cause unclear — possibly a subtle interaction between the
+ // windows-rs crate's bindings and the named-pipe server's thread
+ // security context, possibly a Vec-buffer alignment issue that
+ // doesn't reproduce in standalone testing. RevertToSelf +
+ // OpenProcess(GetCurrentProcessId()) didn't help.
+ //
+ // Pragmatic workaround: shell out to PowerShell and run the same
+ // C# Win32 helper that's verified to return the correct IL. Cost
+ // is one PowerShell spawn (~100-200ms) per check_permissions call,
+ // which is acceptable — check_permissions is a low-frequency,
+ // user-initiated diagnostic, not a hot path.
+ const HIGH_IL_RID: u32 = 0x3000;
+
+ let il_rid: Option<u32> = read_self_integrity_level_via_powershell();
+
+ let is_elevated = il_rid.map(|r| r >= HIGH_IL_RID).unwrap_or(false);
+ let il_name = match il_rid {
+ Some(0x0000) => "Untrusted",
+ Some(0x1000) => "Low",
+ Some(0x2000) => "Medium",
+ Some(0x2100) => "Medium+",
+ Some(0x3000) => "High",
+ Some(0x4000) => "System",
+ Some(_) => "Unknown",
+ None => "Unavailable",
+ };
+ // Distinguish the unavailable case from a real measurement — don't
+ // render a fake RID 0x0000 or a fake elevation status when the lookup
+ // failed. See CodeRabbit review on PR #1641.
+ let status_text = match il_rid {
+ Some(rid) => format!(
+ "Process integrity level: {il_name} (RID 0x{rid:04X}, {})\n\
+ UIA accessibility: available (no special permission needed)\n\
+ PostMessage injection: available",
+ if is_elevated {
+ "elevated"
+ } else {
+ "non-elevated"
+ }
+ ),
+ None => format!(
+ "Process integrity level: {il_name} (token query failed)\n\
+ UIA accessibility: available (no special permission needed)\n\
+ PostMessage injection: available"
+ ),
+ };
+ ToolResult::text(status_text).with_structured(json!({
+ "elevated": is_elevated,
+ "integrity_level": il_name,
+ "integrity_level_rid": il_rid,
+ "uia": true,
+ "post_message": true
+ }))
+ }
+}
+
+/// Read the current process's token integrity level (RID) by shelling out
+/// to PowerShell and running a P/Invoke-based C# helper. See the long
+/// comment in `CheckPermissionsTool::invoke` for why the in-process
+/// windows-rs approach was abandoned (issue #1646 / dogfood iteration 3).
+///
+/// Returns `Some(rid)` on success (e.g. `0x3000` for High), `None` if
+/// PowerShell exits non-zero or stdout isn't a u32. Spawns one short-lived
+/// `powershell.exe` process per call; only used from `check_permissions`,
+/// which is a low-frequency, user-initiated diagnostic.
+fn read_self_integrity_level_via_powershell() -> Option<u32> {
+ let pid = std::process::id();
+ // Inline the same C# helper we verified externally on the dogfood VM.
+ // Heredoc-wrap (`@" ... "@`) avoids shell-escape acrobatics around the
+ // C# braces / double-quotes. The `Write-Output` at the end is parsed
+ // as a plain integer.
+ let script = format!(
+ r#"
+$ErrorActionPreference = 'Stop'
+Add-Type -TypeDefinition @"
+using System;
+using System.Runtime.InteropServices;
+public static class IL {{
+ [DllImport("kernel32.dll")] public static extern IntPtr OpenProcess(int a, bool i, int p);
+ [DllImport("kernel32.dll")] public static extern bool CloseHandle(IntPtr h);
+ [DllImport("advapi32.dll")] public static extern bool OpenProcessToken(IntPtr p, int a, out IntPtr t);
+ [DllImport("advapi32.dll")] public static extern bool GetTokenInformation(IntPtr t, int c, IntPtr i, int l, out int n);
+ [DllImport("advapi32.dll")] public static extern IntPtr GetSidSubAuthority(IntPtr s, int i);
+ [DllImport("advapi32.dll")] public static extern IntPtr GetSidSubAuthorityCount(IntPtr s);
+ public static uint? Rid(int pid) {{
+ var h = OpenProcess(0x1000, false, pid); if (h == IntPtr.Zero) return null;
+ try {{ IntPtr t; if (!OpenProcessToken(h, 0x8, out t)) return null;
+ try {{ int n; GetTokenInformation(t, 25, IntPtr.Zero, 0, out n); if (n == 0) return null;
+ var b = Marshal.AllocHGlobal(n); try {{ if (!GetTokenInformation(t, 25, b, n, out n)) return null;
+ var s = Marshal.ReadIntPtr(b); var c = Marshal.ReadByte(GetSidSubAuthorityCount(s));
+ return (uint)Marshal.ReadInt32(GetSidSubAuthority(s, c - 1)); }}
+ finally {{ Marshal.FreeHGlobal(b); }} }}
+ finally {{ CloseHandle(t); }} }}
+ finally {{ CloseHandle(h); }}
+ }}
+}}
+"@
+$rid = [IL]::Rid({pid})
+if ($null -ne $rid) {{ Write-Output ([int]$rid) }}
+"#,
+ pid = pid
+ );
+ let out = std::process::Command::new("powershell.exe")
+ .args([
+ "-NoProfile",
+ "-NonInteractive",
+ "-ExecutionPolicy",
+ "Bypass",
+ "-Command",
+ &script,
+ ])
+ .output()
+ .ok()?;
+ if !out.status.success() {
+ return None;
+ }
+ let stdout = String::from_utf8_lossy(&out.stdout);
+ stdout.trim().parse::<u32>().ok()
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/click.rs
new file mode 100644
index 0000000000..e76b76ddce
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/click.rs
@@ -0,0 +1,1530 @@
+use async_trait::async_trait;
+use cua_driver_contract::ClickButton;
+use cua_driver_core::{
+ action_record::ActionTransport,
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_legacy_click_input,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ bitmap_to_screen, element_target_rect, is_windowless_desktop_action, minimized_element_refusal,
+ overlay_glide_to, overlay_glide_to_target, pin_overlay_above, resolve_cursor_key,
+ resolve_onscreen_point_with_scroll, restore_foreground_polling_best_effort, screenshot_scale,
+ winui3_background_gesture, ToolState,
+};
+
+fn tool_result_text(result: ToolResult) -> String {
+ result
+ .content
+ .into_iter()
+ .find_map(|content| match content {
+ cua_driver_core::protocol::Content::Text { text, .. } => Some(text),
+ _ => None,
+ })
+ .unwrap_or_else(|| "element action was refused".to_owned())
+}
+
+fn capture_admission_refusal(error: anyhow::Error) -> ToolResult {
+ let code = cua_driver_core::capture_runtime::admission_error_code(&error);
+ ToolResult::error(format!("Capture-bound click refused: {error}")).with_structured(json!({
+ "code": code,
+ "effect": "refused",
+ "detail": error.to_string(),
+ }))
+}
+
+// ── click ─────────────────────────────────────────────────────────────────────
+
+fn posted_pixel_click_result(pid: u32, click_word: &str) -> ToolResult {
+ ToolResult::text(format!("✅ Posted {click_word} to pid {pid}.")).with_structured(json!({
+ "path": "post_message",
+ "verified": false,
+ "effect": "unverifiable"
+ }))
+}
+
+/// Only a completed miss permits another transport. A provider may finish an
+/// activation after the deadline, so errors must not replay the click.
+fn finish_pixel_uia_attempt(
+ outcome: crate::uia::windows_enum::PointInvokeOutcome,
+ pid: u32,
+ sx: i32,
+ sy: i32,
+) -> Option<ToolResult> {
+ use crate::uia::windows_enum::PointInvokeOutcome;
+ let status = match outcome {
+ PointInvokeOutcome::Miss => return None,
+ PointInvokeOutcome::Invoked => {
+ return Some(
+ ToolResult::text(format!(
+ "✅ Performed UIA Invoke at ({sx},{sy}) for pid {pid}."
+ ))
+ .with_structured(json!({
+ "path": "ax", "verified": false, "effect": "unverifiable"
+ })),
+ );
+ }
+ PointInvokeOutcome::Busy => "busy",
+ PointInvokeOutcome::Timeout => "timeout",
+ PointInvokeOutcome::Unavailable => "unavailable",
+ };
+ Some(cua_driver_core::delivery::background_unavailable_result(
+ format!(
+ "UIA pixel click {status} for pid {pid}. No fallback input was sent. \
+ The click effect is unknown; inspect the target state before another action. \
+ If the provider does not recover, retry this action with delivery_mode:\"foreground\"."
+ ),
+ "background_unavailable",
+ format!(
+ "the UIA provider is {status}; retry this action with delivery_mode:\"foreground\"."
+ ),
+ json!({
+ "uia_status": status,
+ "path": "ax",
+ "verified": false,
+ "effect": "unverifiable",
+ }),
+ ))
+}
+
+enum BackgroundElementClick {
+ Semantic {
+ message: String,
+ transport: ActionTransport,
+ failed_calls: Vec<ActionTransport>,
+ },
+ Inject {
+ x: i32,
+ y: i32,
+ failed_calls: Vec<ActionTransport>,
+ },
+ Post {
+ x: i32,
+ y: i32,
+ failed_calls: Vec<ActionTransport>,
+ },
+}
+
+fn background_element_click_result(
+ message: String,
+ transport: ActionTransport,
+ failed_calls: Vec<ActionTransport>,
+) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionAttempt, ActionEffect, ActionExecutionRecord, ActionFallback, ActualDelivery,
+ RequestedDelivery,
+ };
+ let path = match transport {
+ ActionTransport::WindowsTargetedInjection => "pixel",
+ ActionTransport::WindowsPostMessage => "post_message",
+ _ => "ax",
+ };
+ let mut record = ActionExecutionRecord::new(
+ ActionEffect::Unverifiable,
+ transport,
+ RequestedDelivery::Background,
+ );
+ record.actual_delivery = Some(ActualDelivery::Background);
+ for (index, attempted) in failed_calls.iter().copied().enumerate() {
+ record.attempts.push(ActionAttempt {
+ transport: attempted,
+ delivery: ActualDelivery::Background,
+ detail: Some("UIA provider action returned an error".into()),
+ });
+ record.fallbacks.push(ActionFallback {
+ from: attempted,
+ to: failed_calls.get(index + 1).copied().unwrap_or(transport),
+ reason: "Previous UIA provider action failed".into(),
+ });
+ }
+ ToolResult::text(message)
+ .with_structured(json!({ "path": path, "verified": false, "effect": "unverifiable" }))
+ .with_action_record(record)
+}
+
+pub struct ClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ClickTool {
+ fn def(&self) -> &ToolDef {
+ CLICK_DEF.get_or_init(|| ToolDef {
+ name: "click".into(),
+ // Description matches Swift `ClickTool.swift` semantics (left-click
+ // primitive with two addressing modes — element_token via UIA, or
+ // window-local pixel coords via PostMessage). Windows-only schema
+ // extras (`button`; `action` limited to "expand"; no
+ // `debug_image_out`) apply here.
+ description: "Left-click against a target pid. **Prefer `element_token` over \
+ pixel coordinates** — element_token works on backgrounded / minimized / \
+ hidden / off-desktop windows, surfaces a stable handle that survives \
+ rebuilds, and tells you what you're clicking via the cached element's \
+ role + label. Reach for `x, y` only when the target is a canvas / video / \
+ WebGL / custom-drawn surface that doesn't appear in the UIA tree.\n\n\
+ Two addressing modes:\n\n\
+ - `element_token` (from the last `get_window_state` snapshot \
+ of that window) — performs the UIA Invoke pattern on the cached element via \
+ PostMessage. No cursor move, no focus steal. Requires a prior \
+ `get_window_state(pid, window_id)` in this turn; the token is stale once \
+ the next snapshot of the same window replaces it.\n\n\
+ - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
+ returned by `get_window_state`). On `delivery_mode:\"background\"` (the \
+ default), cua-driver FIRST does a UIA hit-test at the resolved \
+ screen position: if the deepest invokable element under that point exposes \
+ `InvokePattern`, it's invoked through the accessibility channel — same \
+ background-safe path as the `element_token` mode, no foreground swap, no \
+ visible flash. **This makes pixel clicks on UWP / WinUI3 / Win11 packaged \
+ apps work flash-free out of the box** — agents should default to \
+ `delivery_mode:\"background\"` even on XAML hosts whose CoreInput dispatcher \
+ drops raw PostMessage. The PostMessage(WM_LBUTTONDOWN/UP) fallback only \
+ runs when the UIA hit-test misses (canvas / video / WebGL / custom-drawn \
+ surfaces with no UIA peer), and on those targets `delivery_mode:\"background\"` \
+ returns a structured `background_unavailable` error if PostMessage is \
+ known to drop too — at which point you switch to `delivery_mode:\"foreground\"`. \
+ **Always try the default `background` click first and let that error be \
+ your signal — do NOT pass `foreground` preemptively because the target \
+ 'looks like' GTK/Chromium/etc. The driver decides when background is \
+ impossible; a guessed foreground click needlessly steals the user's focus.** \
+ `count: 2` posts two down/up pairs for a double-click. Pixel clicks need \
+ a visible on-screen window to anchor the coordinate conversion (errors \
+ with `pid X has no on-screen window` otherwise).\n\n\
+ Exactly one of `element_token` or (`x` AND `y`) must be provided. \
+ `pid` is required for window scope and omitted for desktop scope. After a `zoom` call, \
+ pass `from_zoom=true` to auto-translate zoom-image coords back to \
+ full-window space.\n\n\
+ Windows-only convenience: `button: \"left\"|\"right\"|\"middle\"` switches \
+ the mouse button (Swift exposes right-click as a separate `right_click` \
+ tool). `action: \"expand\"` on an `element_token` opens the element \
+ through UIA ExpandCollapsePattern (the dropdown half of an MSAA split \
+ button); other actions use the default invoke. `debug_image_out` isn't \
+ supported yet.".into(),
+ input_schema: json!({
+ "type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID for window scope. Optional with element_token (the token carries the pid). Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
+ "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "action":{"type":"string","description":"Accessibility action for an element_token target. \"expand\" opens the element through UIA ExpandCollapsePattern, or clicks the dropdown half of an MSAA split button. Omit for the default invoke."},
+ "capture_id":{"type":"string","minLength":1,"description":"Optional ID from get_window_state or get_desktop_state. When present, x and y are admitted against that exact capture and the ID is consumed before native dispatch."},
+ "x":{"type":"number","description":"X in window-local screenshot pixels — same space as the PNG get_window_state returns. Must be provided together with y."},
+ "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
+ "button": cua_driver_core::tool_schema::button_schema(),
+ "count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Coordinate frame (default \"window\"). Pass \"desktop\" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use crate::uia::snapshot::SnapshotKind;
+ use cua_driver_core::tool_args::ArgsExt;
+ let cursor_key = resolve_cursor_key(&args);
+
+ // ── Window-less screen-absolute branch (desktop target) ───────────────
+ // When the caller gives x,y with NO pid/window_id, treat x,y as TRUE
+ // SCREEN pixels. The core registry has already enforced the session's
+ // effective capture scope; this local check validates the action form.
+ let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
+ let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
+ let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
+ && args.get("y").map(|v| v.is_number()).unwrap_or(false);
+ if !has_pid && !has_window_id && has_xy {
+ if !is_windowless_desktop_action(&args) {
+ return ToolResult::error(
+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",
+ )
+ .with_structured(json!({
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
+ }));
+ }
+ let input = match parse_legacy_click_input(&args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let button = input
+ .button
+ .unwrap_or(ClickButton::Left)
+ .as_str()
+ .to_owned();
+ let count = input.count.unwrap_or(1) as usize;
+ if count == 0 {
+ return ToolResult::error("click.count must be at least 1.")
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ let modifiers: Vec<String> = args.str_array("modifier");
+ let capture_point = match self.state.capture_bridge.as_ref() {
+ Some(bridge) => match bridge.admit_click(
+ &args,
+ crate::capture_admission::WindowsCaptureTarget::PrimaryDesktop,
+ input.x,
+ input.y,
+ ) {
+ Ok(point) => point,
+ Err(error) => return capture_admission_refusal(error),
+ },
+ None if args.get("capture_id").is_some() => {
+ return capture_admission_refusal(anyhow::anyhow!(
+ "capture service is unavailable"
+ ))
+ }
+ None => None,
+ };
+ let (sx, sy) = match capture_point {
+ Some((action_x, action_y)) => {
+ match crate::capture_admission::round_action_point(action_x, action_y) {
+ Ok(point) => point,
+ Err(error) => return capture_admission_refusal(error),
+ }
+ }
+ None => (input.x as i32, input.y as i32),
+ };
+
+ // Resolve the application window before moving the agent cursor.
+ // WindowFromPoint can return a transparent layered overlay, and the
+ // cursor overlay is about to occupy this exact screen point.
+ let root = unsafe {
+ use windows::Win32::Foundation::POINT;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetAncestor, WindowFromPoint, GA_ROOT,
+ };
+ let target = WindowFromPoint(POINT { x: sx, y: sy });
+ if target.0.is_null() {
+ return ToolResult::error(format!("No window under screen point ({sx},{sy})."));
+ }
+ let root = GetAncestor(target, GA_ROOT);
+ if root.0.is_null() {
+ target
+ } else {
+ root
+ }
+ };
+ let hwnd_u = root.0 as u64;
+
+ // Animate the agent cursor to the screen point, then click.
+ overlay_glide_to(&cursor_key, sx as f64, sy as f64).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx as f64,
+ y: sy as f64,
+ },
+ );
+
+ // Click the HWND that owned the pixel before the driver overlay
+ // moved there. The active SendInput path performs the foreground
+ // swap and UIPI checks needed for Chromium and retained-mode apps.
+ let send_result = tokio::task::spawn_blocking(move || -> anyhow::Result<u64> {
+ let mod_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_click_synthesized_desktop_mods(
+ hwnd_u, sx, sy, count, &button, &mod_refs,
+ )?;
+ Ok(hwnd_u)
+ })
+ .await;
+ return match send_result {
+ Ok(Ok(hwnd_u)) => {
+ let click_word = match count {
+ 2 => "double-click",
+ 3 => "triple-click",
+ _ => "click",
+ };
+ ToolResult::text(format!(
+ "✅ Sent {click_word} via SendInput at screen ({sx},{sy}) on HWND 0x{hwnd_u:x} (desktop scope)."
+ ))
+ // SendInput dispatched but is not driver-verifiable (no read-back).
+ .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" }))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ let pid = match args.require_u32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let x = args.opt_f64("x");
+ let y = args.opt_f64("y");
+ // Surface 5: explicit rejection of unknown buttons so a typo doesn't fall
+ // through to the SendInput `_ => MOUSEEVENTF_LEFTDOWN/UP` default and
+ // silently emit a left-click. Empty / missing keeps back-compat.
+ let button_raw = args.str_or("button", "left").to_lowercase();
+ if !matches!(button_raw.as_str(), "" | "left" | "right" | "middle") {
+ return ToolResult::error(format!(
+ "click: unknown button \"{button_raw}\" — expected one of left, right, middle."
+ ));
+ }
+ let button = if button_raw.is_empty() {
+ "left".to_string()
+ } else {
+ button_raw
+ };
+ let count = args.u64_or("count", 1) as usize;
+ // macOS click `modifier` surface: held keys (cmd/shift/option/ctrl). Only
+ // the SendInput rung (delivery_mode:"foreground" pixel tap, and the
+ // window-less desktop-scope click above) can hold them; the background
+ // UIA-Invoke / PostMessage rungs have no keyboard state to carry them, so
+ // a modifier passed to a background click is necessarily ignored there.
+ let modifiers: Vec<String> = args.str_array("modifier");
+ let delivery = DeliveryMode::from_args(&args);
+ if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {
+ return ToolResult::error(
+ "click modifiers require delivery_mode:\"foreground\" on Windows; UIA and \
+ PostMessage cannot carry live modifier-key state",
+ )
+ .with_structured(json!({
+ "code": "background_unavailable",
+ "effect": "refused",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "Windows modifier clicks require SendInput so the target observes live modifier state"
+ }
+ }));
+ }
+ // Optional `action` arg picks among the actions exposed in the
+ // accessibility tree. Today this only changes behavior for MSAA
+ // BUTTONDROPDOWN: `"expand"` clicks the right-edge (dropdown arrow
+ // half) instead of the center (press half). Defaults to first
+ // action in the element's `actions=[...]` list, which preserves
+ // existing semantics for UIA elements.
+ let action_req = args
+ .get("action")
+ .and_then(|v| v.as_str())
+ .map(|s| s.to_string());
+
+ // Resolve HWND: explicit, or auto from pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows_via_win32(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ let capture_point = if args.get("capture_id").is_some() {
+ if elem_idx.is_some()
+ || x.is_none()
+ || y.is_none()
+ || hwnd_opt.is_none()
+ || args.get("from_zoom").and_then(Value::as_bool) == Some(true)
+ {
+ return capture_admission_refusal(anyhow::anyhow!(
+ "capture_id requires x, y, and an exact window_id; element and zoom coordinates are not capture-bound"
+ ));
+ }
+ let Some(bridge) = &self.state.capture_bridge else {
+ return capture_admission_refusal(anyhow::anyhow!(
+ "capture service is unavailable"
+ ));
+ };
+ match bridge.admit_click(
+ &args,
+ crate::capture_admission::WindowsCaptureTarget::Window {
+ pid,
+ window_id: hwnd,
+ },
+ x.unwrap(),
+ y.unwrap(),
+ ) {
+ Ok(Some(point)) => Some(point),
+ Ok(None) => unreachable!("capture_id was present"),
+ Err(error) => return capture_admission_refusal(error),
+ }
+ } else {
+ None
+ };
+
+ // Arm this only after capture admission so stale or mismatched IDs have
+ // no native side effect before the click is refused.
+ let _noact = match (delivery != DeliveryMode::Foreground, hwnd_opt) {
+ (true, Some(h)) => Some(crate::input::NoActivateGuard::arm(
+ windows::Win32::Foundation::HWND(h as *mut _),
+ )),
+ _ => None,
+ };
+
+ if let Some(idx) = elem_idx {
+ // ── MSAA dispatch (SAL/VCL targets) ────────────────────────────
+ // For MSAA elements, route by role:
+ // - `action:"expand"` on BUTTONDROPDOWN → SendInput at the
+ // cached rect's right-edge (the dropdown arrow half).
+ // Opens the picker (e.g. LO Writer Font Color → SALTMPSUBFRAME).
+ // - default action / `"invoke"` → SendInput at center
+ // (matches `accDoDefaultAction` semantics for VCL controls
+ // and works through delivery_mode:"foreground"). We DO NOT call
+ // `accDoDefaultAction` here because LO's MSAA impl applies
+ // the change asynchronously and returns S_OK either way,
+ // so the visible behavior is the same as a center click.
+ if let Some((SnapshotKind::Msaa, role)) = admitted
+ .as_ref()
+ .map(|element| (element.kind, element.msaa_role))
+ {
+ const ROLE_BUTTONDROPDOWN: i32 = 0x38;
+ const ROLE_BUTTONMENU: i32 = 0x39;
+ const ROLE_BUTTONDROPDOWNGRID: i32 = 0x3A;
+ const ROLE_SPLITBUTTON: i32 = 0x3E;
+ let want_expand = action_req.as_deref() == Some("expand");
+ let is_dropdown_role = matches!(
+ role,
+ Some(
+ ROLE_BUTTONDROPDOWN
+ | ROLE_BUTTONMENU
+ | ROLE_BUTTONDROPDOWNGRID
+ | ROLE_SPLITBUTTON
+ )
+ );
+ let (tx, ty) = if want_expand && is_dropdown_role {
+ match admitted.as_ref().and_then(|element| element.rect) {
+ Some((_l, t, r, b)) => {
+ // Right-edge of the SplitButton — the dropdown
+ // arrow half. -4 puts the click safely inside
+ // the arrow region for the typical ~12-16 px
+ // arrow width VCL uses.
+ (r - 4, (t + b) / 2)
+ }
+ None => {
+ return ToolResult::error(format!(
+ "MSAA element [{idx}] has no cached rect — \
+ cannot dispatch action:\"expand\". Re-run \
+ get_window_state to refresh the cache."
+ ));
+ }
+ }
+ } else if want_expand && !is_dropdown_role {
+ return ToolResult::error(format!(
+ "action:\"expand\" requested for MSAA element [{idx}] \
+ but its role ({role:?}) is not a dropdown button. \
+ Use action:\"invoke\" or omit the action arg."
+ ));
+ } else {
+ match admitted.as_ref().map(|element| element.center) {
+ Some(v) => v,
+ None => {
+ return ToolResult::error(format!(
+ "MSAA element [{idx}] not in cache for hwnd={hwnd}. \
+ Call get_window_state first."
+ ))
+ }
+ }
+ };
+ if admitted
+ .as_ref()
+ .is_some_and(|element| element.read_while_minimized)
+ {
+ if let Err(refusal) =
+ resolve_onscreen_point_with_scroll(&admitted, hwnd, idx, tx, ty, "clicking")
+ {
+ return refusal;
+ }
+ }
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ tx as f64,
+ ty as f64,
+ element_target_rect(admitted.as_ref(), (0, 0)),
+ )
+ .await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: tx as f64,
+ y: ty as f64,
+ },
+ );
+ let btn_fg = button.clone();
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let mods_owned = modifiers.clone();
+ let activate = delivery == DeliveryMode::Foreground;
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ if activate {
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, tx, ty, count, &btn_fg, &mod_refs,
+ )
+ } else {
+ crate::input::send_click_synthesized_mods(
+ hwnd, tx, ty, count, &btn_fg, &mod_refs,
+ )
+ }
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ let half = if want_expand { "dropdown" } else { "press" };
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Performed SendInput click on MSAA [{idx}] {half} half at ({tx},{ty})."
+ ))
+ .with_structured(
+ json!({ "path": "msaa", "verified": false, "effect": "unverifiable" }),
+ ),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // UIA path: get cached center (no COM call needed — captured at walk time).
+ let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
+ Some(v) => v,
+ None => {
+ return ToolResult::error(format!(
+ "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
+ ))
+ }
+ };
+ // NB: the off-screen guard is applied per-delivery-path below (the
+ // foreground SendInput tap and the background coordinate injection),
+ // NOT here — a background UIA Invoke fires the element's handler
+ // regardless of whether the element is scrolled into view, so a broad
+ // guard would wrongly block legitimate actions like opening a modal
+ // from an off-screen button.
+ // Step 2: pin overlay to target window, then animate to screen coords.
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(admitted.as_ref(), (0, 0)),
+ )
+ .await;
+ // Step 3: click pulse + actual click.
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: cx as f64,
+ y: cy as f64,
+ },
+ );
+ let btn = button.clone();
+
+ // An explicit accessibility action is a semantic request, not a
+ // pixel gesture hint. Route expand through ExpandCollapsePattern
+ // even when foreground delivery was allowed; this reliably opens
+ // WPF/WinUI menus and tree nodes whose visual click target is
+ // transient or scroll-adjusted.
+ if action_req.as_deref() == Some("expand") {
+ let expand = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || -> anyhow::Result<()> {
+ let _admission = &admitted;
+ use windows::core::Interface;
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationExpandCollapsePattern,
+ UIA_ExpandCollapsePatternId,
+ };
+
+ let retained = admitted.as_ref().ok_or_else(|| {
+ anyhow::anyhow!("element [{idx}] is not in the UIA cache")
+ })?;
+ if !retained.is_uia() {
+ anyhow::bail!("element [{idx}] is not a UIA element");
+ }
+ let element = std::mem::ManuallyDrop::new(unsafe {
+ IUIAutomationElement::from_raw(retained.as_ptr() as *mut _)
+ });
+ let pattern = unsafe {
+ element
+ .GetCurrentPattern(UIA_ExpandCollapsePatternId)
+ .and_then(|value| {
+ value.cast::<IUIAutomationExpandCollapsePattern>()
+ })
+ }
+ .map_err(|error| {
+ anyhow::anyhow!("ExpandCollapsePattern unavailable: {error}")
+ })?;
+ let result =
+ crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
+ pattern.Expand()
+ });
+ std::mem::forget(element);
+ result.map_err(|error| {
+ anyhow::anyhow!("ExpandCollapse.Expand failed: {error}")
+ })
+ }
+ })
+ .await;
+ return match expand {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Expanded UIA element [{idx}] via ExpandCollapsePattern."
+ ))
+ .with_structured(json!({ "path": "uia_expand_collapse", "verified": false, "effect": "unverifiable" })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ // delivery_mode:"foreground" — skip UIA Invoke and use SendInput at the
+ // cached element center. The caller explicitly chose foreground
+ // delivery; UIA Invoke would be background-safe (which they
+ // rejected) and might fail on elements that don't support
+ // InvokePattern anyway.
+ if delivery == DeliveryMode::Foreground {
+ // Foreground delivery is a real SendInput tap at (cx,cy); if the
+ // element is off-screen, scroll it into view and re-resolve so the
+ // tap doesn't land on the taskbar, else keep the clean failure.
+ // Modified selection needs one stronger preflight: some WPF
+ // ScrollViewers report a clipped item as on-screen because its
+ // stale rectangle is still inside the outer HWND. Ask the item
+ // to scroll itself into view before trusting that rectangle.
+ let recorded_center = (cx, cy);
+ let (cx, cy) = if !modifiers.is_empty() {
+ admitted
+ .clone()
+ .and_then(|retained| {
+ retained.is_uia().then(|| unsafe {
+ crate::uia::scroll::scroll_into_view_and_recenter(
+ hwnd,
+ retained.as_ptr(),
+ )
+ })
+ })
+ .flatten()
+ .unwrap_or((cx, cy))
+ } else {
+ (cx, cy)
+ };
+ let (cx, cy) = match resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx,
+ cx,
+ cy,
+ "a foreground click",
+ ) {
+ Ok(p) => p,
+ Err(result) => return result,
+ };
+ if (cx, cy) != recorded_center {
+ crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
+ }
+ let btn_fg = btn.clone();
+ let mods_owned = modifiers.clone();
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, cx, cy, count, &btn_fg, &mod_refs,
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Performed SendInput click on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
+ ))
+ .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // UIA and posted-message routes can report success without a
+ // user-visible effect on an iconic target. Refuse before any
+ // background route selects or dispatches its transport.
+ if let Some(result) = minimized_element_refusal(hwnd, idx, cx, cy, "clicking") {
+ return result;
+ }
+ // delivery_mode:"background" on WinUI3 for double / right / middle: single
+ // left falls through to the UIA Invoke path below (already drives
+ // WinUI3). Double-left lands via a double UIA Invoke; right/middle
+ // can't both land and hold the contract → structured error.
+ if delivery == DeliveryMode::Background
+ && (count > 1 || btn == "right" || btn == "middle")
+ {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, Some(idx), count, &btn).await
+ {
+ return r;
+ }
+ }
+ // Chromium's UIA Invoke raises a fully occluded renderer, while a
+ // targeted PostMessage left click reaches the renderer without
+ // changing foreground, z-order, or the real cursor. Keep AX for
+ // target resolution and use the posted-message transport only for
+ // the empirically verified single-left-click shape.
+ if delivery == DeliveryMode::Background
+ && btn == "left"
+ && count == 1
+ && crate::input::is_chromium_target_window(hwnd)
+ {
+ // This route posts at the cached center without
+ // `resolve_onscreen_point_with_scroll`, so check it here.
+ if admitted
+ .as_ref()
+ .is_some_and(|element| element.read_while_minimized)
+ {
+ let message = format!(
+ "Element [{idx}] resolves to ({cx},{cy}), which was read while \
+ window 0x{hwnd:x} was minimized. Re-snapshot with get_window_state \
+ before retrying."
+ );
+ return ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "code": "element_not_visible",
+ "effect": "refused",
+ "refusal": {
+ "code": "element_not_visible",
+ "message": message,
+ }
+ }));
+ }
+ let posted = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, cx, cy, count, &btn)
+ }
+ })
+ .await;
+ return match posted {
+ Ok(Ok(())) => background_element_click_result(
+ format!(
+ "✅ Posted click on Chromium element [{idx}] at screen ({cx},{cy}) \
+ (background, no foreground swap)."
+ ),
+ ActionTransport::WindowsPostMessage,
+ vec![],
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ // Try UIA Invoke first (it works for UWP / modern XAML / web
+ // content where PostMessage(WM_LBUTTONDOWN) hits the outer
+ // HWND but never reaches the inner XAML/composition element).
+ // Fall through to PostMessage when:
+ // - element doesn't support InvokePattern (most text inputs,
+ // non-interactive elements)
+ // - right-click (UIA's ExpandCollapse / ShowContextMenu are
+ // different patterns; keep PostMessage for now)
+ // - count > 1 (double-click semantics aren't an Invoke
+ // concept — PostMessage produces the actual WM_LBUTTONDBLCLK)
+ let use_uia_invoke = (btn == "left" || btn == "middle") && count == 1;
+ let result = tokio::task::spawn_blocking({ let admitted = admitted.clone(); move || -> anyhow::Result<BackgroundElementClick> {
+ let _admission = &admitted;
+ let mut failed_calls = Vec::new();
+ // Direct Chromium UIA Invoke can return S_OK without firing a
+ // DOM event while occluded. Try the honest coordinate actuator
+ // first: it lands while visible and reports occlusion without
+ // raising the window when hidden.
+ if delivery == DeliveryMode::Background
+ && crate::input::is_chromium_target_window(hwnd)
+ {
+ let (cx, cy) = resolve_onscreen_point_with_scroll(
+ &admitted, hwnd, idx, cx, cy, "clicking",
+ )
+ .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
+ return Ok(BackgroundElementClick::Inject { x: cx, y: cy, failed_calls });
+ }
+ if use_uia_invoke {
+ // Retain the element out of the cache (AddRef under the
+ // cache lock) so it can't be freed by a concurrent
+ // get_window_state on the same (pid, hwnd) while this click
+ // is mid-flight (use-after-free → daemon crash). The guard
+ // lives to the end of this `if let` block, past every UIA
+ // pattern dispatch below; its Release fires when it drops.
+ if let Some(element_guard) = admitted.as_ref() {
+ let ptr = element_guard.as_ptr();
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationInvokePattern,
+ IUIAutomationTogglePattern, IUIAutomationSelectionItemPattern,
+ IUIAutomationExpandCollapsePattern,
+ UIA_InvokePatternId, UIA_TogglePatternId,
+ UIA_SelectionItemPatternId, UIA_ExpandCollapsePatternId,
+ };
+ use windows::core::Interface;
+ let elem: IUIAutomationElement =
+ unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
+
+ // Try patterns in order of click-semantics specificity:
+ // 1. Invoke - buttons / hyperlinks (canonical)
+ // 2. Toggle - checkboxes (no Invoke)
+ // 3. SelectionItem - radio buttons, listbox items
+ // 4. ExpandCollapse - combo boxes, tree nodes
+ // Each call is wrapped in the UWP foreground-steal bypass.
+ //
+ // Why this order: PR #1699's WinUI3 control parity tests
+ // surfaced these pattern-dispatch gaps. Without these
+ // fallthroughs, cua-driver couldn't drive WinUI3
+ // CheckBox / RadioButton / ComboBox because PostMessage
+ // doesn't reach their handlers (CoreInput dispatcher).
+ let invoke_result = unsafe { elem.GetCurrentPattern(UIA_InvokePatternId) };
+ if let Ok(pattern) = invoke_result {
+ if let Ok(inv) = pattern.cast::<IUIAutomationInvokePattern>() {
+ let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
+ hwnd as isize, || unsafe { inv.Invoke() });
+ match outcome {
+ Ok(()) => {
+ std::mem::forget(elem);
+ return Ok(BackgroundElementClick::Semantic { message: format!(
+ "✅ Performed UIA Invoke on [{idx}] (screen ({cx},{cy}))."
+ ), transport: ActionTransport::WindowsUiaInvoke, failed_calls });
+ }
+ Err(e) => tracing::debug!(target: "click",
+ "UIA Invoke on [{idx}]: {e}, trying Toggle"),
+ }
+ failed_calls.push(ActionTransport::WindowsUiaInvoke);
+ }
+ }
+ let toggle_result = unsafe { elem.GetCurrentPattern(UIA_TogglePatternId) };
+ if let Ok(pattern) = toggle_result {
+ if let Ok(tg) = pattern.cast::<IUIAutomationTogglePattern>() {
+ let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
+ hwnd as isize, || unsafe { tg.Toggle() });
+ if outcome.is_ok() {
+ std::mem::forget(elem);
+ return Ok(BackgroundElementClick::Semantic { message: format!(
+ "✅ Performed UIA Toggle on [{idx}] (screen ({cx},{cy}))."
+ ), transport: ActionTransport::WindowsUiaToggle, failed_calls });
+ }
+ failed_calls.push(ActionTransport::WindowsUiaToggle);
+ }
+ }
+ let sel_result = unsafe { elem.GetCurrentPattern(UIA_SelectionItemPatternId) };
+ if let Ok(pattern) = sel_result {
+ if let Ok(si) = pattern.cast::<IUIAutomationSelectionItemPattern>() {
+ let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
+ hwnd as isize, || unsafe { si.Select() });
+ if outcome.is_ok() {
+ std::mem::forget(elem);
+ return Ok(BackgroundElementClick::Semantic { message: format!(
+ "✅ Performed UIA SelectionItem.Select on [{idx}] (screen ({cx},{cy}))."
+ ), transport: ActionTransport::WindowsUiaSelection, failed_calls });
+ }
+ failed_calls.push(ActionTransport::WindowsUiaSelection);
+ }
+ }
+ let exp_result = unsafe { elem.GetCurrentPattern(UIA_ExpandCollapsePatternId) };
+ if let Ok(pattern) = exp_result {
+ if let Ok(ec) = pattern.cast::<IUIAutomationExpandCollapsePattern>() {
+ let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
+ hwnd as isize, || unsafe { ec.Expand() });
+ if outcome.is_ok() {
+ std::mem::forget(elem);
+ return Ok(BackgroundElementClick::Semantic { message: format!(
+ "✅ Performed UIA ExpandCollapse.Expand on [{idx}] (screen ({cx},{cy}))."
+ ), transport: ActionTransport::WindowsUiaExpandCollapse, failed_calls });
+ }
+ failed_calls.push(ActionTransport::WindowsUiaExpandCollapse);
+ }
+ }
+ std::mem::forget(elem);
+ }
+ }
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(
+ hwnd,
+ EventKind::MouseClick,
+ )
+ {
+ let (cx, cy) = resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx,
+ cx,
+ cy,
+ "clicking",
+ )
+ .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
+ return Ok(BackgroundElementClick::Inject { x: cx, y: cy, failed_calls });
+ }
+ if admitted.as_ref().is_some_and(|element| element.read_while_minimized) {
+ let (x, y) =
+ resolve_onscreen_point_with_scroll(&admitted, hwnd, idx, cx, cy, "clicking")
+ .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
+ return Ok(BackgroundElementClick::Post { x, y, failed_calls });
+ }
+ Ok(BackgroundElementClick::Post { x: cx, y: cy, failed_calls })
+ } }).await;
+ let plan = match result {
+ Ok(Ok(plan)) => plan,
+ Ok(Err(e)) => return ToolResult::error(e.to_string()),
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ };
+ let (x, y, failed_calls, inject) = match plan {
+ BackgroundElementClick::Semantic {
+ message,
+ transport,
+ failed_calls,
+ } => {
+ return background_element_click_result(message, transport, failed_calls);
+ }
+ BackgroundElementClick::Inject { x, y, failed_calls } => (x, y, failed_calls, true),
+ BackgroundElementClick::Post { x, y, failed_calls } => (x, y, failed_calls, false),
+ };
+ // Return to the dispatch task before capturing: task-local recording
+ // context does not propagate into the blocking scroll resolver.
+ if (x, y) != (cx, cy) {
+ crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, x, y);
+ }
+ let btn = button.clone();
+ let action_name = if btn == "right" {
+ "ShowMenu"
+ } else {
+ "PostMessage click"
+ };
+ let sent = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ if inject {
+ crate::input::inject_click_screen(hwnd, x, y, count, &btn)
+ } else {
+ crate::input::post_click_screen(hwnd, x, y, count, &btn)
+ }
+ }
+ })
+ .await;
+ match sent {
+ Ok(Ok(())) => {
+ let (message, transport) = if inject {
+ (format!("✅ Injected click on [{idx}] (screen ({x},{y}), background, no foreground swap)."),
+ ActionTransport::WindowsTargetedInjection)
+ } else {
+ (
+ format!("✅ Performed {action_name} on [{idx}] (screen ({x},{y}))."),
+ ActionTransport::WindowsPostMessage,
+ )
+ };
+ background_element_click_result(message, transport, failed_calls)
+ }
+ Ok(Err(error)) if inject => {
+ crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ error.to_string(),
+ )
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ }
+ } else if let (Some(mut px), Some(mut py)) = (x, y) {
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if let Some((mapped_x, mapped_y)) = capture_point {
+ let (mapped_x, mapped_y) =
+ match crate::capture_admission::round_action_point(mapped_x, mapped_y) {
+ Ok(point) => point,
+ Err(error) => return capture_admission_refusal(error),
+ };
+ px = f64::from(mapped_x);
+ py = f64::from(mapped_y);
+ } else if from_zoom {
+ match self.state.zoom_context(&args, pid, Some(hwnd)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(px, py);
+ px = wx;
+ py = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else if !args.bool_or("_native_coordinates", false) {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ px *= ratio;
+ py *= ratio;
+ }
+ // px/py are bitmap pixels — see `bitmap_to_screen` for the
+ // mapping (DWM-frame top-left + 1-px inset, NOT ClientToScreen).
+ let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
+ let (sx, sy) = (sx_i as f64, sy_i as f64);
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to(&cursor_key, sx, sy).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ let btn = button.clone();
+ // Vision-mode (x, y) dispatch is **layered**:
+ //
+ // 1. UIA hit-test in the target HWND's subtree. If the
+ // deepest InvokePattern-bearing element at (sx, sy) is
+ // found, fire `Invoke()`. Works occluded, no focus
+ // steal, and is the **only** path that lands on UWP /
+ // WebView2 / DirectComposition surfaces — those route
+ // input through `Windows.UI.Input`, not WM_LBUTTONDOWN.
+ //
+ // 2. If UIA completes with a miss (no Invokable element under the
+ // pixel inside `hwnd`, or `hwnd` has no useful UIA
+ // tree at all), fall through to
+ // `PostMessage(WM_LBUTTONDOWN/UP)` against the deepest
+ // child HWND at the screen point. Covers plain Win32
+ // controls without UIA InvokePattern (most edit
+ // fields, custom-drawn widgets, native containers) and
+ // apps that don't expose UIA at all.
+ //
+ // The combination closes the gap macOS gets for free from
+ // `CGEventPostToPSN` (per-pid event routing): UIA covers
+ // UWP-style targets without z-order tricks, PostMessage
+ // covers plain Win32 without z-order tricks. Neither path
+ // moves the OS cursor or activates the receiver.
+ //
+ // UIA path runs only for single left/middle clicks —
+ // right-click (UIA has no by-coord `ShowContextMenu`) and
+ // multi-click (`Invoke()` is single-fire by definition)
+ // skip directly to PostMessage.
+ // delivery_mode:"foreground" short-circuit — caller explicitly chose
+ // foreground delivery. Skip the UIA hit-test (UIA Invoke is
+ // background-safe and would deliver the click without the
+ // foreground swap the caller asked for).
+ if delivery == DeliveryMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let mods_owned = modifiers.clone();
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ crate::input::send_click_synthesized_active_mods(
+ hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => {
+ let click_word = match count {
+ 2 => "double-click",
+ 3 => "triple-click",
+ _ => "click",
+ };
+ ToolResult::text(format!(
+ "✅ Sent {click_word} via SendInput to pid {pid} at ({sx},{sy}) (delivery_mode:foreground)."
+ ))
+ .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" }))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // delivery_mode:"background" on WinUI3 for double / right / middle (pixel
+ // path, no element_token → no cached element to UIA-Invoke): these
+ // can't both land and hold the contract on WinUI3 (posted input is
+ // ignored, the pen injector steals foreground) → structured error.
+ if delivery == DeliveryMode::Background
+ && (count > 1 || btn == "right" || btn == "middle")
+ {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, None, count, &btn).await
+ {
+ return r;
+ }
+ }
+ // Match the AX-addressed Chromium route above: the point remains
+ // PX-resolved, but transport uses the background-safe posted
+ // message path proven against the fully occluded fixture.
+ if delivery == DeliveryMode::Background
+ && btn == "left"
+ && count == 1
+ && crate::input::is_chromium_target_window(hwnd)
+ {
+ let posted = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, sx_i, sy_i, count, &btn)
+ }
+ })
+ .await;
+ return match posted {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Posted click to Chromium pid {pid} at ({sx},{sy}) \
+ (background, no foreground swap)."
+ ))
+ .with_structured(json!({
+ "path": "post_message",
+ "verified": false,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ // As above, bypass Chromium's false-positive UIA Invoke and use
+ // the coordinate actuator before attempting any accessibility hit
+ // test. The actuator itself distinguishes visible delivery from a
+ // fully occluded structured refusal.
+ if delivery == DeliveryMode::Background && crate::input::is_chromium_target_window(hwnd)
+ {
+ let btn2 = btn.clone();
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, sx as i32, sy as i32, count, &btn2)
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected click to pid {pid} at ({sx},{sy}) (background, no foreground swap)."
+ ))
+ .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
+ Ok(Err(error)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ error.to_string(),
+ ),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+ let use_uia = (btn == "left" || btn == "middle") && count == 1;
+ if use_uia {
+ let outcome = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::uia::windows_enum::try_invoke_in_window_at_point(
+ hwnd as isize,
+ sx as i32,
+ sy as i32,
+ )
+ }
+ })
+ .await
+ .unwrap_or(crate::uia::windows_enum::PointInvokeOutcome::Unavailable);
+ if let Some(result) = finish_pixel_uia_attempt(outcome, pid, sx_i, sy_i) {
+ return result;
+ }
+ }
+
+ // UIA completed with a miss or was not applicable. Known dropped surfaces other than direct
+ // Chromium (handled above) get one targeted injection attempt.
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ let btn2 = btn.clone();
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, sx as i32, sy as i32, count, &btn2)
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected click to pid {pid} at ({sx},{sy}) (background, no foreground swap)."
+ ))
+ .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
+ Ok(Err(error)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ error.to_string(),
+ ),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
+
+ // delivery_mode:"background", non-dropped + non-UIA click. This
+ // includes plain Win32 targets with no invokable UIA element.
+ // Attempt PostMessage without a
+ // foreground swap; the caller must verify the resulting fixture state.
+ // Foreground was handled at the top, and known dropped surfaces use
+ // the targeted-injection path above.
+
+ // bitmap pixels -> screen (DWM-frame origin + inset). Use
+ // post_click_screen so we don't double-ClientToScreen.
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, sx_i, sy_i, count, &btn)
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ // Match Swift text format: "✅ Posted click/double-click/triple-click to pid X."
+ let click_word = match count {
+ 2 => "double-click",
+ 3 => "triple-click",
+ _ => "click",
+ };
+ posted_pixel_click_result(pid, click_word)
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ } else {
+ // Match Swift's error wording for the missing-target case.
+ ToolResult::error("Provide element_token or (x, y) to address the click target.")
+ }
+ }
+}
+
+#[cfg(test)]
+mod pixel_click_transport_tests {
+ use super::{finish_pixel_uia_attempt, posted_pixel_click_result};
+ use crate::uia::windows_enum::PointInvokeOutcome;
+ use cua_driver_core::action_record::{
+ ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
+ };
+
+ #[test]
+ fn uia_unavailable_errors_advertise_foreground_escalation() {
+ use cua_driver_core::action_record::EscalationKind;
+ use cua_driver_core::protocol::Content;
+ for (outcome, status) in [
+ (PointInvokeOutcome::Busy, "busy"),
+ (PointInvokeOutcome::Timeout, "timeout"),
+ (PointInvokeOutcome::Unavailable, "unavailable"),
+ ] {
+ let result = finish_pixel_uia_attempt(outcome, 7, 3, 4)
+ .expect("unavailable UIA must stop the route without fallback");
+ assert!(result.is_error.unwrap_or(false), "{outcome:?}");
+ let data = result
+ .structured_content
+ .as_ref()
+ .expect("structured error");
+ assert_eq!(data["code"], "background_unavailable");
+ assert_eq!(data["uia_status"], status);
+ assert_eq!(data["path"], "ax");
+ assert_eq!(data["effect"], "unverifiable");
+ assert_eq!(
+ data["suggestion"].as_str(),
+ Some("Retry this action with delivery_mode:\"foreground\"."),
+ "{outcome:?}"
+ );
+ assert_eq!(
+ data["escalation"]["recommended"].as_str(),
+ Some("foreground"),
+ "{outcome:?}"
+ );
+ let reason = data["escalation"]["reason"]
+ .as_str()
+ .expect("escalation reason");
+ assert!(
+ reason.contains(status),
+ "reason must name the UIA status: {reason}"
+ );
+ assert!(
+ reason.contains("delivery_mode:\"foreground\""),
+ "reason must name the next rung: {reason}"
+ );
+ let text = match &result.content[0] {
+ Content::Text { text, .. } => text,
+ _ => panic!("expected text content for {outcome:?}"),
+ };
+ assert!(
+ text.contains("No fallback input was sent"),
+ "text must keep the no-replay guarantee: {text}"
+ );
+ assert!(
+ text.contains("delivery_mode:\"foreground\""),
+ "text must surface the escalation: {text}"
+ );
+ let record = ActionExecutionRecord::from_legacy(
+ "click",
+ &serde_json::json!({ "delivery_mode": "background" }),
+ data,
+ )
+ .expect("UIA refusal should normalize into the public action contract");
+ let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
+ .expect("serialize ActionResult");
+ assert_eq!(public["effect"], "unverifiable", "{outcome:?}");
+ }
+ // The completed-miss boundary is unchanged: only Miss falls through,
+ // and a delivered Invoke carries no escalation hint.
+ assert!(finish_pixel_uia_attempt(PointInvokeOutcome::Miss, 7, 3, 4).is_none());
+ let ok = finish_pixel_uia_attempt(PointInvokeOutcome::Invoked, 7, 3, 4)
+ .expect("invoked click reports success");
+ assert!(!ok.is_error.unwrap_or(false));
+ let ok_data = ok.structured_content.as_ref().expect("structured success");
+ assert_eq!(ok_data["path"], "ax");
+ assert!(ok_data.get("escalation").is_none());
+ assert!(ok_data.get("suggestion").is_none());
+ let ok_record = ActionExecutionRecord::from_legacy(
+ "click",
+ &serde_json::json!({ "delivery_mode": "background" }),
+ ok_data,
+ )
+ .expect("UIA invoke should normalize into the public action contract");
+ let ok_public =
+ serde_json::to_value(ok_record.public_result().expect("valid ActionResult"))
+ .expect("serialize ActionResult");
+ assert_eq!(ok_public["effect"], "unverifiable");
+ // The hint must flow into the existing escalation pipeline, not sit
+ // as inert metadata: Timeout (the #3621 case) normalizes to a
+ // foreground-delivery escalation on the internal record.
+ let timeout_data = finish_pixel_uia_attempt(PointInvokeOutcome::Timeout, 7, 3, 4)
+ .expect("timeout stops the route")
+ .structured_content
+ .expect("structured error");
+ let timeout_record = ActionExecutionRecord::from_legacy(
+ "click",
+ &serde_json::json!({ "delivery_mode": "background" }),
+ &timeout_data,
+ )
+ .expect("UIA timeout should normalize into the public action contract");
+ assert_eq!(
+ timeout_record.escalation.map(|escalation| escalation.kind),
+ Some(EscalationKind::RetryWithForegroundDelivery)
+ );
+ }
+
+ #[test]
+ fn post_message_pixel_click_reports_synthetic_transport() {
+ let result = posted_pixel_click_result(42, "click");
+ let structured = result
+ .structured_content
+ .as_ref()
+ .expect("posted click should expose legacy transport metadata");
+ assert_eq!(structured["path"], "post_message");
+
+ let record = ActionExecutionRecord::from_legacy(
+ "click",
+ &serde_json::json!({ "delivery_mode": "background" }),
+ structured,
+ )
+ .expect("posted click should normalize into the public action contract");
+ assert_eq!(record.transport, ActionTransport::WindowsPostMessage);
+ assert_eq!(record.requested_delivery, RequestedDelivery::Background);
+ assert_eq!(record.actual_delivery, Some(ActualDelivery::Background));
+
+ let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
+ .expect("serialize ActionResult");
+ assert_eq!(public["route"], "synthetic_events");
+ assert_eq!(public["delivery"]["mode"], "background");
+ }
+}
+
+#[cfg(test)]
+mod background_element_click_record_tests {
+ use super::{background_element_click_result, ActionTransport};
+
+ #[test]
+ fn raw_fallback_reports_its_physical_transport() {
+ for (transport, path) in [
+ (ActionTransport::WindowsTargetedInjection, "pixel"),
+ (ActionTransport::WindowsPostMessage, "post_message"),
+ ] {
+ let result = background_element_click_result(
+ "physical fallback".into(),
+ transport,
+ vec![ActionTransport::WindowsUiaInvoke],
+ );
+ assert_eq!(result.structured_content.as_ref().unwrap()["path"], path);
+ let record = result.action_record.unwrap();
+ assert_eq!(record.transport, transport);
+ let truth = record.debug_json();
+ assert_ne!(truth["route"], "accessibility");
+ assert_eq!(
+ record.attempts[0].transport,
+ ActionTransport::WindowsUiaInvoke
+ );
+ assert_eq!(record.fallbacks[0].to, transport);
+ }
+ }
+
+ #[test]
+ fn failed_provider_calls_disqualify_single_action_marker_exemption() {
+ let result = background_element_click_result(
+ "semantic success after provider failure".into(),
+ ActionTransport::WindowsUiaToggle,
+ vec![ActionTransport::WindowsUiaInvoke],
+ );
+ let record = result.action_record.unwrap();
+ assert_eq!(record.transport, ActionTransport::WindowsUiaToggle);
+ assert_eq!(record.attempts.len(), 1);
+ assert_eq!(record.fallbacks.len(), 1);
+ assert_eq!(record.fallbacks[0].from, ActionTransport::WindowsUiaInvoke);
+ assert_eq!(record.fallbacks[0].to, ActionTransport::WindowsUiaToggle);
+ // Point-free recording requires an empty fallback journal.
+ assert!(!record.debug_json()["fallbacks"]
+ .as_array()
+ .unwrap()
+ .is_empty());
+ }
+
+ #[test]
+ fn single_provider_success_has_no_invented_attempts() {
+ for transport in [
+ ActionTransport::WindowsUiaInvoke,
+ ActionTransport::WindowsUiaToggle,
+ ActionTransport::WindowsUiaSelection,
+ ActionTransport::WindowsUiaExpandCollapse,
+ ] {
+ let result =
+ background_element_click_result("semantic success".into(), transport, vec![]);
+ assert_eq!(result.structured_content.as_ref().unwrap()["path"], "ax");
+ let record = result.action_record.unwrap();
+ assert_eq!(record.transport, transport);
+ assert!(record.attempts.is_empty());
+ assert!(record.fallbacks.is_empty());
+ }
+ }
+}
+
+#[cfg(test)]
+mod click_capture_id_schema_tests {
+ use super::ClickTool;
+ use cua_driver_core::tool::Tool;
+
+ /// The portable click contract pins `button` and the other accepted
+ /// fields. A live schema may broaden `capture_id`, which the subset gate
+ /// allows, so the non-empty bound is pinned here.
+ #[test]
+ fn schema_requires_non_empty_capture_id() {
+ let tool = ClickTool {
+ state: super::ToolState::new(None),
+ };
+ let d = tool.def();
+ let props = d.input_schema.get("properties").expect("properties");
+ let capture_id = props.get("capture_id").expect("capture_id field present");
+ assert_eq!(capture_id["type"], "string");
+ assert_eq!(capture_id["minLength"], 1);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/cursor_tools.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/cursor_tools.rs
new file mode 100644
index 0000000000..d0f0939ac2
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/cursor_tools.rs
@@ -0,0 +1,243 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{resolve_cursor_key, ToolState};
+
+// ── set_agent_cursor_enabled ──────────────────────────────────────────────────
+
+pub struct SetAgentCursorEnabledV2Tool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_ENABLED_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetAgentCursorEnabledV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_ENABLED_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_enabled"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let enabled = match args.get("enabled").and_then(Value::as_bool) {
+ Some(value) => value,
+ None => return ToolResult::error("Missing required boolean field `enabled`."),
+ };
+ let session = resolve_cursor_key(&args);
+ if session.is_empty() {
+ return ToolResult::error("`session` is required for agent cursor controls.");
+ }
+ self.state.cursor_registry.set_enabled(&session, enabled);
+ crate::overlay::send_command(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetEnabled(enabled),
+ );
+ ToolResult::text(format!(
+ "Agent cursor for session '{session}' {}.",
+ if enabled { "enabled" } else { "disabled" }
+ ))
+ .with_structured(json!({"session":session,"enabled":enabled}))
+ }
+}
+
+pub struct SetAgentCursorMotionV2Tool;
+
+static CURSOR_MOTION_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn cursor_number(value: Option<&Value>) -> Option<f64> {
+ value.and_then(|value| {
+ value
+ .as_f64()
+ .or_else(|| value.as_i64().map(|integer| integer as f64))
+ })
+}
+
+#[async_trait]
+impl Tool for SetAgentCursorMotionV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_MOTION_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_motion"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let session = resolve_cursor_key(&args);
+ if session.is_empty() {
+ return ToolResult::error("`session` is required for agent cursor controls.");
+ }
+ let current = crate::overlay::current_motion(&session);
+ let motion = current.with_overrides(
+ cursor_number(args.get("start_handle")),
+ cursor_number(args.get("end_handle")),
+ cursor_number(args.get("arc_size")),
+ cursor_number(args.get("arc_flow")),
+ cursor_number(args.get("spring")),
+ cursor_number(args.get("glide_duration_ms")),
+ cursor_number(args.get("dwell_after_click_ms")),
+ cursor_number(args.get("idle_hide_ms")),
+ None,
+ cursor_number(args.get("turn_radius")),
+ );
+ let motion = match motion.with_style_args(&args) {
+ Ok(motion) => motion,
+ Err(message) => return ToolResult::error(message),
+ };
+ crate::overlay::send_command(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetMotion(motion.clone()),
+ );
+ ToolResult::text(format!(
+ "Agent cursor motion updated for session '{session}'."
+ ))
+ .with_structured(json!({"session":session,"motion":motion.output_json()}))
+ }
+}
+
+pub struct SetAgentCursorThemeTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_THEME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn cursor_reduced_motion(value: Option<&str>) -> cursor_overlay::ReducedMotion {
+ match value {
+ Some("on") => cursor_overlay::ReducedMotion::On,
+ Some("off") => cursor_overlay::ReducedMotion::Off,
+ _ => cursor_overlay::ReducedMotion::Auto,
+ }
+}
+
+#[async_trait]
+impl Tool for SetAgentCursorThemeTool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_THEME_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_theme"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let Some(theme_id) = args.get("theme_id").and_then(Value::as_str) else {
+ return ToolResult::error("Missing required string field `theme_id`.");
+ };
+ let session = resolve_cursor_key(&args);
+ if session.is_empty() {
+ return ToolResult::error("`session` is required for agent cursor controls.");
+ }
+ let resolved_theme = match cursor_overlay::resolve_theme_selection(theme_id) {
+ Ok(theme) => theme,
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Cursor theme '{theme_id}' cannot be selected: {error}"
+ ));
+ }
+ };
+ let (version, profile) = resolved_theme
+ .as_deref()
+ .map(|theme| (theme.version.as_str(), theme.profile.as_str()))
+ .unwrap_or((
+ cursor_overlay::DEFAULT_THEME_VERSION,
+ cursor_overlay::THEME_PROFILE,
+ ));
+ let reduced_motion =
+ cursor_reduced_motion(args.get("reduced_motion").and_then(Value::as_str));
+ self.state
+ .cursor_registry
+ .update_config(&session, |config| {
+ config.theme_id = theme_id.to_owned();
+ config.reduced_motion = reduced_motion;
+ });
+ crate::overlay::send_command(
+ session.clone(),
+ cursor_overlay::OverlayCommand::SetTheme {
+ theme_id: theme_id.to_owned(),
+ reduced_motion,
+ },
+ );
+ ToolResult::text(format!(
+ "Agent cursor theme for session '{session}' set to '{theme_id}'."
+ ))
+ .with_structured(json!({"session":session,"theme":{
+ "id":theme_id,
+ "version":version,
+ "profile":profile,
+ "reduced_motion":reduced_motion,
+ "fallback":null
+ }}))
+ }
+}
+
+pub struct GetAgentCursorStateV2Tool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static CURSOR_STATE_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetAgentCursorStateV2Tool {
+ fn def(&self) -> &ToolDef {
+ CURSOR_STATE_V2_DEF.get_or_init(|| canonical_cursor_def("get_agent_cursor_state"))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let session = resolve_cursor_key(&args);
+ if session.is_empty() {
+ return ToolResult::error("`session` is required for agent cursor controls.");
+ }
+ let enabled = crate::overlay::is_enabled(&session);
+ let motion = crate::overlay::current_motion(&session);
+ let (theme_id, version, profile, fallback, visual) =
+ crate::overlay::current_theme_state(&session).unwrap_or_else(|| {
+ (
+ cursor_overlay::DEFAULT_THEME_ID.into(),
+ cursor_overlay::DEFAULT_THEME_VERSION.into(),
+ cursor_overlay::THEME_PROFILE.into(),
+ None,
+ cursor_overlay::CursorVisualState::default(),
+ )
+ });
+ let modifiers: Vec<&str> = [
+ visual.delivery.map(|value| value.as_str()),
+ visual.target.map(|value| value.as_str()),
+ ]
+ .into_iter()
+ .flatten()
+ .collect();
+ // The last point this session's cursor was placed at, `null` until
+ // it first moves (same registry source as macOS).
+ let position = self
+ .state
+ .cursor_registry
+ .get(&session)
+ .and_then(|cursor| cursor.x.zip(cursor.y))
+ .map(|(x, y)| json!({"x": x, "y": y}));
+ ToolResult::text(format!("Agent cursor state for session '{session}'.")).with_structured(
+ json!({
+ "session":session,
+ "enabled":enabled,
+ "position":position,
+ "theme":{
+ "id":theme_id,
+ "version":version,
+ "profile":profile,
+ "reduced_motion":visual.reduced_motion,
+ "fallback":fallback
+ },
+ "visual_state":{
+ "requested_action":visual.requested_action,
+ "resolved_action":visual.resolved_action,
+ "modifiers":modifiers,
+ "phase":visual.phase(),
+ "frame":visual.frame(),
+ "preempted_count":visual.preempted_count
+ },
+ "motion":motion.output_json()
+ }),
+ )
+ }
+}
+
+fn canonical_cursor_def(name: &str) -> ToolDef {
+ let contract = cua_driver_contract::tool_contract(name)
+ .unwrap_or_else(|| panic!("missing canonical cursor contract for {name}"));
+ ToolDef::from_contract(&contract)
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/debug_window_info.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/debug_window_info.rs
new file mode 100644
index 0000000000..cc091d7a25
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/debug_window_info.rs
@@ -0,0 +1,264 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use windows::core::Interface as _;
+
+// ── debug_window_info ─────────────────────────────────────────────────────────
+//
+// Diagnostic tool: dump everything the daemon (Session 2 when launched via
+// `cua-driver autostart kick`) sees about a given pid's top-level windows.
+// Includes Win32 class names, owning .exe basename + full path, and — when
+// CUIAutomation is available — the currently-focused UIA element with the
+// list of patterns it supports.
+//
+// Built for CUA-543 investigation: the routing between PostMessage-based
+// input and UIA-pattern-based input depends on what the OS actually reports
+// for modern XAML / UWP / WinUI3 targets. SSH-side PowerShell probes from
+// Session 0 see nothing (cross-session HWNDs return empty class names);
+// this tool gives the same view from inside the daemon process.
+
+pub struct DebugWindowInfoTool;
+static DEBUG_WI_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for DebugWindowInfoTool {
+ fn def(&self) -> &ToolDef {
+ DEBUG_WI_DEF.get_or_init(|| ToolDef {
+ name: "debug_window_info".into(),
+ description: "Diagnostic: dump everything cua-driver sees about a pid's \
+ top-level windows from the daemon's session perspective. Returns \
+ window class names, owning .exe basename + path, and — when \
+ CUIAutomation succeeds — the focused UIA element with the list of \
+ patterns it supports (ValuePattern, InvokePattern, TextPattern, \
+ TogglePattern, etc.). Used to design / debug input routing for \
+ XAML / UWP / WinUI3 targets; see CUA-543."
+ .into(),
+ input_schema: json!({"type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"PID of the process to inspect."}
+ },"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let pid_v: u32 = match args.get("pid").and_then(|v| v.as_u64()) {
+ Some(p) if p > 0 && p <= u32::MAX as u64 => p as u32,
+ _ => {
+ return ToolResult::error("debug_window_info: missing or invalid `pid`".to_string())
+ }
+ };
+
+ let outcome = tokio::task::spawn_blocking(move || -> serde_json::Value {
+ use std::collections::HashSet;
+ use windows::Win32::Foundation::{CloseHandle, HWND, LPARAM, BOOL, TRUE};
+ use windows::Win32::System::Com::{
+ CoCreateInstance, CoInitializeEx, CoUninitialize,
+ CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED,
+ };
+ use windows::Win32::System::RemoteDesktop::ProcessIdToSessionId;
+ use windows::Win32::System::Threading::{
+ GetCurrentProcessId, OpenProcess,
+ QueryFullProcessImageNameW, PROCESS_NAME_FORMAT,
+ PROCESS_QUERY_LIMITED_INFORMATION,
+ };
+ use windows::Win32::UI::Accessibility::{
+ CUIAutomation, IUIAutomation,
+ UIA_InvokePatternId, UIA_TextPatternId, UIA_TogglePatternId,
+ UIA_ValuePatternId, UIA_LegacyIAccessiblePatternId,
+ UIA_SelectionItemPatternId, UIA_ExpandCollapsePatternId,
+ };
+ use windows::Win32::UI::WindowsAndMessaging::{
+ EnumWindows, GetClassNameW, GetWindow, GetWindowTextW,
+ GetWindowThreadProcessId, IsWindowVisible, GW_OWNER,
+ };
+
+ // Helper: read class name + title
+ fn class_name(h: HWND) -> Option<String> {
+ let mut buf = [0u16; 256];
+ let n = unsafe { GetClassNameW(h, &mut buf) };
+ if n <= 0 { None } else { Some(String::from_utf16_lossy(&buf[..n as usize])) }
+ }
+ fn window_title(h: HWND) -> Option<String> {
+ let mut buf = [0u16; 512];
+ let n = unsafe { GetWindowTextW(h, &mut buf) };
+ if n <= 0 { None } else { Some(String::from_utf16_lossy(&buf[..n as usize])) }
+ }
+
+ // ── Process info: session id + exe path
+ let mut session_id: u32 = 0;
+ let _ = unsafe { ProcessIdToSessionId(pid_v, &mut session_id) };
+
+ let (exe_path, exe_basename) = unsafe {
+ match OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, false, pid_v) {
+ Ok(h) => {
+ let mut buf = [0u16; 1024];
+ let mut len: u32 = buf.len() as u32;
+ let ok = QueryFullProcessImageNameW(
+ h, PROCESS_NAME_FORMAT(0),
+ windows::core::PWSTR(buf.as_mut_ptr()),
+ &mut len,
+ );
+ let _ = CloseHandle(h);
+ if ok.is_ok() && len > 0 {
+ let path = String::from_utf16_lossy(&buf[..len as usize]);
+ let base = path
+ .rsplit(|c: char| c == '\\' || c == '/')
+ .next()
+ .unwrap_or(&path)
+ .to_ascii_lowercase();
+ (Some(path), Some(base))
+ } else {
+ (None, None)
+ }
+ }
+ Err(_) => (None, None),
+ }
+ };
+
+ // ── Enumerate top-level visible windows owned by this pid
+ struct Ctx { target_pid: u32, hwnds: Vec<isize> }
+ extern "system" fn enum_cb(hwnd: HWND, lparam: LPARAM) -> BOOL {
+ unsafe {
+ let ctx = &mut *(lparam.0 as *mut Ctx);
+ let mut p: u32 = 0;
+ GetWindowThreadProcessId(hwnd, Some(&mut p));
+ if p == ctx.target_pid && IsWindowVisible(hwnd).as_bool() {
+ // Only true top-level (no owner) windows.
+ let owner = GetWindow(hwnd, GW_OWNER);
+ if owner.unwrap_or_default().is_invalid() {
+ ctx.hwnds.push(hwnd.0 as isize);
+ }
+ }
+ TRUE
+ }
+ }
+ let mut ctx = Ctx { target_pid: pid_v, hwnds: vec![] };
+ let _ = unsafe {
+ EnumWindows(Some(enum_cb), LPARAM(&mut ctx as *mut Ctx as isize))
+ };
+
+ // ── Init UIA (best-effort; non-fatal if it fails)
+ let mut focused_info: Option<serde_json::Value> = None;
+ let coinit_ok = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED).is_ok() };
+ if let Ok(uia) = unsafe {
+ CoCreateInstance::<_, IUIAutomation>(&CUIAutomation, None, CLSCTX_INPROC_SERVER)
+ } {
+ if let Ok(focused) = unsafe { uia.GetFocusedElement() } {
+ // Patterns we care about for input routing
+ let pattern_ids = [
+ (UIA_ValuePatternId.0, "Value"),
+ (UIA_InvokePatternId.0, "Invoke"),
+ (UIA_TextPatternId.0, "Text"),
+ (UIA_TogglePatternId.0, "Toggle"),
+ (UIA_ExpandCollapsePatternId.0, "ExpandCollapse"),
+ (UIA_SelectionItemPatternId.0, "SelectionItem"),
+ (UIA_LegacyIAccessiblePatternId.0, "LegacyIAccessible"),
+ ];
+ let mut patterns: Vec<&str> = vec![];
+ for (id, name) in &pattern_ids {
+ let pid_struct = windows::Win32::UI::Accessibility::UIA_PATTERN_ID(*id);
+ if unsafe { focused.GetCurrentPattern(pid_struct) }.is_ok() {
+ patterns.push(name);
+ }
+ }
+ let name = unsafe { focused.CurrentName() }
+ .ok()
+ .map(|b| b.to_string())
+ .unwrap_or_default();
+ let class = unsafe { focused.CurrentClassName() }
+ .ok()
+ .map(|b| b.to_string())
+ .unwrap_or_default();
+ let ctype = unsafe { focused.CurrentControlType() }
+ .map(|t| t.0)
+ .unwrap_or(0);
+ let value_ro = if patterns.contains(&"Value") {
+ unsafe {
+ focused.GetCurrentPattern(UIA_ValuePatternId)
+ .and_then(|p| p.cast::<windows::Win32::UI::Accessibility::IUIAutomationValuePattern>())
+ .and_then(|vp| vp.CurrentIsReadOnly())
+ .map(|b| b.as_bool())
+ .ok()
+ }
+ } else { None };
+ focused_info = Some(serde_json::json!({
+ "name": name,
+ "class_name": class,
+ "control_type_id": ctype,
+ "supported_patterns": patterns,
+ "value_pattern_is_read_only": value_ro,
+ }));
+ }
+ }
+ if coinit_ok {
+ unsafe { CoUninitialize(); }
+ }
+
+ // ── Build per-window JSON
+ let xaml_classes: HashSet<&str> = [
+ "ApplicationFrameWindow",
+ "WinUIDesktopWin32WindowClass",
+ "Windows.UI.Core.CoreWindow",
+ "Microsoft.UI.Content.DesktopChildSiteBridge",
+ ].iter().copied().collect();
+ let xaml_exes: HashSet<&str> = [
+ "notepad.exe", "calculatorapp.exe", "calc.exe",
+ "applicationframehost.exe", "photos.exe", "systemsettings.exe",
+ ].iter().copied().collect();
+
+ let windows_json: Vec<serde_json::Value> = ctx.hwnds.iter()
+ .map(|&h| {
+ let hwnd = HWND(h as *mut _);
+ let cls = class_name(hwnd);
+ let title = window_title(hwnd);
+ let xaml_class_match = cls.as_deref()
+ .map(|c| xaml_classes.contains(c))
+ .unwrap_or(false);
+ serde_json::json!({
+ "hwnd": h,
+ "class_name": cls,
+ "title": title,
+ "xaml_class_match": xaml_class_match,
+ })
+ })
+ .collect();
+
+ let exe_xaml_match = exe_basename.as_deref()
+ .map(|e| xaml_exes.contains(e))
+ .unwrap_or(false);
+
+ let daemon_pid = unsafe { GetCurrentProcessId() };
+ let mut daemon_session: u32 = 0;
+ let _ = unsafe { ProcessIdToSessionId(daemon_pid, &mut daemon_session) };
+
+ serde_json::json!({
+ "pid": pid_v,
+ "session_id": session_id,
+ "daemon_pid": daemon_pid,
+ "daemon_session_id": daemon_session,
+ "exe_path": exe_path,
+ "exe_basename": exe_basename,
+ "exe_xaml_match": exe_xaml_match,
+ "windows": windows_json,
+ "focused_element": focused_info,
+ "xaml_routing_recommended": exe_xaml_match
+ || windows_json.iter()
+ .any(|w| w.get("xaml_class_match").and_then(|v| v.as_bool()).unwrap_or(false)),
+ })
+ }).await;
+
+ match outcome {
+ Ok(j) => {
+ let pretty = serde_json::to_string_pretty(&j).unwrap_or_else(|_| j.to_string());
+ ToolResult::text(pretty).with_structured(j)
+ }
+ Err(e) => ToolResult::error(format!("debug_window_info: join error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/double_click.rs
new file mode 100644
index 0000000000..c78851c311
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/double_click.rs
@@ -0,0 +1,380 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ bitmap_to_screen, chromium_click_short_circuit, element_target_rect, overlay_glide_to,
+ overlay_glide_to_target, pin_overlay_above, resolve_cursor_key,
+ resolve_onscreen_point_with_scroll, restore_foreground_polling_best_effort, screenshot_scale,
+ winui3_background_gesture, ToolState,
+};
+
+// ── double_click ──────────────────────────────────────────────────────────────
+
+pub struct DoubleClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static DCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for DoubleClickTool {
+ fn def(&self) -> &ToolDef {
+ DCLICK_DEF.get_or_init(|| ToolDef {
+ name: "double_click".into(),
+ // Description ported from Swift `DoubleClickTool.swift` semantics
+ // (two addressing modes, AXOpen analogue on Windows is not yet
+ // wired up — element path always falls back to a stamped pixel
+ // double-click via PostMessage).
+ description: "Double-click against a target pid. Two addressing modes:\n\n\
+ - `element_token` (from the last `get_window_state` snapshot \
+ of that window) — synthesizes a stamped pixel double-click at the element's \
+ cached on-screen center via PostMessage. (Windows has no AXOpen analogue; \
+ Swift's AXOpen-first path falls through to the same pixel recipe when the \
+ element doesn't advertise AXOpen, so the user-visible behavior matches.)\n\n\
+ - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
+ returned by `get_window_state`) — posts two WM_LBUTTONDOWN/UP pairs in \
+ quick succession to the deepest child window at that point.\n\n\
+ Exactly one of `element_token` or (`x` AND `y`) must be provided. \
+ `pid` is required with `x`, `y`; an `element_token` carries its own pid. The macOS-only `modifier` field is accepted for \
+ parity (no-op on Windows — PostMessage doesn't propagate modifier-key state).".into(),
+ input_schema: json!({"type":"object","required":[],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
+ "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ // Swift error wording 1:1.
+ let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(p) => p,
+ None => return ToolResult::error("Missing required integer field pid."),
+ };
+ let pid = raw_pid as u32;
+ use cua_driver_core::tool_args::ArgsExt;
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let x = args.opt_f64("x");
+ let y = args.opt_f64("y");
+ let delivery = DeliveryMode::from_args(&args);
+ let cursor_key = resolve_cursor_key(&args);
+ // Swift validates "both x and y or neither" and "no element_token without window_id".
+ let has_xy = x.is_some() && y.is_some();
+ let partial_xy = x.is_some() != y.is_some();
+ if partial_xy {
+ return ToolResult::error("Provide both x and y together, not just one.");
+ }
+ if elem_idx.is_some() && has_xy {
+ return ToolResult::error("Provide either element_token or (x, y), not both.");
+ }
+ if elem_idx.is_none() && !has_xy {
+ return ToolResult::error(
+ "Provide element_token or (x, y) to address the double-click target.",
+ );
+ }
+
+ // Resolve HWND: explicit, or auto from pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ if let Some(idx) = elem_idx {
+ let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
+ Some(v) => v,
+ None => {
+ return ToolResult::error(format!(
+ "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
+ ))
+ }
+ };
+ let recorded_center = (cx, cy);
+ let (cx, cy) = match resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx,
+ cx,
+ cy,
+ "double-clicking",
+ ) {
+ Ok(p) => p,
+ Err(result) => return result,
+ };
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(
+ admitted.as_ref(),
+ (cx - recorded_center.0, cy - recorded_center.1),
+ ),
+ )
+ .await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: cx as f64,
+ y: cy as f64,
+ },
+ );
+ // delivery_mode:"background" (default) on WinUI3: the pen/touch injector
+ // click-activates the content island (8/8 foreground steals) and
+ // posted WM_*BUTTON to the frame/island no-ops (measured). A double
+ // UIA Invoke on the cached element is the only contract-holding way
+ // to land the double-click — it fires the XAML Click handler twice
+ // → last_action=double_click, held non-activatable so no foreground
+ // swap. If the element has no InvokePattern, a background double-
+ // click isn't expressible → structured error.
+ if delivery == DeliveryMode::Background {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, Some(idx), 2, "left").await
+ {
+ return r;
+ }
+ }
+ // delivery_mode:"background" (default): Chromium/Electron & GTK targets
+ // silently drop posted clicks (#1984) — inject via the coordinate
+ // actuator (system input queue, NO foreground swap), exactly like
+ // ClickTool, instead of refusing. Only error if injection can't
+ // express this click (e.g. right/middle on such a target).
+ if (cx, cy) != recorded_center {
+ crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
+ }
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, cx, cy, 2, "left")
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected double-click to [{idx}] at screen ({cx},{cy}) (background, no foreground swap)."
+ )),
+ Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ e.to_string(),
+ ),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // delivery_mode:"foreground" — route through SendInput at the cached coords.
+ if delivery == DeliveryMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_click_synthesized_active_mods(
+ hwnd,
+ cx,
+ cy,
+ 2,
+ "left",
+ &[],
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent double-click via SendInput on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
+ if let Some(r) =
+ chromium_click_short_circuit(hwnd, cx, cy, 2, "left", pid, "double-click").await
+ {
+ return r;
+ }
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || -> anyhow::Result<String> {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, cx, cy, 2, "left")?;
+ // Swift text format 1:1: `"✅ Posted double-click to [N] role \"title\" at screen-point (X, Y)."`.
+ // UIA role/title placeholder pending element-cache enrichment.
+ Ok(format!(
+ "✅ Posted double-click to [{idx}] at screen-point ({cx}, {cy})."
+ ))
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(msg)) => ToolResult::text(msg),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ } else if let (Some(mut px), Some(mut py)) = (x, y) {
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if from_zoom {
+ match self.state.zoom_context(&args, pid, Some(hwnd)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(px, py);
+ px = wx;
+ py = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ px *= ratio;
+ py *= ratio;
+ }
+ // bitmap pixels -> screen via DWM-frame origin (see
+ // `bitmap_to_screen` doc for why ClientToScreen is wrong).
+ let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
+ let (sx, sy) = (sx_i as f64, sy_i as f64);
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to(&cursor_key, sx, sy).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ // delivery_mode:"background" on WinUI3 (pixel path, no element_token): a
+ // background double-click on WinUI3 only lands via UIA Invoke on a
+ // cached element, which the pixel path doesn't have → structured
+ // error (caller retries with delivery_mode:foreground or uses element_index).
+ if delivery == DeliveryMode::Background {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, None, 2, "left").await
+ {
+ return r;
+ }
+ }
+ // delivery_mode:"background" (default): inject via the coordinate actuator
+ // (no foreground swap) for targets that drop posted clicks (#1984).
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, sx_i, sy_i, 2, "left")
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected double-click to pid {pid} at screen ({sx_i},{sy_i}) (background, no foreground swap)."
+ )),
+ Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ e.to_string(),
+ ),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // delivery_mode:"foreground" — SendInput at screen coords with FG swap.
+ if delivery == DeliveryMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_click_synthesized_active_mods(
+ hwnd,
+ sx_i,
+ sy_i,
+ 2,
+ "left",
+ &[],
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent double-click via SendInput to pid {pid} at screen ({sx_i},{sy_i}) (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ let (xi, yi) = (px as i32, py as i32);
+ // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
+ if let Some(r) =
+ chromium_click_short_circuit(hwnd, sx_i, sy_i, 2, "left", pid, "double-click").await
+ {
+ return r;
+ }
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, sx_i, sy_i, 2, "left")
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ // Match Swift's pixel-path text 1:1.
+ ToolResult::text(format!(
+ "✅ Posted double-click to pid {pid} at window-pixel ({xi}, {yi}) → screen-point ({sx_i}, {sy_i})."))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ } else {
+ // Unreachable — guarded by the earlier validation block.
+ unreachable!()
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/drag.rs
new file mode 100644
index 0000000000..3234bafccd
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/drag.rs
@@ -0,0 +1,500 @@
+use async_trait::async_trait;
+use cua_driver_contract::{ClickButton, DragInput};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ bitmap_to_screen, overlay_glide_to, pin_overlay_above, resolve_cursor_key,
+ restore_foreground_polling_best_effort, screenshot_scale, winui3_background_gesture, ToolState,
+};
+
+async fn track_overlay_drag(
+ key: String,
+ from: (f64, f64),
+ to: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+) {
+ if key.is_empty() || !crate::overlay::is_enabled(&key) {
+ return;
+ }
+ crate::overlay::send_command(
+ key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let steps = steps.max(1);
+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
+ for index in 0..=steps {
+ let t = index as f64 / steps as f64;
+ let x = from.0 + (to.0 - from.0) * t;
+ let y = from.1 + (to.1 - from.1) * t;
+ crate::overlay::send_command(key.clone(), cursor_overlay::track_pointer_command(x, y));
+ if index < steps && !step_delay.is_zero() {
+ tokio::time::sleep(step_delay).await;
+ }
+ }
+ crate::overlay::send_command(key, cursor_overlay::OverlayCommand::SetPressed(false));
+}
+
+// ── drag ─────────────────────────────────────────────────────────────────────
+
+pub struct DragTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static DRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for DragTool {
+ fn def(&self) -> &ToolDef {
+ DRAG_DEF.get_or_init(|| ToolDef {
+ name: "drag".into(),
+ description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. \
+ duration_ms (default 500) is the wall-clock budget; steps (default 20) interpolates intermediate \
+ WM_MOUSEMOVE events along the path. Background delivery does not steal focus; delivery_mode:\"foreground\" \
+ activates the target and refuses with foreground_unavailable, sending nothing, when another window still \
+ covers the start point. After a zoom call, pass from_zoom=true. \
+ Drags that start on a window caption / title bar or resize border (i.e. moving or resizing the \
+ window itself) cannot be delivered in the background — the OS move/resize loop needs real pointer \
+ input — so they return background_unavailable; re-issue those with delivery_mode:\"foreground\". \
+ `modifier` holds keys from press to release in delivery_mode:\"foreground\" and scope:\"desktop\" via \
+ SendInput. Background drags with modifiers return background_unavailable because posted messages and \
+ pen injection cannot carry live modifier-key state.".into(),
+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id for screen-absolute coordinates."},
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"Target window handle (HWND). Optional — driver picks frontmost window of pid when omitted."},
+ "from_x":{"type":"number","description":"Drag-start X in window-local screenshot pixels."},
+ "from_y":{"type":"number","description":"Drag-start Y in window-local screenshot pixels."},
+ "to_x":{"type":"number","description":"Drag-end X in window-local screenshot pixels."},
+ "to_y":{"type":"number","description":"Drag-end Y in window-local screenshot pixels."},
+ "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Wall-clock duration of drag path. Default: 500."},
+ "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Number of intermediate WM_MOUSEMOVE events. Default: 20."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "button": cua_driver_core::tool_schema::button_schema(),
+ "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ let cursor_key = resolve_cursor_key(&args);
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<DragInput>("drag", &args) {
+ Ok(input) => input,
+ Err(error) => return error,
+ };
+ let from_x = input.from_x as i32;
+ let from_y = input.from_y as i32;
+ let to_x = input.to_x as i32;
+ let to_y = input.to_y as i32;
+ let duration_ms = input.duration_ms.unwrap_or(500);
+ let steps = input.steps.unwrap_or(20) as usize;
+ let button = input.button.unwrap_or(ClickButton::Left).as_str();
+ let modifiers = input.modifier.unwrap_or_default();
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let native_drag = tokio::task::spawn_blocking(move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::mouse::send_drag_synthesized(
+ hwnd,
+ from_x,
+ from_y,
+ to_x,
+ to_y,
+ duration_ms,
+ steps,
+ &button,
+ &modifier_refs,
+ )
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(from_x), f64::from(from_y)),
+ (f64::from(to_x), f64::from(to_y)),
+ duration_ms,
+ steps,
+ );
+ let (native_drag, ()) = tokio::join!(native_drag, visual_drag);
+ return match native_drag {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput",
+ "from": {"x": from_x, "y": from_y}, "to": {"x": to_x, "y": to_y},
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),
+ };
+ }
+ // Swift error wording 1:1.
+ let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(p) => p,
+ None => return ToolResult::error("Missing required integer field pid."),
+ };
+ let pid = raw_pid as u32;
+ let delivery = DeliveryMode::from_args(&args);
+ let modifiers: Vec<String> = args.str_array("modifier");
+ if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {
+ return ToolResult::error(
+ "drag modifiers require delivery_mode:\"foreground\" on Windows; posted \
+ messages and pen injection cannot carry live modifier-key state",
+ )
+ .with_structured(json!({
+ "code": "background_unavailable",
+ "effect": "refused",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "Windows modifier drags require SendInput so the target observes live modifier state"
+ }
+ }));
+ }
+
+ // Accepts numeric JSON as either float or integer — coerce both to f64.
+ let coerce = |key: &str| -> Option<f64> {
+ args.opt_f64(key)
+ .or_else(|| args.opt_i64(key).map(|i| i as f64))
+ };
+ let from_x_opt = coerce("from_x");
+ let from_y_opt = coerce("from_y");
+ let to_x_opt = coerce("to_x");
+ let to_y_opt = coerce("to_y");
+ let (mut from_x, mut from_y, mut to_x, mut to_y) =
+ match (from_x_opt, from_y_opt, to_x_opt, to_y_opt) {
+ (Some(a), Some(b), Some(c), Some(d)) => (a, b, c, d),
+ _ => {
+ return ToolResult::error(
+ "from_x, from_y, to_x, and to_y are all required (window-local pixels).",
+ )
+ }
+ };
+
+ let hwnd_opt = args.opt_u64("window_id");
+ let duration_ms = args.u64_or("duration_ms", 500);
+ let steps = args.u64_or("steps", 20) as usize;
+ let button = args.str_or("button", "left");
+ let from_zoom = args.bool_or("from_zoom", false);
+
+ if from_zoom {
+ match self.state.zoom_context(&args, pid, hwnd_opt) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(from_x, from_y);
+ let (wx2, wy2) = ctx.zoom_to_window(to_x, to_y);
+ from_x = wx;
+ from_y = wy;
+ to_x = wx2;
+ to_y = wy2;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, hwnd_opt) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ from_x *= ratio;
+ from_y *= ratio;
+ to_x *= ratio;
+ to_y *= ratio;
+ }
+
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows =
+ tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // Compute screen-coord endpoints. Same correction as the click
+ // tools — bitmap pixels are anchored to the DWM-frame top-left,
+ // not the client area top-left (see `bitmap_to_screen` doc).
+ let (sx_from, sy_from) = bitmap_to_screen(hwnd, from_x as i32, from_y as i32);
+ let (sx_to, sy_to) = bitmap_to_screen(hwnd, to_x as i32, to_y as i32);
+
+ // delivery_mode:"background" on WinUI3: a pointer drag can't both land and
+ // hold the contract — the content island only consumes real
+ // system-queue pointer input (which the pen/touch injector delivers but
+ // at the cost of click-activating the frame: 8/8 foreground steals) and
+ // ignores posted WM_* drag messages; UIA has no drag analogue. Return
+ // the structured error so the caller retries with delivery_mode:foreground
+ // (a WinUI3 Slider can also be set with the set_value tool, which uses
+ // UIA RangeValuePattern and holds the contract).
+ if delivery == DeliveryMode::Background {
+ if let Some(r) = winui3_background_gesture(&None, pid, hwnd, None, 1, &button).await {
+ return r;
+ }
+ }
+
+ // delivery_mode:"background" — if a PostMessage drag would silently drop
+ // (Chromium/WPF/GTK canvas content reads mouse from the system input
+ // queue, not the per-window queue), fall back to coordinate-routed
+ // synthetic-pen drag injection instead of refusing. No foreground swap,
+ // no cursor move; the target is held non-activatable + cloaked for the
+ // stroke (mirrors the click pen path).
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseMove)
+ {
+ let target = hwnd;
+ let btn = button.clone();
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx_from as f64,
+ y: sy_from as f64,
+ },
+ );
+ let inj = tokio::task::spawn_blocking(move || {
+ crate::input::inject::inject_drag_screen(
+ target,
+ sx_from,
+ sy_from,
+ sx_to,
+ sy_to,
+ steps.max(8),
+ &btn,
+ )
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (inj, ()) = tokio::join!(inj, visual_drag);
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \
+ from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \
+ (delivery_mode:background, PostMessage would have been dropped)."
+ )),
+ Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseMove,
+ e.to_string(),
+ ),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // delivery_mode:"foreground" — SendInput-based drag. Required for WPF
+ // Slider thumbs (and any framework that polls GetKeyState during
+ // its drag handler — PostMessage doesn't update per-thread input
+ // state, so those targets see a button-up world during the drag
+ // and never start tracking). Like the foreground click, it activates
+ // the target and refuses rather than drag in a window that covers it.
+ if delivery == DeliveryMode::Foreground {
+ let btn_fg = button.clone();
+ pin_overlay_above(&cursor_key, hwnd);
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking(move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::mouse::send_drag_synthesized_active(
+ hwnd,
+ sx_from,
+ sy_from,
+ sx_to,
+ sy_to,
+ duration_ms,
+ steps,
+ &btn_fg,
+ &modifier_refs,
+ )
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (send_result, ()) = tokio::join!(send_result, visual_drag);
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ let button_suffix = if button == "left" {
+ String::new()
+ } else {
+ format!(" ({} button)", button)
+ };
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent drag{button_suffix} via SendInput on pid {raw_pid} \
+ from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \
+ in {duration_ms}ms / {steps} steps (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => {
+ let message = e.to_string();
+ if message.starts_with("foreground_unavailable:") {
+ // Nothing was sent; give callers the structured code
+ // and its recovery hint rather than bare text.
+ ToolResult::error(message).with_structured(json!({
+ "code": "foreground_unavailable",
+ "effect": "refused",
+ "pid": raw_pid,
+ "window_id": hwnd,
+ }))
+ } else {
+ ToolResult::error(message)
+ }
+ }
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Background posted-message drags cannot move or resize a top-level
+ // window: a caption/border drag is handled by the OS's interactive
+ // move/resize modal loop, which only real pointer input can drive —
+ // the posted WM_MOUSEMOVE stream is discarded and the "success" is a
+ // verified no-op. Hit-test the start point and refuse with the
+ // structured background_unavailable error so callers escalate to
+ // delivery_mode:"foreground" per the documented ladder instead of
+ // trusting a drag that did nothing.
+ let nc_hit = tokio::task::spawn_blocking(move || {
+ crate::input::mouse::non_client_move_resize_hit(hwnd, sx_from, sy_from)
+ })
+ .await
+ .ok()
+ .flatten();
+ if let Some(region) = nc_hit {
+ return crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseMove,
+ format!(
+ "the drag starts on the window's {region} (WM_NCHITTEST), and moving \
+ or resizing a top-level window requires the OS interactive \
+ move/resize loop, which posted mouse messages cannot drive. \
+ Re-issue the same drag with delivery_mode:\"foreground\"."
+ ),
+ );
+ }
+
+ // Pin the agent-cursor overlay above the drag target so the synthetic
+ // cursor stays sandwiched at z+1 of the dragged window for the full
+ // path. Drag stays within a single HWND, so one pin at the start is
+ // sufficient — the 80 ms z-order tick keeps it asserted thereafter.
+ pin_overlay_above(&cursor_key, hwnd);
+
+ // Move the agent cursor to the drag-start, show the press state, and
+ // track the same interpolated step count and duration while the
+ // posted-message drag runs.
+ overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: sx_from as f64,
+ y: sy_from as f64,
+ },
+ );
+
+ let button_c = button.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ // Screen-coord, deepest-child variant: routes the gesture to the
+ // child control under the start point (e.g. a WinForms Panel),
+ // not the top-level frame that would ignore it.
+ crate::input::mouse::post_drag_screen(
+ hwnd,
+ sx_from,
+ sy_from,
+ sx_to,
+ sy_to,
+ duration_ms,
+ steps,
+ &button_c,
+ )
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (result, ()) = tokio::join!(result, visual_drag);
+
+ match result {
+ Ok(Ok(())) => {
+ // Match Swift's text format verbatim.
+ let button_suffix = if button == "left" {
+ String::new()
+ } else {
+ format!(" ({} button)", button)
+ };
+ ToolResult::text(format!(
+ "✅ Posted drag{button_suffix} to pid {raw_pid} \
+ from window-pixel ({}, {}) → ({}, {}), \
+ screen ({sx_from}, {sy_from}) → ({sx_to}, {sy_to}) \
+ in {duration_ms}ms / {steps} steps.",
+ from_x as i32, from_y as i32, to_x as i32, to_y as i32,
+ ))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+#[cfg(test)]
+mod drag_modifier_tests {
+ use super::DragTool;
+ use cua_driver_core::tool::Tool;
+ use serde_json::json;
+
+ // A background drag posts mouse messages or injects pen input, neither of
+ // which carries live modifier-key state. The refusal comes before any window
+ // lookup or input, so an unknown pid never reaches the OS.
+ #[tokio::test]
+ async fn background_drag_with_modifiers_is_refused_before_input() {
+ let tool = DragTool {
+ state: super::ToolState::new(None),
+ };
+ for delivery in [json!(null), json!("background")] {
+ let mut args = json!({
+ "pid": 0x7fff_fff0_i64,
+ "from_x": 10, "from_y": 10, "to_x": 50, "to_y": 10,
+ "modifier": ["ctrl"]
+ });
+ if !delivery.is_null() {
+ args["delivery_mode"] = delivery;
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ result.is_error,
+ Some(true),
+ "modified background drag was not refused"
+ );
+ let structured = result.structured_content.expect("structured refusal");
+ assert_eq!(structured["code"], "background_unavailable");
+ assert_eq!(structured["effect"], "refused");
+ assert_eq!(structured["escalation"]["recommended"], "foreground");
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_accessibility_tree.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_accessibility_tree.rs
new file mode 100644
index 0000000000..c211a11748
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_accessibility_tree.rs
@@ -0,0 +1,77 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── get_accessibility_tree ────────────────────────────────────────────────────
+
+pub struct GetAccessibilityTreeTool;
+
+static GAX_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetAccessibilityTreeTool {
+ fn def(&self) -> &ToolDef {
+ GAX_DEF.get_or_init(|| ToolDef {
+ name: "get_accessibility_tree".into(),
+ description: "Return a lightweight snapshot of the desktop: running processes and \
+ on-screen visible windows with their bounds and owner pid.\n\n\
+ For the full UIA subtree of a single window (with interactive element indices \
+ you can click by), use get_window_state instead — this is a fast discovery read."
+ .into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ let (procs, windows) = tokio::task::spawn_blocking(|| {
+ (
+ crate::win32::list_processes(),
+ crate::win32::list_windows(None),
+ )
+ })
+ .await
+ .unwrap_or_default();
+
+ let mut lines = vec![format!(
+ "{} running process(es), {} visible window(s)",
+ procs.len(),
+ windows.len()
+ )];
+ for p in &procs {
+ lines.push(format!("- {} (pid {})", p.name, p.pid));
+ }
+ if !windows.is_empty() {
+ lines.push(String::new());
+ lines.push("Windows:".to_owned());
+ for w in &windows {
+ let title = if w.title.is_empty() {
+ "(no title)".to_owned()
+ } else {
+ format!("\"{}\"", w.title)
+ };
+ lines.push(format!(
+ "- pid={} {} [window_id: {}] {}x{}+{}+{}",
+ w.pid, title, w.hwnd, w.width, w.height, w.x, w.y
+ ));
+ }
+ lines.push(
+ "→ Call get_window_state(pid, window_id) to inspect a window's UI.".to_owned(),
+ );
+ }
+
+ let structured = json!({
+ "processes": procs.iter().map(|p| json!({"pid":p.pid,"name":p.name})).collect::<Vec<_>>(),
+ "windows": windows.iter().map(|w| json!({
+ "window_id": w.hwnd, "pid": w.pid, "title": w.title,
+ "x": w.x, "y": w.y, "width": w.width, "height": w.height
+ })).collect::<Vec<_>>()
+ });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_config.rs
new file mode 100644
index 0000000000..e7eca6a9a4
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_config.rs
@@ -0,0 +1,66 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{resolve_cursor_key, ToolState};
+
+// ── get_config ────────────────────────────────────────────────────────────────
+
+pub struct GetConfigTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static GCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetConfigTool {
+ fn def(&self) -> &ToolDef {
+ GCFG_DEF.get_or_init(|| ToolDef {
+ name: "get_config".into(),
+ // Description ported from Swift `GetConfigTool.swift` with the
+ // Windows config path. Swift's `agent_cursor.*` subtree is not
+ // mirrored on Windows yet (separate cursor-overlay config path).
+ description: "Report the current persistent driver config.\n\n\
+ Pure read-only. Returns defaults when the underlying state is unset — \
+ same fallback the daemon uses at startup. Sibling to `set_config`.\n\n\
+ Current schema (Windows):\n\n \
+ {\n \"schema_version\": 1,\n \"version\": \"<crate version>\",\n \
+ \"source_sha\": \"<maintainer build commit or null>\",\n \"platform\": \"windows\",\n \
+ \"capture_mode\": \"ax\" | \"vision\" (DEPRECATED, ignored),\n \
+ \"max_image_dimension\": 0\n }\n\n\
+ `capture_mode` is deprecated and no longer affects behavior — \
+ `get_window_state` always returns both the UIA tree and a screenshot.".into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cfg = self.state.config.read().unwrap();
+ // Mirror the macOS agent's parity addition (commit adb9ecca):
+ // nested `agent_cursor.enabled` block so Swift-shaped get_config
+ // consumers can read the cursor's enabled state from one place.
+ // Scope to the CALLING session's cursor (session > cursor_id > default)
+ // and read it from the overlay deterministically — `all_states().first()`
+ // was a nondeterministic HashMap read across sessions (macOS BUG 3).
+ let cursor_enabled = crate::overlay::is_enabled(&resolve_cursor_key(&args));
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
+ let payload = json!({
+ "schema_version": 1,
+ "version": env!("CARGO_PKG_VERSION"),
+ "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
+ "platform": "windows",
+ "capture_mode": cfg.capture_mode,
+ "max_image_dimension": cfg.max_image_dimension,
+ "agent_cursor": { "enabled": cursor_enabled },
+ "cursor": cursor_overlay::cursor_options::saved_config_json(),
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry,
+ });
+ // Match Swift's text format 1:1: `"✅ <pretty JSON>"`.
+ let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
+ ToolResult::text(format!("✅ {pretty}")).with_structured(payload)
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_cursor_position.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_cursor_position.rs
new file mode 100644
index 0000000000..56711b0bdb
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_cursor_position.rs
@@ -0,0 +1,47 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetCursorPositionInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+// ── get_cursor_position ───────────────────────────────────────────────────────
+
+pub struct GetCursorPositionTool;
+static GCP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetCursorPositionTool {
+ fn def(&self) -> &ToolDef {
+ GCP_DEF.get_or_init(|| ToolDef {
+ name: "get_cursor_position".into(),
+ description:
+ "Return the current mouse cursor position in screen points (origin top-left)."
+ .into(),
+ input_schema: json!({"type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema()
+ },"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Err(error) = parse_typed_input::<GetCursorPositionInput>("get_cursor_position", args)
+ {
+ return error;
+ }
+ use windows::Win32::Foundation::POINT;
+ use windows::Win32::UI::WindowsAndMessaging::GetCursorPos;
+ let mut pt = POINT { x: 0, y: 0 };
+ unsafe {
+ let _ = GetCursorPos(&mut pt);
+ }
+ // Text format matches Swift `GetCursorPositionTool` 1:1.
+ ToolResult::text(format!("✅ Cursor at ({}, {})", pt.x, pt.y))
+ .with_structured(json!({ "x": pt.x, "y": pt.y }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_desktop_state.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_desktop_state.rs
new file mode 100644
index 0000000000..73de4e4c01
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_desktop_state.rs
@@ -0,0 +1,262 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetDesktopStateInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{physical_screen_size, ToolState};
+
+#[cfg(test)]
+use super::impl_::is_windowless_desktop_action;
+
+// ── get_desktop_state ─────────────────────────────────────────────────────────
+
+/// `get_desktop_state` — full-display vision screenshot (Windows).
+///
+/// Vision-only desktop capture: grabs the ENTIRE primary display at native
+/// physical-pixel size (no downscale) so screen-absolute pixel picks land
+/// exactly, then reports the true screen size. No UIA walk, no pid/window_id —
+/// this is the capture surface for actions with a primary-display desktop
+/// target and screen-absolute coordinates.
+///
+/// Mirrors the `get_window_state` vision branch's ToolResult shape: an
+/// `image_png` content part (or a written-out file path), a text summary line,
+/// and a `structuredContent` object.
+pub struct GetDesktopStateTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static GDS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetDesktopStateTool {
+ fn def(&self) -> &ToolDef {
+ GDS_DEF.get_or_init(|| ToolDef {
+ name: "get_desktop_state".into(),
+ description: "Capture the full display in true screen pixels, full size unless \
+ `max_image_dimension` caps it. Use its PNG as the coordinate source for actions \
+ whose target is {kind:\"desktop\",display_id:\"primary\"}. Returns the true \
+ screen size. Vision-only: no UIA tree walk.".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
+ "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." },
+ "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()
+ },
+ "additionalProperties": false
+ }),
+ read_only: true, destructive: false, idempotent: false, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::protocol::Content;
+ let binding_args = args.clone();
+ let input = match parse_typed_input::<GetDesktopStateInput>("get_desktop_state", args) {
+ Ok(input) => input,
+ Err(error) => return error,
+ };
+ let screenshot_out_file = input.screenshot_out_file;
+ let max_image_dimension = input.max_image_dimension.filter(|cap| *cap > 0);
+
+ // True screen geometry in physical pixels (same space as the capture).
+ let (screen_width, screen_height) = physical_screen_size();
+
+ // Capture the FULL display at native size — no resize. Run the
+ // blocking GDI capture off the async runtime.
+ let out_file = screenshot_out_file.clone();
+ let res = tokio::task::spawn_blocking(
+ move || -> anyhow::Result<(Option<String>, Option<String>, u32, u32, Vec<u8>, (u32, u32), cursor_overlay::capture_exclusion::AgentOverlayCapture)> {
+ use base64::{engine::general_purpose::STANDARD as BASE64, Engine};
+ // The agent reads this image, so the Driver's own cursor and
+ // session pill must not sit over the controls it is reading.
+ let (png, overlay_capture) =
+ crate::capture::screenshot_display_bytes_excluding_overlay()?;
+ let full = crate::capture::png_dimensions_pub(&png)?;
+ // Opt-in cap; later desktop-scope pixels from the capped
+ // image are mapped back at dispatch and by its capture_id.
+ let png = match max_image_dimension {
+ Some(cap) if full.0.max(full.1) > cap => {
+ crate::capture::resize_png_if_needed(&png, cap)?
+ }
+ _ => png,
+ };
+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;
+ if let Some(ref path) = out_file {
+ std::fs::write(path, &png)?;
+ Ok((None, Some(path.clone()), w, h, png, full, overlay_capture))
+ } else {
+ Ok((Some(BASE64.encode(&png)), None, w, h, png, full, overlay_capture))
+ }
+ },
+ )
+ .await;
+
+ let (
+ b64_opt,
+ file_path,
+ screenshot_width,
+ screenshot_height,
+ png,
+ full_size,
+ overlay_capture,
+ ) = match res {
+ Ok(Ok(v)) => v,
+ Ok(Err(e)) => return ToolResult::error(format!("Desktop screenshot failed: {e}")),
+ Err(e) => return ToolResult::error(format!("Desktop screenshot task error: {e}")),
+ };
+
+ let mut content: Vec<Content> = Vec::new();
+ if let Some(b64) = b64_opt {
+ content.push(Content::image_png(b64));
+ }
+ let mut summary = format!(
+ "desktop screenshot {screenshot_width}x{screenshot_height} px \
+ (screen {screen_width}x{screen_height} px)"
+ );
+ cursor_overlay::capture_exclusion::append_summary_note(&mut summary, &overlay_capture);
+ content.push(Content::text(summary));
+
+ let mut structured = json!({
+ "platform": "windows",
+ "display": "primary",
+ "screenshot_width": screenshot_width,
+ "screenshot_height": screenshot_height,
+ "screen_width": screen_width,
+ "screen_height": screen_height,
+ "scale_factor": unsafe {
+ let dpi = windows::Win32::UI::HiDpi::GetDpiForSystem();
+ if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 }
+ },
+ "screenshot_mime_type": "image/png",
+ "agent_overlay_capture": overlay_capture,
+ });
+ if full_size != (screenshot_width, screenshot_height) {
+ structured["screenshot_original_width"] = json!(full_size.0);
+ structured["screenshot_original_height"] = json!(full_size.1);
+ }
+ if let Some(bridge) = &self.state.capture_bridge {
+ let geometry = match crate::capture_admission::CaptureGeometry::new(
+ screenshot_width,
+ screenshot_height,
+ full_size.0,
+ full_size.1,
+ ) {
+ Ok(geometry) => geometry,
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Desktop capture geometry is invalid: {error}"
+ ))
+ }
+ };
+ match bridge.publish(
+ &binding_args,
+ png,
+ crate::capture_admission::WindowsCaptureTarget::PrimaryDesktop,
+ geometry,
+ ) {
+ Ok(Some(capture_id)) => structured["capture_id"] = json!(capture_id),
+ Ok(None) => {}
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Desktop capture publication failed: {error}"
+ ))
+ }
+ }
+ }
+ if let Some(ref fp) = file_path {
+ structured["screenshot_file_path"] = json!(fp);
+ }
+
+ ToolResult {
+ content,
+ is_error: None,
+ structured_content: Some(structured),
+ action_record: None,
+ }
+ }
+}
+
+#[cfg(test)]
+mod desktop_scope_tests {
+ use super::{is_windowless_desktop_action, GetDesktopStateTool};
+ use cua_driver_core::tool::Tool;
+ use serde_json::json;
+
+ // ── is_windowless_desktop_action ──────────────────────────────────────────
+
+ #[test]
+ fn windowless_true_for_xy_under_desktop_scope_click_shape() {
+ // Click arg shape: {x, y}.
+ assert!(is_windowless_desktop_action(
+ &json!({"x": 10, "y": 20, "scope": "desktop"})
+ ));
+ }
+
+ #[test]
+ fn windowless_true_for_xy_under_desktop_scope_scroll_shape() {
+ // Scroll arg shape: {direction, x, y}.
+ assert!(is_windowless_desktop_action(&json!({
+ "direction": "down", "x": 10, "y": 20, "scope": "desktop"
+ })));
+ }
+
+ #[test]
+ fn windowless_false_when_pid_present() {
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "pid": 5, "scope": "desktop"
+ })));
+ }
+
+ #[test]
+ fn windowless_false_when_window_id_present() {
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "window_id": 99, "scope": "desktop"
+ })));
+ }
+
+ #[test]
+ fn windowless_false_under_window_scope() {
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "scope": "window"
+ })));
+ }
+
+ #[test]
+ fn windowless_false_when_xy_missing() {
+ assert!(!is_windowless_desktop_action(
+ &json!({"x": 10, "scope": "desktop"})
+ ));
+ assert!(!is_windowless_desktop_action(
+ &json!({"y": 20, "scope": "desktop"})
+ ));
+ assert!(!is_windowless_desktop_action(&json!({"scope": "desktop"})));
+ // Non-numeric x/y must not qualify.
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": "10", "y": "20", "scope": "desktop"
+ })));
+ }
+
+ // ── get_desktop_state schema ──────────────────────────────────────────────
+
+ #[test]
+ fn get_desktop_state_schema_rejects_window_scoped_fields() {
+ let tool = GetDesktopStateTool {
+ state: super::ToolState::new(None),
+ };
+ // The portable get_desktop_state contract pins the accepted fields and
+ // annotations; it cannot see a window-scoped field added to live.
+ let d = tool.def();
+ let props = d.input_schema["properties"].as_object().unwrap();
+ assert!(!props.contains_key("pid"), "must not accept pid");
+ assert!(
+ !props.contains_key("window_id"),
+ "must not accept window_id"
+ );
+ assert_eq!(d.input_schema["additionalProperties"], json!(false));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_screen_size.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_screen_size.rs
new file mode 100644
index 0000000000..f2578cf5bc
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_screen_size.rs
@@ -0,0 +1,46 @@
+use async_trait::async_trait;
+use cua_driver_contract::GetScreenSizeInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+use super::impl_::physical_screen_size;
+
+// ── get_screen_size ───────────────────────────────────────────────────────────
+
+pub struct GetScreenSizeTool;
+static GSS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for GetScreenSizeTool {
+ fn def(&self) -> &ToolDef {
+ GSS_DEF.get_or_init(|| ToolDef {
+ name: "get_screen_size".into(),
+ description: "Return the size of the main display in physical pixels plus its display \
+ scale factor. On Windows, screenshots and pixel clicks use this same physical-pixel \
+ coordinate space. Requires no special permissions.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "session": cua_driver_core::tool_schema::session_schema()
+ },"additionalProperties":false}),
+ read_only: true, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Err(error) = parse_typed_input::<GetScreenSizeInput>("get_screen_size", args) {
+ return error;
+ }
+ use windows::Win32::UI::HiDpi::GetDpiForSystem;
+ // With permonitorv2 DPI awareness (set in cua-driver.manifest),
+ // SM_CXSCREEN/SM_CYSCREEN return PHYSICAL pixels — the same
+ // coordinate space screenshots and pixel clicks use on Windows.
+ // Report these as-is, along with the scale factor for reference.
+ let (w, h) = physical_screen_size();
+ let dpi = unsafe { GetDpiForSystem() };
+ let scale = if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 };
+ ToolResult::text(format!("✅ Main display: {w}x{h} pixels @ {scale}x"))
+ .with_structured(json!({ "width": w, "height": h, "scale_factor": scale }))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_window_state.rs
new file mode 100644
index 0000000000..4534e2669a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/get_window_state.rs
@@ -0,0 +1,853 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{bitmap_to_screen, is_chromium_browser_target, ToolState};
+
+#[cfg(test)]
+mod get_window_state_actions_tests {
+ use super::*;
+ use crate::uia::UiaNode;
+
+ fn node(actions: Vec<String>) -> UiaNode {
+ UiaNode {
+ element_index: Some(1),
+ control_type: "Button".to_owned(),
+ name: Some("OK".to_owned()),
+ value: None,
+ automation_id: None,
+ help_text: None,
+ actions,
+ enabled: Some(true),
+ selected: None,
+ element_ptr: 0,
+ center_x: 0,
+ center_y: 0,
+ rect: None,
+ msaa_role: None,
+ depth: 0,
+ parent_element_index: None,
+ in_web_content: false,
+ }
+ }
+
+ #[test]
+ fn element_entry_includes_actions_when_present() {
+ let n = node(vec!["invoke".to_owned(), "toggle".to_owned()]);
+ let entry = build_element_entry(&n, None).unwrap();
+ assert_eq!(entry["actions"], json!(["invoke", "toggle"]));
+ }
+
+ #[test]
+ fn element_entry_omits_actions_when_empty() {
+ let n = node(Vec::new());
+ let entry = build_element_entry(&n, None).unwrap();
+ assert!(entry.get("actions").is_none());
+ }
+}
+
+// ── get_window_state ─────────────────────────────────────────────────────────
+
+/// Build a single structured element entry for `get_window_state`.
+/// Returns `None` when the node has no `element_index` (non-actionable rows).
+fn build_element_entry(
+ n: &crate::uia::UiaNode,
+ snapshot_id: Option<u32>,
+) -> Option<serde_json::Value> {
+ let idx = n.element_index?;
+ // `label`: name → value → automation_id → help_text.
+ let label = n
+ .name
+ .clone()
+ .or_else(|| n.value.clone())
+ .or_else(|| n.automation_id.clone())
+ .or_else(|| n.help_text.clone());
+ let mut entry = json!({
+ "element_index": idx,
+ "role": n.control_type,
+ "depth": n.depth,
+ });
+ if let Some(snapshot_id) = snapshot_id {
+ entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx));
+ }
+ if n.in_web_content {
+ entry["in_web_content"] = json!(true);
+ }
+ if let Some(label) = label {
+ entry["label"] = json!(label);
+ }
+ // Surface the element's value separately from `label` (which collapses
+ // name→value→automation_id→help): a control with both a name AND text
+ // (a ValuePattern edit holding typed content) would otherwise hide the
+ // text from a caller reading the structured side. See the macOS
+ // get_window_state builder for the rationale.
+ if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) {
+ entry["value"] = json!(value);
+ }
+ if let Some(enabled) = n.enabled {
+ entry["enabled"] = json!(enabled);
+ }
+ if let Some(selected) = n.selected {
+ entry["selected"] = json!(selected);
+ }
+ if !n.actions.is_empty() {
+ entry["actions"] = json!(n.actions);
+ }
+ if let Some(parent) = n.parent_element_index {
+ entry["parent_index"] = json!(parent);
+ }
+ if let Some((l, t, r, b)) = n.rect {
+ entry["frame"] = json!({
+ "x": l,
+ "y": t,
+ "w": (r - l).max(0),
+ "h": (b - t).max(0),
+ });
+ }
+ Some(entry)
+}
+
+pub struct GetWindowStateTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static GWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+/// Slack past `timeout_ms` before the walk task is abandoned.
+const UIA_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_millis(500);
+/// Deadline for the window capture, separate from the walk budget.
+const UIA_SCREENSHOT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(4);
+
+#[async_trait]
+impl Tool for GetWindowStateTool {
+ fn def(&self) -> &ToolDef {
+ GWS_DEF.get_or_init(|| ToolDef {
+ name: "get_window_state".into(),
+ // Description ported from Swift `GetWindowStateTool.swift` with
+ // Windows-specific adaptations (UIA instead of AX; no SkyLight
+ // Space validation; no per-call `javascript` field — that's a
+ // macOS AppleScript hook).
+ description: "Walk a running app's UIA tree and return it ONCE, as compact \
+ Markdown by default (`tree_format:\"markdown\"`). Every actionable element is \
+ tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where \
+ `snapshot_id` is printed in the response header and returned in structuredContent. \
+ Pass `tree_format:\"elements\"` for the structured `elements` array instead \
+ (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), \
+ or `\"both\"` for both (about twice the size).\n\n\
+ Size: a read is bounded by default (`max_elements` 250). When the walk stops at \
+ that budget the response says `Tree truncated at max_elements=…` and `truncated: \
+ true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. \
+ `_note` is omitted unless `verbose:true`. `full_output:true` restores the \
+ previous full response: both representations, all metadata, and the platform's \
+ walk limits (≤5 000 elements, depth ≤25).\n\n\
+ DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
+ to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
+ (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
+ their previous `[N]` unless a `reindexed:` line says otherwise. The response \
+ carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
+ other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
+ back to a full read and `since_status` says why. The focused element is not \
+ reported on Windows yet.\n\n\
+ INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \
+ any element action against that window. The next snapshot of the same \
+ (pid, window_id) replaces this one, stales its element tokens, and lists the \
+ replaced ids in `invalidated_snapshot_ids`. A screenshot-only read \
+ (`include_accessibility_tree:false`) keeps the current snapshot's rows and \
+ tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\n\
+ The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \
+ window bounds reported come from the same `window_id`. This is the source of \
+ truth for which window the caller intends to reason about — the driver never \
+ picks a window implicitly.\n\n\
+ `window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. \
+ The driver does not auto-fall-back to a different window.\n\n\
+ Set `query` to a case-insensitive substring to project the tree to matching \
+ rows plus their ancestor chain. \
+ Original element indices are preserved. `total_element_count` reports the \
+ unprojected walked snapshot; `returned_element_count` reports the projection. \
+ `elements_complete` describes the underlying walk and is unaffected by query \
+ projection.\n\n\
+ Always returns BOTH the element tree AND a screenshot — ground on both \
+ and cross-check (the tree lies on some surfaces). Choose the modality at \
+ ACTION time: an element ax action (element_token → \
+ accessibility rung) or an element px action (x,y → pixel rung off this \
+ screenshot). capture_mode is deprecated and ignored.\n\n\
+ The mirror image: pass `include_accessibility_tree:false` to SKIP the \
+ UIA walk entirely and return just the screenshot plus window metadata \
+ (window_bounds, app_name, window_title) — the capture-only path for a \
+ live window preview / picture-in-picture. Setting BOTH \
+ `include_accessibility_tree:false` and `include_screenshot:false` is an \
+ error. Optional `max_image_dimension` overrides the configured screenshot \
+ long-edge limit for this call; pass 0 for native resolution. Legacy \
+ `max_dimension` remains a cap on the configured limit.\n\n\
+ Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a \
+ single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of \
+ timing out at 4s with per-property RPCs).\n\n\
+ Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \
+ apps produce 10k+ element trees). Markdown and structured elements are \
+ truncated identically. `elements_complete` is false when a bound skips a node \
+ or subtree; ending exactly at a bound remains complete. Default: \
+ `max_elements` 250, depth ≤25.\n\n\
+ CHROMIUM COVERAGE: a browser-owned permission bubble can be \
+ composited outside the requested native window. Chromium-family \
+ snapshots therefore describe this limit in structuredContent.capture_coverage. \
+ After a verified ineffective window action, call escalate_session, take a \
+ fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, \
+ then verify with another fresh desktop snapshot. \
+ This is separate from page JavaScript dialogs, which remain on \
+ browser_dialog.\n\n\
+ Windows requires no special permissions.".into(),
+ input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid","window_id"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Process ID from `list_apps`."},
+ "window_id":{"type":"integer","description":"HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array."},
+ "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
+ "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
+ "include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
+ "screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},
+ "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk."},
+ "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
+ "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},
+ "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
+ "max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default."},
+ "max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins."}
+ },"additionalProperties":false})),
+ // Swift annotation: idempotent: false (each call is a fresh snapshot).
+ read_only: true, destructive: false, idempotent: false, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let display_only = match cua_driver_core::window_state_view::display_only(&args) {
+ Ok(display_only) => display_only,
+ Err(refusal) => return refusal,
+ };
+ // Swift error wording 1:1.
+ let pid = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(v) => v as u32,
+ None => return ToolResult::error("Missing required integer field pid."),
+ };
+ let hwnd =
+ match args.get("window_id").and_then(|v| v.as_u64()) {
+ Some(v) => v,
+ None => return ToolResult::error(
+ "Missing required integer field window_id. Use `list_windows` to enumerate \
+ the target app's windows, or read `launch_app`'s `windows` array.",
+ ),
+ };
+ // Validate window belongs to pid — Swift's hard error. The exact HWND is
+ // probed through Win32 first; the desktop-wide UIA union (2 s deadline)
+ // is consulted only on a miss (#4416).
+ use crate::win32::PidWindowLookup;
+ let lookup =
+ tokio::task::spawn_blocking(move || crate::win32::lookup_window_for_pid(pid, hwnd))
+ .await
+ .unwrap_or(PidWindowLookup::Missing);
+ let window = match lookup {
+ PidWindowLookup::Found(window) => window,
+ PidWindowLookup::OtherPid(owner) => {
+ return ToolResult::error(format!(
+ "window_id {hwnd} belongs to pid {owner}, not pid {pid}. Call \
+ `list_windows({{\"pid\": {pid}}})` to get this pid's own windows."
+ ));
+ }
+ PidWindowLookup::Missing => {
+ return ToolResult::error(format!(
+ "No window with window_id {hwnd} exists. Call `list_windows({{\"pid\": \
+ {pid}}})` for candidates."
+ ));
+ }
+ };
+ // Window identity metadata (additive): title + on-screen rectangle from
+ // the lookup we already did, plus the owning process's executable
+ // name. Names the surface on the capture-only path, where no UIA tree
+ // identifies it.
+ let win_geom = Some((
+ window.title.clone(),
+ window.x,
+ window.y,
+ window.width,
+ window.height,
+ ));
+ let app_name = tokio::task::spawn_blocking(move || {
+ crate::win32::list_processes()
+ .into_iter()
+ .find(|p| p.pid == pid)
+ .map(|p| p.name)
+ })
+ .await
+ .ok()
+ .flatten();
+ use cua_driver_core::tool_args::ArgsExt;
+ // The canonical per-call value overrides all configured limits,
+ // including `0` for native resolution. The legacy field retains its
+ // previous cap-folding behavior when the canonical field is omitted.
+ let max_image_dimension = match args.get("max_image_dimension") {
+ Some(value) => match value.as_u64().and_then(|v| u32::try_from(v).ok()) {
+ Some(value) => Some(value),
+ None => {
+ return ToolResult::error(
+ "max_image_dimension must be an integer between 0 and 4294967295.",
+ )
+ }
+ },
+ None => None,
+ };
+ let max_dimension = args
+ .get("max_dimension")
+ .and_then(|v| v.as_u64())
+ .and_then(|v| u32::try_from(v).ok())
+ .map(|v| v.max(1));
+ let max_dim = {
+ let cfg = self.state.config.read().unwrap();
+ cua_driver_core::image_utils::ImageDimensionLimits {
+ configured: cfg.max_image_dimension,
+ legacy_max_dimension: max_dimension,
+ max_image_dimension,
+ }
+ .resolve()
+ };
+ // `capture_mode` is DEPRECATED and ignored — get_window_state always
+ // returns BOTH the UIA tree and a screenshot now, so the agent grounds on
+ // both and cross-checks (the UIA tree lies often enough that a grounding
+ // screenshot should always be present). The modality is chosen at action
+ // time: an element ax action (element_index) or an element px action (x,y).
+ // We don't read the arg; it stays in the schema only so old callers don't
+ // trip additionalProperties:false.
+ let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
+ Ok(view) => view,
+ Err(refusal) => return refusal,
+ };
+ let query = args.opt_str("query");
+ let session_id = args.opt_str("_session_id");
+ let screenshot_out_file = args.opt_str("screenshot_out_file");
+ // `display_only` is the public form of the internal observation-only
+ // read: pixels for a preview, with no snapshot or capture change.
+ let observation_only = display_only
+ || args
+ .get("_observation_only")
+ .and_then(|value| value.as_bool())
+ == Some(true);
+ // Optional caps — when omitted, fall back to the walker's built-in
+ // defaults (#22865). minimum:1 enforced in the schema, but defend
+ // against 0 here too.
+ let max_elements = view.max_elements(
+ &args,
+ crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,
+ observation_only,
+ );
+ let max_depth = args
+ .get("max_depth")
+ .and_then(|v| v.as_u64())
+ .map(|v| v.max(1) as usize)
+ .unwrap_or(crate::uia::DEFAULT_MAX_DEPTH);
+ let timeout_ms = cua_driver_core::tool_schema::resolve_timeout_ms(args.get("timeout_ms"));
+
+ // Always walk the UIA tree. The screenshot is returned by DEFAULT — the
+ // grounding frame the agent cross-checks the (sometimes-lying) tree
+ // against — but `include_screenshot:false` opts out of the grab entirely
+ // (tree only: the cheap path for re-indexing before an element ax action).
+ // `screenshot_out_file` still forces a capture to disk regardless. With
+ // `screenshot_out_file` the bytes go to disk and the path is surfaced
+ // instead of embedding base64; otherwise the base64 PNG is embedded.
+ let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
+ // `include_accessibility_tree` (default true) mirrors include_screenshot:
+ // set false to SKIP the UIA walk and return just the screenshot + window
+ // metadata (the capture-only / preview path).
+ let do_tree = args
+ .get("include_accessibility_tree")
+ .and_then(|v| v.as_bool())
+ != Some(false);
+ let do_shot = include_screenshot != Some(false) || screenshot_out_file.is_some();
+ if !do_tree && !do_shot {
+ return ToolResult::error(
+ "Nothing to return: both include_accessibility_tree:false and \
+ include_screenshot:false. Set at least one to true, or pass \
+ screenshot_out_file to force a capture.",
+ );
+ }
+
+ let state = self.state.clone();
+ let q = query.clone();
+ let out_file = screenshot_out_file.clone();
+ // The tree walk runs under `timeout_ms`. UIA's bulk BuildUpdatedCache
+ // cannot be interrupted (Chrome's provider can block indefinitely on
+ // property reads), so when the walk task itself outlives the budget
+ // the response carries an empty tree flagged `truncated` instead of an
+ // error, and the screenshot below still comes back.
+ let tree_result = if do_tree {
+ let tree_task = tokio::task::spawn_blocking(move || {
+ let mut budget =
+ cua_driver_core::walk_budget::WalkBudget::new(timeout_ms, max_elements);
+ let iconic_before = crate::input::window_is_iconic(hwnd);
+ let tree =
+ crate::uia::walk_tree_budgeted(hwnd, q.as_deref(), max_depth, &mut budget);
+ let kind = if tree.nodes.iter().any(|node| node.msaa_role.is_some()) {
+ crate::uia::snapshot::SnapshotKind::Msaa
+ } else {
+ crate::uia::snapshot::SnapshotKind::Uia
+ };
+ let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(&tree.nodes, kind)
+ .with_read_while_minimized(
+ iconic_before || crate::input::window_is_iconic(hwnd),
+ );
+ (tree, payload, budget.outcome())
+ });
+ let started = std::time::Instant::now();
+ let deadline = std::time::Duration::from_millis(timeout_ms) + UIA_WALK_BACKSTOP_GRACE;
+ match tokio::time::timeout(deadline, tree_task).await {
+ Ok(Ok(tree)) => Some(tree),
+ Ok(Err(e)) => return ToolResult::error(format!("UIA walk task panicked: {e}")),
+ Err(_elapsed) => {
+ let class = crate::input::delivery::read_class_name(hwnd);
+ let walk = cua_driver_core::walk_budget::WalkOutcome::timed_out(
+ timeout_ms,
+ started.elapsed(),
+ );
+ let tree = crate::uia::UiaTreeResult {
+ tree_markdown: format!(
+ "(The UIA provider for hwnd 0x{hwnd:x}, class '{class}', did not \
+ answer. Act by pixel off the screenshot, or send press_key with \
+ delivery_mode:\"foreground\" for a dialog's default accelerator.)"
+ ),
+ nodes: Vec::new(),
+ elements_complete: false,
+ };
+ let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
+ &[],
+ crate::uia::snapshot::SnapshotKind::Uia,
+ );
+ Some((tree, payload, walk))
+ }
+ }
+ } else {
+ None
+ };
+ let blocking = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
+ // Capture screenshot AND any error message so the response can
+ // surface *why* there's no image (the iconic-window guard from
+ // #1973 / PR #1974 is the load-bearing case: minimized windows
+ // legitimately can't be captured, and the caller needs to know
+ // to call `bring_to_front` instead of retrying).
+ // The previous `Err(_) => None` silently dropped the error and
+ // upstream agents saw an empty response with no signal.
+ let (screenshot, screenshot_err) = if do_shot {
+ match crate::capture::screenshot_window_bytes(hwnd) {
+ Ok(raw) => {
+ let (native_w, native_h) = crate::capture::png_dimensions_pub(&raw)?;
+ let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;
+ // `screenshot_out_file` set (any mode) → write to disk and
+ // surface the path, never embed bytes. Otherwise (vision,
+ // no out_file) → embed base64.
+ if let Some(ref path) = out_file {
+ std::fs::write(path, &png)?;
+ (
+ Some((None, Some(path.clone()), w, h, native_w, native_h, png)),
+ None,
+ )
+ } else {
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ (
+ Some((Some(B64.encode(&png)), None, w, h, native_w, native_h, png)),
+ None,
+ )
+ }
+ }
+ Err(e) => (None, Some(format!("{e}"))),
+ }
+ } else {
+ (None, None)
+ };
+ Ok((screenshot, screenshot_err))
+ });
+ let result: anyhow::Result<_> =
+ match tokio::time::timeout(UIA_SCREENSHOT_TIMEOUT, blocking).await {
+ Ok(join_result) => join_result
+ .map_err(|e| anyhow::anyhow!("task panic: {e}"))
+ .and_then(|r| r),
+ Err(_elapsed) => Ok((
+ None,
+ Some(format!(
+ "window capture did not finish within {}s",
+ UIA_SCREENSHOT_TIMEOUT.as_secs()
+ )),
+ )),
+ };
+ let result =
+ result.map(|(screenshot, screenshot_err)| (tree_result, screenshot, screenshot_err));
+
+ match result {
+ Ok((tree_opt, screenshot_opt, screenshot_err)) => {
+ let mut content = Vec::new();
+ let mut structured = json!({ "window_id": hwnd, "pid": pid });
+ let screenshot_scale = screenshot_opt
+ .as_ref()
+ .map(|(_, _, w, _, native_w, _, _)| *native_w as f64 / *w as f64);
+ let mut published_snapshot = false;
+ let mut invalidated = Vec::new();
+
+ if let Some((tr, payload, walk)) = tree_opt {
+ let is_msaa = tr.nodes.iter().any(|node| node.msaa_role.is_some());
+ let count = tr
+ .nodes
+ .iter()
+ .filter(|n| n.element_index.is_some())
+ .count();
+ let mut header = format!("window_id={hwnd} pid={pid} elements={count}\n");
+ if let Some(note) = walk.note() {
+ header.push_str(&note);
+ header.push('\n');
+ }
+ header.push('\n');
+ content.push(cua_driver_core::protocol::Content::text(
+ header + &tr.tree_markdown,
+ ));
+ structured["element_count"] = json!(count);
+ // Verification consumes only structured rows, which still
+ // omit non-actionable UIA nodes on Windows. A complete
+ // native walk cannot prove absence in that projection.
+ structured["elements_complete"] =
+ json!(tr.elements_complete && !observation_only);
+ structured["tree_markdown"] = json!(tr.tree_markdown);
+ walk.apply(&mut structured);
+
+ let (snapshot_id, replaced) = (!observation_only)
+ .then(|| {
+ state.snapshots.publish_for_session(
+ pid as i32,
+ hwnd,
+ payload,
+ session_id.as_deref(),
+ screenshot_scale,
+ )
+ })
+ .flatten()
+ .unzip();
+ published_snapshot = snapshot_id.is_some();
+ invalidated.extend(replaced.into_iter().flatten());
+
+ // Structured `elements` array — preferred consumption
+ // path. Shape matches the cross-platform spec:
+ // `{element_index, element_token, role, label, depth,
+ // actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is
+ // included when UIA reported a usable BoundingRectangle.
+ let elements: Vec<serde_json::Value> = tr
+ .nodes
+ .iter()
+ .filter_map(|n| build_element_entry(n, snapshot_id))
+ .collect();
+ let elements = cua_driver_core::element_query::project_elements_for_query(
+ elements,
+ query.as_deref(),
+ &tr.tree_markdown,
+ );
+ // Screenshot pixels: the capture's top-left is the DWM
+ // bitmap origin (the inverse of the pixel tools'
+ // `bitmap_to_screen`), downsized by delivered/native.
+ let elements = match screenshot_opt.as_ref() {
+ Some((_, _, w, _, native_w, _, _)) if *native_w > 0 => {
+ let (ox, oy) = bitmap_to_screen(hwnd, 0, 0);
+ cua_driver_core::element_frame::with_screenshot_frames(
+ elements,
+ (f64::from(ox), f64::from(oy)),
+ f64::from(*w) / f64::from(*native_w),
+ )
+ }
+ _ => elements,
+ };
+ structured["total_element_count"] = json!(count);
+ structured["returned_element_count"] = json!(elements.len());
+ structured["elements"] = json!(elements);
+ // Surface 6: snapshot id mirror for debug correlation.
+ if let Some(snapshot_id) = snapshot_id {
+ structured["snapshot_id"] = json!(
+ cua_driver_core::element_token::format_snapshot_id(snapshot_id)
+ );
+ }
+ structured["_note"] = json!(
+ "Prefer `elements` — `tree_markdown` will continue to work \
+ but new fields will only be added to the structured side. \
+ Issue #22865: use `max_elements` / `max_depth` to bound the \
+ UIA walk on apps with very large trees."
+ );
+ // Best-effort-background ladder: a UIA walk that ran but found
+ // zero actionable elements is NOT a clean snapshot — the window
+ // may be a non-UIA surface (canvas/WebGL/custom-drawn) or its
+ // tree wasn't ready (Chromium/Electron need an enable + settle).
+ // Mark it degraded so callers don't read `elements: []` as "this
+ // window has no controls", and point at the next rung. An empty
+ // UIA tree means element_index has nothing to bind to, so the
+ // deliberate move is an element px action — read the screenshot
+ // already in this response and click by pixel (x,y).
+ if is_msaa {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!(
+ "msaa_fallback_partial: the UIA provider was unavailable and \
+ Cua Driver used a partial MSAA tree. Treat it as discovery \
+ evidence only; it cannot prove checked state."
+ );
+ } else if let (0, Some(note)) = (count, walk.note()) {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!(note);
+ } else if count == 0 {
+ structured["degraded"] = json!(true);
+ structured["degraded_reason"] = json!(
+ "ax_tree_empty: the UIA walk returned no actionable elements. \
+ The window may be a non-UIA surface (canvas/WebGL/custom-drawn) \
+ or its accessibility tree was not ready (Chromium/Electron \
+ require a UIA-enable + settle). Do not treat element data as \
+ authoritative — re-snapshot if the app just launched, otherwise \
+ switch to the visual path."
+ );
+ structured["escalation"] = json!({
+ "recommended": "px",
+ "reason": "non-AX surface — act by pixel (x,y) off the \
+ screenshot in this response (an element px action)."
+ });
+ }
+ }
+
+ // A screenshot-only read adds an image but no element rows: it
+ // refreshes the current snapshot's pixel frame in place and
+ // keeps its tokens valid. Only a window with no element
+ // snapshot yet gets a new, empty one.
+ let mut refreshed = None;
+ if let (false, false, Some(scale)) =
+ (observation_only, published_snapshot, screenshot_scale)
+ {
+ refreshed = state.snapshots.refresh_screenshot_for_session(
+ pid as i32,
+ hwnd,
+ session_id.as_deref(),
+ scale,
+ None,
+ );
+ if refreshed.is_none() {
+ let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
+ &[],
+ crate::uia::snapshot::SnapshotKind::Uia,
+ );
+ if let Some((_, replaced)) = state.snapshots.publish_for_session(
+ pid as i32,
+ hwnd,
+ payload,
+ session_id.as_deref(),
+ screenshot_scale,
+ ) {
+ invalidated.extend(replaced);
+ }
+ }
+ }
+ if let Some(id) = refreshed {
+ let sid = cua_driver_core::element_token::format_snapshot_id(id);
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
+ ({sid}:N) stay valid."
+ )));
+ structured["snapshot_id"] = json!(sid);
+ structured["screenshot_refreshed_snapshot"] = json!(true);
+ }
+ if !invalidated.is_empty() {
+ let ids: Vec<String> = invalidated
+ .into_iter()
+ .map(cua_driver_core::element_token::format_snapshot_id)
+ .collect();
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "Invalidated snapshots {}: their element_tokens are stale.",
+ ids.join(", ")
+ )));
+ structured["invalidated_snapshot_ids"] = json!(ids);
+ }
+
+ if let Some((b64_opt, file_path, w, h, native_w, native_h, png)) = screenshot_opt {
+ let capture_id = if !observation_only {
+ if let Some(bridge) = &state.capture_bridge {
+ let geometry = match crate::capture_admission::CaptureGeometry::new(
+ w, h, native_w, native_h,
+ ) {
+ Ok(geometry) => geometry,
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Window capture geometry is invalid: {error}"
+ ))
+ }
+ };
+ match bridge.publish(
+ &args,
+ png,
+ crate::capture_admission::WindowsCaptureTarget::Window {
+ pid,
+ window_id: hwnd,
+ },
+ geometry,
+ ) {
+ Ok(capture_id) => capture_id,
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Window capture publication failed: {error}"
+ ))
+ }
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+ // base64 is embedded only when no out_file was given (vision
+ // path). With `screenshot_out_file` the bytes went to disk and
+ // we surface the path instead — never both. Keep a text content
+ // part when the image went to disk so the response is never
+ // empty on the capture-only path (which has no tree markdown).
+ if let Some(b64) = b64_opt {
+ content.push(cua_driver_core::protocol::Content::image_png(b64));
+ } else if let Some(fp) = &file_path {
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "window_id={hwnd} pid={pid} size={w}x{h} screenshot written to {fp}"
+ )));
+ }
+ structured["screenshot_width"] = json!(w);
+ structured["screenshot_height"] = json!(h);
+ // Surface 7: mirror the MCP image part's `mimeType` onto
+ // the structured payload so consumers don't have to sniff
+ // magic bytes off the base64 to know the format.
+ structured["screenshot_mime_type"] = json!("image/png");
+ if let Some(capture_id) = capture_id {
+ structured["capture_id"] = json!(capture_id);
+ }
+ if let Some(fp) = file_path {
+ structured["screenshot_file_path"] = json!(fp);
+ }
+ } else if let Some(err) = screenshot_err {
+ // Capture failed (most commonly because the target is
+ // minimized — see #1973). Surface the reason in BOTH the
+ // human-readable content stream (so the model sees it
+ // alongside the UIA tree it does get) AND structuredContent
+ // (so MCP clients with structured-only parsing can detect
+ // and act on it). Without this the caller saw an empty
+ // response with no clue why and burned turns retrying.
+ content.push(cua_driver_core::protocol::Content::text(format!(
+ "screenshot unavailable: {err}"
+ )));
+ structured["screenshot_error"] = json!(err);
+ }
+
+ // Window identity metadata (additive): title + on-screen
+ // rectangle + owning process name for the requested window_id,
+ // useful on the capture-only path where no UIA tree names it.
+ if let Some((title, x, y, w, h)) = &win_geom {
+ if !title.is_empty() {
+ structured["window_title"] = json!(title);
+ }
+ structured["window_bounds"] =
+ json!({ "x": x, "y": y, "width": w, "height": h });
+ }
+ if let Some(name) = app_name.as_deref().filter(|n| !n.is_empty()) {
+ structured["app_name"] = json!(name);
+ }
+
+ cua_driver_core::window_inspection::mark_browser_chrome_capture_coverage(
+ &mut structured,
+ is_standalone_chromium_browser_process(pid).then_some(
+ cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::NotObservable,
+ ),
+ );
+
+ // The capture-only path (include_accessibility_tree:false) leaves
+ // `content` empty if the screenshot was also unavailable. Return a
+ // structured error rather than a "successful" response with no
+ // content parts (consistent with the tree+screenshot path, which
+ // always carries at least the tree markdown).
+ if content.is_empty() {
+ return ToolResult::error(format!(
+ "No content produced for window_id {hwnd}: the accessibility tree was \
+ skipped (include_accessibility_tree:false) and no screenshot was returned."
+ ))
+ .with_structured(structured);
+ }
+
+ if display_only {
+ cua_driver_core::window_state_view::mark_display_only(&mut structured);
+ }
+ if !observation_only {
+ cua_driver_core::window_state_view::apply(
+ &view,
+ &cua_driver_core::window_state_view::ViewContext {
+ pid: i64::from(pid),
+ window_id: hwnd,
+ key: cua_driver_core::window_state_view::ViewKey {
+ query: query.clone(),
+ max_elements,
+ max_depth: args
+ .get("max_depth")
+ .and_then(|v| v.as_u64())
+ .map(|v| v as usize),
+ },
+ focus_probe: None,
+ },
+ &mut content,
+ &mut structured,
+ );
+ }
+
+ ToolResult {
+ content,
+ is_error: None,
+ structured_content: Some(structured),
+ action_record: None,
+ }
+ }
+ Err(e) => ToolResult::error(format!("Error: {e}")),
+ }
+ }
+}
+
+fn is_standalone_chromium_browser_process(pid: u32) -> bool {
+ crate::win32::list_processes()
+ .into_iter()
+ .find(|process| process.pid == pid)
+ .is_some_and(|process| is_chromium_browser_target(&process.name))
+}
+
+#[cfg(test)]
+mod display_only_tests {
+ use super::{GetWindowStateTool, ToolState};
+ use cua_driver_core::tool::Tool;
+
+ // T3 Code feature-detects the preview mode from this schema property.
+ #[test]
+ fn display_only_is_advertised() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(None),
+ };
+ assert_eq!(
+ tool.def().input_schema["properties"]["display_only"]["type"],
+ "boolean"
+ );
+ }
+
+ #[tokio::test]
+ async fn display_only_refuses_accessibility_snapshot_requests() {
+ let tool = GetWindowStateTool {
+ state: ToolState::new(None),
+ };
+ for tree in [None, Some(true)] {
+ let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
+ if let Some(tree) = tree {
+ args["include_accessibility_tree"] = serde_json::json!(tree);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(
+ serde_json::to_value(result).unwrap(),
+ serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
+ "display_only requires include_accessibility_tree:false"
+ ))
+ .unwrap()
+ );
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/hotkey.rs
new file mode 100644
index 0000000000..c92bbd5753
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/hotkey.rs
@@ -0,0 +1,398 @@
+use async_trait::async_trait;
+use cua_driver_contract::HotkeyInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ focus_by_pixel, focus_cached_element_for_foreground, position_keyboard_cursor, ToolState,
+};
+
+// ── hotkey ────────────────────────────────────────────────────────────────────
+
+fn is_modifier(k: &str) -> bool {
+ matches!(
+ k.to_lowercase().as_str(),
+ "ctrl" | "control" | "shift" | "alt" | "win" | "windows" | "cmd" | "command"
+ )
+}
+
+pub struct HotkeyTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static HOTKEY_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for HotkeyTool {
+ fn def(&self) -> &ToolDef {
+ HOTKEY_DEF.get_or_init(|| ToolDef {
+ name: "hotkey".into(),
+ // Description ported from Swift `HotkeyTool.swift` with the
+ // Windows-specific dispatch (UIA accelerator for XAML; SendInput
+ // for legacy Win32 with modifiers per #1614 / #1618; PostMessage
+ // for modifier-less keys).
+ description: "Press a combination of keys simultaneously — e.g. `[\"ctrl\", \"c\"]` \
+ for Copy, `[\"ctrl\", \"shift\", \"t\"]` for reopen-closed-tab. Dispatch is \
+ target-aware:\n\n\
+ - **Modern XAML / WinUI / UWP targets** route through UI Automation: the driver \
+ walks the target's accessibility subtree, finds a descendant whose AcceleratorKey \
+ matches the combo, and invokes it. No focus steal, no system-queue input.\n\n\
+ - **Legacy Win32 targets with modifiers** (Ctrl+S, Alt+Tab, etc.) route through \
+ `SendInput` against the system input queue, with a brief `SetForegroundWindow` \
+ swap so the events land on the target. This path is necessary because \
+ PostMessage(WM_KEYDOWN, VK_CONTROL) does NOT update the OS-wide modifier state \
+ visible to `GetKeyState` / `TranslateAccelerator`, so Win32 apps that bind \
+ accelerators via `TranslateAccelerator` (LibreOffice, FAR, classic Notepad, etc.) \
+ never see the combo as a real accelerator on the PostMessage-only path. \
+ Trade-off: brief foreground swap (mitigated by restoring the previous foreground \
+ after the keystrokes flush). Requires the daemon to have UIAccess integrity so \
+ `SetForegroundWindow` is permitted — the MCP proxy auto-prefers the \
+ `cua-driver-uia.exe` worker pipe when both daemons are running.\n\n\
+ - **Legacy Win32 targets without modifiers** (plain `enter`, `tab`, `f5`, etc.) \
+ route through `PostMessage(WM_KEYDOWN/UP)` — no focus steal, no need to update \
+ modifier state.\n\n\
+ The target does NOT need to be frontmost in any branch.\n\n\
+ **`window_id`** (optional): explicit HWND when the pid owns more than one \
+ window; otherwise the pid's first visible window is used.\n\n\
+ Recognized modifiers: ctrl/control, shift, alt, win/windows. Non-modifier \
+ keys use the same vocabulary as `press_key` (return, tab, escape, \
+ up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, \
+ letters, digits). Order: modifiers first, one non-modifier last.".into(),
+ input_schema: json!({
+ "type":"object","required":["keys"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the hotkey to the current foreground application."},
+ "pid":{"type":"integer","description":"Target process ID."},
+ "keys":{"type":"array","items":{"type":"string"},"minItems":2,
+ "description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
+ "y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
+ "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<HotkeyInput>("hotkey", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ if input.keys.len() < 2 {
+ return ToolResult::error("hotkey.keys must contain at least two keys.")
+ .with_structured(json!({ "code": "invalid_arguments" }));
+ }
+ let mods: Vec<String> = input
+ .keys
+ .iter()
+ .filter(|key| is_modifier(key))
+ .cloned()
+ .collect();
+ let Some(key) = input
+ .keys
+ .iter()
+ .rev()
+ .find(|key| !is_modifier(key))
+ .cloned()
+ else {
+ return ToolResult::error("keys must include at least one non-modifier key.");
+ };
+ let full_keys = input.keys;
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let key_display = full_keys.join("+");
+ return match tokio::task::spawn_blocking(move || {
+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Sent {key_display} to the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "keys": full_keys,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),
+ };
+ }
+ // Parse keys array (Swift's only path). Legacy key+modifiers shape
+ // still accepted as a Rust-side convenience.
+ let (key, mods, full_keys) = if let Some(arr) = args.get("keys") {
+ let raw = match arr.as_array() {
+ Some(r) => r,
+ None => return ToolResult::error("Missing required array field keys."),
+ };
+ let keys: Vec<String> = raw
+ .iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect();
+ if keys.len() != raw.len() || keys.is_empty() {
+ return ToolResult::error("keys must be a non-empty array of strings.");
+ }
+ let modifiers: Vec<String> = keys.iter().filter(|k| is_modifier(k)).cloned().collect();
+ let non_mods: Vec<String> = keys.iter().filter(|k| !is_modifier(k)).cloned().collect();
+ if non_mods.is_empty() {
+ return ToolResult::error("keys must include at least one non-modifier key.");
+ }
+ (non_mods.last().unwrap().clone(), modifiers, keys)
+ } else if let Some(k) = args.get("key").and_then(|v| v.as_str()) {
+ let m: Vec<String> = args
+ .get("modifiers")
+ .and_then(|v| v.as_array())
+ .map(|a| {
+ a.iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut full = m.clone();
+ full.push(k.to_owned());
+ (k.to_owned(), m, full)
+ } else {
+ return ToolResult::error("Missing required array field keys.");
+ };
+ let raw_pid = match args.require_i64("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let pid = raw_pid as u32;
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(resolved) => resolved,
+ Err(error) => return error,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let delivery = DeliveryMode::from_args(&args);
+
+ // Resolve HWND: use window_id if given, else pick first window for pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let pid2 = pid;
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid2))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // Match Swift's text format exactly: `"✅ Pressed K1+K2+... on pid X."`
+ // where K1+K2+... is `keys.joined(separator: "+")` of the FULL keys
+ // array as the caller supplied it (modifiers + non-modifier in order).
+ let key_display = full_keys.join("+");
+
+ // px form: pixel-click to focus, then the combo acts on the focused field
+ // (e.g. Ctrl+V into a Chromium input). A background pixel focus does not
+ // make Chromium's modifier PostMessage path reliable, so it still goes
+ // through the capability guard below; foreground remains explicit.
+ let px_focus = {
+ let px = args.get("x").and_then(|v| v.as_f64());
+ let py = args.get("y").and_then(|v| v.as_f64());
+ if let (Some(cx), Some(cy)) = (px, py) {
+ if elem_idx.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to hotkey, not both.",
+ );
+ }
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ hwnd_opt,
+ cx,
+ cy,
+ delivery.is_foreground(),
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ false,
+ )
+ .await
+ {
+ return e;
+ }
+ true
+ } else {
+ false
+ }
+ };
+
+ // The px form's focus click already moved the cursor. Otherwise place
+ // a named session's cursor so a keyboard-first session stays visible.
+ if !px_focus {
+ let element_center = admitted
+ .as_ref()
+ .map(|element| (element.center.0 as f64, element.center.1 as f64));
+ position_keyboard_cursor(&self.state, &args, hwnd, element_center).await;
+ }
+
+ if !px_focus && crate::input::is_xaml_host_hwnd(hwnd) {
+ let mut accelerator_keys = mods.clone();
+ accelerator_keys.push(key.clone());
+ let accelerator_combo = accelerator_keys.join("+");
+ let key_display_for_result = key_display.clone();
+ // UIA cross-process subtree walks can wedge on a bad provider —
+ // bound the call so a hung provider returns an error instead of
+ // blocking the daemon indefinitely. 4 s matches the budget
+ // `get_window_state.rs` uses for similar UIA scans (`UIA_SCREENSHOT_TIMEOUT`).
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::uia::windows_enum::try_invoke_accelerator_in_window(
+ hwnd as isize,
+ &accelerator_combo,
+ )
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok((true, _))) => ToolResult::text(format!(
+ "✅ Pressed {key_display_for_result} on pid {raw_pid} via UIA \
+ InvokePattern/TogglePattern (XAML / UWP target)."
+ )),
+ Ok(Ok((false, scanned))) => ToolResult::error(format!(
+ "hotkey on a modern XAML / UWP target (pid {raw_pid}, hwnd {hwnd}) \
+ could not find a UIA AcceleratorKey or `(Ctrl+X)`-style name hint \
+ matching `{key_display_for_result}` (scanned {scanned} element(s)). \
+ PostMessage WM_KEYDOWN/UP is ignored by this target's input pipeline. \
+ Common cause: the action is nested behind a closed menu (e.g. modern \
+ Notepad's Save under the File menu) — its element isn't in the visible \
+ subtree until the menu is opened. Call \
+ `get_window_state(pid={raw_pid}, window_id={hwnd})` to inspect available \
+ UIA actions, then use an exposed accelerator or `click` the matching \
+ element directly."
+ )),
+ Ok(Err(e)) => ToolResult::error(format!("hotkey (UIA path): {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Non-XAML target. Three routing decisions:
+ // 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus
+ // theft. Modifier state is encoded in the LPARAM bits, not
+ // in GetKeyState, so apps that read modifier state directly
+ // from the message (Chromium's `Browser::HandleKeyboardEvent`,
+ // Chromium renderer's input handler, every web app) see
+ // Ctrl+X correctly. Apps that consult `GetKeyState` via
+ // `TranslateAccelerator` (LibreOffice, FAR, classic Notepad)
+ // DON'T fire the accelerator — the modifier looks unset to
+ // them.
+ // 2. SendInput synthesized hotkey — pushes events onto the
+ // *system input queue*. Updates GetKeyState system-wide, so
+ // TranslateAccelerator-based apps see Ctrl+S as a real
+ // accelerator. Trade-off: brief foreground swap so SendInput
+ // lands on the right HWND. **This is the only path that
+ // violates the no-foreground contract** — we minimise its
+ // use to apps that genuinely require it.
+ //
+ // Routing:
+ // - No modifiers → PostMessage (no accelerator concern).
+ // - Modifiers + Chromium-family target → PostMessage. Chromium
+ // processes accelerators in `Browser::HandleKeyboardEvent`,
+ // which reads modifier state from the WM_KEYDOWN LPARAM
+ // directly (not via GetKeyState). Every Chromium browser
+ // (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app
+ // (Slack/VS Code/Discord/Teams/Notion) detects as Chromium
+ // via `is_chromium_target_window` and gets the no-foreground
+ // path.
+ // - Modifiers + non-Chromium Win32 → SendInput (the
+ // TranslateAccelerator path). The foreground swap stays.
+ let has_modifiers = !mods.is_empty();
+ // delivery_mode:"background" — refuse if PostMessage of the key combo
+ // would be silently dropped on this target (Chromium with modifiers).
+ let event_kind = if has_modifiers {
+ EventKind::KeyCombo
+ } else {
+ EventKind::Keystroke
+ };
+ if delivery == DeliveryMode::Background
+ && (px_focus || crate::input::delivery::would_be_silently_dropped(hwnd, event_kind))
+ {
+ // macOS-aligned: a `background` hotkey never fronts. This combo would
+ // be silently dropped (VCL/classic-Win32 TranslateAccelerator, or
+ // Chromium key-combos) and only a foreground/focus grab lands it —
+ // which background must not do. Surface background_unavailable; the
+ // agent escalates to delivery_mode:"foreground".
+ return crate::input::delivery::background_unavailable_error(hwnd, event_kind);
+ }
+ // delivery_mode:"foreground" — explicit SendInput swap (the path that
+ // unblocks TranslateAccelerator-style apps). A Background call that
+ // reaches here means the key combo is NOT silently dropped on this
+ // target (the drop-check above returned early otherwise), so it stays
+ // on PostMessage and the no-foreground contract holds.
+ // Foreground is an explicit request for system-queue delivery. This is
+ // still required after a PX focus click: PostMessage does not update
+ // global modifier state, so Chromium never observes Ctrl+Shift+H as a
+ // chord even though the renderer control is focused.
+ let use_send_input = delivery == DeliveryMode::Foreground;
+ let focus_target = elem_idx.map(|idx| {
+ let point = admitted.as_ref().map(|element| element.center);
+ (idx, point)
+ });
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ if use_send_input {
+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&admitted, hwnd, idx, point)?;
+ }
+ Ok(())
+ })
+ } else {
+ crate::input::post_key(hwnd, &key, &m)
+ }
+ }
+ })
+ .await;
+ let path = if use_send_input {
+ "SendInput"
+ } else {
+ "PostMessage"
+ };
+ match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Pressed {key_display} on pid {raw_pid} via {path} \
+ (Win32 target)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 11ff469021..62cefc5726 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -1,7 +1,6 @@
//! Real Windows tool implementations (compiled only on Windows).
use async_trait::async_trait;
-use cua_driver_core::action_record::ActionTransport;
/// Pin the agent-cursor overlay above `hwnd` in the z-order, normalising to
/// the **root** ancestor so the pin lands on the window that actually appears
@@ -14,7 +13,7 @@ use cua_driver_core::action_record::ActionTransport;
///
/// No-op when the overlay is disabled or `key` is empty (a direct platform
/// call without lifecycle metadata); the render thread drops the command.
-fn pin_overlay_above(key: &str, hwnd: u64) {
+pub(crate) fn pin_overlay_above(key: &str, hwnd: u64) {
use windows::Win32::Foundation::HWND;
use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT};
let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };
@@ -36,7 +35,7 @@ fn pin_overlay_above(key: &str, hwnd: u64) {
/// to the target window's centre, so the first keyboard action of a session
/// is visible. Anonymous calls keep their existing behavior. Geometry and
/// overlay failures are observational and never affect the input result.
-async fn position_keyboard_cursor(
+pub(crate) async fn position_keyboard_cursor(
state: &ToolState,
args: &Value,
hwnd: u64,
@@ -103,7 +102,7 @@ fn keyboard_window_center(hwnd: u64) -> Option<(f64, f64)> {
/// (wrapped in the UWP fg-steal bypass) and does not raise or activate the
/// window. MSAA-walked elements are skipped (no ScrollItemPattern) and keep the
/// existing failure.
-fn resolve_onscreen_point_with_scroll(
+pub(crate) fn resolve_onscreen_point_with_scroll(
admitted: &Option<crate::uia::snapshot::RetainedElement>,
hwnd: u64,
idx: usize,
@@ -188,7 +187,7 @@ fn resolve_onscreen_point_with_scroll(
})))
}
-fn minimized_element_refusal(
+pub(crate) fn minimized_element_refusal(
hwnd: u64,
idx: usize,
cx: i32,
@@ -216,17 +215,6 @@ fn minimized_element_refusal(
})
}
-fn tool_result_text(result: ToolResult) -> String {
- result
- .content
- .into_iter()
- .find_map(|content| match content {
- cua_driver_core::protocol::Content::Text { text, .. } => Some(text),
- _ => None,
- })
- .unwrap_or_else(|| "element action was refused".to_owned())
-}
-
/// Convert (px, py) — "window-local screenshot pixels, top-left origin
/// of the PNG returned by `get_window_state`" — to screen coordinates.
///
@@ -248,7 +236,7 @@ fn tool_result_text(result: ToolResult) -> String {
/// hook) we degrade to `GetWindowRect.top-left + (px, py)` — matches
/// the capture fallback which also keeps the full PrintWindow bitmap
/// without crop.
-fn bitmap_to_screen(hwnd: u64, px: i32, py: i32) -> (i32, i32) {
+pub(crate) fn bitmap_to_screen(hwnd: u64, px: i32, py: i32) -> (i32, i32) {
use windows::Win32::Foundation::{HWND, RECT};
use windows::Win32::Graphics::Dwm::{DwmGetWindowAttribute, DWMWA_EXTENDED_FRAME_BOUNDS};
use windows::Win32::UI::WindowsAndMessaging::GetWindowRect;
@@ -279,11 +267,6 @@ fn bitmap_to_screen(hwnd: u64, px: i32, py: i32) -> (i32, i32) {
}
}
-fn screen_to_bitmap(hwnd: u64, sx: i32, sy: i32) -> (i32, i32) {
- let (origin_x, origin_y) = bitmap_to_screen(hwnd, 0, 0);
- (sx - origin_x, sy - origin_y)
-}
-
/// Animate the agent cursor to (sx, sy) in screen coordinates and wait for the
/// glide to finish before returning. No-op when the overlay is not enabled.
///
@@ -300,13 +283,13 @@ fn screen_to_bitmap(hwnd: u64, sx: i32, sy: i32) -> (i32, i32) {
/// `key` is the session's cursor key (see [`resolve_cursor_key`]). An empty key
/// is possible only for a direct platform call without lifecycle metadata; in
/// that case every overlay operation short-circuits.
-async fn overlay_glide_to(key: &str, sx: f64, sy: f64) {
+pub(crate) async fn overlay_glide_to(key: &str, sx: f64, sy: f64) {
overlay_glide_to_target(key, sx, sy, None).await;
}
/// [`overlay_glide_to`] with the targeted element's screen rect
/// `[x, y, width, height]` (same space as `sx`/`sy`); `None` for pixel actions.
-async fn overlay_glide_to_target(key: &str, sx: f64, sy: f64, target: Option<[f64; 4]>) {
+pub(crate) async fn overlay_glide_to_target(key: &str, sx: f64, sy: f64, target: Option<[f64; 4]>) {
if key.is_empty() {
return;
}
@@ -355,7 +338,7 @@ async fn overlay_glide_to_target(key: &str, sx: f64, sy: f64, target: Option<[f6
/// `[x, y, width, height]`, in the same screen space as its cached `center`
/// (and so as the glide point). `(dx, dy)` shifts it when the glide point
/// moved off the cached center (scroll-into-view re-resolution).
-fn element_target_rect(
+pub(crate) fn element_target_rect(
element: Option<&crate::uia::snapshot::RetainedElement>,
(dx, dy): (i32, i32),
) -> Option<[f64; 4]> {
@@ -370,42 +353,9 @@ fn element_target_rect(
})
}
-async fn track_overlay_drag(
- key: String,
- from: (f64, f64),
- to: (f64, f64),
- duration_ms: u64,
- steps: usize,
-) {
- if key.is_empty() || !crate::overlay::is_enabled(&key) {
- return;
- }
- crate::overlay::send_command(
- key.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
- let steps = steps.max(1);
- let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
- for index in 0..=steps {
- let t = index as f64 / steps as f64;
- let x = from.0 + (to.0 - from.0) * t;
- let y = from.1 + (to.1 - from.1) * t;
- crate::overlay::send_command(key.clone(), cursor_overlay::track_pointer_command(x, y));
- if index < steps && !step_delay.is_zero() {
- tokio::time::sleep(step_delay).await;
- }
- }
- crate::overlay::send_command(key, cursor_overlay::OverlayCommand::SetPressed(false));
-}
-use cua_driver_contract::{
- ClickButton, DragInput, GetCursorPositionInput, GetDesktopStateInput, GetScreenSizeInput,
- HotkeyInput, InvokeMenuInput, MoveCursorInput, PressKeyInput, ScrollDirection, ScrollInput,
- TypeTextInput,
-};
use cua_driver_core::{
protocol::ToolResult,
tool::{Tool, ToolDef, ToolRegistry},
- tool_args::{parse_legacy_click_input, parse_typed_input, parse_typed_projection},
window_target::{PidOnlyWindowTargetGuard, WindowTargetCandidate, WindowTargetCandidates},
};
use serde_json::{json, Value};
@@ -413,7 +363,41 @@ use std::sync::{Arc, RwLock};
use crate::uia::Snapshots;
use cursor_overlay::CursorRegistry;
-use windows::core::Interface as _;
+
+use super::{
+ bring_to_front::BringToFrontTool,
+ check_permissions::CheckPermissionsTool,
+ click::ClickTool,
+ cursor_tools::{
+ GetAgentCursorStateV2Tool, SetAgentCursorEnabledV2Tool, SetAgentCursorMotionV2Tool,
+ SetAgentCursorThemeTool,
+ },
+ debug_window_info::DebugWindowInfoTool,
+ double_click::DoubleClickTool,
+ drag::DragTool,
+ get_accessibility_tree::GetAccessibilityTreeTool,
+ get_config::GetConfigTool,
+ get_cursor_position::GetCursorPositionTool,
+ get_desktop_state::GetDesktopStateTool,
+ get_screen_size::GetScreenSizeTool,
+ get_window_state::GetWindowStateTool,
+ hotkey::HotkeyTool,
+ invoke_menu::InvokeMenuTool,
+ kill_app::KillAppTool,
+ launch_app::LaunchAppTool,
+ list_apps::ListAppsTool,
+ list_windows::ListWindowsTool,
+ move_cursor::MoveCursorTool,
+ press_key::PressKeyTool,
+ right_click::RightClickTool,
+ scroll::ScrollTool,
+ set_config::SetConfigTool,
+ set_value::SetValueTool,
+ set_window_frame::SetWindowFrameTool,
+ type_text::TypeTextTool,
+ type_text_chars::TypeTextCharsTool,
+ zoom::ZoomTool,
+};
fn window_target_candidates_for_pid(
windows: impl IntoIterator<Item = crate::win32::WindowInfo>,
@@ -443,7 +427,7 @@ struct ExactPidWindowTargetGuard {
inner: Box<dyn Tool>,
}
-fn exact_window_ownership_result(
+pub(crate) fn exact_window_ownership_result(
pid: u32,
window_id: u64,
owner_pid: Option<u32>,
@@ -581,7 +565,7 @@ pub(crate) fn resolve_cursor_key(args: &Value) -> String {
/// Pure + arg-shape agnostic (works for both click and scroll args) so it's
/// unit-testable without Win32. When this returns `false` but `x,y` are present
/// with no pid/window_id, the caller returns a coordinate-scope error.
-fn is_windowless_desktop_action(args: &serde_json::Value) -> bool {
+pub(crate) fn is_windowless_desktop_action(args: &serde_json::Value) -> bool {
if args.get("scope").and_then(Value::as_str) != Some("desktop") {
return false;
}
@@ -640,7 +624,7 @@ pub struct ToolState {
pub snapshots: Arc<Snapshots>,
pub cursor_registry: Arc<CursorRegistry>,
pub config: Arc<RwLock<DriverConfig>>,
- capture_bridge: Option<Arc<crate::capture_admission::WindowsCaptureBridge>>,
+ pub(crate) capture_bridge: Option<Arc<crate::capture_admission::WindowsCaptureBridge>>,
}
impl ToolState {
@@ -656,7 +640,7 @@ impl ToolState {
})
}
- fn zoom_context(
+ pub(crate) fn zoom_context(
&self,
args: &Value,
pid: u32,
@@ -670,7 +654,7 @@ impl ToolState {
}
}
-fn screenshot_scale(
+pub(crate) fn screenshot_scale(
state: &ToolState,
args: &Value,
pid: u32,
@@ -681,1192 +665,6 @@ fn screenshot_scale(
.screenshot_scale(pid as i32, window_id, args)
}
-fn capture_admission_refusal(error: anyhow::Error) -> ToolResult {
- let code = cua_driver_core::capture_runtime::admission_error_code(&error);
- ToolResult::error(format!("Capture-bound click refused: {error}")).with_structured(json!({
- "code": code,
- "effect": "refused",
- "detail": error.to_string(),
- }))
-}
-
-// ── list_apps ────────────────────────────────────────────────────────────────
-
-pub struct ListAppsTool;
-
-static LIST_APPS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ListAppsTool {
- fn def(&self) -> &ToolDef {
- LIST_APPS_DEF.get_or_init(|| ToolDef {
- name: "list_apps".into(),
- description:
- "List Windows apps — both currently running and installed-but-not-running — \
- with per-app state flags:\n\n\
- - running: is a process for this app live? (pid is 0 when false)\n\
- - active: is it the system-frontmost app? (implies running)\n\
- - kind: `\"desktop\"` for `.exe`-backed apps (resolved from Start-Menu \
- shortcuts) or `\"uwp\"` for packaged store apps.\n\
- - launch_path: what `launch_app` would consume.\n\
- * desktop: full `.exe` commandline (path + any arguments preserved from \
- the source `.lnk`; the exe is quoted when it contains whitespace).\n\
- * uwp: `shell:appsFolder\\{PackageFamilyName}!{AppId}` where `{AppId}` \
- falls back to `App` when the package manifest does not expose a \
- specific Application.Id.\n\
- - last_used: RFC3339 mtime of the launcher (`.lnk` for desktop, \
- package install location for UWP), when readable.\n\n\
- Running apps are derived from visible top-level windows + the foreground \
- pid (mirrors NSApplicationActivationPolicyRegular's intent: background \
- services and console processes are excluded). Installed apps are merged \
- from Start-Menu `.lnk` enumeration and the WinRT PackageManager — running \
- processes whose executable matches a Start-Menu target are folded into a \
- single entry with `running: true`.\n\n\
- Use this for \"is X installed?\" as well as \"is X running?\". For per-window \
- state — on-screen, minimized, window titles — call list_windows instead. \
- For just opening an app — running or not — call launch_app({path: ...}) \
- directly; list_apps is not a prerequisite."
- .into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
-
- async fn invoke(&self, _args: Value) -> ToolResult {
- // Strategy:
- // 1. Enumerate top-level visible windows → derive the set of running pids
- // that have a user-facing surface.
- // 2. Enumerate the process table to map pid → executable name and full
- // path (used to merge against Start-Menu / UWP launch_paths).
- // 3. Enumerate installed apps from Start-Menu shortcuts + WinRT
- // PackageManager. Each installed entry has a launch_path.
- // 4. Merge: a running pid whose executable path matches an installed
- // desktop launch_path becomes a single entry with running=true and
- // that launch_path. Remaining installed entries are emitted with
- // running=false, pid=0. Remaining running pids (no installed-app
- // match) are emitted as running=true with launch_path=null.
- let apps = tokio::task::spawn_blocking(|| -> Vec<serde_json::Value> {
- use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
- use windows::Win32::UI::WindowsAndMessaging::GetWindowThreadProcessId;
-
- // Per-step timings logged via `tracing::debug!` (silent at default
- // log level — enable with `RUST_LOG=cua_driver=debug` when debugging
- // a slow list_apps invocation).
- let t0 = std::time::Instant::now();
- let wins = crate::win32::list_windows_via_win32(None);
- tracing::debug!(target: "list_apps", "step 1 list_windows: {} wins ({}ms)", wins.len(), t0.elapsed().as_millis());
- let mut seen_pids = std::collections::HashSet::new();
- let mut running_pids: Vec<u32> = Vec::new();
- for w in &wins {
- if seen_pids.insert(w.pid) { running_pids.push(w.pid); }
- }
-
- let t1 = std::time::Instant::now();
- let foreground_pid = unsafe {
- let fg = GetForegroundWindow();
- let mut pid = 0u32;
- let _ = GetWindowThreadProcessId(fg, Some(&mut pid));
- pid
- };
- tracing::debug!(target: "list_apps", "step 2 fg-window: pid={} ({}ms)", foreground_pid, t1.elapsed().as_millis());
-
- let t2 = std::time::Instant::now();
- let procs = crate::win32::list_processes();
- tracing::debug!(target: "list_apps", "step 3 list_processes: {} procs ({}ms)", procs.len(), t2.elapsed().as_millis());
- let pid_to_name: std::collections::HashMap<u32, String> =
- procs.iter().map(|p| (p.pid, p.name.clone())).collect();
-
- let t3 = std::time::Instant::now();
- let installed = crate::win32::list_installed_apps();
- tracing::debug!(target: "list_apps", "step 4 list_installed_apps: {} installed ({}ms)", installed.len(), t3.elapsed().as_millis());
- // Lowercase-exe-basename → installed-app indices. We match running
- // processes by basename rather than full path because the process
- // table's executable name is `notepad.exe` while the Start-Menu
- // shortcut target is `C:\Windows\System32\notepad.exe` — both
- // sides need to be normalised to compare. Multiple Start-Menu
- // shortcuts often target the same basename (different `chrome.exe`
- // launchers, multiple `code.exe` profiles, …), so we bucket as
- // `Vec<usize>` and let `disambiguate_installed_match` pick a
- // winner per running pid rather than letting the last write win.
- let mut by_exe: std::collections::HashMap<String, Vec<usize>> =
- std::collections::HashMap::new();
- for (i, app) in installed.iter().enumerate() {
- if app.kind == "desktop" {
- let basename = std::path::Path::new(&app.launch_path)
- .file_name()
- .and_then(|s| s.to_str())
- .unwrap_or("")
- .to_ascii_lowercase();
- if !basename.is_empty() {
- by_exe.entry(basename).or_default().push(i);
- }
- }
- }
-
- let mut consumed_installed: std::collections::HashSet<usize> =
- std::collections::HashSet::new();
- let mut out: Vec<serde_json::Value> = Vec::new();
-
- for &pid in &running_pids {
- let name = pid_to_name.get(&pid).cloned().unwrap_or_else(|| "<unknown>".into());
- let key = name.to_ascii_lowercase();
- let candidates = by_exe.get(&key).map(|v| v.as_slice()).unwrap_or(&[]);
- let merged = disambiguate_installed_match(candidates, &installed, &name);
- if let Some(idx) = merged { consumed_installed.insert(idx); }
- let (display_name, bundle_id, launch_path, kind, last_used) = match merged {
- Some(idx) => {
- let a = &installed[idx];
- (
- a.name.clone(),
- Some(a.bundle_id.clone()),
- Some(a.launch_path.clone()),
- Some(a.kind.clone()),
- a.last_used.clone(),
- )
- }
- None => (name, None, None, Some("desktop".to_owned()), None),
- };
- out.push(json!({
- "pid": pid,
- "bundle_id": bundle_id,
- "name": display_name,
- "running": true,
- "active": pid == foreground_pid,
- "kind": kind,
- "launch_path": launch_path,
- "last_used": last_used,
- "windows": Vec::<serde_json::Value>::new(),
- }));
- }
- for (i, app) in installed.iter().enumerate() {
- if consumed_installed.contains(&i) { continue }
- out.push(json!({
- "pid": 0,
- "bundle_id": app.bundle_id.clone(),
- "name": app.name.clone(),
- "running": false,
- "active": false,
- "kind": app.kind.clone(),
- "launch_path": app.launch_path.clone(),
- "last_used": app.last_used.clone(),
- "windows": Vec::<serde_json::Value>::new(),
- }));
- }
- out
- }).await.unwrap_or_default();
-
- let running_count = apps
- .iter()
- .filter(|a| a["running"].as_bool().unwrap_or(false))
- .count();
- let total = apps.len();
- let installed_only = total - running_count;
- let mut lines = vec![format!(
- "✅ Found {total} app(s): {running_count} running, {installed_only} installed-not-running."
- )];
- for app in apps
- .iter()
- .filter(|a| a["running"].as_bool().unwrap_or(false))
- {
- let name = app["name"].as_str().unwrap_or("?");
- let pid = app["pid"].as_u64().unwrap_or(0);
- lines.push(format!("- {name} (pid {pid})"));
- }
- // `apps` is the unified shape; `processes` retained for any pre-existing
- // callers that consumed the old running-only shape.
- let structured = json!({
- "apps": apps,
- "processes": apps.iter().filter(|a| a["running"].as_bool().unwrap_or(false))
- .map(|a| json!({
- "pid": a["pid"], "name": a["name"]
- })).collect::<Vec<_>>(),
- });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
-}
-
-/// Pick which of several Start-Menu / UWP-derived installed apps best matches
-/// a running pid whose exe basename collided. Precedence:
-/// 1. exact case-insensitive equality between `proc_exe_basename` and the
-/// installed app's launch_path basename (always true here; left for
-/// future per-path matches if we ever bucket on something coarser)
-/// 2. most recently modified launcher (`last_used` desc — RFC3339 strings
-/// are lexicographically ordered the same as chronologically)
-/// 3. first candidate (deterministic by source order)
-fn disambiguate_installed_match(
- candidates: &[usize],
- installed: &[crate::win32::InstalledApp],
- _proc_exe_basename: &str,
-) -> Option<usize> {
- if candidates.is_empty() {
- return None;
- }
- if candidates.len() == 1 {
- return Some(candidates[0]);
- }
-
- candidates
- .iter()
- .copied()
- .max_by(|&a, &b| {
- installed[a]
- .last_used
- .as_deref()
- .unwrap_or("")
- .cmp(installed[b].last_used.as_deref().unwrap_or(""))
- })
- .or_else(|| candidates.first().copied())
-}
-
-// ── list_windows ─────────────────────────────────────────────────────────────
-
-pub struct ListWindowsTool;
-
-static LIST_WINDOWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ListWindowsTool {
- fn def(&self) -> &ToolDef {
- LIST_WINDOWS_DEF.get_or_init(|| ToolDef {
- name: "list_windows".into(),
- // Description matches Swift `ListWindowsTool.swift` semantics —
- // Windows-specific caveat: no Spaces concept, so on_current_space
- // / space_ids are always omitted and current_space_id is null.
- description: "List every top-level window currently known to the window manager. \
- Each record self-contains its owning app identity so the caller never has to \
- join back against list_apps.\n\n\
- Use this — not list_apps — for any window-level reasoning: \"does this app \
- have a visible window right now?\", \"which of this pid's windows is the main \
- one?\".\n\n\
- Per-record fields: window_id (HWND), pid + app_name, title, \
- bounds {x, y, width, height}, layer (always 0), z_index (integer or null; higher \
- values are closer to the front; null means stacking order is unavailable and \
- callers must not infer one), is_on_screen, minimized. To select a frontmost \
- candidate, take the maximum integer z_index; if every value is null, use an \
- explicit fallback instead of relying on array order. The macOS-specific \
- on_current_space / space_ids fields are \
- omitted on Windows; current_space_id is null.\n\n\
- Inputs: pid (optional pid filter), app (app-name filter), on_screen_only (bool, \
- default false). Filter with app or pid: an unfiltered call lists every window. \
- The driver's own overlay is omitted.".into(),
- input_schema: json!({"type":"object","properties":{
- "pid":{"type":"integer","description":"Optional pid filter. When set, only this pid's windows are returned."},
- "app":{"type":"string","description":cua_driver_core::tool_args::LIST_WINDOWS_APP_FILTER_DESCRIPTION},
- "on_screen_only":{"type":"boolean","description":"When true, drop windows that aren't currently on-screen. Default false."}
- },"additionalProperties":false}),
- read_only: true, destructive: false, idempotent: true, open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let filter_pid = args.opt_u64("pid").map(|v| v as u32);
- let on_screen_only = args.bool_or("on_screen_only", false);
- let (mut windows, pid_to_name) = tokio::task::spawn_blocking(move || {
- let wins = crate::win32::list_windows(filter_pid);
- let procs = crate::win32::list_processes();
- let map: std::collections::HashMap<u32, String> =
- procs.into_iter().map(|p| (p.pid, p.name)).collect();
- (wins, map)
- })
- .await
- .unwrap_or_default();
- if on_screen_only {
- windows.retain(|w| w.is_on_screen);
- }
- drop_own_windows(&mut windows, std::process::id());
- if let Some(filter) = args.opt_str("app") {
- windows.retain(|w| {
- let app_name = pid_to_name.get(&w.pid).map(String::as_str).unwrap_or("");
- cua_driver_core::tool_args::app_filter_matches(&filter, app_name, None)
- });
- }
-
- // Swift surfaces a warning when a pid filter matches nothing.
- let missing_pid_warning = if let Some(fp) = filter_pid {
- if windows.is_empty() {
- use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
- use windows::Win32::UI::WindowsAndMessaging::GetWindowThreadProcessId;
- let fg_pid = unsafe {
- let fg = GetForegroundWindow();
- let mut p = 0u32;
- let _ = GetWindowThreadProcessId(fg, Some(&mut p));
- p
- };
- let fg_name = pid_to_name.get(&fg_pid).map(|s| s.as_str()).unwrap_or("?");
- Some(format!(
- "⚠️ No windows found for pid {fp}. The pid may be wrong or the app may not \
- have created a window yet. The current frontmost app appears to be \
- \"{fg_name}\" (pid {fg_pid})."
- ))
- } else {
- None
- }
- } else {
- None
- };
-
- // z_index: list_windows merges EnumWindows first (which the Win32
- // window manager returns in canonical top-to-bottom z-order), then
- // appends any UIA-only HWNDs UIA found that EnumWindows missed
- // (UIA-only entries land at the bottom of the z-stack — they have
- // no canonical Win32 ordering). Higher list index = farther from
- // front. Swift convention: higher z_index = closer to front.
- // Invert via `(len - 1 - i)` so the front-most window gets the
- // largest z.
- let n = windows.len();
- let records: Vec<serde_json::Value> = windows
- .iter()
- .enumerate()
- .map(|(i, w)| {
- let app_name = pid_to_name.get(&w.pid).cloned().unwrap_or_default();
- let z_index = z_index_from_front_to_back(n, i) as i64;
- json!({
- "window_id": w.hwnd,
- "pid": w.pid,
- "app_name": app_name,
- "title": w.title,
- "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
- "layer": 0,
- "z_index": z_index,
- "is_on_screen": w.is_on_screen,
- "minimized": w.minimized,
- })
- })
- .collect();
-
- let total = records.len();
- let pid_count = windows
- .iter()
- .map(|w| w.pid)
- .collect::<std::collections::HashSet<_>>()
- .len();
- let on_screen = records
- .iter()
- .filter(|r| r["is_on_screen"].as_bool().unwrap_or(false))
- .count();
- // Swift text format 1:1 — append the SPI-unavailable note since
- // Windows has no equivalent of macOS Spaces.
- let header = format!(
- "✅ Found {total} window(s) across {pid_count} app(s); {on_screen} on-screen. \
- (SkyLight Space SPIs unavailable — on_current_space / space_ids omitted.)"
- );
- let mut lines = vec![header];
- if let Some(warning) = missing_pid_warning {
- lines.push(warning);
- }
- for r in &records {
- let app = r["app_name"].as_str().unwrap_or("?");
- let pid = r["pid"].as_u64().unwrap_or(0);
- let tt = r["title"].as_str().unwrap_or("");
- let wid = r["window_id"].as_u64().unwrap_or(0);
- let title_disp = if tt.is_empty() {
- "(no title)".to_owned()
- } else {
- format!("\"{tt}\"")
- };
- let tag = if r["is_on_screen"].as_bool().unwrap_or(false) {
- ""
- } else {
- " [off-screen]"
- };
- lines.push(format!(
- "- {app} (pid {pid}) {title_disp} [window_id: {wid}]{tag}"
- ));
- }
-
- // Legacy alias: keep the old flat fields under each record for any
- // pre-existing Rust callers, but they read from the same source.
- let legacy_windows: Vec<serde_json::Value> = windows
- .iter()
- .map(|w| {
- json!({
- "window_id": w.hwnd, "pid": w.pid, "title": w.title,
- "x": w.x, "y": w.y, "width": w.width, "height": w.height,
- "is_on_screen": w.is_on_screen, "minimized": w.minimized,
- })
- })
- .collect();
- let structured = json!({
- "windows": records,
- "current_space_id": serde_json::Value::Null, // Windows has no Spaces
- "_legacy_windows": legacy_windows,
- });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
-}
-
-use cua_driver_core::window_target::z_index_from_front_to_back;
-
-fn drop_own_windows(windows: &mut Vec<crate::win32::WindowInfo>, own_pid: u32) {
- windows.retain(|w| w.pid != own_pid);
-}
-
-#[cfg(test)]
-mod list_windows_z_index_tests;
-
-#[cfg(test)]
-mod get_window_state_actions_tests;
-
-// ── get_window_state ─────────────────────────────────────────────────────────
-
-/// Build a single structured element entry for `get_window_state`.
-/// Returns `None` when the node has no `element_index` (non-actionable rows).
-fn build_element_entry(
- n: &crate::uia::UiaNode,
- snapshot_id: Option<u32>,
-) -> Option<serde_json::Value> {
- let idx = n.element_index?;
- // `label`: name → value → automation_id → help_text.
- let label = n
- .name
- .clone()
- .or_else(|| n.value.clone())
- .or_else(|| n.automation_id.clone())
- .or_else(|| n.help_text.clone());
- let mut entry = json!({
- "element_index": idx,
- "role": n.control_type,
- "depth": n.depth,
- });
- if let Some(snapshot_id) = snapshot_id {
- entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx));
- }
- if n.in_web_content {
- entry["in_web_content"] = json!(true);
- }
- if let Some(label) = label {
- entry["label"] = json!(label);
- }
- // Surface the element's value separately from `label` (which collapses
- // name→value→automation_id→help): a control with both a name AND text
- // (a ValuePattern edit holding typed content) would otherwise hide the
- // text from a caller reading the structured side. See the macOS
- // get_window_state builder for the rationale.
- if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) {
- entry["value"] = json!(value);
- }
- if let Some(enabled) = n.enabled {
- entry["enabled"] = json!(enabled);
- }
- if let Some(selected) = n.selected {
- entry["selected"] = json!(selected);
- }
- if !n.actions.is_empty() {
- entry["actions"] = json!(n.actions);
- }
- if let Some(parent) = n.parent_element_index {
- entry["parent_index"] = json!(parent);
- }
- if let Some((l, t, r, b)) = n.rect {
- entry["frame"] = json!({
- "x": l,
- "y": t,
- "w": (r - l).max(0),
- "h": (b - t).max(0),
- });
- }
- Some(entry)
-}
-
-pub struct GetWindowStateTool {
- state: Arc<ToolState>,
-}
-
-static GWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-/// Slack past `timeout_ms` before the walk task is abandoned.
-const UIA_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_millis(500);
-/// Deadline for the window capture, separate from the walk budget.
-const UIA_SCREENSHOT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(4);
-
-#[async_trait]
-impl Tool for GetWindowStateTool {
- fn def(&self) -> &ToolDef {
- GWS_DEF.get_or_init(|| ToolDef {
- name: "get_window_state".into(),
- // Description ported from Swift `GetWindowStateTool.swift` with
- // Windows-specific adaptations (UIA instead of AX; no SkyLight
- // Space validation; no per-call `javascript` field — that's a
- // macOS AppleScript hook).
- description: "Walk a running app's UIA tree and return it ONCE, as compact \
- Markdown by default (`tree_format:\"markdown\"`). Every actionable element is \
- tagged `[N]` (an element_index); its element_token is `<snapshot_id>:N`, where \
- `snapshot_id` is printed in the response header and returned in structuredContent. \
- Pass `tree_format:\"elements\"` for the structured `elements` array instead \
- (explicit `element_token`, `frame`, `parent_index`, `value`, `actions` per row), \
- or `\"both\"` for both (about twice the size).\n\n\
- Size: a read is bounded by default (`max_elements` 250). When the walk stops at \
- that budget the response says `Tree truncated at max_elements=…` and `truncated: \
- true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. \
- `_note` is omitted unless `verbose:true`. `full_output:true` restores the \
- previous full response: both representations, all metadata, and the platform's \
- walk limits (≤5 000 elements, depth ≤25).\n\n\
- DIFF READS: pass `since: <snapshot_id>` from an earlier read of the same window \
- to get only what changed: `+` added rows, `~` changed rows, `-` removed rows \
- (their ids are the old snapshot's), or `no change since …`. Rows not listed keep \
- their previous `[N]` unless a `reindexed:` line says otherwise. The response \
- carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, \
- other-window or differently-scoped (query/max_elements/max_depth) `since` falls \
- back to a full read and `since_status` says why. The focused element is not \
- reported on Windows yet.\n\n\
- INVARIANT: call `get_window_state` once per turn per (pid, window_id) before \
- any element action against that window. The next snapshot of the same \
- (pid, window_id) replaces this one, stales its element tokens, and lists the \
- replaced ids in `invalidated_snapshot_ids`. A screenshot-only read \
- (`include_accessibility_tree:false`) keeps the current snapshot's rows and \
- tokens valid, and `display_only:true` also leaves its pixel frame alone.\n\n\
- The UIA tree walked is the window's tree (HWND-scoped); the screenshot and \
- window bounds reported come from the same `window_id`. This is the source of \
- truth for which window the caller intends to reason about — the driver never \
- picks a window implicitly.\n\n\
- `window_id` MUST belong to `pid`; the call returns `isError: true` otherwise. \
- The driver does not auto-fall-back to a different window.\n\n\
- Set `query` to a case-insensitive substring to project the tree to matching \
- rows plus their ancestor chain. \
- Original element indices are preserved. `total_element_count` reports the \
- unprojected walked snapshot; `returned_element_count` reports the projection. \
- `elements_complete` describes the underlying walk and is unaffected by query \
- projection.\n\n\
- Always returns BOTH the element tree AND a screenshot — ground on both \
- and cross-check (the tree lies on some surfaces). Choose the modality at \
- ACTION time: an element ax action (element_token → \
- accessibility rung) or an element px action (x,y → pixel rung off this \
- screenshot). capture_mode is deprecated and ignored.\n\n\
- The mirror image: pass `include_accessibility_tree:false` to SKIP the \
- UIA walk entirely and return just the screenshot plus window metadata \
- (window_bounds, app_name, window_title) — the capture-only path for a \
- live window preview / picture-in-picture. Setting BOTH \
- `include_accessibility_tree:false` and `include_screenshot:false` is an \
- error. Optional `max_image_dimension` overrides the configured screenshot \
- long-edge limit for this call; pass 0 for native resolution. Legacy \
- `max_dimension` remains a cap on the configured limit.\n\n\
- Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a \
- single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of \
- timing out at 4s with per-property RPCs).\n\n\
- Optional `max_elements` / `max_depth` bound the UIA walk (Electron / large web \
- apps produce 10k+ element trees). Markdown and structured elements are \
- truncated identically. `elements_complete` is false when a bound skips a node \
- or subtree; ending exactly at a bound remains complete. Default: \
- `max_elements` 250, depth ≤25.\n\n\
- CHROMIUM COVERAGE: a browser-owned permission bubble can be \
- composited outside the requested native window. Chromium-family \
- snapshots therefore describe this limit in structuredContent.capture_coverage. \
- After a verified ineffective window action, call escalate_session, take a \
- fresh get_desktop_state snapshot, act explicitly in desktop scope if needed, \
- then verify with another fresh desktop snapshot. \
- This is separate from page JavaScript dialogs, which remain on \
- browser_dialog.\n\n\
- Windows requires no special permissions.".into(),
- input_schema: cua_driver_core::window_state_view::extend_input_schema(json!({"type":"object","required":["pid","window_id"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Process ID from `list_apps`."},
- "window_id":{"type":"integer","description":"HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array."},
- "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
- "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
- "include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
- "screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."},
- "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count; elements_complete continues to describe the underlying unprojected walk."},
- "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window."},
- "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},
- "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),
- "max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default."},
- "max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins."}
- },"additionalProperties":false})),
- // Swift annotation: idempotent: false (each call is a fresh snapshot).
- read_only: true, destructive: false, idempotent: false, open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let display_only = match cua_driver_core::window_state_view::display_only(&args) {
- Ok(display_only) => display_only,
- Err(refusal) => return refusal,
- };
- // Swift error wording 1:1.
- let pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(v) => v as u32,
- None => return ToolResult::error("Missing required integer field pid."),
- };
- let hwnd =
- match args.get("window_id").and_then(|v| v.as_u64()) {
- Some(v) => v,
- None => return ToolResult::error(
- "Missing required integer field window_id. Use `list_windows` to enumerate \
- the target app's windows, or read `launch_app`'s `windows` array.",
- ),
- };
- // Validate window belongs to pid — Swift's hard error. The exact HWND is
- // probed through Win32 first; the desktop-wide UIA union (2 s deadline)
- // is consulted only on a miss (#4416).
- use crate::win32::PidWindowLookup;
- let lookup =
- tokio::task::spawn_blocking(move || crate::win32::lookup_window_for_pid(pid, hwnd))
- .await
- .unwrap_or(PidWindowLookup::Missing);
- let window = match lookup {
- PidWindowLookup::Found(window) => window,
- PidWindowLookup::OtherPid(owner) => {
- return ToolResult::error(format!(
- "window_id {hwnd} belongs to pid {owner}, not pid {pid}. Call \
- `list_windows({{\"pid\": {pid}}})` to get this pid's own windows."
- ));
- }
- PidWindowLookup::Missing => {
- return ToolResult::error(format!(
- "No window with window_id {hwnd} exists. Call `list_windows({{\"pid\": \
- {pid}}})` for candidates."
- ));
- }
- };
- // Window identity metadata (additive): title + on-screen rectangle from
- // the lookup we already did, plus the owning process's executable
- // name. Names the surface on the capture-only path, where no UIA tree
- // identifies it.
- let win_geom = Some((
- window.title.clone(),
- window.x,
- window.y,
- window.width,
- window.height,
- ));
- let app_name = tokio::task::spawn_blocking(move || {
- crate::win32::list_processes()
- .into_iter()
- .find(|p| p.pid == pid)
- .map(|p| p.name)
- })
- .await
- .ok()
- .flatten();
- use cua_driver_core::tool_args::ArgsExt;
- // The canonical per-call value overrides all configured limits,
- // including `0` for native resolution. The legacy field retains its
- // previous cap-folding behavior when the canonical field is omitted.
- let max_image_dimension = match args.get("max_image_dimension") {
- Some(value) => match value.as_u64().and_then(|v| u32::try_from(v).ok()) {
- Some(value) => Some(value),
- None => {
- return ToolResult::error(
- "max_image_dimension must be an integer between 0 and 4294967295.",
- )
- }
- },
- None => None,
- };
- let max_dimension = args
- .get("max_dimension")
- .and_then(|v| v.as_u64())
- .and_then(|v| u32::try_from(v).ok())
- .map(|v| v.max(1));
- let max_dim = {
- let cfg = self.state.config.read().unwrap();
- cua_driver_core::image_utils::ImageDimensionLimits {
- configured: cfg.max_image_dimension,
- legacy_max_dimension: max_dimension,
- max_image_dimension,
- }
- .resolve()
- };
- // `capture_mode` is DEPRECATED and ignored — get_window_state always
- // returns BOTH the UIA tree and a screenshot now, so the agent grounds on
- // both and cross-checks (the UIA tree lies often enough that a grounding
- // screenshot should always be present). The modality is chosen at action
- // time: an element ax action (element_index) or an element px action (x,y).
- // We don't read the arg; it stays in the schema only so old callers don't
- // trip additionalProperties:false.
- let view = match cua_driver_core::window_state_view::ViewOptions::from_args(&args) {
- Ok(view) => view,
- Err(refusal) => return refusal,
- };
- let query = args.opt_str("query");
- let session_id = args.opt_str("_session_id");
- let screenshot_out_file = args.opt_str("screenshot_out_file");
- // `display_only` is the public form of the internal observation-only
- // read: pixels for a preview, with no snapshot or capture change.
- let observation_only = display_only
- || args
- .get("_observation_only")
- .and_then(|value| value.as_bool())
- == Some(true);
- // Optional caps — when omitted, fall back to the walker's built-in
- // defaults (#22865). minimum:1 enforced in the schema, but defend
- // against 0 here too.
- let max_elements = view.max_elements(
- &args,
- crate::uia::DEFAULT_MAX_TOTAL_ELEMENTS,
- observation_only,
- );
- let max_depth = args
- .get("max_depth")
- .and_then(|v| v.as_u64())
- .map(|v| v.max(1) as usize)
- .unwrap_or(crate::uia::DEFAULT_MAX_DEPTH);
- let timeout_ms = cua_driver_core::tool_schema::resolve_timeout_ms(args.get("timeout_ms"));
-
- // Always walk the UIA tree. The screenshot is returned by DEFAULT — the
- // grounding frame the agent cross-checks the (sometimes-lying) tree
- // against — but `include_screenshot:false` opts out of the grab entirely
- // (tree only: the cheap path for re-indexing before an element ax action).
- // `screenshot_out_file` still forces a capture to disk regardless. With
- // `screenshot_out_file` the bytes go to disk and the path is surfaced
- // instead of embedding base64; otherwise the base64 PNG is embedded.
- let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
- // `include_accessibility_tree` (default true) mirrors include_screenshot:
- // set false to SKIP the UIA walk and return just the screenshot + window
- // metadata (the capture-only / preview path).
- let do_tree = args
- .get("include_accessibility_tree")
- .and_then(|v| v.as_bool())
- != Some(false);
- let do_shot = include_screenshot != Some(false) || screenshot_out_file.is_some();
- if !do_tree && !do_shot {
- return ToolResult::error(
- "Nothing to return: both include_accessibility_tree:false and \
- include_screenshot:false. Set at least one to true, or pass \
- screenshot_out_file to force a capture.",
- );
- }
-
- let state = self.state.clone();
- let q = query.clone();
- let out_file = screenshot_out_file.clone();
- // The tree walk runs under `timeout_ms`. UIA's bulk BuildUpdatedCache
- // cannot be interrupted (Chrome's provider can block indefinitely on
- // property reads), so when the walk task itself outlives the budget
- // the response carries an empty tree flagged `truncated` instead of an
- // error, and the screenshot below still comes back.
- let tree_result = if do_tree {
- let tree_task = tokio::task::spawn_blocking(move || {
- let mut budget =
- cua_driver_core::walk_budget::WalkBudget::new(timeout_ms, max_elements);
- let iconic_before = crate::input::window_is_iconic(hwnd);
- let tree =
- crate::uia::walk_tree_budgeted(hwnd, q.as_deref(), max_depth, &mut budget);
- let kind = if tree.nodes.iter().any(|node| node.msaa_role.is_some()) {
- crate::uia::snapshot::SnapshotKind::Msaa
- } else {
- crate::uia::snapshot::SnapshotKind::Uia
- };
- let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(&tree.nodes, kind)
- .with_read_while_minimized(
- iconic_before || crate::input::window_is_iconic(hwnd),
- );
- (tree, payload, budget.outcome())
- });
- let started = std::time::Instant::now();
- let deadline = std::time::Duration::from_millis(timeout_ms) + UIA_WALK_BACKSTOP_GRACE;
- match tokio::time::timeout(deadline, tree_task).await {
- Ok(Ok(tree)) => Some(tree),
- Ok(Err(e)) => return ToolResult::error(format!("UIA walk task panicked: {e}")),
- Err(_elapsed) => {
- let class = crate::input::delivery::read_class_name(hwnd);
- let walk = cua_driver_core::walk_budget::WalkOutcome::timed_out(
- timeout_ms,
- started.elapsed(),
- );
- let tree = crate::uia::UiaTreeResult {
- tree_markdown: format!(
- "(The UIA provider for hwnd 0x{hwnd:x}, class '{class}', did not \
- answer. Act by pixel off the screenshot, or send press_key with \
- delivery_mode:\"foreground\" for a dialog's default accelerator.)"
- ),
- nodes: Vec::new(),
- elements_complete: false,
- };
- let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
- &[],
- crate::uia::snapshot::SnapshotKind::Uia,
- );
- Some((tree, payload, walk))
- }
- }
- } else {
- None
- };
- let blocking = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
- // Capture screenshot AND any error message so the response can
- // surface *why* there's no image (the iconic-window guard from
- // #1973 / PR #1974 is the load-bearing case: minimized windows
- // legitimately can't be captured, and the caller needs to know
- // to call `bring_to_front` instead of retrying).
- // The previous `Err(_) => None` silently dropped the error and
- // upstream agents saw an empty response with no signal.
- let (screenshot, screenshot_err) = if do_shot {
- match crate::capture::screenshot_window_bytes(hwnd) {
- Ok(raw) => {
- let (native_w, native_h) = crate::capture::png_dimensions_pub(&raw)?;
- let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
- let (w, h) = crate::capture::png_dimensions_pub(&png)?;
- // `screenshot_out_file` set (any mode) → write to disk and
- // surface the path, never embed bytes. Otherwise (vision,
- // no out_file) → embed base64.
- if let Some(ref path) = out_file {
- std::fs::write(path, &png)?;
- (
- Some((None, Some(path.clone()), w, h, native_w, native_h, png)),
- None,
- )
- } else {
- use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
- (
- Some((Some(B64.encode(&png)), None, w, h, native_w, native_h, png)),
- None,
- )
- }
- }
- Err(e) => (None, Some(format!("{e}"))),
- }
- } else {
- (None, None)
- };
- Ok((screenshot, screenshot_err))
- });
- let result: anyhow::Result<_> =
- match tokio::time::timeout(UIA_SCREENSHOT_TIMEOUT, blocking).await {
- Ok(join_result) => join_result
- .map_err(|e| anyhow::anyhow!("task panic: {e}"))
- .and_then(|r| r),
- Err(_elapsed) => Ok((
- None,
- Some(format!(
- "window capture did not finish within {}s",
- UIA_SCREENSHOT_TIMEOUT.as_secs()
- )),
- )),
- };
- let result =
- result.map(|(screenshot, screenshot_err)| (tree_result, screenshot, screenshot_err));
-
- match result {
- Ok((tree_opt, screenshot_opt, screenshot_err)) => {
- let mut content = Vec::new();
- let mut structured = json!({ "window_id": hwnd, "pid": pid });
- let screenshot_scale = screenshot_opt
- .as_ref()
- .map(|(_, _, w, _, native_w, _, _)| *native_w as f64 / *w as f64);
- let mut published_snapshot = false;
- let mut invalidated = Vec::new();
-
- if let Some((tr, payload, walk)) = tree_opt {
- let is_msaa = tr.nodes.iter().any(|node| node.msaa_role.is_some());
- let count = tr
- .nodes
- .iter()
- .filter(|n| n.element_index.is_some())
- .count();
- let mut header = format!("window_id={hwnd} pid={pid} elements={count}\n");
- if let Some(note) = walk.note() {
- header.push_str(&note);
- header.push('\n');
- }
- header.push('\n');
- content.push(cua_driver_core::protocol::Content::text(
- header + &tr.tree_markdown,
- ));
- structured["element_count"] = json!(count);
- // Verification consumes only structured rows, which still
- // omit non-actionable UIA nodes on Windows. A complete
- // native walk cannot prove absence in that projection.
- structured["elements_complete"] =
- json!(tr.elements_complete && !observation_only);
- structured["tree_markdown"] = json!(tr.tree_markdown);
- walk.apply(&mut structured);
-
- let (snapshot_id, replaced) = (!observation_only)
- .then(|| {
- state.snapshots.publish_for_session(
- pid as i32,
- hwnd,
- payload,
- session_id.as_deref(),
- screenshot_scale,
- )
- })
- .flatten()
- .unzip();
- published_snapshot = snapshot_id.is_some();
- invalidated.extend(replaced.into_iter().flatten());
-
- // Structured `elements` array — preferred consumption
- // path. Shape matches the cross-platform spec:
- // `{element_index, element_token, role, label, depth,
- // actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is
- // included when UIA reported a usable BoundingRectangle.
- let elements: Vec<serde_json::Value> = tr
- .nodes
- .iter()
- .filter_map(|n| build_element_entry(n, snapshot_id))
- .collect();
- let elements = cua_driver_core::element_query::project_elements_for_query(
- elements,
- query.as_deref(),
- &tr.tree_markdown,
- );
- // Screenshot pixels: the capture's top-left is the DWM
- // bitmap origin (the inverse of the pixel tools'
- // `bitmap_to_screen`), downsized by delivered/native.
- let elements = match screenshot_opt.as_ref() {
- Some((_, _, w, _, native_w, _, _)) if *native_w > 0 => {
- let (ox, oy) = bitmap_to_screen(hwnd, 0, 0);
- cua_driver_core::element_frame::with_screenshot_frames(
- elements,
- (f64::from(ox), f64::from(oy)),
- f64::from(*w) / f64::from(*native_w),
- )
- }
- _ => elements,
- };
- structured["total_element_count"] = json!(count);
- structured["returned_element_count"] = json!(elements.len());
- structured["elements"] = json!(elements);
- // Surface 6: snapshot id mirror for debug correlation.
- if let Some(snapshot_id) = snapshot_id {
- structured["snapshot_id"] = json!(
- cua_driver_core::element_token::format_snapshot_id(snapshot_id)
- );
- }
- structured["_note"] = json!(
- "Prefer `elements` — `tree_markdown` will continue to work \
- but new fields will only be added to the structured side. \
- Issue #22865: use `max_elements` / `max_depth` to bound the \
- UIA walk on apps with very large trees."
- );
- // Best-effort-background ladder: a UIA walk that ran but found
- // zero actionable elements is NOT a clean snapshot — the window
- // may be a non-UIA surface (canvas/WebGL/custom-drawn) or its
- // tree wasn't ready (Chromium/Electron need an enable + settle).
- // Mark it degraded so callers don't read `elements: []` as "this
- // window has no controls", and point at the next rung. An empty
- // UIA tree means element_index has nothing to bind to, so the
- // deliberate move is an element px action — read the screenshot
- // already in this response and click by pixel (x,y).
- if is_msaa {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!(
- "msaa_fallback_partial: the UIA provider was unavailable and \
- Cua Driver used a partial MSAA tree. Treat it as discovery \
- evidence only; it cannot prove checked state."
- );
- } else if let (0, Some(note)) = (count, walk.note()) {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!(note);
- } else if count == 0 {
- structured["degraded"] = json!(true);
- structured["degraded_reason"] = json!(
- "ax_tree_empty: the UIA walk returned no actionable elements. \
- The window may be a non-UIA surface (canvas/WebGL/custom-drawn) \
- or its accessibility tree was not ready (Chromium/Electron \
- require a UIA-enable + settle). Do not treat element data as \
- authoritative — re-snapshot if the app just launched, otherwise \
- switch to the visual path."
- );
- structured["escalation"] = json!({
- "recommended": "px",
- "reason": "non-AX surface — act by pixel (x,y) off the \
- screenshot in this response (an element px action)."
- });
- }
- }
-
- // A screenshot-only read adds an image but no element rows: it
- // refreshes the current snapshot's pixel frame in place and
- // keeps its tokens valid. Only a window with no element
- // snapshot yet gets a new, empty one.
- let mut refreshed = None;
- if let (false, false, Some(scale)) =
- (observation_only, published_snapshot, screenshot_scale)
- {
- refreshed = state.snapshots.refresh_screenshot_for_session(
- pid as i32,
- hwnd,
- session_id.as_deref(),
- scale,
- None,
- );
- if refreshed.is_none() {
- let payload = crate::uia::snapshot::UiaSnapshot::from_nodes(
- &[],
- crate::uia::snapshot::SnapshotKind::Uia,
- );
- if let Some((_, replaced)) = state.snapshots.publish_for_session(
- pid as i32,
- hwnd,
- payload,
- session_id.as_deref(),
- screenshot_scale,
- ) {
- invalidated.extend(replaced);
- }
- }
- }
- if let Some(id) = refreshed {
- let sid = cua_driver_core::element_token::format_snapshot_id(id);
- content.push(cua_driver_core::protocol::Content::text(format!(
- "Screenshot only: snapshot {sid} keeps its rows, and its element_tokens \
- ({sid}:N) stay valid."
- )));
- structured["snapshot_id"] = json!(sid);
- structured["screenshot_refreshed_snapshot"] = json!(true);
- }
- if !invalidated.is_empty() {
- let ids: Vec<String> = invalidated
- .into_iter()
- .map(cua_driver_core::element_token::format_snapshot_id)
- .collect();
- content.push(cua_driver_core::protocol::Content::text(format!(
- "Invalidated snapshots {}: their element_tokens are stale.",
- ids.join(", ")
- )));
- structured["invalidated_snapshot_ids"] = json!(ids);
- }
-
- if let Some((b64_opt, file_path, w, h, native_w, native_h, png)) = screenshot_opt {
- let capture_id = if !observation_only {
- if let Some(bridge) = &state.capture_bridge {
- let geometry = match crate::capture_admission::CaptureGeometry::new(
- w, h, native_w, native_h,
- ) {
- Ok(geometry) => geometry,
- Err(error) => {
- return ToolResult::error(format!(
- "Window capture geometry is invalid: {error}"
- ))
- }
- };
- match bridge.publish(
- &args,
- png,
- crate::capture_admission::WindowsCaptureTarget::Window {
- pid,
- window_id: hwnd,
- },
- geometry,
- ) {
- Ok(capture_id) => capture_id,
- Err(error) => {
- return ToolResult::error(format!(
- "Window capture publication failed: {error}"
- ))
- }
- }
- } else {
- None
- }
- } else {
- None
- };
- // base64 is embedded only when no out_file was given (vision
- // path). With `screenshot_out_file` the bytes went to disk and
- // we surface the path instead — never both. Keep a text content
- // part when the image went to disk so the response is never
- // empty on the capture-only path (which has no tree markdown).
- if let Some(b64) = b64_opt {
- content.push(cua_driver_core::protocol::Content::image_png(b64));
- } else if let Some(fp) = &file_path {
- content.push(cua_driver_core::protocol::Content::text(format!(
- "window_id={hwnd} pid={pid} size={w}x{h} screenshot written to {fp}"
- )));
- }
- structured["screenshot_width"] = json!(w);
- structured["screenshot_height"] = json!(h);
- // Surface 7: mirror the MCP image part's `mimeType` onto
- // the structured payload so consumers don't have to sniff
- // magic bytes off the base64 to know the format.
- structured["screenshot_mime_type"] = json!("image/png");
- if let Some(capture_id) = capture_id {
- structured["capture_id"] = json!(capture_id);
- }
- if let Some(fp) = file_path {
- structured["screenshot_file_path"] = json!(fp);
- }
- } else if let Some(err) = screenshot_err {
- // Capture failed (most commonly because the target is
- // minimized — see #1973). Surface the reason in BOTH the
- // human-readable content stream (so the model sees it
- // alongside the UIA tree it does get) AND structuredContent
- // (so MCP clients with structured-only parsing can detect
- // and act on it). Without this the caller saw an empty
- // response with no clue why and burned turns retrying.
- content.push(cua_driver_core::protocol::Content::text(format!(
- "screenshot unavailable: {err}"
- )));
- structured["screenshot_error"] = json!(err);
- }
-
- // Window identity metadata (additive): title + on-screen
- // rectangle + owning process name for the requested window_id,
- // useful on the capture-only path where no UIA tree names it.
- if let Some((title, x, y, w, h)) = &win_geom {
- if !title.is_empty() {
- structured["window_title"] = json!(title);
- }
- structured["window_bounds"] =
- json!({ "x": x, "y": y, "width": w, "height": h });
- }
- if let Some(name) = app_name.as_deref().filter(|n| !n.is_empty()) {
- structured["app_name"] = json!(name);
- }
-
- cua_driver_core::window_inspection::mark_browser_chrome_capture_coverage(
- &mut structured,
- is_standalone_chromium_browser_process(pid).then_some(
- cua_driver_core::window_inspection::BrowserChromeCaptureCoverage::NotObservable,
- ),
- );
-
- // The capture-only path (include_accessibility_tree:false) leaves
- // `content` empty if the screenshot was also unavailable. Return a
- // structured error rather than a "successful" response with no
- // content parts (consistent with the tree+screenshot path, which
- // always carries at least the tree markdown).
- if content.is_empty() {
- return ToolResult::error(format!(
- "No content produced for window_id {hwnd}: the accessibility tree was \
- skipped (include_accessibility_tree:false) and no screenshot was returned."
- ))
- .with_structured(structured);
- }
-
- if display_only {
- cua_driver_core::window_state_view::mark_display_only(&mut structured);
- }
- if !observation_only {
- cua_driver_core::window_state_view::apply(
- &view,
- &cua_driver_core::window_state_view::ViewContext {
- pid: i64::from(pid),
- window_id: hwnd,
- key: cua_driver_core::window_state_view::ViewKey {
- query: query.clone(),
- max_elements,
- max_depth: args
- .get("max_depth")
- .and_then(|v| v.as_u64())
- .map(|v| v as usize),
- },
- focus_probe: None,
- },
- &mut content,
- &mut structured,
- );
- }
-
- ToolResult {
- content,
- is_error: None,
- structured_content: Some(structured),
- action_record: None,
- }
- }
- Err(e) => ToolResult::error(format!("Error: {e}")),
- }
- }
-}
-
/// Split a `launch_path` round-tripped from `list_apps` into a single
/// ShellExecuteEx `lpFile` token plus a trailing arguments string. Handles
/// three input shapes:
@@ -1879,7 +677,7 @@ impl Tool for GetWindowStateTool {
///
/// Args are returned trimmed; their internal quoting is preserved as-is so
/// `--profile-directory="Profile 2"` survives.
-fn split_launchable_target(s: &str) -> (String, String) {
+pub(crate) fn split_launchable_target(s: &str) -> (String, String) {
let trimmed = s.trim();
if trimmed.is_empty() {
return (String::new(), String::new());
@@ -1911,12 +709,6 @@ fn split_launchable_target(s: &str) -> (String, String) {
(trimmed.to_owned(), String::new())
}
-/// Index of the first URL not already consumed as the primary shell target.
-/// An explicit app target consumes no URL; a URLs-only launch consumes index 0.
-fn first_unopened_shell_url_index(has_app_target: bool) -> usize {
- usize::from(!has_app_target)
-}
-
/// Returns `true` when the launch target names a Chromium-based browser
/// — i.e. one whose hidden launch under `SW_SHOWNOACTIVATE` will trigger
/// renderer-occlusion throttling unless we inject the anti-throttling flags
@@ -1927,7 +719,7 @@ fn first_unopened_shell_url_index(has_app_target: bool) -> usize {
/// full paths (`"C:\...\msedge.exe"`), and round-tripped launch paths with
/// trailing arguments (`"\"C:\...\chrome.exe\" --foo"` — the first token is
/// matched via `split_launchable_target`).
-fn is_chromium_browser_target(target: &str) -> bool {
+pub(crate) fn is_chromium_browser_target(target: &str) -> bool {
let (file, _) = split_launchable_target(target);
let candidate = if file.is_empty() {
target
@@ -1956,113 +748,6 @@ fn is_chromium_browser_target(target: &str) -> bool {
)
}
-fn is_standalone_chromium_browser_process(pid: u32) -> bool {
- crate::win32::list_processes()
- .into_iter()
- .find(|process| process.pid == pid)
- .is_some_and(|process| is_chromium_browser_target(&process.name))
-}
-
-/// Anti-throttling flags injected on hidden Chromium launches (#1620).
-///
-/// `launch_app` uses `SW_SHOWNOACTIVATE` so the launched window is
-/// non-foreground from birth. Chromium's `CalculateNativeWinOcclusion`
-/// feature treats that as occluded and suspends the renderer process for
-/// the tab's entire lifetime — UIA tree exposes only browser chrome, no
-/// page DOM, and `PrintWindow` returns a blank body.
-///
-/// `CalculateNativeWinOcclusion` is the root cause; the two
-/// `--disable-backgrounding-*` flags backstop the same effect through the
-/// process-priority and renderer-throttling layers (Chromium suspends
-/// renderers on multiple signals).
-const CHROMIUM_ANTI_THROTTLING_FLAGS: &[&str] = &[
- "--disable-features=CalculateNativeWinOcclusion",
- "--disable-backgrounding-occluded-windows",
- "--disable-renderer-backgrounding",
-];
-
-/// Prepend the Chromium anti-throttling flags to `extra_args` unless the
-/// caller already has each one. Merges into an existing `--disable-features=`
-/// list rather than appending a second `--disable-features=` entry (Chromium
-/// has subtle merging rules across duplicate flags).
-fn inject_chromium_anti_throttling_flags(extra_args: &mut Vec<String>) {
- const OCCLUSION_FEATURE: &str = "CalculateNativeWinOcclusion";
-
- // Merge `CalculateNativeWinOcclusion` into any existing
- // `--disable-features=` entry, or insert a fresh one at the front.
- let has_occlusion = extra_args.iter().any(|a| {
- a.strip_prefix("--disable-features=")
- .map(|v| v.split(',').any(|f| f.trim() == OCCLUSION_FEATURE))
- .unwrap_or(false)
- });
- if !has_occlusion {
- if let Some(idx) = extra_args
- .iter()
- .position(|a| a.starts_with("--disable-features="))
- {
- extra_args[idx] = format!("{},{OCCLUSION_FEATURE}", extra_args[idx]);
- } else {
- extra_args.insert(0, format!("--disable-features={OCCLUSION_FEATURE}"));
- }
- }
-
- // Boolean toggles — insert at front only when not already present.
- for flag in &[
- "--disable-backgrounding-occluded-windows",
- "--disable-renderer-backgrounding",
- ] {
- if !extra_args.iter().any(|a| a == flag) {
- extra_args.insert(0, (*flag).to_string());
- }
- }
- let _ = CHROMIUM_ANTI_THROTTLING_FLAGS; // referenced for docs alignment
-}
-
-// ── launch_app ───────────────────────────────────────────────────────────────
-
-/// Shape of the resolved launch-target params, used to decide whether the
-/// best-effort foreground-restore guard should fire.
-///
-/// The decision is intentionally a pure function of the params (not of any
-/// HWND state) so it can be unit-tested. See
-/// [`should_restore_foreground_after_launch`].
-#[derive(Clone, Copy, Debug)]
-struct LaunchTargetShape {
- has_aumid: bool,
- has_bundle_id: bool,
- has_name: bool,
- has_path: bool,
- has_launch_path: bool,
- has_urls: bool,
-}
-
-/// True iff the launch is "open an app in the background" (so we want to
-/// restore the user's foreground after the spawn activates the new window)
-/// rather than "open a URL in the default browser" (where the user
-/// explicitly asked for that page to come up).
-///
-/// Rule, matching the audit spec:
-/// - URLs-only call (no app-identifying field set) → SKIP restore. The
-/// user asked for `https://…` to be openable in their default browser;
-/// a browser instance freshly launched for that URL is the legitimate
-/// foreground.
-/// - Any app-identifying field present (`aumid` / `bundle_id` / `name` /
-/// `path` / `launch_path`) → RESTORE. The intent is hidden-launch +
-/// background drive; the no-foreground contract applies.
-fn should_restore_foreground_after_launch(shape: LaunchTargetShape) -> bool {
- let has_app_identifier = shape.has_aumid
- || shape.has_bundle_id
- || shape.has_name
- || shape.has_path
- || shape.has_launch_path;
- // URLs-only carve-out: when the caller passed urls AND no
- // app-identifying field, skip the restore.
- if shape.has_urls && !has_app_identifier {
- return false;
- }
- has_app_identifier
-}
-
/// Async polling restore of the prior foreground window, used by
/// `LaunchAppTool` to mirror the macOS `FocusRestoreGuard` for the legacy
/// `ShellExecuteExW` launch path. The UWP/AUMID branch has its own
@@ -2087,7 +772,10 @@ fn should_restore_foreground_after_launch(shape: LaunchTargetShape) -> bool {
/// than `HWND` directly) because `HWND` wraps `*mut c_void` which is
/// `!Send` and would prevent this future from being scheduled on the
/// multi-threaded tokio runtime.
-async fn restore_foreground_polling_best_effort(prior_foreground_addr: usize, spawned_pid: u32) {
+pub(crate) async fn restore_foreground_polling_best_effort(
+ prior_foreground_addr: usize,
+ spawned_pid: u32,
+) {
use windows::Win32::Foundation::{CloseHandle, HWND};
use windows::Win32::System::Threading::{
OpenProcess, WaitForInputIdle, PROCESS_QUERY_LIMITED_INFORMATION, PROCESS_SYNCHRONIZE,
@@ -2214,8227 +902,339 @@ async fn restore_foreground_polling_best_effort(prior_foreground_addr: usize, sp
}
}
-/// Bound on one standard-user shell launch helper. It stays inside the 15 s
-/// launch deadline so a helper stuck on a shell dialog is terminated rather
-/// than left behind.
-const STANDARD_USER_SHELL_LAUNCH_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(12);
-
-pub struct LaunchAppTool;
-static LAUNCH_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for LaunchAppTool {
- fn def(&self) -> &ToolDef {
- LAUNCH_DEF.get_or_init(|| ToolDef {
- name: "launch_app".into(),
- // Description ported from Swift `LaunchAppTool.swift` with
- // Windows-specific notes. `bundle_id` is accepted as an alias
- // for `name` on Windows, with one extra behavior: if the
- // string looks like an AUMID (App User Model ID — contains
- // `!`, e.g. `Microsoft.WindowsNotepad_8wekyb3d8bbwe!App`) we
- // route through `IApplicationActivationManager` so packaged
- // (Microsoft Store / UWP / MSIX) apps return the real
- // process pid instead of a stub-redirect pid.
- description: "Launch a Windows app hidden — the driver never brings the target to \
- the foreground, the target's window is launched with SW_SHOWNOACTIVATE so it \
- does not steal focus from whatever is currently frontmost.\n\n\
- Provide either `bundle_id` / `name` / `aumid` (resolved as below) or `path` \
- (full path to an executable). If both `name` and `path` are given, `path` wins. \
- `urls` opens each URL in the default browser without activating it.\n\n\
- Routing order: explicit `aumid` (or a `bundle_id` containing `!`) activates the \
- packaged app via `IApplicationActivationManager::ActivateApplication`, returning \
- the real packaged-process pid — required on Win11 for built-in apps like \
- Notepad, Calculator, and Paint, which now ship as Microsoft Store packages \
- where the legacy .exe in System32 is a ~7 KB stub that exits immediately. A \
- plain `name` first attempts a `shell:AppsFolder` lookup; packaged matches use \
- the activation manager while desktop registrations are launched through their \
- shell parsing path. A miss falls back to ShellExecuteEx's PATH search.\n\n\
- Returns the launched app's pid, name, active flag, AND a `windows` array — \
- same per-window shape `list_windows` returns. When the launch settles but no \
- window has materialized yet (transient; rare), `windows` comes back empty — \
- call `list_windows(pid)` explicitly a moment later. `bundle_id` in the \
- response is set to the AUMID actually used for packaged-app launches and \
- `null` for ShellExecuteEx launches.\n\n\
- Windows-only field: `path` (Swift uses `bundle_id` since macOS apps resolve via \
- LaunchServices; Windows has no LaunchServices, so `path` is the canonical form). \
- The macOS-specific `webkit_inspector_port`, \
- `creates_new_application_instance`, and `additional_arguments` fields are \
- accepted; `additional_arguments` is honored (forwarded as ShellExecuteEx \
- parameters or as the AUMID activation arguments string). \
- The remaining macOS-only fields currently no-op on Windows.".into(),
- input_schema: json!({"type":"object","properties":{
- "bundle_id":{"type":"string","description":"App User Model ID (AUMID) for a packaged app — pattern `{PackageFamilyName}!{ApplicationId}`, e.g. `Microsoft.WindowsNotepad_8wekyb3d8bbwe!App`. Falls back to a `name` alias if no `!` is present. Either bundle_id, name, aumid, path, or launch_path must be provided."},
- "aumid":{"type":"string","description":"Explicit AUMID for a packaged app. Cleaner alternative to overloading `bundle_id`. Takes precedence over `bundle_id` / `name` when present."},
- "name":{"type":"string","description":"App display name. Tried against the `shell:AppsFolder` index first for packaged-app lookup; on a miss, passed to ShellExecuteEx's PATH search."},
- "path":{"type":"string","description":"Full path to executable. Windows-only; takes precedence over name/bundle_id/aumid (but not over `launch_path`)."},
- "launch_path":{"type":"string","description":"Round-trip the `launch_path` returned by `list_apps`. Highest precedence — when set, this exact string is handed to ShellExecuteEx unchanged. For Windows desktop apps it's the full `.exe` commandline (path + arguments preserved from the source shortcut); for UWP apps it's `shell:appsFolder\\{PackageFamilyName}!{AppId}`. For precise UWP pid capture, prefer `aumid` over `launch_path`."},
- "urls":{"type":"array","items":{"type":"string"},"description":"URLs to open in the default browser via ShellExecuteEx (no activation)."},
- "additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process (or activation arguments for packaged apps)."},
- "webkit_inspector_port":{"type":"integer","description":"Accepted for cross-platform parity; no-op on Windows."},
- "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."},
- "start_minimized":{"type":"boolean","description":"When true, launch the app's window minimized to the taskbar instead of restored-but-not-activated. Use this when the agent wants to drive the app entirely in the background — the user's previously-frontmost window (e.g. terminal) stays visually on top. Desktop launches hold the foreground lock through startup and use SW_SHOWMINNOACTIVE; packaged-app activation remains broker-controlled and receives a best-effort SW_SHOWMINNOACTIVE post-pass. UIA / background dispatch still work on a minimized window; only `screenshot` and `delivery_mode:\"foreground\"` need it restored."}
- },"additionalProperties":false}),
- read_only: false, destructive: false, idempotent: true, open_world: true,
- })
+/// px-focus for the keyboard family (type_text / press_key / hotkey): pixel-click
+/// at (x,y) to establish real renderer focus before a keystroke — the *element px
+/// action* form of a keyboard tool. Reuses ClickTool's exact coordinate
+/// translation + delivery_mode (UIA hit-test → PostMessage fallback), so it lands
+/// on the same pixel a px-click would. `Ok(())` on success; `Err(ToolResult)`
+/// short-circuits the caller. Mirrors macOS `tools::focus_by_pixel`.
+#[allow(clippy::too_many_arguments)]
+pub(crate) fn focus_by_pixel_click_args(
+ pid: u32,
+ window_id: Option<u64>,
+ x: f64,
+ y: f64,
+ foreground: bool,
+ session: Option<String>,
+ session_id: Option<String>,
+ from_zoom: bool,
+ native_coordinates: bool,
+) -> Value {
+ let mut click_args = json!({
+ "pid": pid, "x": x, "y": y,
+ "delivery_mode": if foreground { "foreground" } else { "background" },
+ });
+ if let Some(wid) = window_id {
+ click_args["window_id"] = json!(wid);
}
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let launch_path_opt = args
- .get("launch_path")
- .and_then(|v| v.as_str())
- .map(str::to_owned);
- let path_opt = args.get("path").and_then(|v| v.as_str()).map(str::to_owned);
- let name_opt = args.get("name").and_then(|v| v.as_str()).map(str::to_owned);
- let bundle_id_opt = args
- .get("bundle_id")
- .and_then(|v| v.as_str())
- .map(str::to_owned);
- let aumid_opt = args
- .get("aumid")
- .and_then(|v| v.as_str())
- .map(str::to_owned);
- let urls: Vec<String> = args
- .get("urls")
- .and_then(|v| v.as_array())
- .map(|a| {
- a.iter()
- .filter_map(|v| v.as_str().map(str::to_owned))
- .collect()
- })
- .unwrap_or_default();
- let start_minimized = args
- .get("start_minimized")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- // `nShow` for the ShellExecuteEx-based launches. SW_SHOWNOACTIVATE
- // keeps the new window from grabbing focus but still renders it on
- // top of the previous frontmost window. SW_SHOWMINNOACTIVE both
- // skips activation AND drops the window to the taskbar, which is
- // what an agent doing background work wants — the user's terminal
- // stays visually on top. UIA / PostMessage / set_value all keep
- // working against a minimized window; only screenshot and
- // delivery_mode:"foreground" pay a cost.
- let n_show: i32 = if start_minimized {
- windows::Win32::UI::WindowsAndMessaging::SW_SHOWMINNOACTIVE.0
- } else {
- windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE.0
- };
- // Snapshot the set of currently-running pids BEFORE we launch. The
- // detached start_minimized polling task uses this to detect "any
- // process that came into existence as a result of our launch", so
- // it can minimize their windows too — covers the launcher-stub
- // case (LibreOffice swriter.exe → soffice.bin, GIMP gimp-3.exe →
- // gimp-3.2.exe, etc.) where the launched pid exits and a child
- // process with a different name owns the actual window. Captured
- // here (before the launch) so the new soffice.bin pid won't be
- // in the snapshot.
- let pre_launch_pids: std::sync::Arc<std::collections::HashSet<u32>> = if start_minimized {
- let pids: std::collections::HashSet<u32> = crate::win32::list_processes()
- .into_iter()
- .map(|p| p.pid)
- .collect();
- std::sync::Arc::new(pids)
- } else {
- std::sync::Arc::new(std::collections::HashSet::new())
- };
-
- // Capture the foreground window BEFORE any launch path runs so the
- // post-spawn polling restore (below) has a target HWND to flip back
- // to. Mirrors the macOS `FocusRestoreGuard` behavior. Best-effort —
- // see `restore_foreground_polling_best_effort` for the actual
- // restoration semantics.
- //
- // The UWP/AUMID path already restores foreground synchronously
- // (see `launch_uwp::restore_foreground_best_effort`), so this only
- // gates the legacy ShellExecuteExW path. We capture early so the
- // value is consistent regardless of which sub-path runs.
- //
- // Stored as `usize` (not `HWND`) so this `invoke` future stays
- // `Send` — `HWND` wraps `*mut c_void` and would taint the future
- // for the multi-threaded tokio runtime if held across `.await`.
- let foreground_before_addr: usize =
- unsafe { windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize };
-
- // Pure-function decision on whether to fire the post-launch
- // restore. Shape mirrors the resolved params — see
- // `should_restore_foreground_after_launch` for the rule.
- let launch_shape = LaunchTargetShape {
- has_aumid: aumid_opt.is_some(),
- has_bundle_id: bundle_id_opt.is_some(),
- has_name: name_opt.is_some(),
- has_path: path_opt.is_some(),
- has_launch_path: launch_path_opt.is_some(),
- has_urls: !urls.is_empty(),
- };
- let restore_foreground = should_restore_foreground_after_launch(launch_shape);
- let mut extra_args: Vec<String> = args
- .get("additional_arguments")
- .and_then(|v| v.as_array())
- .map(|a| {
- a.iter()
- .filter_map(|v| v.as_str().map(str::to_owned))
- .collect()
- })
- .unwrap_or_default();
- if args.get("cdp_debugging_port").is_some() {
- return ToolResult::error(
- "cdp_debugging_port moved to browser_prepare so DevTools is never enabled on an unproven user profile",
- );
- }
- if launch_path_opt
- .as_deref()
- .into_iter()
- .chain(path_opt.as_deref())
- .chain(name_opt.as_deref())
- .chain(extra_args.iter().map(String::as_str))
- .any(cua_driver_core::launch_guard::contains_remote_debugging_flag)
- {
- return ToolResult::error(
- cua_driver_core::launch_guard::REMOTE_DEBUGGING_LAUNCH_REFUSAL,
- );
- }
-
- // Resolve target — launch_path > path > aumid > name > bundle_id
- // (alias of name on Windows). launch_path is highest precedence for
- // round-trip with list_apps; the value is handed to ShellExecuteEx
- // unchanged (UWP routing below is bypassed when launch_path is set).
- let target = launch_path_opt
- .clone()
- .or(path_opt.clone())
- .or(aumid_opt.clone())
- .or(name_opt.clone())
- .or(bundle_id_opt.clone());
-
- // #1620: auto-inject Chromium anti-throttling flags when the resolved
- // target names a Chromium-based browser (Edge / Chrome / Brave /
- // Vivaldi / Opera / Chromium / Arc / Thorium / Iridium / Yandex).
- // launch_app hides the window via SW_SHOWNOACTIVATE; Chromium's
- // occlusion logic suspends the renderer in that case, leaving the
- // UIA tree empty and PrintWindow capturing a blank body. The three
- // flags below defeat that for the lifetime of the launched process
- // without affecting normal interactive use.
- //
- // Skipped when `aumid` or `bundle_id` route through the UWP path
- // (packaged Edge is the exception; the desktop Edge channel is what
- // ships now and that goes through the ShellExecuteEx path below).
- if launch_path_opt.is_some() || path_opt.is_some() || name_opt.is_some() {
- if let Some(t) = target.as_deref() {
- if is_chromium_browser_target(t) {
- inject_chromium_anti_throttling_flags(&mut extra_args);
- }
- }
- }
-
- if target.is_none() && urls.is_empty() {
- // Error message lists every field the resolver actually accepts.
- // The Swift-original "bundle_id or name" message predated the Windows
- // additions (aumid, path, launch_path, urls) and made #1635 look like
- // an aumid-specific bug. The actual cause of #1635 was the upstream
- // PS argv quote-stripping bug fixed in #1637; this message just stops
- // misleading anyone who hits the error for unrelated reasons.
- return ToolResult::error(
- "Provide one of: bundle_id, name, aumid, path, launch_path, or urls to identify the app to launch.",
- );
- }
-
- // ── Packaged-app (UWP / MSIX) routing decision ──────────────────────
- //
- // Routing precedence — most explicit signal wins:
- // 1. `launch_path` set: skip UWP routing entirely — round-trip
- // semantics demand the value be handed to ShellExecuteEx
- // unchanged. Callers wanting precise UWP pid capture should
- // pass `aumid` instead.
- // 2. `aumid` parameter (any value): packaged path, AUMID = value.
- // 3. `bundle_id` containing `!`: packaged path, AUMID = value
- // (Win32 PATH lookups never produce a `!` in the executable
- // name, so this is a safe marker of caller intent).
- // 4. `name` with no `path`: try `shell:AppsFolder` lookup. A
- // packaged hit uses the resolved AUMID; a desktop registration
- // uses `shell:AppsFolder\<AppUserModelID>` via ShellExecuteExW.
- // On miss, fall through to the existing Win32 path.
- // 5. Anything else: existing ShellExecuteExW path.
- //
- // `path` is intentionally excluded from packaged routing — when
- // the caller has a concrete executable in mind they want
- // CreateProcess semantics, not Start Menu activation.
- let mut name_lookup_missed = false;
- let mut shell_target_from_name: Option<String> = None;
- let aumid_for_uwp: Option<String> = if launch_path_opt.is_some() || path_opt.is_some() {
- None
- } else if let Some(a) = aumid_opt.clone() {
- Some(a)
- } else if let Some(b) = bundle_id_opt
- .clone()
- .filter(|s| crate::launch_uwp::is_aumid(s))
- {
- Some(b)
- } else if let Some(n) = name_opt.clone() {
- // The cold AppsFolder COM walk can stop making progress when its
- // interactive shell broker is unhealthy. Bound it before the
- // separate ShellExecuteEx deadline below; the resolver keeps a
- // timed-out worker single-flight until COM really returns.
- match crate::launch_uwp::resolve_apps_folder_target_by_name_bounded(n.clone()).await {
- Ok(Some(crate::launch_uwp::AppsFolderLaunchTarget::PackagedAumid(aumid))) => {
- Some(aumid)
- }
- Ok(Some(crate::launch_uwp::AppsFolderLaunchTarget::ShellLaunchPath(
- launch_path,
- ))) => {
- shell_target_from_name = Some(launch_path);
- None
- }
- Ok(None) => {
- name_lookup_missed = true;
- None
- }
- Err(crate::launch_uwp::AppsFolderLookupError::Timeout) => {
- return ToolResult::error(format!(
- "App name lookup for {n:?} is temporarily unavailable: Windows did not respond to the shell:AppsFolder query within 4s. No app was launched; retry after the shell recovers, or pass an explicit path or aumid."
- ))
- }
- Err(crate::launch_uwp::AppsFolderLookupError::Busy) => {
- return ToolResult::error(format!(
- "App name lookup for {n:?} is temporarily unavailable because a prior Windows shell lookup is still running or cooling down. No app was launched; retry later, or pass an explicit path or aumid."
- ))
- }
- Err(crate::launch_uwp::AppsFolderLookupError::Unavailable) => {
- return ToolResult::error(format!(
- "App name lookup for {n:?} is unavailable because the Windows shell lookup worker failed. No app was launched; pass an explicit path or aumid."
- ))
- }
- }
- } else {
- None
- };
- let resolved_target = shell_target_from_name.or(target);
-
- // Single launch path. `display` is the human-readable identifier
- // we report in the success header; for the packaged path it's
- // the AUMID, which is more informative than the display name.
- let display = aumid_for_uwp
- .clone()
- .or_else(|| resolved_target.clone())
- .unwrap_or_else(|| urls.join(", "));
-
- // When the resolved target came from `launch_path` it may carry
- // shortcut arguments (e.g. `"C:\Path\chrome.exe" --profile-directory="Profile 2"`).
- // Split it into a file token + arguments tail so ShellExecuteEx
- // sees them separately — passing the whole string as lpFile would
- // fail because the shell only resolves a single filesystem path
- // there. No-op for AUMID tokens / plain paths without args / UWP
- // `shell:appsFolder\..!..` tokens.
- let (target_file_opt, extra_joined) = {
- let base_extra = extra_args.join(" ");
- if launch_path_opt.is_some() {
- if let Some(t) = resolved_target.as_deref() {
- let (file, args_tail) = split_launchable_target(t);
- let combined = if args_tail.is_empty() {
- base_extra
- } else if base_extra.is_empty() {
- args_tail
- } else {
- format!("{args_tail} {base_extra}")
- };
- let file_opt = if file.is_empty() { None } else { Some(file) };
- (file_opt, combined)
- } else {
- (None, base_extra)
- }
- } else {
- (resolved_target.clone(), base_extra)
- }
- };
-
- // Strict no-activation is available only for the desktop launch path.
- // UWP activation is broker-controlled and retains its existing
- // restore-after-activation behavior.
- let mut foreground_lock = if start_minimized && aumid_for_uwp.is_none() {
- Some(crate::input::ForegroundLockGuard::acquire())
- } else {
- None
- };
- if foreground_lock
- .as_ref()
- .is_some_and(|guard| !guard.acquired())
- {
- return ToolResult::error(
- "Background minimized launch is unavailable because Windows did not grant the \
- foreground lock required to prevent the new process from activating. No process \
- was started. Launch without start_minimized only when a foreground change is \
- acceptable.",
- )
- .with_structured(json!({
- "code": "background_unavailable",
- "delivery_mode": "background",
- "event_kind": "app_launch",
- }));
- }
-
- // Branch: AUMID activation if we resolved one; else legacy
- // ShellExecuteExW. Both branches still need to handle `urls`
- // (additional URLs always go through ShellExecuteExW since the
- // Start Menu doesn't activate by URL).
- let pid = if let Some(aumid) = aumid_for_uwp.clone() {
- let aumid_clone = aumid.clone();
- let args_clone = extra_joined.clone();
- let activation = tokio::task::spawn_blocking(move || {
- crate::launch_uwp::launch_uwp(&aumid_clone, &args_clone)
- })
- .await;
- match activation {
- Ok(Ok(p)) => p,
- Ok(Err(e)) => {
- return ToolResult::error(format!(
- "Failed to activate packaged app {aumid:?}: {e}"
- ))
- }
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- }
- } else {
- // Legacy ShellExecuteExW path for plain Win32 apps and for
- // callers passing an explicit `path` / `launch_path`. An elevated
- // Driver runs these launches with its standard-user token so the
- // app does not inherit administrator rights (#3607).
- let launch_token = match crate::standard_user_launch::app_launch_token() {
- Ok(token) => token,
- Err(message) => return ToolResult::error(message),
- };
- let urls_clone = urls.clone();
- let target_for_shell = target_file_opt.clone();
- let extra_for_shell = extra_joined.clone();
- let n_show_for_shell = n_show;
- let direct_minimized_exe = start_minimized
- && target_for_shell.as_deref().is_some_and(|target| {
- std::path::Path::new(target).is_file()
- && std::path::Path::new(target)
- .extension()
- .is_some_and(|extension| extension.eq_ignore_ascii_case("exe"))
- });
- // Bound the shell launch with a timeout. An unregistered protocol
- // or file association makes `ShellExecuteExW` block on a modal shell
- // dialog ("you'll need a new app to open this …") on the *session*
- // desktop — which a Session-0/headless daemon can neither see nor
- // dismiss — wedging the launch (and that request) indefinitely.
- // `SEE_MASK_FLAG_NO_UI` (below) suppresses the no-association error
- // UI so that case fails fast with an error code; the timeout is the
- // backstop for any *other* blocking broker dialog (SmartScreen, an
- // elevation/consent surface) so a bad target can't hang the daemon.
- let launch = tokio::task::spawn_blocking(move || -> anyhow::Result<u32> {
- use windows::core::{PCWSTR, PWSTR};
- use windows::Win32::Foundation::CloseHandle;
- use windows::Win32::System::Threading::{
- CreateProcessW, GetProcessId, PROCESS_CREATION_FLAGS, PROCESS_INFORMATION,
- STARTF_USESHOWWINDOW, STARTUPINFOW,
- };
- use windows::Win32::UI::Shell::{
- ShellExecuteExW, SEE_MASK_FLAG_NO_UI, SEE_MASK_NOCLOSEPROCESS,
- SHELLEXECUTEINFOW,
- };
-
- fn to_wide(s: &str) -> Vec<u16> {
- s.encode_utf16().chain(std::iter::once(0)).collect()
- }
- let op_w = to_wide("open");
- let file_w = to_wide(if let Some(t) = target_for_shell.as_deref() {
- t
- } else {
- urls_clone.first().map(|s| s.as_str()).unwrap_or("")
- });
- let args_w = to_wide(&extra_for_shell);
-
- use crate::standard_user_launch::AppLaunchToken;
- let pid = if direct_minimized_exe {
- let target = target_for_shell
- .as_deref()
- .expect("checked executable path");
- let command_line_text = if extra_for_shell.is_empty() {
- format!(r#""{target}""#)
- } else {
- format!(r#""{target}" {extra_for_shell}"#)
- };
- if let AppLaunchToken::StandardUser(token) = &launch_token {
- crate::standard_user_launch::create_process(
- token,
- target,
- &command_line_text,
- n_show_for_shell,
- )
- .map_err(anyhow::Error::msg)?
- } else {
- let mut command_line = to_wide(&command_line_text);
- let startup = STARTUPINFOW {
- cb: std::mem::size_of::<STARTUPINFOW>() as u32,
- dwFlags: STARTF_USESHOWWINDOW,
- wShowWindow: n_show_for_shell as u16,
- ..Default::default()
- };
- let mut process = PROCESS_INFORMATION::default();
- unsafe {
- CreateProcessW(
- PCWSTR(file_w.as_ptr()),
- PWSTR(command_line.as_mut_ptr()),
- None,
- None,
- false,
- PROCESS_CREATION_FLAGS(0),
- None,
- PCWSTR::null(),
- &startup,
- &mut process,
- )?;
- let _ = CloseHandle(process.hThread);
- let _ = CloseHandle(process.hProcess);
- }
- process.dwProcessId
- }
- } else if let AppLaunchToken::StandardUser(token) = &launch_token {
- let file = target_for_shell
- .as_deref()
- .or_else(|| urls_clone.first().map(String::as_str))
- .unwrap_or("");
- crate::standard_user_launch::shell_execute(
- token,
- file,
- &extra_for_shell,
- n_show_for_shell,
- STANDARD_USER_SHELL_LAUNCH_TIMEOUT,
- )
- .map_err(anyhow::Error::msg)?
- } else {
- let mut info = SHELLEXECUTEINFOW {
- cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
- fMask: SEE_MASK_NOCLOSEPROCESS | SEE_MASK_FLAG_NO_UI,
- lpVerb: PCWSTR(op_w.as_ptr()),
- lpFile: PCWSTR(file_w.as_ptr()),
- lpParameters: if extra_for_shell.is_empty() {
- PCWSTR::null()
- } else {
- PCWSTR(args_w.as_ptr())
- },
- nShow: n_show_for_shell,
- ..Default::default()
- };
- unsafe {
- ShellExecuteExW(&mut info)?;
- }
- if !info.hProcess.is_invalid() {
- let p = unsafe { GetProcessId(info.hProcess) };
- unsafe {
- let _ = CloseHandle(info.hProcess);
- }
- p
- } else {
- 0
- }
- };
-
- // Open any additional URLs in the default browser (no focus
- // steal, no pid capture for these — Swift's NSWorkspace flow
- // behaves the same way for secondary URLs).
- // When an app target was launched above, none of the URLs has
- // been consumed yet. URLs-only launches use the first URL as
- // `lpFile`, so only their remaining URLs belong here.
- let first_unopened_url = first_unopened_shell_url_index(target_for_shell.is_some());
- for url in &urls_clone[first_unopened_url.min(urls_clone.len())..] {
- if let AppLaunchToken::StandardUser(token) = &launch_token {
- let _ = crate::standard_user_launch::shell_execute(
- token,
- url,
- "",
- n_show_for_shell,
- STANDARD_USER_SHELL_LAUNCH_TIMEOUT,
- );
- continue;
- }
- let file = to_wide(url);
- let mut url_info = SHELLEXECUTEINFOW {
- cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
- fMask: SEE_MASK_FLAG_NO_UI,
- lpVerb: PCWSTR(op_w.as_ptr()),
- lpFile: PCWSTR(file.as_ptr()),
- nShow: n_show_for_shell,
- ..Default::default()
- };
- unsafe {
- let _ = ShellExecuteExW(&mut url_info);
- }
- }
-
- Ok(pid)
- });
- // 15s is generous: ShellExecuteExW returns once activation starts
- // (process created), not when the window appears, so a healthy
- // launch resolves in well under a second. On timeout we abandon the
- // blocking task (a spawn_blocking thread can't be cancelled — it
- // unblocks if/when the modal is dismissed) and return an error so
- // the daemon stays responsive instead of wedging on the request.
- let result = tokio::time::timeout(std::time::Duration::from_secs(15), launch).await;
-
- match result {
- Ok(Ok(Ok(p))) => p,
- Ok(Ok(Err(e))) if name_lookup_missed => {
- return ToolResult::error(format!(
- "App {:?} was not found in shell:AppsFolder or by Windows PATH/association lookup: {e}",
- name_opt.as_deref().unwrap_or("")
- ))
- }
- Ok(Ok(Err(e))) => return ToolResult::error(format!("Failed to launch: {e}")),
- Ok(Err(e)) => return ToolResult::error(format!("Task error: {e}")),
- Err(_elapsed) => {
- return ToolResult::error(format!(
- "Launch of {:?} timed out after 15s — the target likely has no \
- registered handler and a blocking shell dialog appeared on the \
- session desktop; aborted to keep the daemon responsive.",
- target_file_opt
- .as_deref()
- .or_else(|| urls.first().map(|s| s.as_str()))
- .unwrap_or("")
- ))
- }
- }
- };
-
- // Best-effort foreground-restore for the legacy ShellExecuteExW
- // path. The UWP/AUMID branch already restores foreground
- // synchronously inside `launch_uwp::launch_uwp`, so we only spawn
- // the polling task when we went through the ShellExecuteExW
- // branch. URLs-only invocations are excluded via `restore_foreground`
- // (see `should_restore_foreground_after_launch`).
- //
- // `HWND` wraps a raw `*mut c_void` which is `!Send`, so we marshal
- // the foreground handle across the spawn boundary as a `usize` and
- // reconstruct the `HWND` only inside the synchronous probes (see
- // `restore_foreground_polling_best_effort`). Mirrors how
- // `launch_uwp.rs` carries its restore target.
- if aumid_for_uwp.is_none() && restore_foreground {
- let spawned_pid_for_restore = pid;
- let target_foreground_addr = foreground_before_addr;
- tokio::spawn(async move {
- restore_foreground_polling_best_effort(
- target_foreground_addr,
- spawned_pid_for_restore,
- )
- .await;
- });
- }
-
- // When the packaged path was taken we still need to fan out any
- // extra URLs to the default browser (ShellExecuteEx — no pid
- // capture, mirroring Swift's NSWorkspace secondary-URL flow).
- if aumid_for_uwp.is_some() && !urls.is_empty() {
- let urls_clone = urls.clone();
- let n_show_for_urls = n_show;
- let _ = tokio::task::spawn_blocking(move || {
- use windows::core::PCWSTR;
- use windows::Win32::UI::Shell::{ShellExecuteExW, SHELLEXECUTEINFOW};
- fn to_wide(s: &str) -> Vec<u16> {
- s.encode_utf16().chain(std::iter::once(0)).collect()
- }
- let op_w = to_wide("open");
- for url in &urls_clone {
- let file = to_wide(url);
- let mut url_info = SHELLEXECUTEINFOW {
- cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
- lpVerb: PCWSTR(op_w.as_ptr()),
- lpFile: PCWSTR(file.as_ptr()),
- nShow: n_show_for_urls,
- ..Default::default()
- };
- unsafe {
- let _ = ShellExecuteExW(&mut url_info);
- }
- }
- })
- .await;
- }
-
- // Resolve the pid's windows so the caller can skip a list_windows
- // round-trip — same approach as Swift's `resolveWindows`. Retry
- // 5×200ms; Win32 window registration can lag the launch.
- //
- // Launcher-stub fallback: when the launched binary is a wrapper that
- // re-execs and exits (GIMP's `gimp-3.exe` → `gimp-3.2.exe`; LO's
- // `swriter.exe` → `soffice.bin`), the launched pid never gets a
- // window — list_windows(Some(pid)) stays empty forever. After the
- // primary retry budget we fall back to scanning the launched pid's
- // descendant + name-related processes and pick the first one with a
- // window. The resolved pid is reflected in the response's `pid` field
- // so callers can target it with subsequent calls. See #1615.
- let mut windows_json: Vec<serde_json::Value> = Vec::new();
- let mut resolved_pid: u32 = pid;
-
- // UWP / packaged-app host-window resolution. `launch_uwp` returns the
- // real packaged-app pid (e.g. CalculatorApp.exe), but a UWP app's
- // top-level window — the HWND a caller must actually drive — is owned
- // by `ApplicationFrameHost.exe`, not by that process. The app process
- // only owns a `CoreWindow` reparented as a child of the AFH frame, so
- // `list_windows(Some(app_pid))` is empty and the generic retry loop
- // below would either spin out to a stub-fallback miss or return no
- // windows at all (the original "launch_app of a UWP app returns empty
- // windows[] / a pid you can't drive" bug). Map app_pid → its AFH host
- // frame and report `(frame_hwnd, afh_pid)` so the returned handles
- // resolve in `get_window_state` and element actions land. Retry: the
- // frame can lag the activation by a few hundred ms.
- if aumid_for_uwp.is_some() {
- for _ in 0..10 {
- let host =
- tokio::task::spawn_blocking(move || crate::win32::resolve_uwp_host_window(pid))
- .await
- .unwrap_or(None);
- if let Some(w) = host {
- windows_json = vec![json!({
- "window_id": w.hwnd, "title": w.title,
- "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
- "layer": 0,
- "z_index": 0,
- "is_on_screen": true,
- })];
- // Report the AFH pid: that's the pid that owns the frame
- // HWND, so `get_window_state(resolved_pid, window_id)`
- // validates and `list_windows(Some(resolved_pid))` lists it.
- resolved_pid = w.pid;
- break;
- }
- tokio::time::sleep(std::time::Duration::from_millis(200)).await;
- }
- }
-
- for _ in 0..5 {
- if !windows_json.is_empty() {
- break;
- }
- let wins =
- tokio::task::spawn_blocking(move || crate::win32::list_windows_win32_first(pid))
- .await
- .unwrap_or_default();
- if !wins.is_empty() {
- let window_count = wins.len();
- windows_json = wins.iter().enumerate().map(|(position, w)| json!({
- "window_id": w.hwnd, "title": w.title,
- "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
- "layer": 0,
- "z_index": z_index_from_front_to_back(window_count, position),
- "is_on_screen": w.is_on_screen,
- })).collect();
- break;
- }
- tokio::time::sleep(std::time::Duration::from_millis(200)).await;
- }
- if windows_json.is_empty() {
- // Launcher-stub fallback. Compute the exe basename from the
- // launchable target so name-based matching has something to work
- // with (e.g. "gimp-3.exe" → prefix "gimp" matches "gimp-3.2.exe").
- let basename_for_match = target_file_opt
- .as_deref()
- .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())
- .unwrap_or("")
- .to_owned();
- // Known-slow launchers get an extended retry budget. GIMP 3.x in
- // particular spends 10-20s on its first launch (font cache rebuild,
- // plugin scan, etc.) before the main window registers. We don't
- // want to wait 20s for every launcher — gate on basename prefix
- // matching known-slow apps. Add to this list as encountered.
- let bn_lower = basename_for_match.to_ascii_lowercase();
- let is_slow_launcher = bn_lower.starts_with("gimp")
- || bn_lower.starts_with("blender") // OpenGL init can stall
- || bn_lower.starts_with("inkscape") // similar GTK pattern
- || bn_lower.starts_with("krita")
- || bn_lower.starts_with("freecad");
- let max_candidate_attempts: usize = if is_slow_launcher { 30 } else { 3 };
-
- let basename_clone = basename_for_match.clone();
- let candidates_initial = tokio::task::spawn_blocking(move || {
- crate::win32::related_processes(pid, &basename_clone)
- })
- .await
- .unwrap_or_default();
-
- // For slow launchers we may also need to RE-SCAN candidates over
- // time, because the wrapper may not have spawned its child yet
- // when we first scanned. Cap total wait at ~12s (slow) / 0.6s (fast).
- let mut tried: std::collections::HashSet<u32> = std::collections::HashSet::new();
- tried.insert(pid); // already tried in the primary loop
- let mut candidate_queue: Vec<u32> = candidates_initial
- .into_iter()
- .filter(|p| tried.insert(*p))
- .collect();
- let mut total_attempts: usize = 0;
- 'outer: loop {
- while let Some(candidate_pid) = candidate_queue.pop() {
- for _ in 0..max_candidate_attempts {
- total_attempts += 1;
- let wins = tokio::task::spawn_blocking(move || {
- crate::win32::list_windows_win32_first(candidate_pid)
- })
- .await
- .unwrap_or_default();
- if !wins.is_empty() {
- let window_count = wins.len();
- windows_json = wins.iter().enumerate().map(|(position, w)| json!({
- "window_id": w.hwnd, "title": w.title,
- "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
- "layer": 0,
- "z_index": z_index_from_front_to_back(window_count, position),
- "is_on_screen": w.is_on_screen,
- })).collect();
- resolved_pid = candidate_pid;
- break 'outer;
- }
- tokio::time::sleep(std::time::Duration::from_millis(200)).await;
- }
- }
- // For slow launchers, keep re-scanning descendants — the
- // wrapper may not have spawned its child yet. Cap total
- // wait at ~12s (60 × 200ms) for the slow path.
- if !is_slow_launcher || total_attempts > 60 {
- break;
- }
- // Give the wrapper a moment to spawn before re-scanning.
- tokio::time::sleep(std::time::Duration::from_millis(500)).await;
- total_attempts += 3; // count the 500ms wait as 3 attempts
- let basename_rescan = basename_for_match.clone();
- let fresh = tokio::task::spawn_blocking(move || {
- crate::win32::related_processes(pid, &basename_rescan)
- })
- .await
- .unwrap_or_default();
- let new_ones: Vec<u32> = fresh.into_iter().filter(|p| tried.insert(*p)).collect();
- candidate_queue = new_ones;
- // Continue the outer loop regardless — even with empty
- // new_ones we want another iteration that hits the
- // total_attempts cap. The loop body handles the empty queue
- // by falling through to the re-scan again.
- }
- }
- let pid = resolved_pid;
-
- // start_minimized post-pass: drop every resolved window to the
- // taskbar. The desktop ShellExecuteEx path already minimizes via
- // nShow=SW_SHOWMINNOACTIVE, but three cases still need this
- // fallback:
- // 1. AUMID / UWP activations come up un-minimized —
- // ActivateApplication has no nShow.
- // 2. Apps that ignore the nShow hint and decide their own geometry
- // (notable offenders: LibreOffice, some Electron apps).
- // 3. Launcher-stub flows (LO swriter.exe → soffice.bin, GIMP, etc.)
- // where the launched pid exits before the real window appears,
- // so `windows_json` is empty at this point.
- //
- // To cover (3) we spawn a background polling task that keeps
- // looking for windows under the launched pid family for up to 5 s
- // and minimizes anything that materializes. The task runs detached
- // so the launch_app response isn't held up by it.
- //
- // SW_SHOWMINNOACTIVE preserves the foreground while minimizing. Using
- // SW_MINIMIZE here would itself activate the next z-order window and
- // force a visible restore-after-steal cycle.
- if start_minimized {
- // First, minimize anything already resolved (covers the common
- // single-process path where windows_json was populated).
- let immediate_hwnds: Vec<u64> = windows_json
- .iter()
- .filter_map(|w| w["window_id"].as_u64())
- .collect();
- // Capture pid + basename for the post-launch polling task to
- // pick up launcher-stub child processes.
- let stub_basename = target_file_opt
- .as_deref()
- .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())
- .unwrap_or("")
- .to_owned();
- // parent_pid is the launched pid family root — used below to
- // constrain the post-launch minimize sweep to descendants /
- // name-relatives of the launched app, so we don't accidentally
- // minimize a user's unrelated app that started during the 5 s
- // poll window.
- let parent_pid = pid;
- let immediate_hwnds_for_poll = immediate_hwnds.clone();
- let _ = tokio::task::spawn_blocking(move || {
- use windows::Win32::Foundation::HWND;
- use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_SHOWMINNOACTIVE};
- for h in immediate_hwnds {
- unsafe {
- let _ = ShowWindow(HWND(h as *mut _), SW_SHOWMINNOACTIVE);
- }
- }
- })
- .await;
- // Poll launcher-stub late-window cases before returning. The
- // start_minimized contract is observable at response time; a
- // detached task allowed callers to see a transient restored
- // window immediately after a successful launch.
- // Strategy: every 200 ms for 5 s, find pids that
- // (a) weren't in the pre-launch snapshot, AND
- // (b) are part of the launched app's family — either a
- // descendant of the launched root pid, OR a process
- // sharing the launched binary's name prefix (covers
- // renamed-binary chains like swriter.exe → soffice.bin
- // where the child's name doesn't match but it's a
- // direct descendant).
- // Minimize THEIR visible windows only. Without the family
- // constraint, anything the user opens during the 5 s window
- // (Chrome, terminal, IDE) would also get minimized — that's
- // the regression CodeRabbit flagged.
- //
- // Loop ends early once we've minimized the first set of
- // windows AND they remain minimized for three ticks — the typical
- // app has its main window up within ~2 s of launch.
- let pre_pids = pre_launch_pids.clone();
- let basename_for_poll = stub_basename.clone();
- let launch_foreground_lock = foreground_lock.take();
- (async move {
- use std::collections::HashSet;
- use windows::Win32::Foundation::HWND;
- use windows::Win32::UI::WindowsAndMessaging::{
- IsIconic, ShowWindow, SW_SHOWMINNOACTIVE,
- };
- let _foreground_lock = launch_foreground_lock;
- let mut minimized: HashSet<u64> = immediate_hwnds_for_poll.into_iter().collect();
- let mut idle_ticks_after_any_hit: u8 = 0;
- let mut hit_count_total = minimized.len();
- for _ in 0..25 {
- let pre_pids_clone = pre_pids.clone();
- let basename_clone = basename_for_poll.clone();
- // Build the family pid set fresh each tick — both
- // descendant graph and name-relatives can grow as
- // launcher-stub chains spawn deeper children.
- let family_new_pids: Vec<u32> = tokio::task::spawn_blocking(move || {
- let related: std::collections::HashSet<u32> =
- crate::win32::related_processes(parent_pid, &basename_clone)
- .into_iter()
- .chain(std::iter::once(parent_pid))
- .collect();
- crate::win32::list_processes()
- .into_iter()
- .filter(|p| {
- !pre_pids_clone.contains(&p.pid) && related.contains(&p.pid)
- })
- .map(|p| p.pid)
- .collect()
- })
- .await
- .unwrap_or_default();
- let mut tick_hits: usize = 0;
- for cpid in family_new_pids {
- let wins = tokio::task::spawn_blocking(move || {
- crate::win32::list_windows(Some(cpid))
- })
- .await
- .unwrap_or_default();
- for w in wins {
- let is_new = minimized.insert(w.hwnd);
- if is_new {
- hit_count_total += 1;
- }
- let hwnd_iso = w.hwnd as usize;
- let restored = tokio::task::spawn_blocking(move || unsafe {
- let hwnd = HWND(hwnd_iso as *mut _);
- if IsIconic(hwnd).as_bool() {
- false
- } else {
- let _ = ShowWindow(hwnd, SW_SHOWMINNOACTIVE);
- true
- }
- })
- .await
- .unwrap_or(false);
- if is_new || restored {
- tick_hits += 1;
- }
- }
- }
- if tick_hits == 0 {
- idle_ticks_after_any_hit += 1;
- if hit_count_total > 0 && idle_ticks_after_any_hit >= 3 {
- // Stable: known windows remained minimized and no
- // new window appeared for 600 ms.
- break;
- }
- } else {
- idle_ticks_after_any_hit = 0;
- }
- tokio::time::sleep(std::time::Duration::from_millis(200)).await;
- }
- })
- .await;
- }
-
- // Match Swift text format 1:1.
- let mut summary = format!("✅ Launched {display} (pid {pid}) in background.");
- if !windows_json.is_empty() {
- summary.push_str("\n\nWindows:");
- for w in &windows_json {
- let title = w["title"].as_str().unwrap_or("");
- let title_disp = if title.is_empty() {
- "(no title)".to_owned()
- } else {
- format!("\"{title}\"")
- };
- let wid = w["window_id"].as_u64().unwrap_or(0);
- summary.push_str(&format!("\n- {title_disp} [window_id: {wid}]"));
- }
- summary.push_str(&format!(
- "\n→ Call get_window_state(pid: {pid}, window_id) to inspect."
- ));
- }
- // `bundle_id` is the AUMID when we went through the packaged-app
- // path (so the caller can round-trip the same value to relaunch),
- // and `null` for plain Win32 launches (which truly have no bundle
- // identifier concept on Windows).
- let bundle_id_response = match aumid_for_uwp.as_ref() {
- Some(a) => serde_json::Value::String(a.clone()),
- None => serde_json::Value::Null,
- };
- let structured = json!({
- "pid": pid,
- "bundle_id": bundle_id_response,
- "name": display,
- "running": true,
- "active": false, // SW_SHOWNOACTIVATE — Swift's background-launch invariant
- "windows": windows_json,
- });
- ToolResult::text(summary).with_structured(structured)
- }
-}
-
-// ── click ─────────────────────────────────────────────────────────────────────
-
-fn posted_pixel_click_result(pid: u32, click_word: &str) -> ToolResult {
- ToolResult::text(format!("✅ Posted {click_word} to pid {pid}.")).with_structured(json!({
- "path": "post_message",
- "verified": false,
- "effect": "unverifiable"
- }))
-}
-
-/// Only a completed miss permits another transport. A provider may finish an
-/// activation after the deadline, so errors must not replay the click.
-fn finish_pixel_uia_attempt(
- outcome: crate::uia::windows_enum::PointInvokeOutcome,
- pid: u32,
- sx: i32,
- sy: i32,
-) -> Option<ToolResult> {
- use crate::uia::windows_enum::PointInvokeOutcome;
- let status = match outcome {
- PointInvokeOutcome::Miss => return None,
- PointInvokeOutcome::Invoked => {
- return Some(
- ToolResult::text(format!(
- "✅ Performed UIA Invoke at ({sx},{sy}) for pid {pid}."
- ))
- .with_structured(json!({
- "path": "ax", "verified": false, "effect": "unverifiable"
- })),
- );
- }
- PointInvokeOutcome::Busy => "busy",
- PointInvokeOutcome::Timeout => "timeout",
- PointInvokeOutcome::Unavailable => "unavailable",
- };
- Some(cua_driver_core::delivery::background_unavailable_result(
- format!(
- "UIA pixel click {status} for pid {pid}. No fallback input was sent. \
- The click effect is unknown; inspect the target state before another action. \
- If the provider does not recover, retry this action with delivery_mode:\"foreground\"."
- ),
- "background_unavailable",
- format!(
- "the UIA provider is {status}; retry this action with delivery_mode:\"foreground\"."
- ),
- json!({
- "uia_status": status,
- "path": "ax",
- "verified": false,
- "effect": "unverifiable",
- }),
- ))
-}
-
-enum BackgroundElementClick {
- Semantic {
- message: String,
- transport: ActionTransport,
- failed_calls: Vec<ActionTransport>,
- },
- Inject {
- x: i32,
- y: i32,
- failed_calls: Vec<ActionTransport>,
- },
- Post {
- x: i32,
- y: i32,
- failed_calls: Vec<ActionTransport>,
- },
-}
-
-fn background_element_click_result(
- message: String,
- transport: ActionTransport,
- failed_calls: Vec<ActionTransport>,
-) -> ToolResult {
- use cua_driver_core::action_record::{
- ActionAttempt, ActionEffect, ActionExecutionRecord, ActionFallback, ActualDelivery,
- RequestedDelivery,
- };
- let path = match transport {
- ActionTransport::WindowsTargetedInjection => "pixel",
- ActionTransport::WindowsPostMessage => "post_message",
- _ => "ax",
- };
- let mut record = ActionExecutionRecord::new(
- ActionEffect::Unverifiable,
- transport,
- RequestedDelivery::Background,
- );
- record.actual_delivery = Some(ActualDelivery::Background);
- for (index, attempted) in failed_calls.iter().copied().enumerate() {
- record.attempts.push(ActionAttempt {
- transport: attempted,
- delivery: ActualDelivery::Background,
- detail: Some("UIA provider action returned an error".into()),
- });
- record.fallbacks.push(ActionFallback {
- from: attempted,
- to: failed_calls.get(index + 1).copied().unwrap_or(transport),
- reason: "Previous UIA provider action failed".into(),
- });
- }
- ToolResult::text(message)
- .with_structured(json!({ "path": path, "verified": false, "effect": "unverifiable" }))
- .with_action_record(record)
-}
-
-pub struct ClickTool {
- state: Arc<ToolState>,
-}
-
-static CLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ClickTool {
- fn def(&self) -> &ToolDef {
- CLICK_DEF.get_or_init(|| ToolDef {
- name: "click".into(),
- // Description matches Swift `ClickTool.swift` semantics (left-click
- // primitive with two addressing modes — element_token via UIA, or
- // window-local pixel coords via PostMessage). Windows-only schema
- // extras (`button`; `action` limited to "expand"; no
- // `debug_image_out`) apply here.
- description: "Left-click against a target pid. **Prefer `element_token` over \
- pixel coordinates** — element_token works on backgrounded / minimized / \
- hidden / off-desktop windows, surfaces a stable handle that survives \
- rebuilds, and tells you what you're clicking via the cached element's \
- role + label. Reach for `x, y` only when the target is a canvas / video / \
- WebGL / custom-drawn surface that doesn't appear in the UIA tree.\n\n\
- Two addressing modes:\n\n\
- - `element_token` (from the last `get_window_state` snapshot \
- of that window) — performs the UIA Invoke pattern on the cached element via \
- PostMessage. No cursor move, no focus steal. Requires a prior \
- `get_window_state(pid, window_id)` in this turn; the token is stale once \
- the next snapshot of the same window replaces it.\n\n\
- - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
- returned by `get_window_state`). On `delivery_mode:\"background\"` (the \
- default), cua-driver FIRST does a UIA hit-test at the resolved \
- screen position: if the deepest invokable element under that point exposes \
- `InvokePattern`, it's invoked through the accessibility channel — same \
- background-safe path as the `element_token` mode, no foreground swap, no \
- visible flash. **This makes pixel clicks on UWP / WinUI3 / Win11 packaged \
- apps work flash-free out of the box** — agents should default to \
- `delivery_mode:\"background\"` even on XAML hosts whose CoreInput dispatcher \
- drops raw PostMessage. The PostMessage(WM_LBUTTONDOWN/UP) fallback only \
- runs when the UIA hit-test misses (canvas / video / WebGL / custom-drawn \
- surfaces with no UIA peer), and on those targets `delivery_mode:\"background\"` \
- returns a structured `background_unavailable` error if PostMessage is \
- known to drop too — at which point you switch to `delivery_mode:\"foreground\"`. \
- **Always try the default `background` click first and let that error be \
- your signal — do NOT pass `foreground` preemptively because the target \
- 'looks like' GTK/Chromium/etc. The driver decides when background is \
- impossible; a guessed foreground click needlessly steals the user's focus.** \
- `count: 2` posts two down/up pairs for a double-click. Pixel clicks need \
- a visible on-screen window to anchor the coordinate conversion (errors \
- with `pid X has no on-screen window` otherwise).\n\n\
- Exactly one of `element_token` or (`x` AND `y`) must be provided. \
- `pid` is required for window scope and omitted for desktop scope. After a `zoom` call, \
- pass `from_zoom=true` to auto-translate zoom-image coords back to \
- full-window space.\n\n\
- Windows-only convenience: `button: \"left\"|\"right\"|\"middle\"` switches \
- the mouse button (Swift exposes right-click as a separate `right_click` \
- tool). `action: \"expand\"` on an `element_token` opens the element \
- through UIA ExpandCollapsePattern (the dropdown half of an MSAA split \
- button); other actions use the default invoke. `debug_image_out` isn't \
- supported yet.".into(),
- input_schema: json!({
- "type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID for window scope. Optional with element_token (the token carries the pid). Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
- "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "action":{"type":"string","description":"Accessibility action for an element_token target. \"expand\" opens the element through UIA ExpandCollapsePattern, or clicks the dropdown half of an MSAA split button. Omit for the default invoke."},
- "capture_id":{"type":"string","minLength":1,"description":"Optional ID from get_window_state or get_desktop_state. When present, x and y are admitted against that exact capture and the ID is consumed before native dispatch."},
- "x":{"type":"number","description":"X in window-local screenshot pixels — same space as the PNG get_window_state returns. Must be provided together with y."},
- "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
- "button": cua_driver_core::tool_schema::button_schema(),
- "count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Coordinate frame (default \"window\"). Pass \"desktop\" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use crate::uia::snapshot::SnapshotKind;
- use cua_driver_core::tool_args::ArgsExt;
- let cursor_key = resolve_cursor_key(&args);
-
- // ── Window-less screen-absolute branch (desktop target) ───────────────
- // When the caller gives x,y with NO pid/window_id, treat x,y as TRUE
- // SCREEN pixels. The core registry has already enforced the session's
- // effective capture scope; this local check validates the action form.
- let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
- let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
- let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
- && args.get("y").map(|v| v.is_number()).unwrap_or(false);
- if !has_pid && !has_window_id && has_xy {
- if !is_windowless_desktop_action(&args) {
- return ToolResult::error(
- "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
- use get_desktop_state to pick coordinates, or pass a pid/window_id.",
- )
- .with_structured(json!({
- "code": "desktop_coordinate_scope_required",
- "suggestion": "pass scope=desktop",
- }));
- }
- let input = match parse_legacy_click_input(&args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let button = input
- .button
- .unwrap_or(ClickButton::Left)
- .as_str()
- .to_owned();
- let count = input.count.unwrap_or(1) as usize;
- if count == 0 {
- return ToolResult::error("click.count must be at least 1.")
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- let modifiers: Vec<String> = args.str_array("modifier");
- let capture_point = match self.state.capture_bridge.as_ref() {
- Some(bridge) => match bridge.admit_click(
- &args,
- crate::capture_admission::WindowsCaptureTarget::PrimaryDesktop,
- input.x,
- input.y,
- ) {
- Ok(point) => point,
- Err(error) => return capture_admission_refusal(error),
- },
- None if args.get("capture_id").is_some() => {
- return capture_admission_refusal(anyhow::anyhow!(
- "capture service is unavailable"
- ))
- }
- None => None,
- };
- let (sx, sy) = match capture_point {
- Some((action_x, action_y)) => {
- match crate::capture_admission::round_action_point(action_x, action_y) {
- Ok(point) => point,
- Err(error) => return capture_admission_refusal(error),
- }
- }
- None => (input.x as i32, input.y as i32),
- };
-
- // Resolve the application window before moving the agent cursor.
- // WindowFromPoint can return a transparent layered overlay, and the
- // cursor overlay is about to occupy this exact screen point.
- let root = unsafe {
- use windows::Win32::Foundation::POINT;
- use windows::Win32::UI::WindowsAndMessaging::{
- GetAncestor, WindowFromPoint, GA_ROOT,
- };
- let target = WindowFromPoint(POINT { x: sx, y: sy });
- if target.0.is_null() {
- return ToolResult::error(format!("No window under screen point ({sx},{sy})."));
- }
- let root = GetAncestor(target, GA_ROOT);
- if root.0.is_null() {
- target
- } else {
- root
- }
- };
- let hwnd_u = root.0 as u64;
-
- // Animate the agent cursor to the screen point, then click.
- overlay_glide_to(&cursor_key, sx as f64, sy as f64).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx as f64,
- y: sy as f64,
- },
- );
-
- // Click the HWND that owned the pixel before the driver overlay
- // moved there. The active SendInput path performs the foreground
- // swap and UIPI checks needed for Chromium and retained-mode apps.
- let send_result = tokio::task::spawn_blocking(move || -> anyhow::Result<u64> {
- let mod_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::send_click_synthesized_desktop_mods(
- hwnd_u, sx, sy, count, &button, &mod_refs,
- )?;
- Ok(hwnd_u)
- })
- .await;
- return match send_result {
- Ok(Ok(hwnd_u)) => {
- let click_word = match count {
- 2 => "double-click",
- 3 => "triple-click",
- _ => "click",
- };
- ToolResult::text(format!(
- "✅ Sent {click_word} via SendInput at screen ({sx},{sy}) on HWND 0x{hwnd_u:x} (desktop scope)."
- ))
- // SendInput dispatched but is not driver-verifiable (no read-back).
- .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" }))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let x = args.opt_f64("x");
- let y = args.opt_f64("y");
- // Surface 5: explicit rejection of unknown buttons so a typo doesn't fall
- // through to the SendInput `_ => MOUSEEVENTF_LEFTDOWN/UP` default and
- // silently emit a left-click. Empty / missing keeps back-compat.
- let button_raw = args.str_or("button", "left").to_lowercase();
- if !matches!(button_raw.as_str(), "" | "left" | "right" | "middle") {
- return ToolResult::error(format!(
- "click: unknown button \"{button_raw}\" — expected one of left, right, middle."
- ));
- }
- let button = if button_raw.is_empty() {
- "left".to_string()
- } else {
- button_raw
- };
- let count = args.u64_or("count", 1) as usize;
- // macOS click `modifier` surface: held keys (cmd/shift/option/ctrl). Only
- // the SendInput rung (delivery_mode:"foreground" pixel tap, and the
- // window-less desktop-scope click above) can hold them; the background
- // UIA-Invoke / PostMessage rungs have no keyboard state to carry them, so
- // a modifier passed to a background click is necessarily ignored there.
- let modifiers: Vec<String> = args.str_array("modifier");
- let delivery = DeliveryMode::from_args(&args);
- if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {
- return ToolResult::error(
- "click modifiers require delivery_mode:\"foreground\" on Windows; UIA and \
- PostMessage cannot carry live modifier-key state",
- )
- .with_structured(json!({
- "code": "background_unavailable",
- "effect": "refused",
- "escalation": {
- "recommended": "foreground",
- "reason": "Windows modifier clicks require SendInput so the target observes live modifier state"
- }
- }));
- }
- // Optional `action` arg picks among the actions exposed in the
- // accessibility tree. Today this only changes behavior for MSAA
- // BUTTONDROPDOWN: `"expand"` clicks the right-edge (dropdown arrow
- // half) instead of the center (press half). Defaults to first
- // action in the element's `actions=[...]` list, which preserves
- // existing semantics for UIA elements.
- let action_req = args
- .get("action")
- .and_then(|v| v.as_str())
- .map(|s| s.to_string());
-
- // Resolve HWND: explicit, or auto from pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows_via_win32(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- let capture_point = if args.get("capture_id").is_some() {
- if elem_idx.is_some()
- || x.is_none()
- || y.is_none()
- || hwnd_opt.is_none()
- || args.get("from_zoom").and_then(Value::as_bool) == Some(true)
- {
- return capture_admission_refusal(anyhow::anyhow!(
- "capture_id requires x, y, and an exact window_id; element and zoom coordinates are not capture-bound"
- ));
- }
- let Some(bridge) = &self.state.capture_bridge else {
- return capture_admission_refusal(anyhow::anyhow!(
- "capture service is unavailable"
- ));
- };
- match bridge.admit_click(
- &args,
- crate::capture_admission::WindowsCaptureTarget::Window {
- pid,
- window_id: hwnd,
- },
- x.unwrap(),
- y.unwrap(),
- ) {
- Ok(Some(point)) => Some(point),
- Ok(None) => unreachable!("capture_id was present"),
- Err(error) => return capture_admission_refusal(error),
- }
- } else {
- None
- };
-
- // Arm this only after capture admission so stale or mismatched IDs have
- // no native side effect before the click is refused.
- let _noact = match (delivery != DeliveryMode::Foreground, hwnd_opt) {
- (true, Some(h)) => Some(crate::input::NoActivateGuard::arm(
- windows::Win32::Foundation::HWND(h as *mut _),
- )),
- _ => None,
- };
-
- if let Some(idx) = elem_idx {
- // ── MSAA dispatch (SAL/VCL targets) ────────────────────────────
- // For MSAA elements, route by role:
- // - `action:"expand"` on BUTTONDROPDOWN → SendInput at the
- // cached rect's right-edge (the dropdown arrow half).
- // Opens the picker (e.g. LO Writer Font Color → SALTMPSUBFRAME).
- // - default action / `"invoke"` → SendInput at center
- // (matches `accDoDefaultAction` semantics for VCL controls
- // and works through delivery_mode:"foreground"). We DO NOT call
- // `accDoDefaultAction` here because LO's MSAA impl applies
- // the change asynchronously and returns S_OK either way,
- // so the visible behavior is the same as a center click.
- if let Some((SnapshotKind::Msaa, role)) = admitted
- .as_ref()
- .map(|element| (element.kind, element.msaa_role))
- {
- const ROLE_BUTTONDROPDOWN: i32 = 0x38;
- const ROLE_BUTTONMENU: i32 = 0x39;
- const ROLE_BUTTONDROPDOWNGRID: i32 = 0x3A;
- const ROLE_SPLITBUTTON: i32 = 0x3E;
- let want_expand = action_req.as_deref() == Some("expand");
- let is_dropdown_role = matches!(
- role,
- Some(
- ROLE_BUTTONDROPDOWN
- | ROLE_BUTTONMENU
- | ROLE_BUTTONDROPDOWNGRID
- | ROLE_SPLITBUTTON
- )
- );
- let (tx, ty) = if want_expand && is_dropdown_role {
- match admitted.as_ref().and_then(|element| element.rect) {
- Some((_l, t, r, b)) => {
- // Right-edge of the SplitButton — the dropdown
- // arrow half. -4 puts the click safely inside
- // the arrow region for the typical ~12-16 px
- // arrow width VCL uses.
- (r - 4, (t + b) / 2)
- }
- None => {
- return ToolResult::error(format!(
- "MSAA element [{idx}] has no cached rect — \
- cannot dispatch action:\"expand\". Re-run \
- get_window_state to refresh the cache."
- ));
- }
- }
- } else if want_expand && !is_dropdown_role {
- return ToolResult::error(format!(
- "action:\"expand\" requested for MSAA element [{idx}] \
- but its role ({role:?}) is not a dropdown button. \
- Use action:\"invoke\" or omit the action arg."
- ));
- } else {
- match admitted.as_ref().map(|element| element.center) {
- Some(v) => v,
- None => {
- return ToolResult::error(format!(
- "MSAA element [{idx}] not in cache for hwnd={hwnd}. \
- Call get_window_state first."
- ))
- }
- }
- };
- if admitted
- .as_ref()
- .is_some_and(|element| element.read_while_minimized)
- {
- if let Err(refusal) =
- resolve_onscreen_point_with_scroll(&admitted, hwnd, idx, tx, ty, "clicking")
- {
- return refusal;
- }
- }
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- tx as f64,
- ty as f64,
- element_target_rect(admitted.as_ref(), (0, 0)),
- )
- .await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: tx as f64,
- y: ty as f64,
- },
- );
- let btn_fg = button.clone();
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let mods_owned = modifiers.clone();
- let activate = delivery == DeliveryMode::Foreground;
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- if activate {
- crate::input::send_click_synthesized_active_mods(
- hwnd, tx, ty, count, &btn_fg, &mod_refs,
- )
- } else {
- crate::input::send_click_synthesized_mods(
- hwnd, tx, ty, count, &btn_fg, &mod_refs,
- )
- }
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- let half = if want_expand { "dropdown" } else { "press" };
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Performed SendInput click on MSAA [{idx}] {half} half at ({tx},{ty})."
- ))
- .with_structured(
- json!({ "path": "msaa", "verified": false, "effect": "unverifiable" }),
- ),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // UIA path: get cached center (no COM call needed — captured at walk time).
- let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
- Some(v) => v,
- None => {
- return ToolResult::error(format!(
- "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
- ))
- }
- };
- // NB: the off-screen guard is applied per-delivery-path below (the
- // foreground SendInput tap and the background coordinate injection),
- // NOT here — a background UIA Invoke fires the element's handler
- // regardless of whether the element is scrolled into view, so a broad
- // guard would wrongly block legitimate actions like opening a modal
- // from an off-screen button.
- // Step 2: pin overlay to target window, then animate to screen coords.
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(admitted.as_ref(), (0, 0)),
- )
- .await;
- // Step 3: click pulse + actual click.
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: cx as f64,
- y: cy as f64,
- },
- );
- let btn = button.clone();
-
- // An explicit accessibility action is a semantic request, not a
- // pixel gesture hint. Route expand through ExpandCollapsePattern
- // even when foreground delivery was allowed; this reliably opens
- // WPF/WinUI menus and tree nodes whose visual click target is
- // transient or scroll-adjusted.
- if action_req.as_deref() == Some("expand") {
- let expand = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || -> anyhow::Result<()> {
- let _admission = &admitted;
- use windows::core::Interface;
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationExpandCollapsePattern,
- UIA_ExpandCollapsePatternId,
- };
-
- let retained = admitted.as_ref().ok_or_else(|| {
- anyhow::anyhow!("element [{idx}] is not in the UIA cache")
- })?;
- if !retained.is_uia() {
- anyhow::bail!("element [{idx}] is not a UIA element");
- }
- let element = std::mem::ManuallyDrop::new(unsafe {
- IUIAutomationElement::from_raw(retained.as_ptr() as *mut _)
- });
- let pattern = unsafe {
- element
- .GetCurrentPattern(UIA_ExpandCollapsePatternId)
- .and_then(|value| {
- value.cast::<IUIAutomationExpandCollapsePattern>()
- })
- }
- .map_err(|error| {
- anyhow::anyhow!("ExpandCollapsePattern unavailable: {error}")
- })?;
- let result =
- crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- pattern.Expand()
- });
- std::mem::forget(element);
- result.map_err(|error| {
- anyhow::anyhow!("ExpandCollapse.Expand failed: {error}")
- })
- }
- })
- .await;
- return match expand {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Expanded UIA element [{idx}] via ExpandCollapsePattern."
- ))
- .with_structured(json!({ "path": "uia_expand_collapse", "verified": false, "effect": "unverifiable" })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
-
- // delivery_mode:"foreground" — skip UIA Invoke and use SendInput at the
- // cached element center. The caller explicitly chose foreground
- // delivery; UIA Invoke would be background-safe (which they
- // rejected) and might fail on elements that don't support
- // InvokePattern anyway.
- if delivery == DeliveryMode::Foreground {
- // Foreground delivery is a real SendInput tap at (cx,cy); if the
- // element is off-screen, scroll it into view and re-resolve so the
- // tap doesn't land on the taskbar, else keep the clean failure.
- // Modified selection needs one stronger preflight: some WPF
- // ScrollViewers report a clipped item as on-screen because its
- // stale rectangle is still inside the outer HWND. Ask the item
- // to scroll itself into view before trusting that rectangle.
- let recorded_center = (cx, cy);
- let (cx, cy) = if !modifiers.is_empty() {
- admitted
- .clone()
- .and_then(|retained| {
- retained.is_uia().then(|| unsafe {
- crate::uia::scroll::scroll_into_view_and_recenter(
- hwnd,
- retained.as_ptr(),
- )
- })
- })
- .flatten()
- .unwrap_or((cx, cy))
- } else {
- (cx, cy)
- };
- let (cx, cy) = match resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx,
- cx,
- cy,
- "a foreground click",
- ) {
- Ok(p) => p,
- Err(result) => return result,
- };
- if (cx, cy) != recorded_center {
- crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
- }
- let btn_fg = btn.clone();
- let mods_owned = modifiers.clone();
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- crate::input::send_click_synthesized_active_mods(
- hwnd, cx, cy, count, &btn_fg, &mod_refs,
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Performed SendInput click on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
- ))
- .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // UIA and posted-message routes can report success without a
- // user-visible effect on an iconic target. Refuse before any
- // background route selects or dispatches its transport.
- if let Some(result) = minimized_element_refusal(hwnd, idx, cx, cy, "clicking") {
- return result;
- }
- // delivery_mode:"background" on WinUI3 for double / right / middle: single
- // left falls through to the UIA Invoke path below (already drives
- // WinUI3). Double-left lands via a double UIA Invoke; right/middle
- // can't both land and hold the contract → structured error.
- if delivery == DeliveryMode::Background
- && (count > 1 || btn == "right" || btn == "middle")
- {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, Some(idx), count, &btn).await
- {
- return r;
- }
- }
- // Chromium's UIA Invoke raises a fully occluded renderer, while a
- // targeted PostMessage left click reaches the renderer without
- // changing foreground, z-order, or the real cursor. Keep AX for
- // target resolution and use the posted-message transport only for
- // the empirically verified single-left-click shape.
- if delivery == DeliveryMode::Background
- && btn == "left"
- && count == 1
- && crate::input::is_chromium_target_window(hwnd)
- {
- // This route posts at the cached center without
- // `resolve_onscreen_point_with_scroll`, so check it here.
- if admitted
- .as_ref()
- .is_some_and(|element| element.read_while_minimized)
- {
- let message = format!(
- "Element [{idx}] resolves to ({cx},{cy}), which was read while \
- window 0x{hwnd:x} was minimized. Re-snapshot with get_window_state \
- before retrying."
- );
- return ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "code": "element_not_visible",
- "effect": "refused",
- "refusal": {
- "code": "element_not_visible",
- "message": message,
- }
- }));
- }
- let posted = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, cx, cy, count, &btn)
- }
- })
- .await;
- return match posted {
- Ok(Ok(())) => background_element_click_result(
- format!(
- "✅ Posted click on Chromium element [{idx}] at screen ({cx},{cy}) \
- (background, no foreground swap)."
- ),
- ActionTransport::WindowsPostMessage,
- vec![],
- ),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- // Try UIA Invoke first (it works for UWP / modern XAML / web
- // content where PostMessage(WM_LBUTTONDOWN) hits the outer
- // HWND but never reaches the inner XAML/composition element).
- // Fall through to PostMessage when:
- // - element doesn't support InvokePattern (most text inputs,
- // non-interactive elements)
- // - right-click (UIA's ExpandCollapse / ShowContextMenu are
- // different patterns; keep PostMessage for now)
- // - count > 1 (double-click semantics aren't an Invoke
- // concept — PostMessage produces the actual WM_LBUTTONDBLCLK)
- let use_uia_invoke = (btn == "left" || btn == "middle") && count == 1;
- let result = tokio::task::spawn_blocking({ let admitted = admitted.clone(); move || -> anyhow::Result<BackgroundElementClick> {
- let _admission = &admitted;
- let mut failed_calls = Vec::new();
- // Direct Chromium UIA Invoke can return S_OK without firing a
- // DOM event while occluded. Try the honest coordinate actuator
- // first: it lands while visible and reports occlusion without
- // raising the window when hidden.
- if delivery == DeliveryMode::Background
- && crate::input::is_chromium_target_window(hwnd)
- {
- let (cx, cy) = resolve_onscreen_point_with_scroll(
- &admitted, hwnd, idx, cx, cy, "clicking",
- )
- .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
- return Ok(BackgroundElementClick::Inject { x: cx, y: cy, failed_calls });
- }
- if use_uia_invoke {
- // Retain the element out of the cache (AddRef under the
- // cache lock) so it can't be freed by a concurrent
- // get_window_state on the same (pid, hwnd) while this click
- // is mid-flight (use-after-free → daemon crash). The guard
- // lives to the end of this `if let` block, past every UIA
- // pattern dispatch below; its Release fires when it drops.
- if let Some(element_guard) = admitted.as_ref() {
- let ptr = element_guard.as_ptr();
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationInvokePattern,
- IUIAutomationTogglePattern, IUIAutomationSelectionItemPattern,
- IUIAutomationExpandCollapsePattern,
- UIA_InvokePatternId, UIA_TogglePatternId,
- UIA_SelectionItemPatternId, UIA_ExpandCollapsePatternId,
- };
- use windows::core::Interface;
- let elem: IUIAutomationElement =
- unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
-
- // Try patterns in order of click-semantics specificity:
- // 1. Invoke - buttons / hyperlinks (canonical)
- // 2. Toggle - checkboxes (no Invoke)
- // 3. SelectionItem - radio buttons, listbox items
- // 4. ExpandCollapse - combo boxes, tree nodes
- // Each call is wrapped in the UWP foreground-steal bypass.
- //
- // Why this order: PR #1699's WinUI3 control parity tests
- // surfaced these pattern-dispatch gaps. Without these
- // fallthroughs, cua-driver couldn't drive WinUI3
- // CheckBox / RadioButton / ComboBox because PostMessage
- // doesn't reach their handlers (CoreInput dispatcher).
- let invoke_result = unsafe { elem.GetCurrentPattern(UIA_InvokePatternId) };
- if let Ok(pattern) = invoke_result {
- if let Ok(inv) = pattern.cast::<IUIAutomationInvokePattern>() {
- let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
- hwnd as isize, || unsafe { inv.Invoke() });
- match outcome {
- Ok(()) => {
- std::mem::forget(elem);
- return Ok(BackgroundElementClick::Semantic { message: format!(
- "✅ Performed UIA Invoke on [{idx}] (screen ({cx},{cy}))."
- ), transport: ActionTransport::WindowsUiaInvoke, failed_calls });
- }
- Err(e) => tracing::debug!(target: "click",
- "UIA Invoke on [{idx}]: {e}, trying Toggle"),
- }
- failed_calls.push(ActionTransport::WindowsUiaInvoke);
- }
- }
- let toggle_result = unsafe { elem.GetCurrentPattern(UIA_TogglePatternId) };
- if let Ok(pattern) = toggle_result {
- if let Ok(tg) = pattern.cast::<IUIAutomationTogglePattern>() {
- let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
- hwnd as isize, || unsafe { tg.Toggle() });
- if outcome.is_ok() {
- std::mem::forget(elem);
- return Ok(BackgroundElementClick::Semantic { message: format!(
- "✅ Performed UIA Toggle on [{idx}] (screen ({cx},{cy}))."
- ), transport: ActionTransport::WindowsUiaToggle, failed_calls });
- }
- failed_calls.push(ActionTransport::WindowsUiaToggle);
- }
- }
- let sel_result = unsafe { elem.GetCurrentPattern(UIA_SelectionItemPatternId) };
- if let Ok(pattern) = sel_result {
- if let Ok(si) = pattern.cast::<IUIAutomationSelectionItemPattern>() {
- let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
- hwnd as isize, || unsafe { si.Select() });
- if outcome.is_ok() {
- std::mem::forget(elem);
- return Ok(BackgroundElementClick::Semantic { message: format!(
- "✅ Performed UIA SelectionItem.Select on [{idx}] (screen ({cx},{cy}))."
- ), transport: ActionTransport::WindowsUiaSelection, failed_calls });
- }
- failed_calls.push(ActionTransport::WindowsUiaSelection);
- }
- }
- let exp_result = unsafe { elem.GetCurrentPattern(UIA_ExpandCollapsePatternId) };
- if let Ok(pattern) = exp_result {
- if let Ok(ec) = pattern.cast::<IUIAutomationExpandCollapsePattern>() {
- let outcome = crate::uia::fg_bypass::run_with_uwp_bypass(
- hwnd as isize, || unsafe { ec.Expand() });
- if outcome.is_ok() {
- std::mem::forget(elem);
- return Ok(BackgroundElementClick::Semantic { message: format!(
- "✅ Performed UIA ExpandCollapse.Expand on [{idx}] (screen ({cx},{cy}))."
- ), transport: ActionTransport::WindowsUiaExpandCollapse, failed_calls });
- }
- failed_calls.push(ActionTransport::WindowsUiaExpandCollapse);
- }
- }
- std::mem::forget(elem);
- }
- }
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(
- hwnd,
- EventKind::MouseClick,
- )
- {
- let (cx, cy) = resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx,
- cx,
- cy,
- "clicking",
- )
- .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
- return Ok(BackgroundElementClick::Inject { x: cx, y: cy, failed_calls });
- }
- if admitted.as_ref().is_some_and(|element| element.read_while_minimized) {
- let (x, y) =
- resolve_onscreen_point_with_scroll(&admitted, hwnd, idx, cx, cy, "clicking")
- .map_err(|result| anyhow::anyhow!(tool_result_text(result)))?;
- return Ok(BackgroundElementClick::Post { x, y, failed_calls });
- }
- Ok(BackgroundElementClick::Post { x: cx, y: cy, failed_calls })
- } }).await;
- let plan = match result {
- Ok(Ok(plan)) => plan,
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("Task error: {e}")),
- };
- let (x, y, failed_calls, inject) = match plan {
- BackgroundElementClick::Semantic {
- message,
- transport,
- failed_calls,
- } => {
- return background_element_click_result(message, transport, failed_calls);
- }
- BackgroundElementClick::Inject { x, y, failed_calls } => (x, y, failed_calls, true),
- BackgroundElementClick::Post { x, y, failed_calls } => (x, y, failed_calls, false),
- };
- // Return to the dispatch task before capturing: task-local recording
- // context does not propagate into the blocking scroll resolver.
- if (x, y) != (cx, cy) {
- crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, x, y);
- }
- let btn = button.clone();
- let action_name = if btn == "right" {
- "ShowMenu"
- } else {
- "PostMessage click"
- };
- let sent = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- if inject {
- crate::input::inject_click_screen(hwnd, x, y, count, &btn)
- } else {
- crate::input::post_click_screen(hwnd, x, y, count, &btn)
- }
- }
- })
- .await;
- match sent {
- Ok(Ok(())) => {
- let (message, transport) = if inject {
- (format!("✅ Injected click on [{idx}] (screen ({x},{y}), background, no foreground swap)."),
- ActionTransport::WindowsTargetedInjection)
- } else {
- (
- format!("✅ Performed {action_name} on [{idx}] (screen ({x},{y}))."),
- ActionTransport::WindowsPostMessage,
- )
- };
- background_element_click_result(message, transport, failed_calls)
- }
- Ok(Err(error)) if inject => {
- crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- error.to_string(),
- )
- }
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- }
- } else if let (Some(mut px), Some(mut py)) = (x, y) {
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if let Some((mapped_x, mapped_y)) = capture_point {
- let (mapped_x, mapped_y) =
- match crate::capture_admission::round_action_point(mapped_x, mapped_y) {
- Ok(point) => point,
- Err(error) => return capture_admission_refusal(error),
- };
- px = f64::from(mapped_x);
- py = f64::from(mapped_y);
- } else if from_zoom {
- match self.state.zoom_context(&args, pid, Some(hwnd)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(px, py);
- px = wx;
- py = wy;
- }
- Err(refusal) => return refusal,
- }
- } else if !args.bool_or("_native_coordinates", false) {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- px *= ratio;
- py *= ratio;
- }
- // px/py are bitmap pixels — see `bitmap_to_screen` for the
- // mapping (DWM-frame top-left + 1-px inset, NOT ClientToScreen).
- let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
- let (sx, sy) = (sx_i as f64, sy_i as f64);
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to(&cursor_key, sx, sy).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- let btn = button.clone();
- // Vision-mode (x, y) dispatch is **layered**:
- //
- // 1. UIA hit-test in the target HWND's subtree. If the
- // deepest InvokePattern-bearing element at (sx, sy) is
- // found, fire `Invoke()`. Works occluded, no focus
- // steal, and is the **only** path that lands on UWP /
- // WebView2 / DirectComposition surfaces — those route
- // input through `Windows.UI.Input`, not WM_LBUTTONDOWN.
- //
- // 2. If UIA completes with a miss (no Invokable element under the
- // pixel inside `hwnd`, or `hwnd` has no useful UIA
- // tree at all), fall through to
- // `PostMessage(WM_LBUTTONDOWN/UP)` against the deepest
- // child HWND at the screen point. Covers plain Win32
- // controls without UIA InvokePattern (most edit
- // fields, custom-drawn widgets, native containers) and
- // apps that don't expose UIA at all.
- //
- // The combination closes the gap macOS gets for free from
- // `CGEventPostToPSN` (per-pid event routing): UIA covers
- // UWP-style targets without z-order tricks, PostMessage
- // covers plain Win32 without z-order tricks. Neither path
- // moves the OS cursor or activates the receiver.
- //
- // UIA path runs only for single left/middle clicks —
- // right-click (UIA has no by-coord `ShowContextMenu`) and
- // multi-click (`Invoke()` is single-fire by definition)
- // skip directly to PostMessage.
- // delivery_mode:"foreground" short-circuit — caller explicitly chose
- // foreground delivery. Skip the UIA hit-test (UIA Invoke is
- // background-safe and would deliver the click without the
- // foreground swap the caller asked for).
- if delivery == DeliveryMode::Foreground {
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let mods_owned = modifiers.clone();
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let mod_refs: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- crate::input::send_click_synthesized_active_mods(
- hwnd, sx as i32, sy as i32, count, &btn, &mod_refs,
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => {
- let click_word = match count {
- 2 => "double-click",
- 3 => "triple-click",
- _ => "click",
- };
- ToolResult::text(format!(
- "✅ Sent {click_word} via SendInput to pid {pid} at ({sx},{sy}) (delivery_mode:foreground)."
- ))
- .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" }))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // delivery_mode:"background" on WinUI3 for double / right / middle (pixel
- // path, no element_token → no cached element to UIA-Invoke): these
- // can't both land and hold the contract on WinUI3 (posted input is
- // ignored, the pen injector steals foreground) → structured error.
- if delivery == DeliveryMode::Background
- && (count > 1 || btn == "right" || btn == "middle")
- {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, None, count, &btn).await
- {
- return r;
- }
- }
- // Match the AX-addressed Chromium route above: the point remains
- // PX-resolved, but transport uses the background-safe posted
- // message path proven against the fully occluded fixture.
- if delivery == DeliveryMode::Background
- && btn == "left"
- && count == 1
- && crate::input::is_chromium_target_window(hwnd)
- {
- let posted = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, sx_i, sy_i, count, &btn)
- }
- })
- .await;
- return match posted {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Posted click to Chromium pid {pid} at ({sx},{sy}) \
- (background, no foreground swap)."
- ))
- .with_structured(json!({
- "path": "post_message",
- "verified": false,
- "effect": "unverifiable"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- // As above, bypass Chromium's false-positive UIA Invoke and use
- // the coordinate actuator before attempting any accessibility hit
- // test. The actuator itself distinguishes visible delivery from a
- // fully occluded structured refusal.
- if delivery == DeliveryMode::Background && crate::input::is_chromium_target_window(hwnd)
- {
- let btn2 = btn.clone();
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, sx as i32, sy as i32, count, &btn2)
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected click to pid {pid} at ({sx},{sy}) (background, no foreground swap)."
- ))
- .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
- Ok(Err(error)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- error.to_string(),
- ),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
- let use_uia = (btn == "left" || btn == "middle") && count == 1;
- if use_uia {
- let outcome = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::uia::windows_enum::try_invoke_in_window_at_point(
- hwnd as isize,
- sx as i32,
- sy as i32,
- )
- }
- })
- .await
- .unwrap_or(crate::uia::windows_enum::PointInvokeOutcome::Unavailable);
- if let Some(result) = finish_pixel_uia_attempt(outcome, pid, sx_i, sy_i) {
- return result;
- }
- }
-
- // UIA completed with a miss or was not applicable. Known dropped surfaces other than direct
- // Chromium (handled above) get one targeted injection attempt.
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
- {
- let btn2 = btn.clone();
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, sx as i32, sy as i32, count, &btn2)
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected click to pid {pid} at ({sx},{sy}) (background, no foreground swap)."
- ))
- .with_structured(json!({ "path": "pixel", "verified": false, "effect": "unverifiable" })),
- Ok(Err(error)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- error.to_string(),
- ),
- Err(error) => ToolResult::error(format!("Task error: {error}")),
- };
- }
-
- // delivery_mode:"background", non-dropped + non-UIA click. This
- // includes plain Win32 targets with no invokable UIA element.
- // Attempt PostMessage without a
- // foreground swap; the caller must verify the resulting fixture state.
- // Foreground was handled at the top, and known dropped surfaces use
- // the targeted-injection path above.
-
- // bitmap pixels -> screen (DWM-frame origin + inset). Use
- // post_click_screen so we don't double-ClientToScreen.
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, sx_i, sy_i, count, &btn)
- }
- })
- .await;
- match result {
- Ok(Ok(())) => {
- // Match Swift text format: "✅ Posted click/double-click/triple-click to pid X."
- let click_word = match count {
- 2 => "double-click",
- 3 => "triple-click",
- _ => "click",
- };
- posted_pixel_click_result(pid, click_word)
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- } else {
- // Match Swift's error wording for the missing-target case.
- ToolResult::error("Provide element_token or (x, y) to address the click target.")
- }
- }
-}
-
-#[cfg(test)]
-mod pixel_click_transport_tests;
-
-#[cfg(test)]
-mod background_element_click_record_tests;
-
-// ── type_text ─────────────────────────────────────────────────────────────────
-
-/// px-focus for the keyboard family (type_text / press_key / hotkey): pixel-click
-/// at (x,y) to establish real renderer focus before a keystroke — the *element px
-/// action* form of a keyboard tool. Reuses ClickTool's exact coordinate
-/// translation + delivery_mode (UIA hit-test → PostMessage fallback), so it lands
-/// on the same pixel a px-click would. `Ok(())` on success; `Err(ToolResult)`
-/// short-circuits the caller. Mirrors macOS `tools::focus_by_pixel`.
-#[allow(clippy::too_many_arguments)]
-fn focus_by_pixel_click_args(
- pid: u32,
- window_id: Option<u64>,
- x: f64,
- y: f64,
- foreground: bool,
- session: Option<String>,
- session_id: Option<String>,
- from_zoom: bool,
- native_coordinates: bool,
-) -> Value {
- let mut click_args = json!({
- "pid": pid, "x": x, "y": y,
- "delivery_mode": if foreground { "foreground" } else { "background" },
- });
- if let Some(wid) = window_id {
- click_args["window_id"] = json!(wid);
- }
- if let Some(s) = session {
- click_args["session"] = json!(s);
- }
- if let Some(s) = session_id {
- click_args["_session_id"] = json!(s);
- }
- if from_zoom {
- click_args["from_zoom"] = json!(true);
- }
- if native_coordinates {
- click_args["_native_coordinates"] = json!(true);
- }
- click_args
-}
-
-#[allow(clippy::too_many_arguments)]
-async fn focus_by_pixel(
- state: &Arc<ToolState>,
- pid: u32,
- window_id: Option<u64>,
- x: f64,
- y: f64,
- foreground: bool,
- session: Option<String>,
- session_id: Option<String>,
- from_zoom: bool,
- native_coordinates: bool,
-) -> Result<(), ToolResult> {
- let click_args = focus_by_pixel_click_args(
- pid,
- window_id,
- x,
- y,
- foreground,
- session,
- session_id,
- from_zoom,
- native_coordinates,
- );
- let focus = ClickTool {
- state: state.clone(),
- }
- .invoke(click_args)
- .await;
- if focus.is_error == Some(true) {
- // Preserve the click tool's structured background refusal (for example
- // background_occluded / background_uipi_blocked). Re-wrapping it as a
- // text-only error made keyboard-family PX calls lose the actionable
- // capability result produced by the actual actuator.
- return Err(focus);
- }
- // Brief settle so the renderer registers focus before the keystrokes.
- tokio::time::sleep(std::time::Duration::from_millis(120)).await;
- Ok(())
-}
-
-pub struct TypeTextTool {
- state: Arc<ToolState>,
-}
-
-fn wait_for_cached_element_keyboard_focus(
- admitted: &Option<crate::uia::snapshot::RetainedElement>,
- timeout: std::time::Duration,
-) -> bool {
- let deadline = std::time::Instant::now() + timeout;
- loop {
- if admitted
- .as_ref()
- .and_then(|element| element.element_has_keyboard_focus())
- == Some(true)
- {
- return true;
- }
- if std::time::Instant::now() >= deadline {
- return false;
- }
- std::thread::sleep(std::time::Duration::from_millis(10));
- }
-}
-
-/// Establish exact UIA child focus after the owning top-level HWND is already
-/// confirmed foreground. Chromium providers sometimes reject UIA `SetFocus`;
-/// in that case a real click at the accessibility-derived center is the
-/// bounded fallback. Both routes require `CurrentHasKeyboardFocus` read-back
-/// before any keyboard input is allowed to leave the driver.
-fn focus_cached_element_for_foreground(
- admitted: &Option<crate::uia::snapshot::RetainedElement>,
- hwnd: u64,
- element_index: usize,
- click_point: Option<(i32, i32)>,
-) -> anyhow::Result<()> {
- // `fg_bypass` is a background-only shield: for Chromium it disables the
- // top-level HWND while UIA runs so the provider cannot self-activate. A
- // disabled foreground window immediately loses foreground on Windows,
- // which would make the enclosing verify-before-SendInput transaction
- // fail closed. The foreground route has already activated the exact HWND,
- // so focus the cached element directly and verify it below.
- let set_focus = admitted
- .as_ref()
- .ok_or_else(|| anyhow::anyhow!("missing admitted element"))
- .and_then(|element| element.focus_element());
- if set_focus.is_ok()
- && wait_for_cached_element_keyboard_focus(admitted, std::time::Duration::from_millis(350))
- {
- return Ok(());
- }
-
- if let Some((x, y)) = click_point {
- crate::input::send_click_synthesized_active_mods(hwnd, x, y, 1, "left", &[])?;
- if wait_for_cached_element_keyboard_focus(admitted, std::time::Duration::from_millis(500)) {
- return Ok(());
- }
- }
-
- let detail = set_focus
- .err()
- .map(|error| format!("; UIA SetFocus failed: {error}"))
- .unwrap_or_default();
- anyhow::bail!(
- "foreground_unavailable: Windows did not confirm keyboard focus on exact UIA element \
- [{element_index}] in HWND 0x{hwnd:x}{detail}; no keyboard input was sent"
- )
-}
-
-static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for TypeTextTool {
- fn def(&self) -> &ToolDef {
- TYPE_DEF.get_or_init(|| ToolDef {
- name: "type_text".into(),
- // Description ported from Swift `TypeTextTool.swift` with the
- // Windows transport note. Swift tries AXSetAttribute(kAXSelectedText)
- // first, falls back to character-by-character CGEvent synthesis.
- // Windows uses character-by-character PostMessage(WM_CHAR) to the
- // window's focused child — there's no UIA equivalent of
- // AXSelectedText wired up yet, so it's always the synthesis path.
- description: "Insert text into the target pid via character-by-character \
- PostMessage(WM_CHAR) to the focused window. No focus steal.\n\n\
- Special keys (Return, Escape, arrows, Tab) go through `press_key` / \
- `hotkey` — they are not text.\n\n\
- **Routing on Windows.** `element_token` selects a specific UIA element \
- and writes its ValuePattern value. For no-element foreground typing, a \
- separate capability-gated path can use the exact focused UIA Document \
- only when it is writable and empty. \
- Eligibility depends on control capabilities and live focus, not app names. \
- Chromium/Electron windows keep their existing boundary because a UIA \
- value echo does not prove that the renderer accepted the text. The \
- path reads the value and caret back; after SetValue starts, an \
- unverifiable result requires a fresh observation before any retry and \
- is never replayed through native input. This replaces the full \
- ValuePattern value, so rich-text formatting and native undo behavior \
- are not guaranteed; single-line Edit controls \
- keep the existing input route. For other XAML / WinUI3 / UWP cases, \
- use `element_token` from `get_window_state`; background WM_CHAR does \
- not reach their CoreInput dispatcher. Native ConsoleHost on Windows ARM64 is \
- hard-refused because it can \
- accept synthesized Unicode events without delivering them; use a process \
- or PTY setup channel instead. Legacy Win32 apps still use the PostMessage \
- path, preserving the no-focus-steal property.\n\n\
- `delay_ms` (0–200, default 30) spaces successive characters on the \
- PostMessage path so autocomplete and IME can keep up. Ignored on the \
- UIA path (SetValue is atomic).".into(),
- input_schema: json!({
- "type":"object","required":["text"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to type into the current foreground application."},
- "pid":{"type":"integer","description":"Target process ID."},
- "text":{"type":"string","description":"Text to insert at the focused element's cursor."},
- "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied (the token carries it)."},
- "element_token":{"type":"string","pattern":"^s[0-9a-f]{8}:[0-9]+$","description":"Opaque element handle from get_window_state. When supplied, type_text writes through UIA ValuePattern and reads that exact element back by handle. The shared ActionResult is confirmed only when the complete expected value is synchronously visible. If SetValue succeeds but read-back is stale or unavailable, the result is unverifiable with no escalation; take a fresh snapshot before retrying because deferred providers may publish only after this call returns."},
- "x":{"type":"number","description":"Window-local screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the UIA/WM_CHAR path can't reach. Read straight off the get_window_state PNG, same convention as click."},
- "y":{"type":"number","description":"Window-local screenshot-pixel Y of the field (see x)."},
- "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use cua_driver_core::tool_args::ArgsExt;
- let cursor_key = resolve_cursor_key(&args);
- if args.get("scope").and_then(Value::as_str) == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<TypeTextInput>("type_text", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let text =
- cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&input.text)
- .into_owned();
- let hwnd = match crate::input::mouse::foreground_window() {
- Ok(hwnd) => hwnd,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- let text_len = text.chars().count();
- if args.get("element_token").is_none()
- && args.get("x").is_none()
- && args.get("y").is_none()
- {
- if let Some(result) = try_focused_document_route(hwnd, None, &text, text_len).await
- {
- return result;
- }
- }
- return match tokio::task::spawn_blocking(move || {
- crate::input::send_text_synthesized(hwnd, &text)
- })
- .await
- {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) into the foreground application (scope:desktop)."
- ))
- .with_structured(json!({
- "scope": "desktop", "path": "SendInput", "characters": text_len,
- "effect": "unverifiable"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("desktop type task failed: {error}")),
- };
- }
- let text_raw = match args.require_str("text") {
- Ok(v) => v,
- Err(e) => return e,
- };
- // Strip trailing agent-protocol closing tags before delivery —
- // catches the case where an LLM hallucinated its own tool-
- // invocation tags into the text param (see text_sanitize docs).
- // Returns Cow::Borrowed on the no-match fast path so the common
- // case is allocation-free.
- let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
- .into_owned();
- let raw_pid = match args.require_i64("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let pid = raw_pid as u32;
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let delivery = DeliveryMode::from_args(&args);
-
- // ── px form: focus by pixel-click, then type into the focused element ──
- // Pass x,y (no element_token) for an *element px action*: pixel-click the
- // field to give the Chromium/Electron renderer the real keyboard focus the
- // UIA/WM_CHAR path can't establish on its own, then fall through to the
- // focused-window WM_CHAR path below (which lands once focused). Reuses
- // ClickTool's exact coordinate translation + delivery_mode. Mirrors macOS.
- {
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
- if let (Some(cx), Some(cy)) = (px, py) {
- if elem_idx.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to type_text, not both.",
- );
- }
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- hwnd_opt,
- cx,
- cy,
- delivery.is_foreground(),
- args.opt_str("session"),
- args.opt_str("_session_id"),
- from_zoom,
- false,
- )
- .await
- {
- return e;
- }
- // elem_idx stays None → the type path below writes to the now-
- // focused element via the WM_CHAR (key_events) rung.
- }
- }
-
- // Same no-raise guard as click: a XAML/WPF ValuePattern.SetValue handler
- // calls UIElement.Focus()→SetForegroundWindow; WS_EX_NOACTIVATE on the
- // target makes that a no-op while the value still gets set. Safe because
- // type_text never uses the SendInput foreground-swap path. No-op for
- // delivery_mode:"foreground" and when no window_id was given.
- let _noact = match (delivery != DeliveryMode::Foreground, hwnd_opt) {
- (true, Some(h)) => Some(crate::input::NoActivateGuard::arm(
- windows::Win32::Foundation::HWND(h as *mut _),
- )),
- _ => None,
- };
- let _delay_ms = args.u64_or("delay_ms", 30);
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- if let Some(element) = admitted.as_ref() {
- let (cx, cy) = element.center;
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(Some(element), (0, 0)),
- )
- .await;
- self.state
- .cursor_registry
- .update_position(&cursor_key, cx as f64, cy as f64);
- } else if args.get("x").and_then(Value::as_f64).is_none()
- || args.get("y").and_then(Value::as_f64).is_none()
- {
- // Untargeted text still gives a named session visible feedback:
- // its remembered position, or the window centre on the first action.
- position_keyboard_cursor(&self.state, &args, hwnd, None).await;
- }
- let text_len = text.chars().count();
-
- // Native ConsoleHost can accept every synthesized Unicode event while
- // dropping the text at the prompt (observed on Windows 11 ARM64).
- // Classify terminal delivery before the foreground SendInput branch so
- // an explicit foreground retry cannot bypass the hard refusal.
- match crate::terminal::type_text_policy(hwnd, delivery.is_foreground()) {
- crate::terminal::TypeTextPolicy::Unsupported => {
- return crate::terminal::native_consolehost_type_text_error(hwnd);
- }
- crate::terminal::TypeTextPolicy::ForegroundRequired => {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::TextInput,
- );
- }
- crate::terminal::TypeTextPolicy::Supported => {}
- }
-
- // Refuse known background drops before the final WM_CHAR path. WPF is
- // conditional: indexed text still has a working UIA ValuePattern route,
- // while unindexed text would be posted to the top-level and disappear.
- if delivery == DeliveryMode::Background
- && (crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::TextInput)
- || (elem_idx.is_none() && crate::input::delivery::is_wpf_target_window(hwnd)))
- {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::TextInput,
- );
- }
- // delivery_mode:"foreground" — explicit SendInput Unicode with a brief
- // foreground swap (send_text_synthesized), symmetric with press_key's
- // foreground path. This is the rung that lands on VCL/LibreOffice
- // document grids and other targets where PostMessage WM_CHAR is
- // silently dropped. Honest by construction: if the foreground swap is
- // rejected (daemon not at UIAccess integrity), it returns an error
- // rather than a false success.
- if delivery == DeliveryMode::Foreground {
- if elem_idx.is_none()
- && args.get("element_token").is_none()
- && args.get("x").is_none()
- && args.get("y").is_none()
- {
- if let Some(result) =
- try_focused_document_route(hwnd, Some(pid), &text, text_len).await
- {
- return result;
- }
- }
- // Resolve the optional click fallback before entering the atomic
- // activation/focus/input transaction. The focus itself happens
- // only after exact top-level foreground is confirmed.
- let focus_target = if let Some(idx) = elem_idx {
- let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
- Some(center) => center,
- None => {
- return ToolResult::error(format!(
- "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
- ))
- }
- };
- let (cx, cy) = match resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx as usize,
- cx,
- cy,
- "foreground typing",
- ) {
- Ok(point) => point,
- Err(result) => return result,
- };
- Some((idx as usize, (cx, cy)))
- } else {
- None
- };
- let text_fg = text.clone();
- let r = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_text_synthesized_after_focus(hwnd, &text_fg, || {
- if let Some((idx, point)) = focus_target {
- focus_cached_element_for_foreground(&admitted, hwnd, idx, Some(point))?;
- }
- Ok(())
- })
- }
- })
- .await;
- return match r {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Typed {text_len} char(s) on pid {raw_pid} via SendInput (delivery_mode:foreground)."
- ))
- .with_structured(serde_json::json!({
- "path": "key_events",
- "characters": text_len,
- // Foreground SendInput delivered, but with no read-back the
- // driver can't confirm the effect → unverifiable. No escalation:
- // foreground IS the escalated rung (mirrors macOS PATH_KEY_EVENTS_FG).
- "effect": "unverifiable",
- })),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Pin the agent-cursor overlay above the target window so the synthetic
- // cursor stays sandwiched at z+1 of the type target for the full
- // duration of the keystrokes (both XAML/UIA and PostMessage paths).
- pin_overlay_above(&cursor_key, hwnd);
-
- // Glide the agent cursor onto the field being typed into, so the viewer
- // can see *where* the agent is typing — same visual feedback as a click.
- // Only when an element_index is supplied (we have its cached center);
- // the focused-element path has no resolvable position to point at.
- if let Some(element) = admitted.as_ref() {
- let (cx, cy) = element.center;
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(Some(element), (0, 0)),
- )
- .await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: cx as f64,
- y: cy as f64,
- },
- );
- }
-
- // CUA-543 routing: PostMessage WM_CHAR doesn't reach modern
- // XAML / WinUI3 hosts. When the target is one of those AND the
- // caller has supplied an element_index, route through UIA
- // `ValuePattern.SetValue` — same code path the `set_value` tool
- // uses, which we've verified works on modern Notepad / WinUI3.
- // Legacy Win32 stays on the PostMessage path so the no-focus-
- // steal property is preserved.
- // ── Automatic routing — the caller need not know the framework. ──
- // 1. With an element_index, try UIA ValuePattern.SetValue first: it works
- // for WPF / WinForms / UWP / XAML and many web inputs, sets the value
- // through the accessibility channel (no keystrokes), and the `_noact`
- // guard blocks any self-foreground — so it never raises. Auto-falls-
- // back to the WM_CHAR path below if the element has no ValuePattern
- // (most legacy Win32 EDITs consume WM_CHAR fine without focus steal).
- if let Some(idx) = elem_idx {
- let text_for_uia = text.clone();
- let set_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- // Retain the element under the cache lock so a concurrent
- // get_window_state snapshot-replace on the same (pid, hwnd)
- // can't Release it to zero while this SetValue is in flight.
- // The guard is held for the whole closure.
- let element_guard = admitted.as_ref().filter(|element| element.is_uia())?;
- let ptr = element_guard.as_ptr();
- use windows::core::{Interface, BSTR};
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationValuePattern, UIA_ValuePatternId,
- };
- let elem: IUIAutomationElement =
- unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
- let result = (|| -> anyhow::Result<(String, String)> {
- let pattern = unsafe { elem.GetCurrentPattern(UIA_ValuePatternId) }?;
- let vp: IUIAutomationValuePattern = pattern.cast()?;
- // Shield the SetValue against host self-foreground. A
- // Chromium/Electron (or XAML) ValuePattern.SetValue handler
- // calls SetForegroundWindow(self), which WS_EX_NOACTIVATE
- // does NOT stop; the EnableWindow shield does (disabled
- // top-level can't be foregrounded) while the a11y-channel
- // SetValue still lands. Same fix as the UIA Invoke path.
- crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- // ValuePattern has no caret/selection insertion API. Preserve
- // the current document and append as the deterministic
- // background fallback; replacing the whole value violates
- // type_text's insertion contract for editors.
- let current = vp
- .CurrentValue()
- .map(|value| value.to_string())
- .unwrap_or_default();
- let inserted = format!("{current}{text_for_uia}");
- vp.SetValue(&BSTR::from(inserted.as_str()))?;
- Ok((current, inserted))
- })
- })()
- .ok();
- std::mem::forget(elem);
- result
- }
- })
- .await
- .unwrap_or(None);
- if let Some((before, expected)) = set_result {
- // Read the value back by element handle — focus-independent
- // (works regardless of which window is foreground), proving the
- // a11y write actually took rather than trusting SetValue's
- // return alone.
- let verify = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let value = read_cached_element_value(&admitted);
- classify_value_write_readback(value.as_deref(), &before, &expected)
- }
- })
- .await
- .unwrap_or("pending");
- // SURFACE-AWARE VERIFICATION (mirrors macOS type_text). On
- // Electron/Chromium web inputs the UIA layer accepts a
- // `ValuePattern.SetValue` and echoes it straight back through
- // `ValuePattern.CurrentValue` while the renderer/DOM never observes
- // it — so a read-back "confirm" is a shim echo, not ground truth
- // (the Windows analogue of the Slack-search AX false-confirm).
- // Refuse to claim verified there: the screenshot is the only truth
- // and the next rung is the element px action. Probe ONLY when it
- // would otherwise confirm via the UIA value path, so native UIA
- // types (WPF / WinForms / UWP / XAML) pay nothing. The signal is
- // the window class — Chromium and every Electron host expose
- // `Chrome_WidgetWin_*` (see `is_chromium_target_window`).
- let ax_echo_surface =
- verify == "confirmed" && crate::input::is_chromium_target_window(hwnd);
- let verified = verify == "confirmed" && !ax_echo_surface;
- let (mark, note) = if verified {
- ("✅ Wrote", String::new())
- } else if ax_echo_surface {
- (
- "📨 Sent (unverified)",
- " — Electron/web surface: the UIA layer accepts and echoes the \
- write but the renderer may not have observed it, so the driver \
- cannot confirm via UIA. Verify via the screenshot; if it didn't \
- land, re-type with the px form (pass x,y to pixel-focus the \
- field)."
- .to_string(),
- )
- } else {
- (
- "📨 Accepted (verification pending)",
- " — the provider may publish its new value only after this call returns. Take a fresh snapshot before retrying."
- .to_string(),
- )
- };
- return ToolResult::text(format!(
- "{mark} {text_len} char(s) on pid {raw_pid} via UIA ValuePattern \
- (element_index=[{idx}]; verify: {verify}).{note}"
- ))
- .with_structured({
- // `effect` mirrors the UIA read-back: a TRUSTED positive
- // read-back is "confirmed"; a deferred provider or an Electron UIA echo
- // that we refuse to trust is "unverifiable". Only the echo
- // recommends pixel escalation.
- let mut s = value_write_structured_result(text_len, verify, verified);
- if ax_echo_surface {
- // Electron UIA echo → the element px action is the next rung,
- // not foreground (it's a renderer-focus problem, not an
- // activation one).
- s["escalation"] = serde_json::json!({
- "recommended": "px",
- "reason": "Electron/web surface — the UIA write was echoed \
- but the renderer may not have observed it. \
- Confirm via the screenshot; if it didn't land, \
- re-type with the element px action (pass x,y to \
- pixel-focus the field, then type)."
- });
- }
- s
- });
- }
- // ValuePattern unavailable → fall through to the WM_CHAR path.
- }
-
- // 2. No element_index on a WPF target: WM_CHAR is dropped and there's no
- // element to SetValue. The only background path was a cloaked focus
- // grab (SendInput), which the macOS-aligned contract forbids in
- // background. Prefer supplying an element_index (path 1, UIA SetValue —
- // never raises); otherwise surface background_unavailable so the agent
- // escalates to delivery_mode:"foreground".
- if elem_idx.is_none() && crate::input::delivery::is_wpf_target_window(hwnd) {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::TextInput,
- );
- }
-
- // 3. Legacy Win32 / GDI / Chromium-IME — PostMessage WM_CHAR, no focus steal.
- //
- // Read-back verification (mirrors macOS type_text): WM_CHAR is silently
- // dropped by some focused targets (notably a VCL/LibreOffice Calc grid
- // not in cell-edit mode), and PostMessage reports success regardless. So
- // snapshot the focused element's UIA value before and after; if it did
- // not change we can't confirm the text landed and report "Sent
- // (unverified)" with the foreground-escalation hint, instead of a blind
- // success. If the focused element exposes no ValuePattern we likewise
- // can't confirm (conservative: under-claim rather than false-succeed).
- let text_for_post = text.clone();
- let verify_pid = pid;
- let verify_idx = elem_idx.map(|i| i as usize);
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- // Prefer a focus-independent read of the *specific* cached element
- // when we have its index; only fall back to the (flaky, focus-
- // dependent) system focused element when typing into "whatever is
- // focused" with no element_token.
- let read = |idx: Option<usize>| match idx {
- Some(_) => read_cached_element_value(&admitted),
- None => read_focused_value_uia(verify_pid),
- };
- let before = read(verify_idx);
- let post_res = crate::input::post_type_text(hwnd, &text_for_post);
- std::thread::sleep(std::time::Duration::from_millis(40));
- let after = read(verify_idx);
- let observed = post_message_readback_observed(
- before.as_deref(),
- after.as_deref(),
- &text_for_post,
- );
- (post_res, before, after, observed)
- }
- })
- .await;
- match result {
- Ok((Ok(()), before, after, observed)) => {
- // Three-way verdict. Crucially, "couldn't read the field" is NOT
- // the same as "the text didn't land": on Windows, UIA's
- // GetFocusedElement is system-wide (unlike macOS's per-app
- // AXFocusedUIElement), so when the target isn't the foreground
- // app we cannot read it back even though a PostMessage WM_CHAR
- // may have landed fine. Treating unreadable as failure would
- // spuriously push agents to foreground and break the
- // background-first contract. A changed value containing the
- // requested text is useful evidence, but the insertion point is
- // unknowable on WM_CHAR. Keep the action unverifiable so callers
- // prove final state from a fresh snapshot before any retry.
- match (&before, &after) {
- (Some(_), Some(_)) if observed => ToolResult::text(format!(
- "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage \
- ({_delay_ms}ms delay; the field changed and now contains the \
- requested text, but the insertion point is not known). Take a \
- fresh snapshot to verify the intended final value before retrying."
- ))
- .with_structured(changed_contains_post_message_result(text_len)),
- (Some(_), Some(_)) => ToolResult::text(format!(
- "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage, but \
- the focused field's value did not contain the requested text \
- after dispatch — delivery was incomplete or dropped (e.g. a \
- VCL/LibreOffice grid not in cell-edit mode). \
- Retry with delivery_mode:\"foreground\"."
- ))
- .with_structured(serde_json::json!({
- "path": "key_events", "characters": text_len,
- "verified": false, "verify": "unchanged",
- "effect": "unverifiable",
- "escalation": {
- "recommended": "foreground",
- "reason": "background insert could not be confirmed — re-call \
- with delivery_mode:\"foreground\" if a screenshot \
- shows the text didn't land."
- },
- })),
- _ => ToolResult::text(format!(
- "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage \
- ({_delay_ms}ms delay; not verified — could not read \
- the focused field back, e.g. the target isn't foreground or \
- exposes no ValuePattern). If it didn't land, retry with \
- delivery_mode:\"foreground\"."
- ))
- .with_structured(serde_json::json!({
- "path": "key_events", "characters": text_len,
- "verified": false, "verify": "unreadable",
- "effect": "unverifiable",
- "escalation": {
- "recommended": "foreground",
- "reason": "background insert could not be confirmed — re-call \
- with delivery_mode:\"foreground\" if a screenshot \
- shows the text didn't land."
- },
- })),
- }
- }
- Ok((Err(e), _, _, _)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-async fn try_focused_document_route(
- hwnd: u64,
- expected_pid: Option<u32>,
- text: &str,
- text_len: usize,
-) -> Option<ToolResult> {
- if crate::input::is_chromium_target_window(hwnd) {
- return None;
- }
- let text = text.to_owned();
- let outcome = tokio::task::spawn_blocking(move || {
- crate::uia::focused_document::try_type_focused_document_append(hwnd, expected_pid, &text)
- })
- .await
- .unwrap_or(
- crate::uia::focused_document::FocusedDocumentTypeOutcome::Unknown(
- "focused UIA task failed",
- ),
- );
- focused_document_tool_result(outcome, text_len)
-}
-
-fn focused_document_tool_result(
- outcome: crate::uia::focused_document::FocusedDocumentTypeOutcome,
- text_len: usize,
-) -> Option<ToolResult> {
- use crate::uia::focused_document::FocusedDocumentTypeOutcome;
-
- match outcome {
- FocusedDocumentTypeOutcome::NotApplicable => None,
- FocusedDocumentTypeOutcome::Confirmed => Some(
- ToolResult::text(format!(
- "Wrote and verified {text_len} character(s) in the focused Document; caret is at the document end."
- ))
- .with_structured(json!({
- "path": "ax",
- "characters": text_len,
- "verified": true,
- "verify": "confirmed",
- "effect": "confirmed",
- })),
- ),
- FocusedDocumentTypeOutcome::Unknown(stage) => Some(
- ToolResult::text(format!(
- "Focused Document text may have been written but could not be verified ({stage}). Inspect the current document before retrying."
- ))
- .with_structured(json!({
- "path": "ax",
- "characters": text_len,
- "verified": false,
- "verify": "unverifiable",
- "effect": "unverifiable",
- "retryable": false,
- })),
- ),
- }
-}
-
-#[cfg(test)]
-mod focused_document_result_tests {
- use super::focused_document_tool_result;
- use crate::uia::focused_document::FocusedDocumentTypeOutcome;
- use cua_driver_contract::{ActionEffect, ActionRoute};
- use cua_driver_core::action_record::{ActionExecutionRecord, ActionTransport};
- use cua_driver_core::protocol::ToolResult;
- use serde_json::json;
-
- fn normalize(result: ToolResult) -> cua_driver_contract::ActionResult {
- let structured = result
- .structured_content
- .expect("focused-document result includes structured evidence");
- let record = ActionExecutionRecord::from_legacy(
- "type_text",
- &json!({"scope": "desktop"}),
- &structured,
- )
- .expect("the action dispatcher recognizes the UIA value transport");
- assert_eq!(record.transport, ActionTransport::WindowsUiaValue);
- record
- .public_result()
- .expect("focused-document action result normalizes")
- }
-
- #[test]
- fn focused_document_results_normalize_to_accessibility_with_truthful_effects() {
- let confirmed = focused_document_tool_result(FocusedDocumentTypeOutcome::Confirmed, 12)
- .expect("confirmed focused-document output");
- let confirmed = normalize(confirmed);
- assert_eq!(confirmed.route, ActionRoute::Accessibility);
- assert_eq!(confirmed.effect, ActionEffect::Confirmed);
-
- let unknown = focused_document_tool_result(
- FocusedDocumentTypeOutcome::Unknown("value read-back failed"),
- 12,
- )
- .expect("unknown focused-document output");
- assert_eq!(
- unknown.structured_content.as_ref().unwrap()["retryable"],
- false
- );
- let unknown = normalize(unknown);
- assert_eq!(unknown.route, ActionRoute::Accessibility);
- assert_eq!(unknown.effect, ActionEffect::Unverifiable);
- }
-}
-
-fn post_message_readback_observed(
- before: Option<&str>,
- after: Option<&str>,
- requested_text: &str,
-) -> bool {
- matches!(
- (before, after),
- (Some(before), Some(after))
- if before != after && after.contains(requested_text)
- )
-}
-
-fn value_write_structured_result(
- text_len: usize,
- verify: &str,
- verified: bool,
-) -> serde_json::Value {
- serde_json::json!({
- "path": "ax",
- "characters": text_len,
- "verified": verified,
- "verify": verify,
- "effect": if verified { "confirmed" } else { "unverifiable" },
- })
-}
-
-fn changed_contains_post_message_result(text_len: usize) -> serde_json::Value {
- serde_json::json!({
- "path": "key_events",
- "characters": text_len,
- "verified": false,
- "verify": "changed_contains",
- "effect": "unverifiable",
- })
-}
-
-#[cfg(test)]
-mod post_message_readback_tests;
-
-fn classify_value_write_readback(
- value: Option<&str>,
- before: &str,
- expected: &str,
-) -> &'static str {
- match value {
- Some(value) if value == expected && value != before => "confirmed",
- // SetValue returned success. Some providers publish their new Value only
- // after this tool call unwinds, so an old or unreadable synchronous value
- // is pending verification, not evidence that the write failed.
- _ => "pending",
- }
-}
-
-/// Read a **specific cached element's** text value by its `element_index`,
-/// for type_text read-back verification. Tries `ValuePattern.CurrentValue`
-/// then falls back to `TextPattern` document text. Unlike
-/// [`read_focused_value_uia`] this is **focus-independent**: it reads the exact
-/// element the caller typed into (by handle, from the per-(pid,window) cache),
-/// so it works whether or not the target is the foreground window. Returns
-/// `None` if the index isn't cached or the element exposes no readable text
-/// pattern. Mirrors the cache-retain + `mem::forget` discipline of the
-/// ValuePattern.SetValue path so the cached COM ref isn't released.
-fn read_cached_element_value(
- admitted: &Option<crate::uia::snapshot::RetainedElement>,
-) -> Option<String> {
- use windows::core::Interface;
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationTextPattern, IUIAutomationValuePattern,
- UIA_TextPatternId, UIA_ValuePatternId,
- };
- let guard = admitted.as_ref().filter(|element| element.is_uia())?;
- let ptr = guard.as_ptr();
- let elem: IUIAutomationElement = unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
- let val = unsafe {
- elem.GetCurrentPattern(UIA_ValuePatternId)
- .ok()
- .and_then(|p| p.cast::<IUIAutomationValuePattern>().ok())
- .and_then(|vp| vp.CurrentValue().ok())
- .map(|b| b.to_string())
- .or_else(|| {
- elem.GetCurrentPattern(UIA_TextPatternId)
- .ok()
- .and_then(|p| p.cast::<IUIAutomationTextPattern>().ok())
- .and_then(|tp| tp.DocumentRange().ok())
- .and_then(|r| r.GetText(-1).ok())
- .map(|b| b.to_string())
- })
- };
- // The cache guard owns the ref; don't let `elem`'s Drop release it.
- std::mem::forget(elem);
- val
-}
-
-/// Read the currently-focused UIA element's `ValuePattern` value **iff it
-/// belongs to `expect_pid`**, for type_text read-back verification.
-/// Self-contained COM (STA) init/teardown so it can be called from a
-/// `spawn_blocking` worker. Returns `None` if UIA is unreachable, nothing is
-/// focused, the focused element is owned by a different process (e.g. the
-/// target is backgrounded and system focus is on the user's foreground app —
-/// we must NOT read that element's value), or the focused element exposes no
-/// `ValuePattern` (most rich-text/document surfaces). Callers treat `None` as
-/// "could not confirm" rather than failure. Mirrors the focused-element read in
-/// the `debug_window_info` tool.
-fn read_focused_value_uia(expect_pid: u32) -> Option<String> {
- use windows::core::Interface;
- use windows::Win32::System::Com::{
- CoCreateInstance, CoInitializeEx, CoUninitialize, CLSCTX_INPROC_SERVER,
- COINIT_APARTMENTTHREADED,
- };
- use windows::Win32::UI::Accessibility::{
- CUIAutomation, IUIAutomation, IUIAutomationValuePattern, UIA_ValuePatternId,
- };
-
- let coinit_ok = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED).is_ok() };
- let value = (|| -> Option<String> {
- let uia: IUIAutomation =
- unsafe { CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER) }.ok()?;
- let focused = unsafe { uia.GetFocusedElement() }.ok()?;
- // Scope to the target: system focus may be on a different app when the
- // target is backgrounded; reading that element would give a bogus verdict.
- if unsafe { focused.CurrentProcessId() }.ok()? as u32 != expect_pid {
- return None;
- }
- let pattern = unsafe { focused.GetCurrentPattern(UIA_ValuePatternId) }.ok()?;
- let vp: IUIAutomationValuePattern = pattern.cast().ok()?;
- Some(unsafe { vp.CurrentValue() }.ok()?.to_string())
- })();
- if coinit_ok {
- unsafe {
- CoUninitialize();
- }
- }
- value
-}
-
-// ── press_key ─────────────────────────────────────────────────────────────────
-
-pub struct PressKeyTool {
- state: Arc<ToolState>,
-}
-static PRESS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for PressKeyTool {
- fn def(&self) -> &ToolDef {
- PRESS_DEF.get_or_init(|| ToolDef {
- name: "press_key".into(),
- // Description ported from Swift `PressKeyTool.swift` with
- // Windows-specific transport note (PostMessage WM_KEYDOWN/UP).
- description: "Press and release a single key, delivered directly to the target pid's \
- top-level window via PostMessage(WM_KEYDOWN/WM_KEYUP). The target does NOT need \
- to be frontmost — no focus steal.\n\n\
- Optional `window_id` selects a specific HWND when the pid owns more than one; \
- without it the first visible top-level window for the pid is used.\n\n\
- Key vocabulary: return, tab, escape, up/down/left/right, space, delete, home, \
- end, pageup, pagedown, f1-f12, plus any letter or digit. Optional `modifiers` \
- array takes ctrl/shift/alt/win. For true combinations (ctrl+c), `hotkey` is a \
- cleaner surface.\n\n\
- `element_token` focuses the cached UIA element before sending the key; the \
- top-level window remains backgrounded when delivery_mode is background.".into(),
- input_schema: json!({
- "type":"object","required":["key"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the key to the current foreground application."},
- "pid":{"type":"integer","description":"Target process ID."},
- "key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},
- "modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the UIA path can't focus. Pass with y, no element_token."},
- "y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
- "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use cua_driver_core::tool_args::ArgsExt;
- if args.get("scope").and_then(Value::as_str) == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<PressKeyInput>("press_key", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let key = input.key;
- let mods = input.modifiers.unwrap_or_default();
- let hwnd = match crate::input::mouse::foreground_window() {
- Ok(hwnd) => hwnd,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- let key_display = key.clone();
- return match tokio::task::spawn_blocking(move || {
- let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
- crate::input::send_key_synthesized(hwnd, &key, &modifiers)
- })
- .await
- {
- Ok(Ok(())) => ToolResult::text(format!(
- "Sent {key_display} to the foreground application (scope:desktop)."
- ))
- .with_structured(json!({
- "scope": "desktop", "path": "SendInput", "key": key_display,
- "effect": "unverifiable"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("desktop key task failed: {error}")),
- };
- }
- let key = match args.require_str("key") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let mods: Vec<String> = args.str_array("modifiers");
- let raw_pid = match args.require_i64("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let pid = raw_pid as u32;
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let delivery = DeliveryMode::from_args(&args);
- // Swift requires window_id when element_index is supplied — ports the
- // same validation.
-
- // px form: pixel-click to focus, then the key goes to the focused element.
- // Reuses click's translation + delivery_mode; after it, deliver via the
- // plain background post path (the focus-click already handled fronting if
- // foreground), so skip the background_unavailable check + foreground swap
- // below. Mirrors macOS press_key.
- let px_focus = {
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
- if let (Some(cx), Some(cy)) = (px, py) {
- if elem_idx.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to press_key, not both.",
- );
- }
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- hwnd_opt,
- cx,
- cy,
- delivery.is_foreground(),
- args.opt_str("session"),
- args.opt_str("_session_id"),
- from_zoom,
- false,
- )
- .await
- {
- return e;
- }
- true
- } else {
- false
- }
- };
-
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // Classify known background drops before touching UIA focus. Focusing
- // first made an honest Chromium refusal transiently activate the target.
- let event_kind = if mods.is_empty() {
- EventKind::Keystroke
- } else {
- EventKind::KeyCombo
- };
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, event_kind)
- {
- return crate::input::delivery::background_unavailable_error(hwnd, event_kind);
- }
-
- // The px form's focus click already moved the cursor. Otherwise place
- // a named session's cursor so a keyboard-first session stays visible.
- if !px_focus {
- let element_center = admitted
- .as_ref()
- .map(|element| (element.center.0 as f64, element.center.1 as f64));
- position_keyboard_cursor(&self.state, &args, hwnd, element_center).await;
- }
-
- // W1: an element-addressed key needs the control's actual focus
- // target, not merely its owning top-level HWND. Embedded WebView hosts
- // can activate their frame from UIA SetFocus even under
- // WS_EX_NOACTIVATE. Their proven-safe pixel route establishes renderer
- // focus with a posted click, so reuse that route at the AX element's
- // cached center.
- let background_webview_focus = elem_idx.is_some()
- && delivery == DeliveryMode::Background
- && crate::input::has_chromium_descendant(hwnd);
- let mut noact = if elem_idx.is_some() && delivery == DeliveryMode::Background {
- Some(crate::input::NoActivateGuard::arm(
- windows::Win32::Foundation::HWND(hwnd as *mut _),
- ))
- } else {
- None
- };
- if let Some(idx) = elem_idx.filter(|_| background_webview_focus) {
- let Some((cx, cy)) = admitted.as_ref().map(|element| element.center) else {
- return ToolResult::error(format!(
- "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
- ));
- };
- let (cx, cy) = match resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx as usize,
- cx,
- cy,
- "focusing for key delivery",
- ) {
- Ok(point) => point,
- Err(result) => return result,
- };
- let (px, py) = screen_to_bitmap(hwnd, cx, cy);
- // Release the outer guard before the shared pixel helper. ClickTool
- // owns a guard around the click itself, then releases it before its
- // renderer settle period. This is the exact route already proven by
- // the PX background cell, including targeted injection fallback.
- drop(noact.take());
- if let Err(error) = focus_by_pixel(
- &self.state,
- pid,
- Some(hwnd),
- px as f64,
- py as f64,
- false,
- args.opt_str("session"),
- args.opt_str("_session_id"),
- false,
- true,
- )
- .await
- {
- return error;
- }
- } else if elem_idx.is_some() && delivery != DeliveryMode::Foreground {
- let focused = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || {
- admitted
- .as_ref()
- .ok_or_else(|| anyhow::anyhow!("missing admitted element"))
- .and_then(|element| element.focus_element())
- })
- }
- })
- .await;
- match focused {
- Ok(Ok(())) => {
- if delivery == DeliveryMode::Background {
- let _ = crate::input::wait_for_focused_descendant(
- hwnd,
- std::time::Duration::from_millis(500),
- );
- }
- }
- Ok(Err(e)) => return ToolResult::error(e.to_string()),
- Err(e) => return ToolResult::error(format!("UIA focus task failed: {e}")),
- }
- }
- let key_display = key.clone();
- // Foreground: send_key_synthesized takes the SetForegroundWindow path.
- // Skipped when px-focus already fronted/clicked the target — the key then
- // goes via the plain background post path below.
- if !px_focus && delivery == DeliveryMode::Foreground {
- let focus_target = elem_idx.map(|idx| {
- let point = admitted.as_ref().map(|element| element.center);
- (idx as usize, point)
- });
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
- if let Some((idx, point)) = focus_target {
- focus_cached_element_for_foreground(&admitted, hwnd, idx, point)?;
- }
- Ok(())
- })
- }
- })
- .await;
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent {key_display} via SendInput on pid {raw_pid} (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- crate::input::post_key(hwnd, &key, &m)
- }
- })
- .await;
- match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "📨 Sent {key_display} to pid {raw_pid} via PostMessage (not verified). \
- If it didn't land, retry with delivery_mode:\"foreground\"."
- ))
- .with_structured(serde_json::json!({
- "path": "key_events",
- "key": key_display,
- "verified": false,
- "verify": "unreadable",
- "effect": "unverifiable",
- "escalation": {
- "recommended": "foreground",
- "reason": "background key delivery could not be confirmed — re-call \
- with delivery_mode:\"foreground\" if a screenshot shows the key didn't land."
- },
- })),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── hotkey ────────────────────────────────────────────────────────────────────
-
-fn is_modifier(k: &str) -> bool {
- matches!(
- k.to_lowercase().as_str(),
- "ctrl" | "control" | "shift" | "alt" | "win" | "windows" | "cmd" | "command"
- )
-}
-
-pub struct HotkeyTool {
- state: Arc<ToolState>,
-}
-static HOTKEY_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for HotkeyTool {
- fn def(&self) -> &ToolDef {
- HOTKEY_DEF.get_or_init(|| ToolDef {
- name: "hotkey".into(),
- // Description ported from Swift `HotkeyTool.swift` with the
- // Windows-specific dispatch (UIA accelerator for XAML; SendInput
- // for legacy Win32 with modifiers per #1614 / #1618; PostMessage
- // for modifier-less keys).
- description: "Press a combination of keys simultaneously — e.g. `[\"ctrl\", \"c\"]` \
- for Copy, `[\"ctrl\", \"shift\", \"t\"]` for reopen-closed-tab. Dispatch is \
- target-aware:\n\n\
- - **Modern XAML / WinUI / UWP targets** route through UI Automation: the driver \
- walks the target's accessibility subtree, finds a descendant whose AcceleratorKey \
- matches the combo, and invokes it. No focus steal, no system-queue input.\n\n\
- - **Legacy Win32 targets with modifiers** (Ctrl+S, Alt+Tab, etc.) route through \
- `SendInput` against the system input queue, with a brief `SetForegroundWindow` \
- swap so the events land on the target. This path is necessary because \
- PostMessage(WM_KEYDOWN, VK_CONTROL) does NOT update the OS-wide modifier state \
- visible to `GetKeyState` / `TranslateAccelerator`, so Win32 apps that bind \
- accelerators via `TranslateAccelerator` (LibreOffice, FAR, classic Notepad, etc.) \
- never see the combo as a real accelerator on the PostMessage-only path. \
- Trade-off: brief foreground swap (mitigated by restoring the previous foreground \
- after the keystrokes flush). Requires the daemon to have UIAccess integrity so \
- `SetForegroundWindow` is permitted — the MCP proxy auto-prefers the \
- `cua-driver-uia.exe` worker pipe when both daemons are running.\n\n\
- - **Legacy Win32 targets without modifiers** (plain `enter`, `tab`, `f5`, etc.) \
- route through `PostMessage(WM_KEYDOWN/UP)` — no focus steal, no need to update \
- modifier state.\n\n\
- The target does NOT need to be frontmost in any branch.\n\n\
- **`window_id`** (optional): explicit HWND when the pid owns more than one \
- window; otherwise the pid's first visible window is used.\n\n\
- Recognized modifiers: ctrl/control, shift, alt, win/windows. Non-modifier \
- keys use the same vocabulary as `press_key` (return, tab, escape, \
- up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, \
- letters, digits). Order: modifiers first, one non-modifier last.".into(),
- input_schema: json!({
- "type":"object","required":["keys"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the hotkey to the current foreground application."},
- "pid":{"type":"integer","description":"Target process ID."},
- "keys":{"type":"array","items":{"type":"string"},"minItems":2,
- "description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
- "y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
- "window_id":{"type":"integer","description":"Explicit HWND when the pid owns multiple windows."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use cua_driver_core::tool_args::ArgsExt;
- if args.get("scope").and_then(Value::as_str) == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<HotkeyInput>("hotkey", &args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- if input.keys.len() < 2 {
- return ToolResult::error("hotkey.keys must contain at least two keys.")
- .with_structured(json!({ "code": "invalid_arguments" }));
- }
- let mods: Vec<String> = input
- .keys
- .iter()
- .filter(|key| is_modifier(key))
- .cloned()
- .collect();
- let Some(key) = input
- .keys
- .iter()
- .rev()
- .find(|key| !is_modifier(key))
- .cloned()
- else {
- return ToolResult::error("keys must include at least one non-modifier key.");
- };
- let full_keys = input.keys;
- let hwnd = match crate::input::mouse::foreground_window() {
- Ok(hwnd) => hwnd,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- let key_display = full_keys.join("+");
- return match tokio::task::spawn_blocking(move || {
- let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
- crate::input::send_key_synthesized(hwnd, &key, &modifiers)
- })
- .await
- {
- Ok(Ok(())) => ToolResult::text(format!(
- "Sent {key_display} to the foreground application (scope:desktop)."
- ))
- .with_structured(json!({
- "scope": "desktop", "path": "SendInput", "keys": full_keys,
- "effect": "unverifiable"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),
- };
- }
- // Parse keys array (Swift's only path). Legacy key+modifiers shape
- // still accepted as a Rust-side convenience.
- let (key, mods, full_keys) = if let Some(arr) = args.get("keys") {
- let raw = match arr.as_array() {
- Some(r) => r,
- None => return ToolResult::error("Missing required array field keys."),
- };
- let keys: Vec<String> = raw
- .iter()
- .filter_map(|v| v.as_str().map(str::to_owned))
- .collect();
- if keys.len() != raw.len() || keys.is_empty() {
- return ToolResult::error("keys must be a non-empty array of strings.");
- }
- let modifiers: Vec<String> = keys.iter().filter(|k| is_modifier(k)).cloned().collect();
- let non_mods: Vec<String> = keys.iter().filter(|k| !is_modifier(k)).cloned().collect();
- if non_mods.is_empty() {
- return ToolResult::error("keys must include at least one non-modifier key.");
- }
- (non_mods.last().unwrap().clone(), modifiers, keys)
- } else if let Some(k) = args.get("key").and_then(|v| v.as_str()) {
- let m: Vec<String> = args
- .get("modifiers")
- .and_then(|v| v.as_array())
- .map(|a| {
- a.iter()
- .filter_map(|v| v.as_str().map(str::to_owned))
- .collect()
- })
- .unwrap_or_default();
- let mut full = m.clone();
- full.push(k.to_owned());
- (k.to_owned(), m, full)
- } else {
- return ToolResult::error("Missing required array field keys.");
- };
- let raw_pid = match args.require_i64("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let pid = raw_pid as u32;
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(resolved) => resolved,
- Err(error) => return error,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let delivery = DeliveryMode::from_args(&args);
-
- // Resolve HWND: use window_id if given, else pick first window for pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let pid2 = pid;
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid2))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // Match Swift's text format exactly: `"✅ Pressed K1+K2+... on pid X."`
- // where K1+K2+... is `keys.joined(separator: "+")` of the FULL keys
- // array as the caller supplied it (modifiers + non-modifier in order).
- let key_display = full_keys.join("+");
-
- // px form: pixel-click to focus, then the combo acts on the focused field
- // (e.g. Ctrl+V into a Chromium input). A background pixel focus does not
- // make Chromium's modifier PostMessage path reliable, so it still goes
- // through the capability guard below; foreground remains explicit.
- let px_focus = {
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
- if let (Some(cx), Some(cy)) = (px, py) {
- if elem_idx.is_some() {
- return ToolResult::error(
- "Pass either element_token (ax) or x,y (px) to hotkey, not both.",
- );
- }
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if let Err(e) = focus_by_pixel(
- &self.state,
- pid,
- hwnd_opt,
- cx,
- cy,
- delivery.is_foreground(),
- args.opt_str("session"),
- args.opt_str("_session_id"),
- from_zoom,
- false,
- )
- .await
- {
- return e;
- }
- true
- } else {
- false
- }
- };
-
- // The px form's focus click already moved the cursor. Otherwise place
- // a named session's cursor so a keyboard-first session stays visible.
- if !px_focus {
- let element_center = admitted
- .as_ref()
- .map(|element| (element.center.0 as f64, element.center.1 as f64));
- position_keyboard_cursor(&self.state, &args, hwnd, element_center).await;
- }
-
- if !px_focus && crate::input::is_xaml_host_hwnd(hwnd) {
- let mut accelerator_keys = mods.clone();
- accelerator_keys.push(key.clone());
- let accelerator_combo = accelerator_keys.join("+");
- let key_display_for_result = key_display.clone();
- // UIA cross-process subtree walks can wedge on a bad provider —
- // bound the call so a hung provider returns an error instead of
- // blocking the daemon indefinitely. 4 s matches the budget the
- // rest of this file uses for similar UIA scans.
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::uia::windows_enum::try_invoke_accelerator_in_window(
- hwnd as isize,
- &accelerator_combo,
- )
- }
- })
- .await;
- return match result {
- Ok(Ok((true, _))) => ToolResult::text(format!(
- "✅ Pressed {key_display_for_result} on pid {raw_pid} via UIA \
- InvokePattern/TogglePattern (XAML / UWP target)."
- )),
- Ok(Ok((false, scanned))) => ToolResult::error(format!(
- "hotkey on a modern XAML / UWP target (pid {raw_pid}, hwnd {hwnd}) \
- could not find a UIA AcceleratorKey or `(Ctrl+X)`-style name hint \
- matching `{key_display_for_result}` (scanned {scanned} element(s)). \
- PostMessage WM_KEYDOWN/UP is ignored by this target's input pipeline. \
- Common cause: the action is nested behind a closed menu (e.g. modern \
- Notepad's Save under the File menu) — its element isn't in the visible \
- subtree until the menu is opened. Call \
- `get_window_state(pid={raw_pid}, window_id={hwnd})` to inspect available \
- UIA actions, then use an exposed accelerator or `click` the matching \
- element directly."
- )),
- Ok(Err(e)) => ToolResult::error(format!("hotkey (UIA path): {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Non-XAML target. Three routing decisions:
- // 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus
- // theft. Modifier state is encoded in the LPARAM bits, not
- // in GetKeyState, so apps that read modifier state directly
- // from the message (Chromium's `Browser::HandleKeyboardEvent`,
- // Chromium renderer's input handler, every web app) see
- // Ctrl+X correctly. Apps that consult `GetKeyState` via
- // `TranslateAccelerator` (LibreOffice, FAR, classic Notepad)
- // DON'T fire the accelerator — the modifier looks unset to
- // them.
- // 2. SendInput synthesized hotkey — pushes events onto the
- // *system input queue*. Updates GetKeyState system-wide, so
- // TranslateAccelerator-based apps see Ctrl+S as a real
- // accelerator. Trade-off: brief foreground swap so SendInput
- // lands on the right HWND. **This is the only path that
- // violates the no-foreground contract** — we minimise its
- // use to apps that genuinely require it.
- //
- // Routing:
- // - No modifiers → PostMessage (no accelerator concern).
- // - Modifiers + Chromium-family target → PostMessage. Chromium
- // processes accelerators in `Browser::HandleKeyboardEvent`,
- // which reads modifier state from the WM_KEYDOWN LPARAM
- // directly (not via GetKeyState). Every Chromium browser
- // (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app
- // (Slack/VS Code/Discord/Teams/Notion) detects as Chromium
- // via `is_chromium_target_window` and gets the no-foreground
- // path.
- // - Modifiers + non-Chromium Win32 → SendInput (the
- // TranslateAccelerator path). The foreground swap stays.
- let has_modifiers = !mods.is_empty();
- // delivery_mode:"background" — refuse if PostMessage of the key combo
- // would be silently dropped on this target (Chromium with modifiers).
- let event_kind = if has_modifiers {
- EventKind::KeyCombo
- } else {
- EventKind::Keystroke
- };
- if delivery == DeliveryMode::Background
- && (px_focus || crate::input::delivery::would_be_silently_dropped(hwnd, event_kind))
- {
- // macOS-aligned: a `background` hotkey never fronts. This combo would
- // be silently dropped (VCL/classic-Win32 TranslateAccelerator, or
- // Chromium key-combos) and only a foreground/focus grab lands it —
- // which background must not do. Surface background_unavailable; the
- // agent escalates to delivery_mode:"foreground".
- return crate::input::delivery::background_unavailable_error(hwnd, event_kind);
- }
- // delivery_mode:"foreground" — explicit SendInput swap (the path that
- // unblocks TranslateAccelerator-style apps). A Background call that
- // reaches here means the key combo is NOT silently dropped on this
- // target (the drop-check above returned early otherwise), so it stays
- // on PostMessage and the no-foreground contract holds.
- // Foreground is an explicit request for system-queue delivery. This is
- // still required after a PX focus click: PostMessage does not update
- // global modifier state, so Chromium never observes Ctrl+Shift+H as a
- // chord even though the renderer control is focused.
- let use_send_input = delivery == DeliveryMode::Foreground;
- let focus_target = elem_idx.map(|idx| {
- let point = admitted.as_ref().map(|element| element.center);
- (idx, point)
- });
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let m: Vec<&str> = mods.iter().map(String::as_str).collect();
- if use_send_input {
- crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
- if let Some((idx, point)) = focus_target {
- focus_cached_element_for_foreground(&admitted, hwnd, idx, point)?;
- }
- Ok(())
- })
- } else {
- crate::input::post_key(hwnd, &key, &m)
- }
- }
- })
- .await;
- let path = if use_send_input {
- "SendInput"
- } else {
- "PostMessage"
- };
- match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Pressed {key_display} on pid {raw_pid} via {path} \
- (Win32 target)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── set_value ─────────────────────────────────────────────────────────────────
-
-pub struct SetValueTool {
- state: Arc<ToolState>,
-}
-
-static SET_VALUE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetValueTool {
- fn def(&self) -> &ToolDef {
- SET_VALUE_DEF.get_or_init(|| ToolDef {
- name: "set_value".into(),
- // Description ported from Swift `SetValueTool.swift` with the
- // Windows transport (UIA ValuePattern instead of AXValue).
- // Native Win32 combo boxes are handled by crate::win32::combo
- // (the analogue of Swift's AXPopUpButton special case).
- description: "Set a value on a UIA element via the ValuePattern interface.\n\n\
- Two semantic modes (matching Swift's split):\n\
- - **Standard input** (text fields, sliders, combo box edit): writes the value \
- directly through UIA `IUIAutomationValuePattern::SetValue`. This is the \
- canonical Windows write path, equivalent to Swift's `AXValue` write.\n\
- - **ComboBox**: on a native Win32 combo box (WinForms, MFC, dialogs) selects \
- the option whose text matches `value` and sends the app the notifications a \
- user's choice sends (CBN_SELENDOK, CBN_SELCHANGE), without focusing the window; \
- an editable combo gets the text typed into its edit field. A drop-down list \
- without that option is refused with the list of options. Other combo boxes get \
- a UIA value write, which the app may not treat as a selection.\n\n\
- For free-form text entry that the target accepts via keystrokes only, prefer \
- `type_text` — UIA ValuePattern writes are ignored by some web inputs (same \
- caveat as Swift's `AXValue`-vs-WebKit).".into(),
- input_schema: json!({
- "type":"object","required":["value"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"HWND of the window. Omit when element_token is supplied (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "value":{"type":"string","description":"New value. UIA will coerce to the element's native type."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- // Swift: idempotent: true (setting same value twice is idempotent).
- read_only: false, destructive: true, idempotent: true, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let cursor_key = resolve_cursor_key(&args);
- let raw_pid = args.get("pid").and_then(|v| v.as_i64());
- if raw_pid.is_none() {
- return ToolResult::error("Missing required integer field pid.");
- }
- let pid = raw_pid.unwrap() as u32;
- let value = match args.get("value").and_then(|v| v.as_str()) {
- Some(v) => v.to_owned(),
- None => return ToolResult::error("Missing required string field value."),
- };
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (index, window_id, admitted) = resolved.into_parts(None);
- let (Some(hwnd), Some(idx), Some(admitted)) = (window_id, index, admitted) else {
- return ToolResult::error("set_value requires an admitted element target.");
- };
- if !admitted.is_uia() {
- return ToolResult::error(
- "set_value requires a UIA element; MSAA value writes are unsupported.",
- );
- }
-
- // No-raise guard: a WPF/XAML automation peer calls UIElement.Focus() →
- // SetForegroundWindow during ValuePattern.SetValue. WS_EX_NOACTIVATE on
- // the target makes that a no-op while the value is still set, so a
- // background SetValue can't steal the user's foreground. Held across the
- // whole write. (No-op for delivery_mode:"foreground".)
- let _noact = if crate::input::delivery::DeliveryMode::from_args(&args)
- != crate::input::delivery::DeliveryMode::Foreground
- {
- Some(crate::input::NoActivateGuard::arm(
- windows::Win32::Foundation::HWND(hwnd as *mut _),
- ))
- } else {
- None
- };
-
- // Glide the agent cursor onto the target element before writing its
- // value, so a value write gets the same visual feedback as a click —
- // the viewer can see *where* the agent is acting. No-op when the
- // overlay is disabled or the element has no cached center.
- {
- let (cx, cy) = admitted.center;
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(Some(&admitted), (0, 0)),
- )
- .await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: cx as f64,
- y: cy as f64,
- },
- );
- }
-
- let value_for_task = value.clone();
- let result = tokio::task::spawn_blocking({
- move || -> anyhow::Result<SetValueRoute> {
- let value = value_for_task;
- let _noact = _noact;
- let ptr = admitted.as_ptr();
- use windows::core::{Interface, BSTR};
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationValuePattern, UIA_ComboBoxControlTypeId,
- UIA_ValuePatternId,
- };
- let elem = std::mem::ManuallyDrop::new(unsafe {
- IUIAutomationElement::from_raw(ptr as *mut _)
- });
- // A native Win32 combo box (or its edit field): select the
- // option and notify the app the way the control does. A UIA
- // value write would only change the displayed text.
- let native = unsafe { elem.CurrentNativeWindowHandle() }
- .ok()
- .and_then(|h| crate::win32::combo::combo_for_window(h.0 as isize));
- if let Some(combo) = native {
- let write = crate::win32::combo::set_value(combo, &value)?;
- return Ok(SetValueRoute::Win32Combo { combo, write });
- }
- let is_combo = unsafe { elem.CurrentControlType() }
- .is_ok_and(|control_type| control_type == UIA_ComboBoxControlTypeId);
- // Try ValuePattern first (text inputs, editable combos, etc).
- // The SetValue is shielded by the EnableWindow bypass: a
- // Chromium/Electron (or XAML) SetValue handler self-foregrounds via
- // SetForegroundWindow, which WS_EX_NOACTIVATE alone does not stop;
- // disabling the host for the duration does, while the a11y-channel
- // write still lands. (No-op shield for non-XAML/non-Chromium hosts.)
- if let Ok(pattern) = unsafe { elem.GetCurrentPattern(UIA_ValuePatternId) } {
- if let Ok(vp) = pattern.cast::<IUIAutomationValuePattern>() {
- let set =
- crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- vp.SetValue(&BSTR::from(value.as_str()))
- });
- if set.is_ok() {
- return Ok(if is_combo {
- SetValueRoute::ComboValuePattern
- } else {
- SetValueRoute::Pattern("ValuePattern")
- });
- }
- }
- }
- // Fall through to RangeValuePattern for Sliders / ProgressBars /
- // numeric ranges. RangeValue.SetValue takes a double, so coerce
- // the string. Documented gap-closer (PR #1699 harness exposed).
- use windows::Win32::UI::Accessibility::{
- IUIAutomationRangeValuePattern, UIA_RangeValuePatternId,
- };
- if let Ok(pattern) = unsafe { elem.GetCurrentPattern(UIA_RangeValuePatternId) } {
- if let Ok(rv) = pattern.cast::<IUIAutomationRangeValuePattern>() {
- let parsed: f64 = value.parse().map_err(|_| {
- anyhow::anyhow!(
- "set_value: target element exposes RangeValuePattern (Slider / \
- ProgressBar / numeric range). `value` must be a parseable f64; \
- got {value:?}."
- )
- })?;
- crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- rv.SetValue(parsed)
- })?;
- return Ok(SetValueRoute::Pattern("RangeValuePattern"));
- }
- }
- anyhow::bail!(
- "set_value: element [{idx}] does not implement ValuePattern or \
- RangeValuePattern. For controls with TogglePattern (CheckBox) or \
- SelectionItemPattern (RadioButton / ComboBoxItem), use the `click` \
- tool instead."
- );
- }
- })
- .await;
- match result {
- Ok(Ok(SetValueRoute::Pattern(pattern_name))) => {
- ToolResult::text(format!("✅ Set AXValue on [{idx}] (UIA {pattern_name})."))
- }
- Ok(Ok(SetValueRoute::ComboValuePattern)) => ToolResult::text(format!(
- "Set the UIA value of combo box [{idx}] to '{value}'. This combo box is not a \
- native Win32 one, and a UIA value write may change the shown text without \
- the app treating it as a selection. Check with get_window_state; if the app \
- did not react, click the combo box and then the option."
- ))
- .with_structured(json!({ "path": "ax", "verified": false, "effect": "unverifiable" })),
- Ok(Ok(SetValueRoute::Win32Combo { combo, write })) => {
- win32_combo_set_value_result(idx, &value, combo, write, &args)
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-/// How `set_value` wrote the value.
-enum SetValueRoute {
- /// A UIA pattern on a non-combo element (`ValuePattern`, `RangeValuePattern`).
- Pattern(&'static str),
- /// UIA `ValuePattern` on a combo box that has no native Win32 window.
- ComboValuePattern,
- /// Win32 combo messages plus the app's change notifications.
- Win32Combo {
- combo: crate::win32::combo::Combo,
- write: crate::win32::combo::ComboWrite,
- },
-}
-
-fn win32_combo_set_value_result(
- idx: usize,
- value: &str,
- combo: crate::win32::combo::Combo,
- write: crate::win32::combo::ComboWrite,
- args: &Value,
-) -> ToolResult {
- use crate::win32::combo::ComboWrite;
- use cua_driver_core::action_record::{
- ActionEffect, ActionEvidence, ActionExecutionRecordBuilder, ActionTransport,
- ActualDelivery, EvidenceKind, RequestedDelivery,
- };
- let kind = if combo.editable {
- "editable combo box"
- } else {
- "drop-down combo box"
- };
- let (selected, unchanged, readback_text, readback_index) = match &write {
- ComboWrite::NoSuchOption { options } => {
- let message = format!(
- "set_value: the {kind} [{idx}] has no option '{value}'; nothing was changed. \
- Options: {options:?}."
- );
- return ToolResult::error(message).with_structured(json!({
- "code": "option_not_found",
- "effect": "none",
- "options": options,
- }));
- }
- ComboWrite::Applied {
- selected,
- unchanged,
- readback_text,
- readback_index,
- } => (
- *selected,
- *unchanged,
- readback_text.clone(),
- *readback_index,
- ),
- };
- let verified = crate::win32::combo::write_verified(&write, value);
- let seen = readback_text.clone().unwrap_or_default();
- let how = match (combo.editable, selected) {
- (false, _) => "CB_SETCURSEL, then CBN_SELENDOK and CBN_SELCHANGE to the app",
- (true, Some(_)) => {
- "typed into its edit field, then CB_SETCURSEL with CBN_SELENDOK and CBN_SELCHANGE \
- to the app"
- }
- (true, None) => "typed into its edit field (the app gets CBN_EDITCHANGE)",
- };
- let text = if unchanged {
- format!("✅ The {kind} [{idx}] already reads '{seen}'; nothing was sent.")
- } else if verified {
- format!("✅ Set the {kind} [{idx}] to '{seen}' ({how}); read back '{seen}'.")
- } else {
- format!(
- "Sent '{value}' to the {kind} [{idx}] ({how}), but it reads back '{seen}' \
- (selected index {readback_index:?}). Check with get_window_state."
- )
- };
- let requested = match args.get("delivery_mode").and_then(Value::as_str) {
- Some("foreground") => RequestedDelivery::Foreground,
- _ => RequestedDelivery::Background,
- };
- let effect = if verified {
- ActionEffect::Confirmed
- } else {
- ActionEffect::Unverifiable
- };
- let mut record =
- ActionExecutionRecordBuilder::new(effect, ActionTransport::WindowsPostMessage, requested)
- // Window messages reach the control without focusing or raising it.
- .actual_delivery(ActualDelivery::Background)
- .detail(text.clone());
- if verified {
- record = record.evidence(ActionEvidence {
- kind: EvidenceKind::ValueReadback,
- detail: format!("combo box reads back '{seen}' (index {readback_index:?})"),
- });
- }
- match record.build() {
- Ok(record) => ToolResult::text(text).with_action_record(record),
- Err(error) => ToolResult::error(format!("set_value: invalid action record: {error:?}")),
- }
-}
-
-// ── scroll ────────────────────────────────────────────────────────────────────
-
-pub struct ScrollTool {
- state: Arc<ToolState>,
-}
-static SCROLL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ScrollTool {
- fn def(&self) -> &ToolDef {
- SCROLL_DEF.get_or_init(|| ToolDef {
- name: "scroll".into(),
- // Description ported from Swift `ScrollTool.swift` with the
- // Windows-specific transport note (WM_VSCROLL/WM_HSCROLL).
- description: "Scroll the target pid's focused region.\n\n\
- Windows transport: WM_VSCROLL / WM_HSCROLL posted to the window — the same \
- events the OS sends when scrollbars or trackpads scroll, so any window that \
- handles scrollbars correctly responds to this. Swift uses synthesized \
- keystrokes (PageDown / arrow keys) via auth-signed SLEventPostToPid; both \
- approaches reach backgrounded windows.\n\n\
- Mapping: `by: \"page\"` → SB_PAGEDOWN/UP/LEFT/RIGHT × amount; \
- `by: \"line\"` → SB_LINEDOWN/UP/LEFT/RIGHT × amount.\n\n\
- Note: `element_token` is accepted for cross-platform parity but currently \
- no-op on Windows (UIA SetFocus not wired up yet — same caveat as `press_key`).".into(),
- input_schema: json!({
- "type":"object","required":["direction"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID for window scope. Optional with element_token (the token carries the pid). Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
- "direction":{"type":"string","enum":["up","down","left","right"],"description":"Scroll direction."},
- "by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},
- "amount":{"type":"integer","minimum":1,"maximum":50,
- "description":"Number of scroll ticks. Default 3."},
- "x":{"type":"number","description":"With pid/window_id: window-local screenshot X used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute X for desktop scope. Must be paired with y."},
- "y":{"type":"number","description":"With pid/window_id: window-local screenshot Y used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute Y for desktop scope. Must be paired with x."},
- "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with x,y and no pid/window_id for screen-absolute coordinates from get_desktop_state. Default \"window\"."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false
- }),
- read_only: false, destructive: false, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use cua_driver_core::tool_args::ArgsExt;
- // ── Window-less screen-absolute branch (desktop target) ───────────────
- // No pid/window_id + numeric x,y + desktop scope → synthesize a wheel
- // event at the screen point via SendInput. The wheel routes to whatever
- // window is under (x,y). up/down map to a vertical wheel (sign), and
- // left/right to a horizontal wheel; `amount` is the tick count.
- let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
- let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
- let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
- && args.get("y").map(|v| v.is_number()).unwrap_or(false);
- if !has_pid && !has_window_id && has_xy {
- if !is_windowless_desktop_action(&args) {
- return ToolResult::error(
- "scroll: x,y given with no pid/window_id requires scope=\"desktop\"; \
- use get_desktop_state to pick coordinates, or pass a pid/window_id.",
- )
- .with_structured(serde_json::json!({
- "code": "desktop_coordinate_scope_required",
- "suggestion": "pass scope=desktop",
- }));
- }
- let input = match parse_typed_projection::<ScrollInput>("scroll", &args) {
- Ok(input) => input,
- Err(error) => return error,
- };
- let sx = input.x as i32;
- let sy = input.y as i32;
- let direction = input.direction;
- let amount = input.amount.unwrap_or(3) as u32;
- // Direction → (horizontal?, sign). Positive ticks = up / right.
- let (horizontal, sign) = match direction {
- ScrollDirection::Up => (false, 1),
- ScrollDirection::Down => (false, -1),
- ScrollDirection::Right => (true, 1),
- ScrollDirection::Left => (true, -1),
- };
- let ticks = sign * amount as i32;
- let dir_disp = direction.as_str();
- let result = tokio::task::spawn_blocking(move || {
- crate::input::send_wheel_synthesized(sx, sy, ticks, horizontal)
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Scrolled {dir_disp} via SendInput wheel ({amount} tick(s)) at screen ({sx},{sy}) (desktop scope)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Window-scoped arguments retain the richer platform contract.
- let direction = match args.get("direction").and_then(|v| v.as_str()) {
- Some(d) => d.to_owned(),
- None => return ToolResult::error("Missing required string field direction."),
- };
- let amount = args.u64_or("amount", 3).clamp(1, 50) as u32;
-
- // Swift error wording 1:1.
- let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(p) => p,
- None => return ToolResult::error("Missing required integer field pid."),
- };
- let pid = raw_pid as u32;
- let delivery = DeliveryMode::from_args(&args);
- let by = args.str_or("by", "line");
- let direction_display = direction.clone();
- let by_display = by.clone();
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
-
- // Resolve HWND: use window_id if given, else first window for pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // WebView2/Tauri hosts can expose the indexed scroll container through
- // UIA while their top-level HWND ignores WM_VSCROLL. Prefer the
- // accessibility channel for an indexed target; the message path below
- // remains the fallback for native Win32 scrollbars.
- if delivery == DeliveryMode::Background && crate::input::is_chromium_target_window(hwnd) {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::MouseScroll,
- );
- }
- if let Some(idx) = elem_idx {
- let prev_fg_addr = if delivery == DeliveryMode::Background {
- Some(unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as isize
- })
- } else {
- None
- };
- let _noact = if delivery == DeliveryMode::Background {
- Some(crate::input::NoActivateGuard::arm(
- windows::Win32::Foundation::HWND(hwnd as *mut _),
- ))
- } else {
- None
- };
- let direction_for_uia = direction.clone();
- let uia_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- let retained = admitted.as_ref().ok_or_else(|| {
- anyhow::anyhow!("element [{idx}] is not in the UIA cache")
- })?;
- if !retained.is_uia() {
- anyhow::bail!("element [{idx}] is not a UIA scroll element");
- }
- unsafe {
- crate::uia::scroll::scroll_element(
- retained.as_ptr(),
- &direction_for_uia,
- amount,
- )
- }
- }
- })
- .await;
- if matches!(uia_result, Ok(Ok(()))) {
- if delivery == DeliveryMode::Background {
- // Keep WS_EX_NOACTIVATE armed through any WebView handler
- // queued by the UIA operation.
- tokio::time::sleep(std::time::Duration::from_millis(50)).await;
- if let Some(previous_addr) = prev_fg_addr {
- use windows::Win32::Foundation::HWND;
- use windows::Win32::UI::WindowsAndMessaging::{
- GetAncestor, GetForegroundWindow, GA_ROOT,
- };
- let previous = HWND(previous_addr as *mut _);
- let current = unsafe { GetForegroundWindow() };
- let current_root = unsafe { GetAncestor(current, GA_ROOT) };
- let target_root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };
- if !previous.0.is_null()
- && current != previous
- && !target_root.0.is_null()
- && current_root == target_root
- {
- unsafe {
- crate::input::force_foreground_attached(previous);
- std::thread::sleep(std::time::Duration::from_millis(12));
- crate::input::force_foreground_attached(previous);
- }
- }
- }
- }
- return ToolResult::text(format!(
- "Scrolled {direction} {amount} ticks via UIA (delivery_mode:background)."
- ))
- .with_structured(serde_json::json!({
- "path": "uia",
- "verified": false,
- "delivery_mode": "background"
- }));
- }
- }
-
- if delivery == DeliveryMode::Background
- && args.get("x").is_some_and(serde_json::Value::is_number)
- && args.get("y").is_some_and(serde_json::Value::is_number)
- {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::MouseScroll,
- );
- }
-
- // delivery_mode:"background" — WM_VSCROLL/HSCROLL is silently dropped by
- // Chromium and may be by GTK. Surface the standard structured
- // background_unavailable error: its remediation (bring_to_front +
- // delivery_mode:"foreground") is now a real recovery because scroll's
- // foreground path is implemented below (SendInput wheel reaches the
- // Chromium/GTK targets PostMessage can't).
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseScroll)
- {
- return crate::input::delivery::background_unavailable_error(
- hwnd,
- EventKind::MouseScroll,
- );
- }
- // delivery_mode:"foreground" — SendInput MOUSEEVENTF_WHEEL at the target
- // window's screen center (send_wheel_synthesized). The wheel routes to
- // the window under the cursor — queue-origin input Chromium/GTK accept —
- // so it lands where PostMessage WM_*SCROLL is dropped. Mirrors the
- // desktop-scope wheel branch above; `by` (line/page) folds into the tick
- // count (page ≈ 3 detents).
- if delivery == DeliveryMode::Foreground {
- let (horizontal, sign) = match direction.as_str() {
- "up" => (false, 1),
- "down" => (false, -1),
- "right" => (true, 1),
- "left" => (true, -1),
- other => {
- return ToolResult::error(format!(
- "scroll: unknown direction \"{other}\" — expected up, down, left, right."
- ))
- }
- };
- let per: i32 = if by == "page" { 3 } else { 1 };
- let ticks = sign * (amount as i32) * per;
- // A supplied PX target is window-local in the get_window_state
- // bitmap. Route the wheel there so nested web scrollers receive it;
- // otherwise retain the whole-window center fallback.
- let px = args.get("x").and_then(|value| value.as_f64());
- let py = args.get("y").and_then(|value| value.as_f64());
- if px.is_some() != py.is_some() {
- return ToolResult::error("scroll requires x and y together.");
- }
- let center = if let (Some(x), Some(y)) = (px, py) {
- let (x, y) = match super::scroll_coordinates::window_bitmap_point(
- &self.state.snapshots,
- &args,
- pid,
- hwnd,
- x,
- y,
- ) {
- Ok(point) => point,
- Err(refusal) => return refusal,
- };
- Some(bitmap_to_screen(hwnd, x, y))
- } else {
- tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- .into_iter()
- .find(|w| w.hwnd == hwnd)
- .map(|w| (w.x + w.width / 2, w.y + w.height / 2))
- }
- })
- .await
- .ok()
- .flatten()
- };
- let (cx, cy) = match center {
- Some(c) => c,
- None => {
- return ToolResult::error(format!(
- "scroll: could not resolve on-screen bounds for window {hwnd:#x} \
- (pid {pid}) to target the wheel. The window may be minimized or \
- off-screen — call bring_to_front first."
- ))
- }
- };
- let dir_disp = direction.clone();
- let tick_disp = ticks.abs();
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_wheel_synthesized(cx, cy, ticks, horizontal)
- }
- })
- .await;
- return match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Scrolled {dir_disp} via SendInput wheel ({tick_disp} tick(s)) at \
- screen ({cx},{cy}) (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || -> anyhow::Result<()> {
- let _admission = &admitted;
- use windows::Win32::Foundation::{HWND, LPARAM, WPARAM};
- use windows::Win32::UI::WindowsAndMessaging::{
- PostMessageW, SB_LINEDOWN, SB_LINELEFT, SB_LINERIGHT, SB_LINEUP, SB_PAGEDOWN,
- SB_PAGELEFT, SB_PAGERIGHT, SB_PAGEUP, WM_HSCROLL, WM_VSCROLL,
- };
-
- let hwnd_win = HWND(hwnd as *mut _);
- let use_page = by == "page";
- let (msg, code) = match (direction.as_str(), use_page) {
- ("up", false) => (WM_VSCROLL, SB_LINEUP),
- ("up", true) => (WM_VSCROLL, SB_PAGEUP),
- ("down", false) => (WM_VSCROLL, SB_LINEDOWN),
- ("down", true) => (WM_VSCROLL, SB_PAGEDOWN),
- ("left", false) => (WM_HSCROLL, SB_LINELEFT),
- ("left", true) => (WM_HSCROLL, SB_PAGELEFT),
- ("right", false) => (WM_HSCROLL, SB_LINERIGHT),
- ("right", true) => (WM_HSCROLL, SB_PAGERIGHT),
- _ => (WM_VSCROLL, SB_LINEDOWN),
- };
- for _ in 0..amount {
- unsafe {
- PostMessageW(hwnd_win, msg, WPARAM(code.0 as usize), LPARAM(0))?;
- }
- }
- Ok(())
- }
- })
- .await;
-
- match result {
- Ok(Ok(())) => {
- // Match Swift's text format shape 1:1 (Swift uses key names
- // like "pagedown"; Windows uses Win32 SB_* constants — show
- // the actual mechanism for traceability).
- let mech_name = match (direction_display.as_str(), by_display.as_str()) {
- ("up", "page") => "SB_PAGEUP",
- ("down", "page") => "SB_PAGEDOWN",
- ("left", "page") => "SB_PAGELEFT",
- ("right", "page") => "SB_PAGERIGHT",
- ("up", _) => "SB_LINEUP",
- ("down", _) => "SB_LINEDOWN",
- ("left", _) => "SB_LINELEFT",
- ("right", _) => "SB_LINERIGHT",
- _ => "SB_LINEDOWN",
- };
- ToolResult::text(format!(
- "✅ Scrolled pid {raw_pid} {direction_display} via {amount}× {mech_name} message(s)."
- ))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// `ScreenshotTool` and `ScreenshotCompatTool` were removed in PR #1692 —
-// `get_window_state` (which now always returns a screenshot) is the single
-// canonical screenshot path. The underlying capture functions
-// (`crate::capture::screenshot_window_bytes_with_occlusion`,
-// `screenshot_display_bytes`, etc.) and the WGC backend
-// (`crate::wgc::screenshot_window_via_wgc`) are still in use by
-// `GetWindowStateTool` — that's where the actual screenshot machinery
-// lives now.
-
-/// Chromium/Electron windows silently drop `PostMessage` mouse events — their
-/// input thread only honors SendInput-origin events (#1623). `ClickTool`
-/// auto-routes Chromium targets through SendInput, but `DoubleClickTool` /
-/// `RightClickTool` did not, so element/pixel gestures on Electron apps
-/// (Obsidian, VS Code, Slack, …) reached `post_click_screen` and no-op'd
-/// silently. This mirrors that short-circuit for those tools (#1984): when
-/// `hwnd` is a Chromium window, deliver `count` clicks of `button` at screen
-/// `(sx, sy)` via SendInput with async foreground restore and return
-/// `Some(result)`. Returns `None` for non-Chromium targets so the caller
-/// proceeds to its normal PostMessage path.
-async fn chromium_click_short_circuit(
- hwnd: u64,
- sx: i32,
- sy: i32,
- count: usize,
- button: &str,
- pid: u32,
- gesture: &str,
-) -> Option<ToolResult> {
- let is_chromium =
- tokio::task::spawn_blocking(move || crate::input::is_chromium_target_window(hwnd))
- .await
- .unwrap_or(false);
- if !is_chromium {
- return None;
- }
- // Capture the pre-click foreground so the poller can restore it even if
- // Chromium re-activates itself from a renderer-side handler (same pattern
- // and rationale as the ClickTool Chromium branch).
- let prev_fg_addr =
- unsafe { windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize };
- let button_owned = button.to_string();
- let send_result = tokio::task::spawn_blocking(move || {
- crate::input::send_click_synthesized(hwnd, sx, sy, count, &button_owned)
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- Some(match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent {gesture} via SendInput to pid {pid} at screen ({sx},{sy}) (Chromium target)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- })
-}
-
-/// Fire UIA `InvokePattern.Invoke()` `count` times (min 2) in rapid succession
-/// on a cached WinUI3 element. Two rapid Invokes land within the system
-/// double-click time, so a Button's Click handler folds them into a
-/// double-click (the harness records `last_action=double_click`).
-///
-/// This is the **only** background actuator that lands on a WinUI3 element: the
-/// content island ignores posted `WM_*BUTTON` messages (measured: a posted
-/// secondary click never fires `RightTapped`), and the synthetic pen/touch
-/// injector click-activates the frame (8/8 foreground steals). UIA delivery
-/// goes through the accessibility channel, not the input queue.
-///
-/// Contract: a XAML `Invoke` handler calls `SetFocus()` → `SetForegroundWindow(self)`,
-/// which the `EnableWindow` UWP bypass does NOT gate (measured: a bare double
-/// Invoke stole foreground). We therefore hold the frame **non-activatable**
-/// (`WS_EX_NOACTIVATE`, via `NoActivateGuard`) for the whole burst, which is
-/// what makes Windows refuse the self-activation — the same guard ClickTool
-/// already relies on for single-click. Runs blocking COM, so call inside
-/// `spawn_blocking`. Returns `Some(Ok)` on success, `Some(Err)` if Invoke
-/// failed, `None` if the element isn't cached or has no InvokePattern.
-fn winui3_uia_multi_invoke(
- admitted: &Option<crate::uia::snapshot::RetainedElement>,
- hwnd: u64,
- count: usize,
-) -> Option<anyhow::Result<()>> {
- let guard = admitted.as_ref().filter(|element| element.is_uia())?;
- let ptr = guard.as_ptr();
- use windows::core::Interface;
- use windows::Win32::UI::Accessibility::{
- IUIAutomationElement, IUIAutomationInvokePattern, UIA_InvokePatternId,
- };
- // Hold the frame non-activatable so the XAML Invoke handler's
- // SetFocus()→SetForegroundWindow(self) is refused for the whole burst.
- let _noact =
- crate::input::NoActivateGuard::arm(windows::Win32::Foundation::HWND(hwnd as *mut _));
- let elem: IUIAutomationElement = unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
- let outcome: Option<anyhow::Result<()>> = (|| {
- let pattern = unsafe { elem.GetCurrentPattern(UIA_InvokePatternId) }.ok()?;
- let inv = pattern.cast::<IUIAutomationInvokePattern>().ok()?;
- let n = count.max(2);
- let mut res = Ok(());
- for i in 0..n {
- let r = crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- inv.Invoke()
- });
- if let Err(e) = r {
- res = Err(anyhow::anyhow!("UIA Invoke #{i}: {e}"));
- break;
- }
- if i + 1 < n {
- std::thread::sleep(std::time::Duration::from_millis(40));
- }
- }
- Some(res)
- })();
- // The cache owns this element's ref (AddRef'd under the cache lock by
- // get_element_retained); `from_raw` took ownership without AddRef, so forget
- // it to avoid double-releasing the cache's ref.
- std::mem::forget(elem);
- outcome
-}
-
-/// WinUI3 background gesture resolution, shared by click / double_click /
-/// right_click / drag. For a WinUI3 (`WinUIDesktopWin32WindowClass`) target:
-///
-/// - **double-click (left)** on a cached InvokePattern element → a double UIA
-/// Invoke (lands + holds the no-foreground contract; see
-/// [`winui3_uia_multi_invoke`]).
-/// - **everything else** — right/middle click, double-click without an
-/// invokable cached element, any drag — cannot both land AND hold the
-/// contract on WinUI3: UIA has no right-click/drag analogue, the content
-/// island ignores posted `WM_*BUTTON`, and the pointer injector
-/// click-activates the frame. So we return the structured
-/// `background_unavailable` error (the honest result — the caller can retry
-/// that action with `delivery_mode:"foreground"`) instead of a silent
-/// no-op that reports false success.
-///
-/// Returns `None` for non-WinUI3 targets (caller falls through to its normal
-/// routing).
-async fn winui3_background_gesture(
- admitted: &Option<crate::uia::snapshot::RetainedElement>,
- pid: u32,
- hwnd: u64,
- idx: Option<usize>,
- count: usize,
- button: &str,
-) -> Option<ToolResult> {
- let is_w = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::delivery::is_winui3_target_window(hwnd)
- }
- })
- .await
- .unwrap_or(false);
- if !is_w {
- return None;
- }
- if count >= 2 && button == "left" {
- if let Some(idx) = idx {
- let uia = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- winui3_uia_multi_invoke(&admitted, hwnd, count)
- }
- })
- .await
- .ok()
- .flatten();
- if let Some(Ok(())) = uia {
- return Some(ToolResult::text(format!(
- "✅ Double-invoked WinUI3 element [{idx}] x{count} via UIA \
- (pid {pid}, background, no foreground swap)."
- )));
- }
- }
- }
- Some(crate::input::delivery::background_unavailable_error(
- hwnd,
- crate::input::delivery::EventKind::MouseClick,
- ))
-}
-
-// ── double_click ──────────────────────────────────────────────────────────────
-
-pub struct DoubleClickTool {
- state: Arc<ToolState>,
-}
-
-static DCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for DoubleClickTool {
- fn def(&self) -> &ToolDef {
- DCLICK_DEF.get_or_init(|| ToolDef {
- name: "double_click".into(),
- // Description ported from Swift `DoubleClickTool.swift` semantics
- // (two addressing modes, AXOpen analogue on Windows is not yet
- // wired up — element path always falls back to a stamped pixel
- // double-click via PostMessage).
- description: "Double-click against a target pid. Two addressing modes:\n\n\
- - `element_token` (from the last `get_window_state` snapshot \
- of that window) — synthesizes a stamped pixel double-click at the element's \
- cached on-screen center via PostMessage. (Windows has no AXOpen analogue; \
- Swift's AXOpen-first path falls through to the same pixel recipe when the \
- element doesn't advertise AXOpen, so the user-visible behavior matches.)\n\n\
- - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
- returned by `get_window_state`) — posts two WM_LBUTTONDOWN/UP pairs in \
- quick succession to the deepest child window at that point.\n\n\
- Exactly one of `element_token` or (`x` AND `y`) must be provided. \
- `pid` is required with `x`, `y`; an `element_token` carries its own pid. The macOS-only `modifier` field is accepted for \
- parity (no-op on Windows — PostMessage doesn't propagate modifier-key state).".into(),
- input_schema: json!({"type":"object","required":[],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
- "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- // Swift error wording 1:1.
- let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(p) => p,
- None => return ToolResult::error("Missing required integer field pid."),
- };
- let pid = raw_pid as u32;
- use cua_driver_core::tool_args::ArgsExt;
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let x = args.opt_f64("x");
- let y = args.opt_f64("y");
- let delivery = DeliveryMode::from_args(&args);
- let cursor_key = resolve_cursor_key(&args);
- // Swift validates "both x and y or neither" and "no element_token without window_id".
- let has_xy = x.is_some() && y.is_some();
- let partial_xy = x.is_some() != y.is_some();
- if partial_xy {
- return ToolResult::error("Provide both x and y together, not just one.");
- }
- if elem_idx.is_some() && has_xy {
- return ToolResult::error("Provide either element_token or (x, y), not both.");
- }
- if elem_idx.is_none() && !has_xy {
- return ToolResult::error(
- "Provide element_token or (x, y) to address the double-click target.",
- );
- }
-
- // Resolve HWND: explicit, or auto from pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- if let Some(idx) = elem_idx {
- let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
- Some(v) => v,
- None => {
- return ToolResult::error(format!(
- "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
- ))
- }
- };
- let recorded_center = (cx, cy);
- let (cx, cy) = match resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx,
- cx,
- cy,
- "double-clicking",
- ) {
- Ok(p) => p,
- Err(result) => return result,
- };
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(
- admitted.as_ref(),
- (cx - recorded_center.0, cy - recorded_center.1),
- ),
- )
- .await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: cx as f64,
- y: cy as f64,
- },
- );
- // delivery_mode:"background" (default) on WinUI3: the pen/touch injector
- // click-activates the content island (8/8 foreground steals) and
- // posted WM_*BUTTON to the frame/island no-ops (measured). A double
- // UIA Invoke on the cached element is the only contract-holding way
- // to land the double-click — it fires the XAML Click handler twice
- // → last_action=double_click, held non-activatable so no foreground
- // swap. If the element has no InvokePattern, a background double-
- // click isn't expressible → structured error.
- if delivery == DeliveryMode::Background {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, Some(idx), 2, "left").await
- {
- return r;
- }
- }
- // delivery_mode:"background" (default): Chromium/Electron & GTK targets
- // silently drop posted clicks (#1984) — inject via the coordinate
- // actuator (system input queue, NO foreground swap), exactly like
- // ClickTool, instead of refusing. Only error if injection can't
- // express this click (e.g. right/middle on such a target).
- if (cx, cy) != recorded_center {
- crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
- }
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
- {
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, cx, cy, 2, "left")
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected double-click to [{idx}] at screen ({cx},{cy}) (background, no foreground swap)."
- )),
- Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- e.to_string(),
- ),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // delivery_mode:"foreground" — route through SendInput at the cached coords.
- if delivery == DeliveryMode::Foreground {
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_click_synthesized_active_mods(
- hwnd,
- cx,
- cy,
- 2,
- "left",
- &[],
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent double-click via SendInput on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
- if let Some(r) =
- chromium_click_short_circuit(hwnd, cx, cy, 2, "left", pid, "double-click").await
- {
- return r;
- }
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || -> anyhow::Result<String> {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, cx, cy, 2, "left")?;
- // Swift text format 1:1: `"✅ Posted double-click to [N] role \"title\" at screen-point (X, Y)."`.
- // UIA role/title placeholder pending element-cache enrichment.
- Ok(format!(
- "✅ Posted double-click to [{idx}] at screen-point ({cx}, {cy})."
- ))
- }
- })
- .await;
- match result {
- Ok(Ok(msg)) => ToolResult::text(msg),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- } else if let (Some(mut px), Some(mut py)) = (x, y) {
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if from_zoom {
- match self.state.zoom_context(&args, pid, Some(hwnd)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(px, py);
- px = wx;
- py = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- px *= ratio;
- py *= ratio;
- }
- // bitmap pixels -> screen via DWM-frame origin (see
- // `bitmap_to_screen` doc for why ClientToScreen is wrong).
- let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
- let (sx, sy) = (sx_i as f64, sy_i as f64);
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to(&cursor_key, sx, sy).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- // delivery_mode:"background" on WinUI3 (pixel path, no element_token): a
- // background double-click on WinUI3 only lands via UIA Invoke on a
- // cached element, which the pixel path doesn't have → structured
- // error (caller retries with delivery_mode:foreground or uses element_index).
- if delivery == DeliveryMode::Background {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, None, 2, "left").await
- {
- return r;
- }
- }
- // delivery_mode:"background" (default): inject via the coordinate actuator
- // (no foreground swap) for targets that drop posted clicks (#1984).
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
- {
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, sx_i, sy_i, 2, "left")
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected double-click to pid {pid} at screen ({sx_i},{sy_i}) (background, no foreground swap)."
- )),
- Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- e.to_string(),
- ),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // delivery_mode:"foreground" — SendInput at screen coords with FG swap.
- if delivery == DeliveryMode::Foreground {
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_click_synthesized_active_mods(
- hwnd,
- sx_i,
- sy_i,
- 2,
- "left",
- &[],
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent double-click via SendInput to pid {pid} at screen ({sx_i},{sy_i}) (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- let (xi, yi) = (px as i32, py as i32);
- // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
- if let Some(r) =
- chromium_click_short_circuit(hwnd, sx_i, sy_i, 2, "left", pid, "double-click").await
- {
- return r;
- }
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, sx_i, sy_i, 2, "left")
- }
- })
- .await;
- match result {
- Ok(Ok(())) => {
- // Match Swift's pixel-path text 1:1.
- ToolResult::text(format!(
- "✅ Posted double-click to pid {pid} at window-pixel ({xi}, {yi}) → screen-point ({sx_i}, {sy_i})."))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- } else {
- // Unreachable — guarded by the earlier validation block.
- unreachable!()
- }
- }
-}
-
-// ── right_click ───────────────────────────────────────────────────────────────
-
-pub struct RightClickTool {
- state: Arc<ToolState>,
-}
-
-static RCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for RightClickTool {
- fn def(&self) -> &ToolDef {
- RCLICK_DEF.get_or_init(|| ToolDef {
- name: "right_click".into(),
- // Description ported from Swift `RightClickTool.swift` semantics.
- // Windows uses PostMessage(WM_RBUTTONDOWN/UP); the AXShowMenu
- // analogue (UIA ShowContextMenu) is not yet wired up, so element
- // path falls through to the pixel recipe at the cached center.
- description: "Right-click against a target pid. Two addressing modes:\n\n\
- - `element_token` (from the last `get_window_state` snapshot \
- of that window) — posts WM_RBUTTONDOWN/UP at the element's cached on-screen \
- center via PostMessage. (Windows has no AXShowMenu analogue wired up yet; \
- Swift's AX-action path falls through to the same pixel recipe on non-\
- advertising elements, so user-visible behavior matches.)\n\n\
- - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
- returned by `get_window_state`) — posts WM_RBUTTONDOWN/UP to the deepest \
- child window at that point.\n\n\
- Exactly one of `element_token` or (`x` AND `y`) must be provided. `pid` is \
- required with `x`, `y`; an `element_token` carries its own pid. \
- `modifier` is accepted for parity (no-op on Windows — PostMessage doesn't \
- propagate modifier-key state).".into(),
- input_schema: json!({"type":"object","required":[],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
- "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied (the token carries it)."},
- "element_token": cua_driver_core::tool_schema::element_token_schema(),
- "x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
- "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- // Swift error wording 1:1.
- let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(p) => p,
- None => return ToolResult::error("Missing required integer field pid."),
- };
- let pid = raw_pid as u32;
- use cua_driver_core::tool_args::ArgsExt;
- let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
- Ok(r) => r,
- Err(e) => return e,
- };
- let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
- let x = args.opt_f64("x");
- let y = args.opt_f64("y");
- let delivery = DeliveryMode::from_args(&args);
- let cursor_key = resolve_cursor_key(&args);
- // Port Swift's full validation set.
- let has_xy = x.is_some() && y.is_some();
- let partial_xy = x.is_some() != y.is_some();
- if partial_xy {
- return ToolResult::error("Provide both x and y together, not just one.");
- }
- if elem_idx.is_some() && has_xy {
- return ToolResult::error("Provide either element_token or (x, y), not both.");
- }
- if elem_idx.is_none() && !has_xy {
- return ToolResult::error(
- "Provide element_token or (x, y) to address the right-click target.",
- );
- }
-
- // Resolve HWND: explicit, or auto from pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::win32::list_windows(Some(pid))
- }
- })
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- if let Some(idx) = elem_idx {
- let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
- Some(v) => v,
- None => {
- return ToolResult::error(format!(
- "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
- ))
- }
- };
- let recorded_center = (cx, cy);
- let (cx, cy) = match resolve_onscreen_point_with_scroll(
- &admitted,
- hwnd,
- idx,
- cx,
- cy,
- "right-clicking",
- ) {
- Ok(p) => p,
- Err(result) => return result,
- };
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to_target(
- &cursor_key,
- cx as f64,
- cy as f64,
- element_target_rect(
- admitted.as_ref(),
- (cx - recorded_center.0, cy - recorded_center.1),
- ),
- )
- .await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: cx as f64,
- y: cy as f64,
- },
- );
- // delivery_mode:"background" on WinUI3: a right-click can't both land and
- // hold the contract — UIA has no right-click pattern, the content
- // island ignores posted WM_RBUTTON (measured: RightTapped never
- // fired), and the pen-barrel injector click-activates the frame.
- // Return the structured error (retry with delivery_mode:foreground).
- if delivery == DeliveryMode::Background {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, Some(idx), 1, "right").await
- {
- return r;
- }
- }
- // delivery_mode:"background" (default): try coordinate injection (no
- // foreground swap) for drop-prone targets (#1984); a right-click
- // injection that the actuator can't express falls back to the error.
- if (cx, cy) != recorded_center {
- crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
- }
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
- {
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, cx, cy, 1, "right")
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected right-click to [{idx}] at screen ({cx},{cy}) (background, no foreground swap)."
- )),
- Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- e.to_string(),
- ),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- if delivery == DeliveryMode::Foreground {
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_click_synthesized_active_mods(
- hwnd,
- cx,
- cy,
- 1,
- "right",
- &[],
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent right-click via SendInput on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
- if let Some(r) =
- chromium_click_short_circuit(hwnd, cx, cy, 1, "right", pid, "right-click").await
- {
- return r;
- }
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || -> anyhow::Result<String> {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, cx, cy, 1, "right")?;
- // Match Swift's element-path text 1:1
- // (`"✅ Shown menu for [N] role \"title\"."`). UIA role/title
- // placeholder pending element-cache enrichment.
- Ok(format!("✅ Shown menu for [{idx}] (screen ({cx}, {cy}))."))
- }
- })
- .await;
- match result {
- Ok(Ok(msg)) => ToolResult::text(msg),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- } else if let (Some(mut px), Some(mut py)) = (x, y) {
- let from_zoom = args
- .get("from_zoom")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if from_zoom {
- match self.state.zoom_context(&args, pid, Some(hwnd)) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(px, py);
- px = wx;
- py = wy;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- px *= ratio;
- py *= ratio;
- }
- // bitmap pixels -> screen via DWM-frame origin (see
- // `bitmap_to_screen` doc).
- let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
- let (sx, sy) = (sx_i as f64, sy_i as f64);
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to(&cursor_key, sx, sy).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
- // delivery_mode:"background" on WinUI3: right-click can't land while
- // holding the contract (no UIA right-click, posted WM_RBUTTON
- // ignored, pen-barrel injector steals foreground) → structured error.
- if delivery == DeliveryMode::Background {
- if let Some(r) =
- winui3_background_gesture(&admitted, pid, hwnd, None, 1, "right").await
- {
- return r;
- }
- }
- // delivery_mode:"background" (default): try coordinate injection (no
- // foreground swap) for drop-prone targets (#1984).
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
- {
- let inj = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::inject_click_screen(hwnd, sx_i, sy_i, 1, "right")
- }
- })
- .await;
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Injected right-click to pid {pid} at screen ({sx_i},{sy_i}) (background, no foreground swap)."
- )),
- Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseClick,
- e.to_string(),
- ),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- if delivery == DeliveryMode::Foreground {
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::send_click_synthesized_active_mods(
- hwnd,
- sx_i,
- sy_i,
- 1,
- "right",
- &[],
- )
- }
- })
- .await;
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent right-click via SendInput to pid {pid} at screen ({sx_i},{sy_i}) (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- let (xi, yi) = (px as i32, py as i32);
- // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
- if let Some(r) =
- chromium_click_short_circuit(hwnd, sx_i, sy_i, 1, "right", pid, "right-click").await
- {
- return r;
- }
- let result = tokio::task::spawn_blocking({
- let admitted = admitted.clone();
- move || {
- let _admission = &admitted;
- crate::input::post_click_screen(hwnd, sx_i, sy_i, 1, "right")
- }
- })
- .await;
- match result {
- Ok(Ok(())) => {
- // Swift pixel-path text 1:1.
- ToolResult::text(format!(
- "✅ Posted right-click to pid {pid} at window-pixel ({xi}, {yi}) → screen-point ({sx_i}, {sy_i})."))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- } else {
- // Guarded above by the early validation block — unreachable.
- unreachable!()
- }
- }
-}
-
-// ── drag ─────────────────────────────────────────────────────────────────────
-
-pub struct DragTool {
- state: Arc<ToolState>,
-}
-
-static DRAG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for DragTool {
- fn def(&self) -> &ToolDef {
- DRAG_DEF.get_or_init(|| ToolDef {
- name: "drag".into(),
- description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. \
- duration_ms (default 500) is the wall-clock budget; steps (default 20) interpolates intermediate \
- WM_MOUSEMOVE events along the path. Background delivery does not steal focus; delivery_mode:\"foreground\" \
- activates the target and refuses with foreground_unavailable, sending nothing, when another window still \
- covers the start point. After a zoom call, pass from_zoom=true. \
- Drags that start on a window caption / title bar or resize border (i.e. moving or resizing the \
- window itself) cannot be delivered in the background — the OS move/resize loop needs real pointer \
- input — so they return background_unavailable; re-issue those with delivery_mode:\"foreground\". \
- `modifier` holds keys from press to release in delivery_mode:\"foreground\" and scope:\"desktop\" via \
- SendInput. Background drags with modifiers return background_unavailable because posted messages and \
- pen injection cannot carry live modifier-key state.".into(),
- input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id for screen-absolute coordinates."},
- "pid":{"type":"integer","description":"Target process ID."},
- "window_id":{"type":"integer","description":"Target window handle (HWND). Optional — driver picks frontmost window of pid when omitted."},
- "from_x":{"type":"number","description":"Drag-start X in window-local screenshot pixels."},
- "from_y":{"type":"number","description":"Drag-start Y in window-local screenshot pixels."},
- "to_x":{"type":"number","description":"Drag-end X in window-local screenshot pixels."},
- "to_y":{"type":"number","description":"Drag-end Y in window-local screenshot pixels."},
- "duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Wall-clock duration of drag path. Default: 500."},
- "steps":{"type":"integer","minimum":1,"maximum":200,"description":"Number of intermediate WM_MOUSEMOVE events. Default: 20."},
- "modifier": cua_driver_core::tool_schema::modifier_schema(),
- "button": cua_driver_core::tool_schema::button_schema(),
- "from_zoom":{"type":"boolean","description":"When true, coordinates are in the last zoom image for this pid."},
- "delivery_mode": crate::input::delivery::delivery_mode_schema()
- },"additionalProperties":false}),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use crate::input::delivery::{DeliveryMode, EventKind};
- use cua_driver_core::tool_args::ArgsExt;
- let cursor_key = resolve_cursor_key(&args);
- if args.get("scope").and_then(Value::as_str) == Some("desktop")
- && args.get("pid").is_none()
- && args.get("window_id").is_none()
- {
- let input = match parse_typed_projection::<DragInput>("drag", &args) {
- Ok(input) => input,
- Err(error) => return error,
- };
- let from_x = input.from_x as i32;
- let from_y = input.from_y as i32;
- let to_x = input.to_x as i32;
- let to_y = input.to_y as i32;
- let duration_ms = input.duration_ms.unwrap_or(500);
- let steps = input.steps.unwrap_or(20) as usize;
- let button = input.button.unwrap_or(ClickButton::Left).as_str();
- let modifiers = input.modifier.unwrap_or_default();
- let hwnd = match crate::input::mouse::foreground_window() {
- Ok(hwnd) => hwnd,
- Err(error) => return ToolResult::error(error.to_string()),
- };
- let native_drag = tokio::task::spawn_blocking(move || {
- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::mouse::send_drag_synthesized(
- hwnd,
- from_x,
- from_y,
- to_x,
- to_y,
- duration_ms,
- steps,
- &button,
- &modifier_refs,
- )
- });
- let visual_drag = track_overlay_drag(
- cursor_key.clone(),
- (f64::from(from_x), f64::from(from_y)),
- (f64::from(to_x), f64::from(to_y)),
- duration_ms,
- steps,
- );
- let (native_drag, ()) = tokio::join!(native_drag, visual_drag);
- return match native_drag {
- Ok(Ok(())) => ToolResult::text(format!(
- "Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."
- ))
- .with_structured(json!({
- "scope": "desktop", "path": "SendInput",
- "from": {"x": from_x, "y": from_y}, "to": {"x": to_x, "y": to_y},
- "effect": "unverifiable"
- })),
- Ok(Err(error)) => ToolResult::error(error.to_string()),
- Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),
- };
- }
- // Swift error wording 1:1.
- let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(p) => p,
- None => return ToolResult::error("Missing required integer field pid."),
- };
- let pid = raw_pid as u32;
- let delivery = DeliveryMode::from_args(&args);
- let modifiers: Vec<String> = args.str_array("modifier");
- if !modifiers.is_empty() && delivery != DeliveryMode::Foreground {
- return ToolResult::error(
- "drag modifiers require delivery_mode:\"foreground\" on Windows; posted \
- messages and pen injection cannot carry live modifier-key state",
- )
- .with_structured(json!({
- "code": "background_unavailable",
- "effect": "refused",
- "escalation": {
- "recommended": "foreground",
- "reason": "Windows modifier drags require SendInput so the target observes live modifier state"
- }
- }));
- }
-
- // Accepts numeric JSON as either float or integer — coerce both to f64.
- let coerce = |key: &str| -> Option<f64> {
- args.opt_f64(key)
- .or_else(|| args.opt_i64(key).map(|i| i as f64))
- };
- let from_x_opt = coerce("from_x");
- let from_y_opt = coerce("from_y");
- let to_x_opt = coerce("to_x");
- let to_y_opt = coerce("to_y");
- let (mut from_x, mut from_y, mut to_x, mut to_y) =
- match (from_x_opt, from_y_opt, to_x_opt, to_y_opt) {
- (Some(a), Some(b), Some(c), Some(d)) => (a, b, c, d),
- _ => {
- return ToolResult::error(
- "from_x, from_y, to_x, and to_y are all required (window-local pixels).",
- )
- }
- };
-
- let hwnd_opt = args.opt_u64("window_id");
- let duration_ms = args.u64_or("duration_ms", 500);
- let steps = args.u64_or("steps", 20) as usize;
- let button = args.str_or("button", "left");
- let from_zoom = args.bool_or("from_zoom", false);
-
- if from_zoom {
- match self.state.zoom_context(&args, pid, hwnd_opt) {
- Ok(ctx) => {
- let (wx, wy) = ctx.zoom_to_window(from_x, from_y);
- let (wx2, wy2) = ctx.zoom_to_window(to_x, to_y);
- from_x = wx;
- from_y = wy;
- to_x = wx2;
- to_y = wy2;
- }
- Err(refusal) => return refusal,
- }
- } else {
- let ratio = match screenshot_scale(&self.state, &args, pid, hwnd_opt) {
- Ok(ratio) => ratio,
- Err(refusal) => return refusal,
- };
- from_x *= ratio;
- from_y *= ratio;
- to_x *= ratio;
- to_y *= ratio;
- }
-
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows =
- tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // Compute screen-coord endpoints. Same correction as the click
- // tools — bitmap pixels are anchored to the DWM-frame top-left,
- // not the client area top-left (see `bitmap_to_screen` doc).
- let (sx_from, sy_from) = bitmap_to_screen(hwnd, from_x as i32, from_y as i32);
- let (sx_to, sy_to) = bitmap_to_screen(hwnd, to_x as i32, to_y as i32);
-
- // delivery_mode:"background" on WinUI3: a pointer drag can't both land and
- // hold the contract — the content island only consumes real
- // system-queue pointer input (which the pen/touch injector delivers but
- // at the cost of click-activating the frame: 8/8 foreground steals) and
- // ignores posted WM_* drag messages; UIA has no drag analogue. Return
- // the structured error so the caller retries with delivery_mode:foreground
- // (a WinUI3 Slider can also be set with the set_value tool, which uses
- // UIA RangeValuePattern and holds the contract).
- if delivery == DeliveryMode::Background {
- if let Some(r) = winui3_background_gesture(&None, pid, hwnd, None, 1, &button).await {
- return r;
- }
- }
-
- // delivery_mode:"background" — if a PostMessage drag would silently drop
- // (Chromium/WPF/GTK canvas content reads mouse from the system input
- // queue, not the per-window queue), fall back to coordinate-routed
- // synthetic-pen drag injection instead of refusing. No foreground swap,
- // no cursor move; the target is held non-activatable + cloaked for the
- // stroke (mirrors the click pen path).
- if delivery == DeliveryMode::Background
- && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseMove)
- {
- let target = hwnd;
- let btn = button.clone();
- pin_overlay_above(&cursor_key, hwnd);
- overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx_from as f64,
- y: sy_from as f64,
- },
- );
- let inj = tokio::task::spawn_blocking(move || {
- crate::input::inject::inject_drag_screen(
- target,
- sx_from,
- sy_from,
- sx_to,
- sy_to,
- steps.max(8),
- &btn,
- )
- });
- let visual_drag = track_overlay_drag(
- cursor_key.clone(),
- (f64::from(sx_from), f64::from(sy_from)),
- (f64::from(sx_to), f64::from(sy_to)),
- duration_ms,
- steps,
- );
- let (inj, ()) = tokio::join!(inj, visual_drag);
- return match inj {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \
- from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \
- (delivery_mode:background, PostMessage would have been dropped)."
- )),
- Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseMove,
- e.to_string(),
- ),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
- // delivery_mode:"foreground" — SendInput-based drag. Required for WPF
- // Slider thumbs (and any framework that polls GetKeyState during
- // its drag handler — PostMessage doesn't update per-thread input
- // state, so those targets see a button-up world during the drag
- // and never start tracking). Like the foreground click, it activates
- // the target and refuses rather than drag in a window that covers it.
- if delivery == DeliveryMode::Foreground {
- let btn_fg = button.clone();
- pin_overlay_above(&cursor_key, hwnd);
- let prev_fg_addr = unsafe {
- windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
- };
- let send_result = tokio::task::spawn_blocking(move || {
- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::mouse::send_drag_synthesized_active(
- hwnd,
- sx_from,
- sy_from,
- sx_to,
- sy_to,
- duration_ms,
- steps,
- &btn_fg,
- &modifier_refs,
- )
- });
- let visual_drag = track_overlay_drag(
- cursor_key.clone(),
- (f64::from(sx_from), f64::from(sy_from)),
- (f64::from(sx_to), f64::from(sy_to)),
- duration_ms,
- steps,
- );
- let (send_result, ()) = tokio::join!(send_result, visual_drag);
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
- let button_suffix = if button == "left" {
- String::new()
- } else {
- format!(" ({} button)", button)
- };
- return match send_result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Sent drag{button_suffix} via SendInput on pid {raw_pid} \
- from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \
- in {duration_ms}ms / {steps} steps (delivery_mode:foreground)."
- )),
- Ok(Err(e)) => {
- let message = e.to_string();
- if message.starts_with("foreground_unavailable:") {
- // Nothing was sent; give callers the structured code
- // and its recovery hint rather than bare text.
- ToolResult::error(message).with_structured(json!({
- "code": "foreground_unavailable",
- "effect": "refused",
- "pid": raw_pid,
- "window_id": hwnd,
- }))
- } else {
- ToolResult::error(message)
- }
- }
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- };
- }
-
- // Background posted-message drags cannot move or resize a top-level
- // window: a caption/border drag is handled by the OS's interactive
- // move/resize modal loop, which only real pointer input can drive —
- // the posted WM_MOUSEMOVE stream is discarded and the "success" is a
- // verified no-op. Hit-test the start point and refuse with the
- // structured background_unavailable error so callers escalate to
- // delivery_mode:"foreground" per the documented ladder instead of
- // trusting a drag that did nothing.
- let nc_hit = tokio::task::spawn_blocking(move || {
- crate::input::mouse::non_client_move_resize_hit(hwnd, sx_from, sy_from)
- })
- .await
- .ok()
- .flatten();
- if let Some(region) = nc_hit {
- return crate::input::delivery::background_unavailable_error_with_cause(
- hwnd,
- EventKind::MouseMove,
- format!(
- "the drag starts on the window's {region} (WM_NCHITTEST), and moving \
- or resizing a top-level window requires the OS interactive \
- move/resize loop, which posted mouse messages cannot drive. \
- Re-issue the same drag with delivery_mode:\"foreground\"."
- ),
- );
- }
-
- // Pin the agent-cursor overlay above the drag target so the synthetic
- // cursor stays sandwiched at z+1 of the dragged window for the full
- // path. Drag stays within a single HWND, so one pin at the start is
- // sufficient — the 80 ms z-order tick keeps it asserted thereafter.
- pin_overlay_above(&cursor_key, hwnd);
-
- // Move the agent cursor to the drag-start, show the press state, and
- // track the same interpolated step count and duration while the
- // posted-message drag runs.
- overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx_from as f64,
- y: sy_from as f64,
- },
- );
-
- let button_c = button.clone();
- let result = tokio::task::spawn_blocking(move || {
- // Screen-coord, deepest-child variant: routes the gesture to the
- // child control under the start point (e.g. a WinForms Panel),
- // not the top-level frame that would ignore it.
- crate::input::mouse::post_drag_screen(
- hwnd,
- sx_from,
- sy_from,
- sx_to,
- sy_to,
- duration_ms,
- steps,
- &button_c,
- )
- });
- let visual_drag = track_overlay_drag(
- cursor_key.clone(),
- (f64::from(sx_from), f64::from(sy_from)),
- (f64::from(sx_to), f64::from(sy_to)),
- duration_ms,
- steps,
- );
- let (result, ()) = tokio::join!(result, visual_drag);
-
- match result {
- Ok(Ok(())) => {
- // Match Swift's text format verbatim.
- let button_suffix = if button == "left" {
- String::new()
- } else {
- format!(" ({} button)", button)
- };
- ToolResult::text(format!(
- "✅ Posted drag{button_suffix} to pid {raw_pid} \
- from window-pixel ({}, {}) → ({}, {}), \
- screen ({sx_from}, {sy_from}) → ({sx_to}, {sy_to}) \
- in {duration_ms}ms / {steps} steps.",
- from_x as i32, from_y as i32, to_x as i32, to_y as i32,
- ))
- }
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── get_screen_size ───────────────────────────────────────────────────────────
-
-/// Read the primary display size in PHYSICAL pixels.
-///
-/// With permonitorv2 DPI awareness (set in cua-driver.manifest),
-/// `SM_CXSCREEN` / `SM_CYSCREEN` already return physical pixels — the same
-/// coordinate space screenshots and pixel clicks use on Windows. Shared by
-/// `get_screen_size` and `get_desktop_state`.
-fn physical_screen_size() -> (i32, i32) {
- use windows::Win32::UI::WindowsAndMessaging::{GetSystemMetrics, SM_CXSCREEN, SM_CYSCREEN};
- unsafe { (GetSystemMetrics(SM_CXSCREEN), GetSystemMetrics(SM_CYSCREEN)) }
-}
-
-pub struct GetScreenSizeTool;
-static GSS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetScreenSizeTool {
- fn def(&self) -> &ToolDef {
- GSS_DEF.get_or_init(|| ToolDef {
- name: "get_screen_size".into(),
- description: "Return the size of the main display in physical pixels plus its display \
- scale factor. On Windows, screenshots and pixel clicks use this same physical-pixel \
- coordinate space. Requires no special permissions.".into(),
- input_schema: json!({"type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema()
- },"additionalProperties":false}),
- read_only: true, destructive: false, idempotent: true, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Err(error) = parse_typed_input::<GetScreenSizeInput>("get_screen_size", args) {
- return error;
- }
- use windows::Win32::UI::HiDpi::GetDpiForSystem;
- // With permonitorv2 DPI awareness (set in cua-driver.manifest),
- // SM_CXSCREEN/SM_CYSCREEN return PHYSICAL pixels — the same
- // coordinate space screenshots and pixel clicks use on Windows.
- // Report these as-is, along with the scale factor for reference.
- let (w, h) = physical_screen_size();
- let dpi = unsafe { GetDpiForSystem() };
- let scale = if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 };
- ToolResult::text(format!("✅ Main display: {w}x{h} pixels @ {scale}x"))
- .with_structured(json!({ "width": w, "height": h, "scale_factor": scale }))
- }
-}
-
-// ── get_desktop_state ─────────────────────────────────────────────────────────
-
-/// `get_desktop_state` — full-display vision screenshot (Windows).
-///
-/// Vision-only desktop capture: grabs the ENTIRE primary display at native
-/// physical-pixel size (no downscale) so screen-absolute pixel picks land
-/// exactly, then reports the true screen size. No UIA walk, no pid/window_id —
-/// this is the capture surface for actions with a primary-display desktop
-/// target and screen-absolute coordinates.
-///
-/// Mirrors the `get_window_state` vision branch's ToolResult shape: an
-/// `image_png` content part (or a written-out file path), a text summary line,
-/// and a `structuredContent` object.
-pub struct GetDesktopStateTool {
- state: Arc<ToolState>,
-}
-static GDS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetDesktopStateTool {
- fn def(&self) -> &ToolDef {
- GDS_DEF.get_or_init(|| ToolDef {
- name: "get_desktop_state".into(),
- description: "Capture the full display in true screen pixels, full size unless \
- `max_image_dimension` caps it. Use its PNG as the coordinate source for actions \
- whose target is {kind:\"desktop\",display_id:\"primary\"}. Returns the true \
- screen size. Vision-only: no UIA tree walk.".into(),
- input_schema: json!({
- "type": "object",
- "properties": {
- "session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },
- "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." },
- "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()
- },
- "additionalProperties": false
- }),
- read_only: true, destructive: false, idempotent: false, open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::protocol::Content;
- let binding_args = args.clone();
- let input = match parse_typed_input::<GetDesktopStateInput>("get_desktop_state", args) {
- Ok(input) => input,
- Err(error) => return error,
- };
- let screenshot_out_file = input.screenshot_out_file;
- let max_image_dimension = input.max_image_dimension.filter(|cap| *cap > 0);
-
- // True screen geometry in physical pixels (same space as the capture).
- let (screen_width, screen_height) = physical_screen_size();
-
- // Capture the FULL display at native size — no resize. Run the
- // blocking GDI capture off the async runtime.
- let out_file = screenshot_out_file.clone();
- let res = tokio::task::spawn_blocking(
- move || -> anyhow::Result<(Option<String>, Option<String>, u32, u32, Vec<u8>, (u32, u32), cursor_overlay::capture_exclusion::AgentOverlayCapture)> {
- use base64::{engine::general_purpose::STANDARD as BASE64, Engine};
- // The agent reads this image, so the Driver's own cursor and
- // session pill must not sit over the controls it is reading.
- let (png, overlay_capture) =
- crate::capture::screenshot_display_bytes_excluding_overlay()?;
- let full = crate::capture::png_dimensions_pub(&png)?;
- // Opt-in cap; later desktop-scope pixels from the capped
- // image are mapped back at dispatch and by its capture_id.
- let png = match max_image_dimension {
- Some(cap) if full.0.max(full.1) > cap => {
- crate::capture::resize_png_if_needed(&png, cap)?
- }
- _ => png,
- };
- let (w, h) = crate::capture::png_dimensions_pub(&png)?;
- if let Some(ref path) = out_file {
- std::fs::write(path, &png)?;
- Ok((None, Some(path.clone()), w, h, png, full, overlay_capture))
- } else {
- Ok((Some(BASE64.encode(&png)), None, w, h, png, full, overlay_capture))
- }
- },
- )
- .await;
-
- let (
- b64_opt,
- file_path,
- screenshot_width,
- screenshot_height,
- png,
- full_size,
- overlay_capture,
- ) = match res {
- Ok(Ok(v)) => v,
- Ok(Err(e)) => return ToolResult::error(format!("Desktop screenshot failed: {e}")),
- Err(e) => return ToolResult::error(format!("Desktop screenshot task error: {e}")),
- };
-
- let mut content: Vec<Content> = Vec::new();
- if let Some(b64) = b64_opt {
- content.push(Content::image_png(b64));
- }
- let mut summary = format!(
- "desktop screenshot {screenshot_width}x{screenshot_height} px \
- (screen {screen_width}x{screen_height} px)"
- );
- cursor_overlay::capture_exclusion::append_summary_note(&mut summary, &overlay_capture);
- content.push(Content::text(summary));
-
- let mut structured = json!({
- "platform": "windows",
- "display": "primary",
- "screenshot_width": screenshot_width,
- "screenshot_height": screenshot_height,
- "screen_width": screen_width,
- "screen_height": screen_height,
- "scale_factor": unsafe {
- let dpi = windows::Win32::UI::HiDpi::GetDpiForSystem();
- if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 }
- },
- "screenshot_mime_type": "image/png",
- "agent_overlay_capture": overlay_capture,
- });
- if full_size != (screenshot_width, screenshot_height) {
- structured["screenshot_original_width"] = json!(full_size.0);
- structured["screenshot_original_height"] = json!(full_size.1);
- }
- if let Some(bridge) = &self.state.capture_bridge {
- let geometry = match crate::capture_admission::CaptureGeometry::new(
- screenshot_width,
- screenshot_height,
- full_size.0,
- full_size.1,
- ) {
- Ok(geometry) => geometry,
- Err(error) => {
- return ToolResult::error(format!(
- "Desktop capture geometry is invalid: {error}"
- ))
- }
- };
- match bridge.publish(
- &binding_args,
- png,
- crate::capture_admission::WindowsCaptureTarget::PrimaryDesktop,
- geometry,
- ) {
- Ok(Some(capture_id)) => structured["capture_id"] = json!(capture_id),
- Ok(None) => {}
- Err(error) => {
- return ToolResult::error(format!(
- "Desktop capture publication failed: {error}"
- ))
- }
- }
- }
- if let Some(ref fp) = file_path {
- structured["screenshot_file_path"] = json!(fp);
- }
-
- ToolResult {
- content,
- is_error: None,
- structured_content: Some(structured),
- action_record: None,
- }
- }
-}
-
-// ── get_cursor_position ───────────────────────────────────────────────────────
-
-pub struct GetCursorPositionTool;
-static GCP_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetCursorPositionTool {
- fn def(&self) -> &ToolDef {
- GCP_DEF.get_or_init(|| ToolDef {
- name: "get_cursor_position".into(),
- description:
- "Return the current mouse cursor position in screen points (origin top-left)."
- .into(),
- input_schema: json!({"type":"object","properties":{
- "session": cua_driver_core::tool_schema::session_schema()
- },"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Err(error) = parse_typed_input::<GetCursorPositionInput>("get_cursor_position", args)
- {
- return error;
- }
- use windows::Win32::Foundation::POINT;
- use windows::Win32::UI::WindowsAndMessaging::GetCursorPos;
- let mut pt = POINT { x: 0, y: 0 };
- unsafe {
- let _ = GetCursorPos(&mut pt);
- }
- // Text format matches Swift `GetCursorPositionTool` 1:1.
- ToolResult::text(format!("✅ Cursor at ({}, {})", pt.x, pt.y))
- .with_structured(json!({ "x": pt.x, "y": pt.y }))
- }
-}
-
-// ── move_cursor ───────────────────────────────────────────────────────────────
-
-pub struct MoveCursorTool {
- state: Arc<ToolState>,
-}
-
-static MCURSOR_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for MoveCursorTool {
- fn def(&self) -> &ToolDef {
- MCURSOR_DEF.get_or_init(|| ToolDef {
- name: "move_cursor".into(),
- description:
- "Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor."
- .into(),
- input_schema: json!({"type":"object","required":["x","y"],"properties":{
- "session": cua_driver_core::tool_schema::session_schema(),
- "scope":{"type":"string","enum":["window","desktop"],"description":"desktop moves the real OS pointer; window moves only the agent overlay."},
- "x":{"type":"number","description":"Destination X. Window scope: screen coordinates of the agent cursor overlay. Desktop scope: get_desktop_state screenshot pixels."},
- "y":{"type":"number","description":"Destination Y, in the same space as x."},
- "cursor_id":{"type":"string","description":"Cursor instance to move. Default: 'default'."}
- },"additionalProperties":false}),
- read_only: false,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- if args.get("scope").and_then(Value::as_str) == Some("desktop") {
- let input = match parse_typed_projection::<MoveCursorInput>("move_cursor", &args) {
- Ok(input) => input,
- Err(error) => return error,
- };
- let x = input.x;
- let y = input.y;
- return match crate::input::mouse::move_cursor_desktop(x as i32, y as i32) {
- Ok(()) => ToolResult::text(format!(
- "Moved the OS pointer to ({x:.1}, {y:.1}) (scope:desktop)."
- ))
- .with_structured(json!({
- "scope": "desktop", "path": "SetCursorPos", "x": x, "y": y
- })),
- Err(error) => ToolResult::error(error.to_string()),
- };
- }
- let x = args.f64_or("x", 0.0);
- let y = args.f64_or("y", 0.0);
- // The trusted dispatch boundary always supplies a lifecycle key,
- // including for an unnamed implicit session.
- let cursor_key = resolve_cursor_key(&args);
- if !cursor_key.is_empty() {
- self.state
- .cursor_registry
- .update_position(&cursor_key, x, y);
- }
- // End pointing upper-left (45°) — matches Swift's
- // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so
- // the cursor settles to the natural macOS-style pose.
- // Use the acknowledged animation path so a first-ever move seeds and
- // displays the session cursor just as reliably as a coordinate click.
- crate::overlay::animate_cursor_to(cursor_key.clone(), x, y).await;
- let shown = if cursor_key.is_empty() {
- "default"
- } else {
- cursor_key.as_str()
- };
- ToolResult::text(format!("Agent cursor '{shown}' moved to ({x:.1}, {y:.1})."))
- }
-}
-
-// ── set_agent_cursor_enabled ──────────────────────────────────────────────────
-
-pub struct SetAgentCursorEnabledV2Tool {
- state: Arc<ToolState>,
-}
-
-static CURSOR_ENABLED_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetAgentCursorEnabledV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_ENABLED_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_enabled"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let enabled = match args.get("enabled").and_then(Value::as_bool) {
- Some(value) => value,
- None => return ToolResult::error("Missing required boolean field `enabled`."),
- };
- let session = resolve_cursor_key(&args);
- if session.is_empty() {
- return ToolResult::error("`session` is required for agent cursor controls.");
- }
- self.state.cursor_registry.set_enabled(&session, enabled);
- crate::overlay::send_command(
- session.clone(),
- cursor_overlay::OverlayCommand::SetEnabled(enabled),
- );
- ToolResult::text(format!(
- "Agent cursor for session '{session}' {}.",
- if enabled { "enabled" } else { "disabled" }
- ))
- .with_structured(json!({"session":session,"enabled":enabled}))
- }
-}
-
-pub struct SetAgentCursorMotionV2Tool;
-
-static CURSOR_MOTION_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-fn cursor_number(value: Option<&Value>) -> Option<f64> {
- value.and_then(|value| {
- value
- .as_f64()
- .or_else(|| value.as_i64().map(|integer| integer as f64))
- })
-}
-
-#[async_trait]
-impl Tool for SetAgentCursorMotionV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_MOTION_V2_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_motion"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let session = resolve_cursor_key(&args);
- if session.is_empty() {
- return ToolResult::error("`session` is required for agent cursor controls.");
- }
- let current = crate::overlay::current_motion(&session);
- let motion = current.with_overrides(
- cursor_number(args.get("start_handle")),
- cursor_number(args.get("end_handle")),
- cursor_number(args.get("arc_size")),
- cursor_number(args.get("arc_flow")),
- cursor_number(args.get("spring")),
- cursor_number(args.get("glide_duration_ms")),
- cursor_number(args.get("dwell_after_click_ms")),
- cursor_number(args.get("idle_hide_ms")),
- None,
- cursor_number(args.get("turn_radius")),
- );
- let motion = match motion.with_style_args(&args) {
- Ok(motion) => motion,
- Err(message) => return ToolResult::error(message),
- };
- crate::overlay::send_command(
- session.clone(),
- cursor_overlay::OverlayCommand::SetMotion(motion.clone()),
- );
- ToolResult::text(format!(
- "Agent cursor motion updated for session '{session}'."
- ))
- .with_structured(json!({"session":session,"motion":motion.output_json()}))
- }
-}
-
-pub struct SetAgentCursorThemeTool {
- state: Arc<ToolState>,
-}
-
-static CURSOR_THEME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-fn cursor_reduced_motion(value: Option<&str>) -> cursor_overlay::ReducedMotion {
- match value {
- Some("on") => cursor_overlay::ReducedMotion::On,
- Some("off") => cursor_overlay::ReducedMotion::Off,
- _ => cursor_overlay::ReducedMotion::Auto,
- }
-}
-
-#[async_trait]
-impl Tool for SetAgentCursorThemeTool {
- fn def(&self) -> &ToolDef {
- CURSOR_THEME_DEF.get_or_init(|| canonical_cursor_def("set_agent_cursor_theme"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let Some(theme_id) = args.get("theme_id").and_then(Value::as_str) else {
- return ToolResult::error("Missing required string field `theme_id`.");
- };
- let session = resolve_cursor_key(&args);
- if session.is_empty() {
- return ToolResult::error("`session` is required for agent cursor controls.");
- }
- let resolved_theme = match cursor_overlay::resolve_theme_selection(theme_id) {
- Ok(theme) => theme,
- Err(error) => {
- return ToolResult::error(format!(
- "Cursor theme '{theme_id}' cannot be selected: {error}"
- ));
- }
- };
- let (version, profile) = resolved_theme
- .as_deref()
- .map(|theme| (theme.version.as_str(), theme.profile.as_str()))
- .unwrap_or((
- cursor_overlay::DEFAULT_THEME_VERSION,
- cursor_overlay::THEME_PROFILE,
- ));
- let reduced_motion =
- cursor_reduced_motion(args.get("reduced_motion").and_then(Value::as_str));
- self.state
- .cursor_registry
- .update_config(&session, |config| {
- config.theme_id = theme_id.to_owned();
- config.reduced_motion = reduced_motion;
- });
- crate::overlay::send_command(
- session.clone(),
- cursor_overlay::OverlayCommand::SetTheme {
- theme_id: theme_id.to_owned(),
- reduced_motion,
- },
- );
- ToolResult::text(format!(
- "Agent cursor theme for session '{session}' set to '{theme_id}'."
- ))
- .with_structured(json!({"session":session,"theme":{
- "id":theme_id,
- "version":version,
- "profile":profile,
- "reduced_motion":reduced_motion,
- "fallback":null
- }}))
- }
-}
-
-pub struct GetAgentCursorStateV2Tool {
- state: Arc<ToolState>,
-}
-
-static CURSOR_STATE_V2_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetAgentCursorStateV2Tool {
- fn def(&self) -> &ToolDef {
- CURSOR_STATE_V2_DEF.get_or_init(|| canonical_cursor_def("get_agent_cursor_state"))
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let session = resolve_cursor_key(&args);
- if session.is_empty() {
- return ToolResult::error("`session` is required for agent cursor controls.");
- }
- let enabled = crate::overlay::is_enabled(&session);
- let motion = crate::overlay::current_motion(&session);
- let (theme_id, version, profile, fallback, visual) =
- crate::overlay::current_theme_state(&session).unwrap_or_else(|| {
- (
- cursor_overlay::DEFAULT_THEME_ID.into(),
- cursor_overlay::DEFAULT_THEME_VERSION.into(),
- cursor_overlay::THEME_PROFILE.into(),
- None,
- cursor_overlay::CursorVisualState::default(),
- )
- });
- let modifiers: Vec<&str> = [
- visual.delivery.map(|value| value.as_str()),
- visual.target.map(|value| value.as_str()),
- ]
- .into_iter()
- .flatten()
- .collect();
- // The last point this session's cursor was placed at, `null` until
- // it first moves (same registry source as macOS).
- let position = self
- .state
- .cursor_registry
- .get(&session)
- .and_then(|cursor| cursor.x.zip(cursor.y))
- .map(|(x, y)| json!({"x": x, "y": y}));
- ToolResult::text(format!("Agent cursor state for session '{session}'.")).with_structured(
- json!({
- "session":session,
- "enabled":enabled,
- "position":position,
- "theme":{
- "id":theme_id,
- "version":version,
- "profile":profile,
- "reduced_motion":visual.reduced_motion,
- "fallback":fallback
- },
- "visual_state":{
- "requested_action":visual.requested_action,
- "resolved_action":visual.resolved_action,
- "modifiers":modifiers,
- "phase":visual.phase(),
- "frame":visual.frame(),
- "preempted_count":visual.preempted_count
- },
- "motion":motion.output_json()
- }),
- )
- }
-}
-
-fn canonical_cursor_def(name: &str) -> ToolDef {
- let contract = cua_driver_contract::tool_contract(name)
- .unwrap_or_else(|| panic!("missing canonical cursor contract for {name}"));
- ToolDef::from_contract(&contract)
-}
-
-// ── check_permissions ─────────────────────────────────────────────────────────
-
-pub struct CheckPermissionsTool;
-static PERMS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for CheckPermissionsTool {
- fn def(&self) -> &ToolDef {
- PERMS_DEF.get_or_init(|| ToolDef {
- name: "check_permissions".into(),
- description: "Check required permissions for cua-driver-rs on Windows.".into(),
- input_schema: json!({"type":"object","properties":{"prompt":{"type":"boolean","description":"Ignored: Windows has no permission prompts."}},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
- async fn invoke(&self, _args: Value) -> ToolResult {
- // `elevated` reports the daemon's actual token integrity level, NOT the
- // legacy `TokenIsElevated` UAC-elevation flag. The two diverge in two
- // important cases that matter to cua-driver:
- //
- // 1. RID-500 built-in Administrator — runs at High IL all the time
- // because RID 500 has no UAC split token (FilterAdministratorToken=0
- // default). `TokenIsElevated` returns false (never UAC-prompted)
- // but the IL is High and admin-level access works fine.
- //
- // 2. Task-Scheduler-spawned processes registered at RunLevel=Highest
- // and triggered by AtLogon — Task Scheduler hands out the user's
- // full admin token without going through the UAC prompt path, so
- // `TokenIsElevated` returns false while IL is High. This is the
- // common case for cua-driver's autostart task on non-RID-500
- // admin users — the daemon IS at High IL (cross-AppContainer UIA
- // works, Calculator-class UWP apps drive cleanly) but the legacy
- // flag reports it as a "standard user", which confused everyone
- // tracking #1602 / #1601. See issue #1640.
- //
- // #1646 / dogfood-iteration-3: PR #1647's in-process windows-rs
- // implementation of TokenIntegrityLevel STILL misreported the IL.
- // Verified externally on the cuademo dogfood VM: a daemon at
- // demonstrably High IL (RID 0x3000 by direct PowerShell-C# Win32
- // read against the daemon's pid) was reported by the daemon's own
- // check_permissions as Medium IL (RID 0x2000). Both code paths used
- // identical Win32 APIs (OpenProcess + OpenProcessToken +
- // GetTokenInformation(TokenIntegrityLevel=25)), yet they disagreed.
- // Root cause unclear — possibly a subtle interaction between the
- // windows-rs crate's bindings and the named-pipe server's thread
- // security context, possibly a Vec-buffer alignment issue that
- // doesn't reproduce in standalone testing. RevertToSelf +
- // OpenProcess(GetCurrentProcessId()) didn't help.
- //
- // Pragmatic workaround: shell out to PowerShell and run the same
- // C# Win32 helper that's verified to return the correct IL. Cost
- // is one PowerShell spawn (~100-200ms) per check_permissions call,
- // which is acceptable — check_permissions is a low-frequency,
- // user-initiated diagnostic, not a hot path.
- const HIGH_IL_RID: u32 = 0x3000;
-
- let il_rid: Option<u32> = read_self_integrity_level_via_powershell();
-
- let is_elevated = il_rid.map(|r| r >= HIGH_IL_RID).unwrap_or(false);
- let il_name = match il_rid {
- Some(0x0000) => "Untrusted",
- Some(0x1000) => "Low",
- Some(0x2000) => "Medium",
- Some(0x2100) => "Medium+",
- Some(0x3000) => "High",
- Some(0x4000) => "System",
- Some(_) => "Unknown",
- None => "Unavailable",
- };
- // Distinguish the unavailable case from a real measurement — don't
- // render a fake RID 0x0000 or a fake elevation status when the lookup
- // failed. See CodeRabbit review on PR #1641.
- let status_text = match il_rid {
- Some(rid) => format!(
- "Process integrity level: {il_name} (RID 0x{rid:04X}, {})\n\
- UIA accessibility: available (no special permission needed)\n\
- PostMessage injection: available",
- if is_elevated {
- "elevated"
- } else {
- "non-elevated"
- }
- ),
- None => format!(
- "Process integrity level: {il_name} (token query failed)\n\
- UIA accessibility: available (no special permission needed)\n\
- PostMessage injection: available"
- ),
- };
- ToolResult::text(status_text).with_structured(json!({
- "elevated": is_elevated,
- "integrity_level": il_name,
- "integrity_level_rid": il_rid,
- "uia": true,
- "post_message": true
- }))
- }
-}
-
-/// Read the current process's token integrity level (RID) by shelling out
-/// to PowerShell and running a P/Invoke-based C# helper. See the long
-/// comment in `CheckPermissionsTool::invoke` for why the in-process
-/// windows-rs approach was abandoned (issue #1646 / dogfood iteration 3).
-///
-/// Returns `Some(rid)` on success (e.g. `0x3000` for High), `None` if
-/// PowerShell exits non-zero or stdout isn't a u32. Spawns one short-lived
-/// `powershell.exe` process per call; only used from `check_permissions`,
-/// which is a low-frequency, user-initiated diagnostic.
-fn read_self_integrity_level_via_powershell() -> Option<u32> {
- let pid = std::process::id();
- // Inline the same C# helper we verified externally on the dogfood VM.
- // Heredoc-wrap (`@" ... "@`) avoids shell-escape acrobatics around the
- // C# braces / double-quotes. The `Write-Output` at the end is parsed
- // as a plain integer.
- let script = format!(
- r#"
-$ErrorActionPreference = 'Stop'
-Add-Type -TypeDefinition @"
-using System;
-using System.Runtime.InteropServices;
-public static class IL {{
- [DllImport("kernel32.dll")] public static extern IntPtr OpenProcess(int a, bool i, int p);
- [DllImport("kernel32.dll")] public static extern bool CloseHandle(IntPtr h);
- [DllImport("advapi32.dll")] public static extern bool OpenProcessToken(IntPtr p, int a, out IntPtr t);
- [DllImport("advapi32.dll")] public static extern bool GetTokenInformation(IntPtr t, int c, IntPtr i, int l, out int n);
- [DllImport("advapi32.dll")] public static extern IntPtr GetSidSubAuthority(IntPtr s, int i);
- [DllImport("advapi32.dll")] public static extern IntPtr GetSidSubAuthorityCount(IntPtr s);
- public static uint? Rid(int pid) {{
- var h = OpenProcess(0x1000, false, pid); if (h == IntPtr.Zero) return null;
- try {{ IntPtr t; if (!OpenProcessToken(h, 0x8, out t)) return null;
- try {{ int n; GetTokenInformation(t, 25, IntPtr.Zero, 0, out n); if (n == 0) return null;
- var b = Marshal.AllocHGlobal(n); try {{ if (!GetTokenInformation(t, 25, b, n, out n)) return null;
- var s = Marshal.ReadIntPtr(b); var c = Marshal.ReadByte(GetSidSubAuthorityCount(s));
- return (uint)Marshal.ReadInt32(GetSidSubAuthority(s, c - 1)); }}
- finally {{ Marshal.FreeHGlobal(b); }} }}
- finally {{ CloseHandle(t); }} }}
- finally {{ CloseHandle(h); }}
- }}
-}}
-"@
-$rid = [IL]::Rid({pid})
-if ($null -ne $rid) {{ Write-Output ([int]$rid) }}
-"#,
- pid = pid
- );
- let out = std::process::Command::new("powershell.exe")
- .args([
- "-NoProfile",
- "-NonInteractive",
- "-ExecutionPolicy",
- "Bypass",
- "-Command",
- &script,
- ])
- .output()
- .ok()?;
- if !out.status.success() {
- return None;
- }
- let stdout = String::from_utf8_lossy(&out.stdout);
- stdout.trim().parse::<u32>().ok()
-}
-
-// ── get_config ────────────────────────────────────────────────────────────────
-
-pub struct GetConfigTool {
- state: Arc<ToolState>,
-}
-static GCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetConfigTool {
- fn def(&self) -> &ToolDef {
- GCFG_DEF.get_or_init(|| ToolDef {
- name: "get_config".into(),
- // Description ported from Swift `GetConfigTool.swift` with the
- // Windows config path. Swift's `agent_cursor.*` subtree is not
- // mirrored on Windows yet (separate cursor-overlay config path).
- description: "Report the current persistent driver config.\n\n\
- Pure read-only. Returns defaults when the underlying state is unset — \
- same fallback the daemon uses at startup. Sibling to `set_config`.\n\n\
- Current schema (Windows):\n\n \
- {\n \"schema_version\": 1,\n \"version\": \"<crate version>\",\n \
- \"source_sha\": \"<maintainer build commit or null>\",\n \"platform\": \"windows\",\n \
- \"capture_mode\": \"ax\" | \"vision\" (DEPRECATED, ignored),\n \
- \"max_image_dimension\": 0\n }\n\n\
- `capture_mode` is deprecated and no longer affects behavior — \
- `get_window_state` always returns both the UIA tree and a screenshot.".into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true, destructive: false, idempotent: true, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- let cfg = self.state.config.read().unwrap();
- // Mirror the macOS agent's parity addition (commit adb9ecca):
- // nested `agent_cursor.enabled` block so Swift-shaped get_config
- // consumers can read the cursor's enabled state from one place.
- // Scope to the CALLING session's cursor (session > cursor_id > default)
- // and read it from the overlay deterministically — `all_states().first()`
- // was a nondeterministic HashMap read across sessions (macOS BUG 3).
- let cursor_enabled = crate::overlay::is_enabled(&resolve_cursor_key(&args));
- let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
- let payload = json!({
- "schema_version": 1,
- "version": env!("CARGO_PKG_VERSION"),
- "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
- "platform": "windows",
- "capture_mode": cfg.capture_mode,
- "max_image_dimension": cfg.max_image_dimension,
- "agent_cursor": { "enabled": cursor_enabled },
- "cursor": cursor_overlay::cursor_options::saved_config_json(),
- "experimental_pip": pip_enabled,
- "experimental_pip_geometry": pip_geometry,
- });
- // Match Swift's text format 1:1: `"✅ <pretty JSON>"`.
- let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
- ToolResult::text(format!("✅ {pretty}")).with_structured(payload)
- }
-}
-
-// ── set_config ────────────────────────────────────────────────────────────────
-
-fn with_cursor_motion_config_properties(mut schema: Value) -> Value {
- if let Some(properties) = schema.get_mut("properties").and_then(Value::as_object_mut) {
- properties.extend(cursor_overlay::motion_defaults::config_schema_properties());
- properties.extend(cursor_overlay::badge_mode::config_schema_properties());
- properties.extend(cursor_overlay::cursor_options::config_schema_properties());
- }
- schema
-}
-
-pub struct SetConfigTool {
- state: Arc<ToolState>,
-}
-static SCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetConfigTool {
- fn def(&self) -> &ToolDef {
- SCFG_DEF.get_or_init(|| ToolDef {
- name: "set_config".into(),
- description: "Write a setting into the persistent driver config. Values take \
- effect immediately. Pass each setting as its own field.\n\n\
- Known keys:\n\
- - `capture_mode` (string: `vision` | `ax` | `som`) — DEPRECATED and ignored; \
- `get_window_state` always returns both the UIA tree and a screenshot. Still \
- accepted/persisted for back-compat but has no effect.\n\
- - `max_image_dimension` (integer)\n\
- - `experimental_pip` (boolean; persisted to config.json, applies on next daemon restart — Windows backend stubbed today, see issue #1729)\n\
- - `experimental_pip_geometry` (string `WxH` or `WxH+X+Y`; persisted; applies on next daemon restart)\n\n\
- - `cursor.motion.style`, `cursor.motion.timing`, `cursor.motion.effects.<name>` (`on`, `off`, or `default`) — saved default cursor motion for sessions started afterwards. `cursor.motion: \"default\"` clears every saved cursor motion default.\n\
- - `cursor.badges` (`minimal` default, `verbose`, or `off`) — agent cursor badge; applies to every cursor at once.\n\
- - `cursor.float` (resting float): `on` (default), `off`, or `default`. `cursor.marks` (action marks), `cursor.rotate` (turn toward travel): `on`, `off` (default), or `default`. All apply to every cursor at once.\n\n\
- Returns the full updated config in the same shape as `get_config`.".into(),
- input_schema: with_cursor_motion_config_properties(json!({"type":"object","properties":{
- "capture_mode":{"type":"string","enum":["ax","vision"],"description":"DEPRECATED and ignored — get_window_state always returns both the UIA tree and a screenshot. Still accepted/persisted for back-compat but has no effect. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
- "max_image_dimension":{"type":"integer","description":"Max dimension for screenshot resizing (0 = no limit)."},
- "experimental_pip":{"type":"boolean","description":"Enables PiP preview (applies next restart)."},
- "experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position."}
- },"additionalProperties":false})),
- read_only: false, destructive: false, idempotent: true, open_world: false,
- })
- }
- async fn invoke(&self, args: Value) -> ToolResult {
- if args.get("capture_scope").is_some() {
- return ToolResult::error(
- "config key 'capture_scope' is retired; select a window or desktop target on each action",
- )
- .with_structured(json!({
- "code": "config_key_retired",
- "key": "capture_scope",
- "replacement": "action.target",
- }));
- }
- let motion_keys = match cursor_overlay::motion_defaults::apply_config_args(&args) {
- Ok(keys) => keys,
- Err(message) => return ToolResult::error(message),
- };
- let mut cfg = self.state.config.write().unwrap();
- let badge_key = match cursor_overlay::badge_mode::apply_config_args(&args) {
- Ok(key) => key,
- Err(message) => return ToolResult::error(message),
- };
- let option_keys = match cursor_overlay::cursor_options::apply_config_args(&args) {
- Ok(keys) => keys,
- Err(message) => return ToolResult::error(message),
- };
- let mut applied = !motion_keys.is_empty() || badge_key.is_some() || !option_keys.is_empty();
- if let Some(mode) = args.get("capture_mode").and_then(|v| v.as_str()) {
- cfg.capture_mode = mode.to_owned();
- if let Err(e) =
- pip_preview::write_config_key("capture_mode", Value::String(mode.to_owned()))
- {
- tracing::warn!("set_config: failed to persist capture_mode: {e}");
- }
- applied = true;
- }
- if let Some(dim) = args.get("max_image_dimension").and_then(|v| v.as_u64()) {
- cfg.max_image_dimension = dim as u32;
- if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
- tracing::warn!("set_config: failed to persist max_image_dimension: {e}");
- }
- applied = true;
- }
- if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
- if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
- {
- return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
- }
- applied = true;
- }
- if let Some(geom) = args
- .get("experimental_pip_geometry")
- .and_then(|v| v.as_str())
- {
- if pip_preview::PipGeometry::parse(geom).is_none() {
- return ToolResult::error(format!(
- "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
- ));
- }
- if let Err(e) = pip_preview::write_config_key(
- "experimental_pip_geometry",
- Value::String(geom.to_owned()),
- ) {
- return ToolResult::error(format!(
- "failed to persist experimental_pip_geometry: {e}"
- ));
- }
- applied = true;
- }
- if !applied {
- return ToolResult::error("set_config did not contain a known field.");
- }
- // Emit the same pretty-JSON payload as `get_config` (matches Swift's
- // `set_config` return shape — both tools echo the full config after).
- let cursor_enabled = self
- .state
- .cursor_registry
- .all_states()
- .first()
- .map(|s| s.config.enabled)
- .unwrap_or(true);
- let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
- let payload = json!({
- "schema_version": 1,
- "version": env!("CARGO_PKG_VERSION"),
- "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
- "platform": "windows",
- "capture_mode": cfg.capture_mode,
- "max_image_dimension": cfg.max_image_dimension,
- "agent_cursor": { "enabled": cursor_enabled },
- "experimental_pip": pip_enabled,
- "experimental_pip_geometry": pip_geometry,
- });
- let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
- ToolResult::text(format!("✅ {pretty}")).with_structured(payload)
- }
-}
-
-// ── get_accessibility_tree ────────────────────────────────────────────────────
-
-pub struct GetAccessibilityTreeTool;
-
-static GAX_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for GetAccessibilityTreeTool {
- fn def(&self) -> &ToolDef {
- GAX_DEF.get_or_init(|| ToolDef {
- name: "get_accessibility_tree".into(),
- description: "Return a lightweight snapshot of the desktop: running processes and \
- on-screen visible windows with their bounds and owner pid.\n\n\
- For the full UIA subtree of a single window (with interactive element indices \
- you can click by), use get_window_state instead — this is a fast discovery read."
- .into(),
- input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
- async fn invoke(&self, _args: Value) -> ToolResult {
- let (procs, windows) = tokio::task::spawn_blocking(|| {
- (
- crate::win32::list_processes(),
- crate::win32::list_windows(None),
- )
- })
- .await
- .unwrap_or_default();
-
- let mut lines = vec![format!(
- "{} running process(es), {} visible window(s)",
- procs.len(),
- windows.len()
- )];
- for p in &procs {
- lines.push(format!("- {} (pid {})", p.name, p.pid));
- }
- if !windows.is_empty() {
- lines.push(String::new());
- lines.push("Windows:".to_owned());
- for w in &windows {
- let title = if w.title.is_empty() {
- "(no title)".to_owned()
- } else {
- format!("\"{}\"", w.title)
- };
- lines.push(format!(
- "- pid={} {} [window_id: {}] {}x{}+{}+{}",
- w.pid, title, w.hwnd, w.width, w.height, w.x, w.y
- ));
- }
- lines.push(
- "→ Call get_window_state(pid, window_id) to inspect a window's UI.".to_owned(),
- );
- }
-
- let structured = json!({
- "processes": procs.iter().map(|p| json!({"pid":p.pid,"name":p.name})).collect::<Vec<_>>(),
- "windows": windows.iter().map(|w| json!({
- "window_id": w.hwnd, "pid": w.pid, "title": w.title,
- "x": w.x, "y": w.y, "width": w.width, "height": w.height
- })).collect::<Vec<_>>()
- });
- ToolResult::text(lines.join("\n")).with_structured(structured)
- }
-}
-
-// ── zoom ──────────────────────────────────────────────────────────────────────
-
-pub struct ZoomTool {
- state: Arc<ToolState>,
-}
-static ZOOM_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for ZoomTool {
- fn def(&self) -> &ToolDef {
- ZOOM_DEF.get_or_init(|| ToolDef {
- name: "zoom".into(),
- // Description ported from Swift `ZoomTool.swift`.
- description: "Zoom into a rectangular region of a window screenshot at full (native) \
- resolution. Use this when `get_window_state` returned a resized image and you \
- need to read small text, identify icons, or verify UI details.\n\n\
- Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted). \
- Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot \
- returned by `get_window_state` (i.e. the resized image if `max_image_dimension` \
- is active). The maximum zoom region width is 500 px in scaled-image coordinates.\n\n\
- A 20% padding is automatically added on every side of the requested region so \
- the target remains visible even if the caller's coordinates are slightly off.\n\n\
- After a zoom, pass `from_zoom=true` to click/type_text to auto-translate \
- coordinates back to full-window space. Coordinate actions return \
- `screenshot_context_missing` when no current snapshot contains a \
- screenshot owned by this session. `from_zoom` actions return \
- `zoom_context_missing` when the zoom was never created or was replaced; call \
- `get_window_state`, then `zoom`, again on the same connection.\n\n\
- Windows-specific: `window_id` is the canonical addressing field (HWND). \
- `pid` is preferred when available; otherwise the driver resolves the unique \
- current snapshot owned by this session and window.".into(),
- input_schema: json!({
- "type":"object","required":["window_id","x1","y1","x2","y2"],"properties":{
- "pid":{"type":"integer","description":"Optional target process ID. When omitted, the unique current snapshot for this session and HWND is used."},
- "window_id":{"type":"integer","description":"HWND of the target window."},
- "x1":{"type":"number","description":"Left edge of the region (resized-image pixels)."},
- "y1":{"type":"number","description":"Top edge of the region (resized-image pixels)."},
- "x2":{"type":"number","description":"Right edge of the region (resized-image pixels)."},
- "y2":{"type":"number","description":"Bottom edge of the region (resized-image pixels)."}
- },"additionalProperties":false
- }),
- // Swift annotation: idempotent: false.
- read_only: true, destructive: false, idempotent: false, open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let requested_pid = match args.get("pid") {
- None => None,
- Some(value) => match value.as_i64().and_then(|pid| i32::try_from(pid).ok()) {
- Some(pid) => Some(pid),
- None => return ToolResult::error("pid must be a 32-bit integer"),
- },
- };
- let hwnd = match args.get("window_id").and_then(|v| v.as_u64()) {
- Some(v) => v,
- None => return ToolResult::error("Missing required integer field window_id."),
- };
- use cua_driver_core::tool_args::ArgsExt;
- let coerce = |k: &str| {
- args.opt_f64(k)
- .or_else(|| args.opt_i64(k).map(|i| i as f64))
- };
- let (x1, y1, x2, y2) = match (coerce("x1"), coerce("y1"), coerce("x2"), coerce("y2")) {
- (Some(a), Some(b), Some(c), Some(d)) => (a, b, c, d),
- _ => return ToolResult::error("Missing required region coordinates (x1, y1, x2, y2)."),
- };
- if x2 <= x1 || y2 <= y1 {
- return ToolResult::error("Invalid region: x2 must be > x1 and y2 must be > y1.");
- }
- if x2 - x1 > 500.0 {
- return ToolResult::error(format!(
- "Zoom region too wide: {} px > 500 px max. Use a narrower region (max 500 px in scaled-image coordinates).",
- (x2 - x1) as i64
- ));
- }
-
- // Issue #1880: the caller's coordinates are in the (possibly downscaled)
- // screenshot space of the last `get_window_state` call, but the crop below
- // runs on a fresh native-resolution capture. Scale by the stored resize
- // ratio so the crop lands on the intended region; the zoom context then
- // holds native-pixel values, which is what `from_zoom` clicks expect.
- let session_id = args.opt_str("_session_id");
- let (pid, screenshot) = match self.state.snapshots.screenshot_context_for_zoom(
- requested_pid,
- hwnd,
- session_id.as_deref(),
- ) {
- Ok(context) => context,
- Err(refusal) => return refusal,
- };
- let (nx1, ny1, nx2, ny2) = (
- x1 * screenshot.scale,
- y1 * screenshot.scale,
- x2 * screenshot.scale,
- y2 * screenshot.scale,
- );
-
- let state = self.state.clone();
- let result = tokio::task::spawn_blocking(move || {
- let png = crate::capture::screenshot_window_bytes(hwnd)?;
- cursor_overlay::capture_utils::crop_png_to_jpeg(&png, nx1, ny1, nx2, ny2, 500)
- })
- .await;
-
- match result {
- Ok(Ok(crop)) => {
- if let Err(refusal) = state.snapshots.set_zoom(
- pid,
- session_id.as_deref(),
- ZoomContext {
- screenshot,
- origin_x: crop.origin_x,
- origin_y: crop.origin_y,
- scale_inv: crop.scale_inv,
- },
- ) {
- return refusal;
- }
- use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
- let b64 = B64.encode(&crop.jpeg_bytes);
- let (w, h) = (crop.out_w, crop.out_h);
- use cua_driver_core::protocol::Content;
- // Match Swift's text format 1:1.
- let summary = "✅ Zoomed region captured at native resolution. \
- To click a target in this image, use \
- `click(pid, x, y, from_zoom=true)` where x,y are pixel \
- coordinates in THIS zoomed image — the driver maps them \
- back automatically."
- .to_owned();
- ToolResult {
- content: vec![Content::image_jpeg(b64), Content::text(summary)],
- is_error: None,
- // Surface 7: `mime_type` mirrors the MCP image part's `mimeType`
- // onto the structured payload (additive — `format` stays).
- structured_content: Some(json!({
- "width": w, "height": h, "format": "jpeg",
- "mime_type": "image/jpeg"
- })),
- action_record: None,
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("Zoom failed: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── type_text_chars ───────────────────────────────────────────────────────────
-
-pub struct TypeTextCharsTool {
- _state: Arc<ToolState>,
-}
-static TYPE_CHARS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for TypeTextCharsTool {
- fn def(&self) -> &ToolDef {
- TYPE_CHARS_DEF.get_or_init(|| ToolDef {
- name: "type_text_chars".into(),
- description: "Type text character-by-character with a configurable inter-character \
- delay (default 30 ms). Useful for apps that miss keystrokes. \
- Otherwise identical to type_text (WM_CHAR, no focus steal).".into(),
- input_schema: json!({
- "type":"object","required":["pid","text"],"properties":{
- "pid":{"type":"integer","description":"Target process ID."},
- "window_id":{"type":"integer","description":"HWND of the target window. Required with element_index."},
- "text":{"type":"string","description":"Text to type, one character at a time."},
- "delay_ms":{"type":"integer","description":"Milliseconds between chars (default 30)."},
- "type_chars_only":{"type":"boolean","description":"Skip element focus, type directly. Default false."}
- },"additionalProperties":false
- }),
- read_only: false, destructive: true, idempotent: false, open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let pid = args.u64_or("pid", 0) as u32;
- let text_raw = match args.require_str("text") {
- Ok(v) => v,
- Err(e) => return e,
- };
- // Same trailing-protocol-tag scrub as the main TypeTextTool — see
- // cua_driver_core::text_sanitize for rationale.
- let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
- .into_owned();
- let delay_ms = args.u64_or("delay_ms", 30);
- let hwnd_opt = args.opt_u64("window_id");
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows =
- tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "No windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
- let text_len = text.chars().count();
- let result = tokio::task::spawn_blocking(move || {
- crate::input::post_type_text_with_delay(hwnd, &text, delay_ms)
- })
- .await;
- match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "Typed {text_len} character(s) with {delay_ms}ms delay."
- )),
- Ok(Err(e)) => ToolResult::error(e.to_string()),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
- }
- }
-}
-
-// ── kill_app ──────────────────────────────────────────────────────────────────
-//
-// Force-terminate a process by pid. Background:
-// * Win32 / GDI apps respond cleanly to `WM_CLOSE` (Alt+F4 via PostMessage),
-// so `click` on the X button or a hand-rolled hotkey gesture suffices.
-// * UWP / WinUI3 apps (Calculator, Photos, modern Settings, Win11 Notepad)
-// have backgrounding semantics — they accept `WM_CLOSE` but route it
-// into a "minimize to suspended" path instead of exiting. Cooperative
-// close-button automation produces orphan processes that survive every
-// polite eviction attempt. See CUA-541 for the live repro from
-// WINDOWS_CLAUDE_CODE_TEST_PLAN.md Test G (3 leftover CalculatorApp.exe).
-//
-// `kill_app` is the force-terminate escalation — `taskkill /F` equivalent —
-// keyed on pid. Marked `destructive: true` so MCP clients with permission
-// gating prompt before invoking.
-
-// ── invoke_menu ───────────────────────────────────────────────────────────
-
-pub struct InvokeMenuTool;
-static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
- if path.is_empty() || path.len() > 16 {
- return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ if let Some(s) = session {
+ click_args["session"] = json!(s);
}
- path.into_iter()
- .enumerate()
- .map(|(index, segment)| {
- let segment = segment.trim();
- if segment.is_empty() {
- Err(format!("invoke_menu: path segment {index} is empty"))
- } else {
- Ok(segment.to_owned())
- }
- })
- .collect()
-}
-
-fn menu_refusal(message: String) -> ToolResult {
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "refusal": { "code": "menu_path_unavailable", "message": message }
- }))
-}
-
-/// Activate a window for a visible native-menu operation. Windows may reject
-/// the first foreground request after the test/user session has been idle even
-/// when we attach to the current foreground thread. A reserved no-name key has
-/// no application meaning but makes this process the owner of the most recent
-/// input, allowing a bounded retry without sending a real shortcut or text to
-/// whichever application currently has focus.
-fn activate_window_for_menu(hwnd: windows::Win32::Foundation::HWND) -> Result<bool, String> {
- let (activated, assisted) = unsafe { crate::input::force_foreground_assisted(hwnd) };
- if activated {
- Ok(assisted)
- } else {
- Err("invoke_menu: target window could not be activated".to_owned())
+ if let Some(s) = session_id {
+ click_args["_session_id"] = json!(s);
}
-}
-
-#[async_trait]
-impl Tool for InvokeMenuTool {
- fn def(&self) -> &ToolDef {
- INVOKE_MENU_DEF.get_or_init(|| {
- let contract =
- cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
- ToolDef {
- name: contract.name,
- description: contract.description,
- input_schema: contract.input_schema,
- read_only: contract.annotations.read_only,
- destructive: contract.annotations.destructive,
- idempotent: contract.annotations.idempotent,
- open_world: contract.annotations.open_world,
- }
- })
+ if from_zoom {
+ click_args["from_zoom"] = json!(true);
}
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::action_record::{
- ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
- EvidenceKind, RequestedDelivery,
- };
- use windows::Win32::Foundation::HWND;
- use windows::Win32::UI::WindowsAndMessaging::{
- GetForegroundWindow, GetWindowThreadProcessId, IsWindow,
- };
-
- let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let path = match normalized_menu_path(input.path) {
- Ok(path) => path,
- Err(error) => return menu_refusal(error),
- };
- let hwnd_value = input.window_id;
- let hwnd = HWND(hwnd_value as usize as *mut _);
- if !unsafe { IsWindow(hwnd) }.as_bool() {
- return menu_refusal("invoke_menu: window_id is closed, stale, or invalid".into());
- }
- let mut owner_pid = 0_u32;
- unsafe { GetWindowThreadProcessId(hwnd, Some(&mut owner_pid)) };
- if owner_pid != input.pid {
- return menu_refusal("invoke_menu: window_id does not belong to pid".into());
- }
-
- let outcome = tokio::task::spawn_blocking(move || {
- // `HWND` wraps a raw pointer and is intentionally not `Send`.
- // Move the integer handle into the blocking worker and rebuild the
- // process-local wrapper there instead of carrying it across threads.
- let hwnd = HWND(hwnd_value as usize as *mut _);
- let prior = unsafe { GetForegroundWindow() };
- let needs_activation = prior != hwnd;
- let used_activation_retry = if needs_activation {
- activate_window_for_menu(hwnd)?
- } else {
- false
- };
- if needs_activation {
- std::thread::sleep(std::time::Duration::from_millis(100));
- }
- let result = crate::uia::invoke_menu_path(hwnd_value, &path);
- if needs_activation && !prior.0.is_null() {
- let _ = unsafe { crate::input::force_foreground_attached(prior) };
- }
- result.map(|()| used_activation_retry)
- })
- .await;
-
- match outcome {
- Ok(Ok(used_activation_retry)) => ToolResult::text(
- "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",
- )
- .with_action_record(
- ActionExecutionRecord::builder(
- ActionEffect::Unverifiable,
- ActionTransport::WindowsUiaInvoke,
- RequestedDelivery::Foreground,
- )
- .actual_delivery(ActualDelivery::Foreground)
- .evidence(ActionEvidence {
- kind: EvidenceKind::NativeApiResult,
- detail: "Every menu hop resolved uniquely and UI Automation accepted the final action"
- .into(),
- })
- .detail(if used_activation_retry {
- "target activation required a bounded reserved-key foreground-lock retry"
- } else {
- "target was already foreground or activated without a foreground-lock retry"
- })
- .build()
- .expect("invoke_menu record is valid"),
- ),
- Ok(Err(error)) => menu_refusal(error),
- Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
- }
+ if native_coordinates {
+ click_args["_native_coordinates"] = json!(true);
}
+ click_args
}
-// ── set_window_frame ──────────────────────────────────────────────────────
-
-fn outer_frame_for_visible_request(
- requested: (i32, i32, i32, i32),
- outer_before: (i32, i32, i32, i32),
- visible_before: (i32, i32, i32, i32),
-) -> Result<(i32, i32, i32, i32), String> {
- let (requested_x, requested_y, requested_width, requested_height) = requested;
- let (outer_x, outer_y, outer_width, outer_height) = outer_before;
- let (visible_x, visible_y, visible_width, visible_height) = visible_before;
- let left = i64::from(visible_x) - i64::from(outer_x);
- let top = i64::from(visible_y) - i64::from(outer_y);
- let right = i64::from(outer_x) + i64::from(outer_width)
- - (i64::from(visible_x) + i64::from(visible_width));
- let bottom = i64::from(outer_y) + i64::from(outer_height)
- - (i64::from(visible_y) + i64::from(visible_height));
- let adjusted = (
- i64::from(requested_x) - left,
- i64::from(requested_y) - top,
- i64::from(requested_width) + left + right,
- i64::from(requested_height) + top + bottom,
+#[allow(clippy::too_many_arguments)]
+pub(crate) async fn focus_by_pixel(
+ state: &Arc<ToolState>,
+ pid: u32,
+ window_id: Option<u64>,
+ x: f64,
+ y: f64,
+ foreground: bool,
+ session: Option<String>,
+ session_id: Option<String>,
+ from_zoom: bool,
+ native_coordinates: bool,
+) -> Result<(), ToolResult> {
+ let click_args = focus_by_pixel_click_args(
+ pid,
+ window_id,
+ x,
+ y,
+ foreground,
+ session,
+ session_id,
+ from_zoom,
+ native_coordinates,
);
- if adjusted.2 <= 0 || adjusted.3 <= 0 {
- return Err("visible-frame adjustment produced a non-positive outer size".to_owned());
- }
- let to_i32 = |value: i64| {
- i32::try_from(value)
- .map_err(|_| "visible-frame adjustment exceeded Windows coordinate range".to_owned())
- };
- Ok((
- to_i32(adjusted.0)?,
- to_i32(adjusted.1)?,
- to_i32(adjusted.2)?,
- to_i32(adjusted.3)?,
- ))
-}
-
-pub struct SetWindowFrameTool;
-
-static SET_WINDOW_FRAME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for SetWindowFrameTool {
- fn def(&self) -> &ToolDef {
- SET_WINDOW_FRAME_DEF.get_or_init(|| {
- let contract = cua_driver_contract::tool_contract("set_window_frame")
- .expect("set_window_frame contract");
- ToolDef {
- name: contract.name,
- description: contract.description,
- input_schema: contract.input_schema,
- read_only: contract.annotations.read_only,
- destructive: contract.annotations.destructive,
- idempotent: contract.annotations.idempotent,
- open_world: contract.annotations.open_world,
- }
- })
+ let focus = ClickTool {
+ state: state.clone(),
}
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_contract::SetWindowFrameInput;
- use cua_driver_core::action_record::{
- effect_from_value_readback, ActionEvidence, ActionExecutionRecord, ActionTransport,
- ActualDelivery, EvidenceKind, RequestedDelivery,
- };
-
- let input: SetWindowFrameInput =
- match cua_driver_core::tool_args::parse_typed_input("set_window_frame", args) {
- Ok(input) => input,
- Err(result) => return result,
- };
- let outcome = tokio::task::spawn_blocking(move || {
- use windows::Win32::{
- Foundation::{HWND, RECT},
- Graphics::Dwm::{DwmGetWindowAttribute, DWMWA_EXTENDED_FRAME_BOUNDS},
- UI::WindowsAndMessaging::{
- GetWindowRect, GetWindowThreadProcessId, IsIconic, IsWindow, IsZoomed,
- SetWindowPos, SWP_NOACTIVATE, SWP_NOZORDER,
- },
- };
-
- let values = [input.x, input.y, input.width, input.height];
- if values.iter().any(|value| !value.is_finite())
- || input.width <= 0.0
- || input.height <= 0.0
- {
- return Err(
- "x/y must be finite and width/height must be finite positive numbers"
- .to_owned(),
- );
- }
- let to_i32 = |name: &str, value: f64| -> Result<i32, String> {
- let rounded = value.round();
- if rounded < i32::MIN as f64 || rounded > i32::MAX as f64 {
- Err(format!("{name} is out of range for a Windows window coordinate"))
- } else {
- Ok(rounded as i32)
- }
- };
- let x = to_i32("x", input.x)?;
- let y = to_i32("y", input.y)?;
- let width = to_i32("width", input.width)?;
- let height = to_i32("height", input.height)?;
- if width <= 0 || height <= 0 {
- return Err("width/height must round to positive Windows window sizes".to_owned());
- }
- let hwnd = HWND(input.window_id as usize as *mut std::ffi::c_void);
- if !unsafe { IsWindow(hwnd).as_bool() } {
- return Err(format!("window_id {} is closed, stale, or invalid", input.window_id));
- }
- let mut owner_pid = 0_u32;
- unsafe { GetWindowThreadProcessId(hwnd, Some(&mut owner_pid)) };
- if owner_pid != input.pid {
- return Err(format!(
- "window_id {} belongs to pid {owner_pid}, not pid {}",
- input.window_id, input.pid
- ));
- }
- if unsafe { IsIconic(hwnd).as_bool() } {
- return Err(format!(
- "window_id {} is minimized; restore it with bring_to_front before setting its frame",
- input.window_id
- ));
- }
- if unsafe { IsZoomed(hwnd).as_bool() } {
- return Err(format!(
- "window_id {} is maximized; restore it before setting its frame",
- input.window_id
- ));
- }
- let read_frames = |hwnd: HWND| -> Result<
- ((i32, i32, i32, i32), (i32, i32, i32, i32)),
- String,
- > {
- let mut outer = RECT::default();
- unsafe { GetWindowRect(hwnd, &mut outer) }
- .map_err(|error| format!("could not read the current window frame: {error}"))?;
- let outer = (
- outer.left,
- outer.top,
- outer.right - outer.left,
- outer.bottom - outer.top,
- );
- let mut visible = RECT::default();
- let visible = unsafe {
- DwmGetWindowAttribute(
- hwnd,
- DWMWA_EXTENDED_FRAME_BOUNDS,
- &mut visible as *mut RECT as *mut _,
- std::mem::size_of::<RECT>() as u32,
- )
- }
- .ok()
- .map(|()| {
- (
- visible.left,
- visible.top,
- visible.right - visible.left,
- visible.bottom - visible.top,
- )
- })
- .filter(|(_, _, width, height)| *width > 0 && *height > 0)
- .unwrap_or(outer);
- Ok((outer, visible))
- };
- let (outer_before, before) = read_frames(hwnd)?;
- let requested = (x, y, width, height);
- let outer_requested =
- outer_frame_for_visible_request(requested, outer_before, before)?;
- let mutation_error = unsafe {
- SetWindowPos(
- hwnd,
- HWND::default(),
- outer_requested.0,
- outer_requested.1,
- outer_requested.2,
- outer_requested.3,
- SWP_NOACTIVATE | SWP_NOZORDER,
- )
- }
- .err()
- .map(|error| format!("SetWindowPos failed: {error}"));
-
- let mut observed = None;
- for _ in 0..6 {
- if let Ok((_, visible)) = read_frames(hwnd) {
- observed = Some(visible);
- if observed == Some(requested) {
- break;
- }
- }
- std::thread::sleep(std::time::Duration::from_millis(40));
- }
- let changed = observed.is_some_and(|observed| before != observed);
- Ok((
- requested,
- outer_requested,
- observed,
- changed,
- mutation_error,
- ))
- })
- .await;
-
- let (requested, outer_requested, observed, changed, mutation_error) = match outcome {
- Ok(Ok(outcome)) => outcome,
- Ok(Err(error)) => return ToolResult::error(format!("set_window_frame: {error}")),
- Err(error) => {
- return ToolResult::error(format!(
- "set_window_frame: blocking task failed: {error}"
- ));
- }
- };
- let confirmed = observed == Some(requested);
- let effect = effect_from_value_readback(confirmed, changed, observed.is_some());
- let mut record = ActionExecutionRecord::builder(
- effect,
- ActionTransport::WindowsSetWindowPos,
- RequestedDelivery::NotApplicable,
- )
- .actual_delivery(ActualDelivery::NotApplicable)
- .detail(format!(
- "requested_visible={requested:?} requested_outer={outer_requested:?} observed_visible={observed:?} mutation_error={mutation_error:?}"
- ));
- if observed.is_some() {
- record = record.evidence(ActionEvidence {
- kind: EvidenceKind::ValueReadback,
- detail: if confirmed {
- "DWM visible-frame bounds matched the requested frame".into()
- } else {
- "DWM visible-frame bounds did not match the requested frame".into()
- },
- });
- }
- ToolResult::text(if confirmed {
- "Set and verified the requested window frame."
- } else if observed.is_none() {
- "The native window mutation was attempted, but its resulting frame could not be read back."
- } else {
- "The window frame did not settle at the requested geometry."
- })
- .with_action_record(record.build().expect("set_window_frame record is valid"))
+ .invoke(click_args)
+ .await;
+ if focus.is_error == Some(true) {
+ // Preserve the click tool's structured background refusal (for example
+ // background_occluded / background_uipi_blocked). Re-wrapping it as a
+ // text-only error made keyboard-family PX calls lose the actionable
+ // capability result produced by the actual actuator.
+ return Err(focus);
}
+ // Brief settle so the renderer registers focus before the keystrokes.
+ tokio::time::sleep(std::time::Duration::from_millis(120)).await;
+ Ok(())
}
-// ── bring_to_front ────────────────────────────────────────────────────────
-
-pub struct BringToFrontTool;
-static BTF_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for BringToFrontTool {
- fn def(&self) -> &ToolDef {
- BTF_DEF.get_or_init(|| ToolDef {
- name: "bring_to_front".into(),
- description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \
- the OS foreground. \n\n\
- **This deliberately breaks the no-foreground contract.** It is not part of \
- the normal input ladder. For an ordinary `background_unavailable` response, \
- retry only the refused action with `delivery_mode:\"foreground\"`; the input \
- tool performs its own activate, act, and restore sequence. Use \
- `bring_to_front` only for a focus-proxy surface that must remain foreground \
- across multiple calls, such as an RDP or Windows App session, or when repeated \
- action-scoped activation prevents the remote surface from accepting input. \n\n\
- Implementation uses the `AttachThreadInput` trick to bypass Windows' \
- foreground-lock when the daemon is not at UIAccess integrity. Returns \
- structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \
- Windows only; macOS / Linux return an error pointing at platform-native \
- alternatives.".into(),
- input_schema: json!({"type":"object","required":["pid"],"properties":{
- "pid":{"type":"integer","description":"Target process ID."},
- "window_id":{"type":"integer","description":"Optional HWND. Defaults to pid's first visible top-level window."}
- },"additionalProperties":false}),
- read_only: false,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- use cua_driver_core::tool_args::ArgsExt;
- let pid = match args.require_u32("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let hwnd_opt = args.opt_u64("window_id");
-
- // Resolve hwnd: explicit, or auto from pid.
- let hwnd = match hwnd_opt {
- Some(h) => h,
- None => {
- let windows =
- tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
- .await
- .unwrap_or_default();
- match windows.first() {
- Some(w) => w.hwnd,
- None => {
- return ToolResult::error(format!(
- "bring_to_front: no windows found for pid {pid}. Provide window_id."
- ))
- }
- }
- }
- };
-
- // Run the foreground swap on a blocking thread. The AttachThreadInput
- // trick mirrors `send_key_synthesized` (input/keyboard.rs:313-345)
- // and is validated by `flash-repro/16-edge-launch-fg.ps1` for the
- // Edge launch focus-steal recovery case.
- let outcome =
- tokio::task::spawn_blocking(move || -> Result<(u64, u64, bool, bool), String> {
- use windows::Win32::Foundation::HWND;
- use windows::Win32::Graphics::Dwm::DwmFlush;
- use windows::Win32::UI::WindowsAndMessaging::{
- GetForegroundWindow, IsIconic, IsWindow, SetWindowPos, ShowWindowAsync,
- HWND_NOTOPMOST, HWND_TOPMOST, SWP_NOACTIVATE, SWP_NOMOVE, SWP_NOSIZE,
- SW_RESTORE,
- };
-
- let target = HWND(hwnd as *mut _);
- if !unsafe { IsWindow(target) }.as_bool() {
- return Err(format!("hwnd 0x{hwnd:x} is not a valid window"));
- }
-
- let prev_fg = unsafe { GetForegroundWindow() };
- let prev_fg_addr = prev_fg.0 as u64;
-
- // Iconic windows have no rendered pixels and live at the sentinel
- // (-32000, -32000) position. Restore before changing z-order so
- // bring_to_front is also the advertised recovery path for capture.
- let was_minimized = unsafe { IsIconic(target) }.as_bool();
- if was_minimized {
- let _ = unsafe { ShowWindowAsync(target, SW_RESTORE) };
- for _ in 0..20 {
- if !unsafe { IsIconic(target) }.as_bool() {
- break;
- }
- std::thread::sleep(std::time::Duration::from_millis(25));
- }
- if unsafe { IsIconic(target) }.as_bool() {
- return Err(format!(
- "restore request did not complete for minimized hwnd 0x{hwnd:x}"
- ));
- }
- }
-
- // Lock-free z-order raise FIRST: bring the window to the top of the
- // normal band (the HWND_TOPMOST→HWND_NOTOPMOST force-to-front trick)
- // so it's brought to the VISIBLE front even when the foreground-lock
- // denies focus. SWP_NOACTIVATE → no focus steal; SetWindowPos z-order
- // is not gated by the foreground-lock / UIAccess. This is the same
- // technique the delivery_mode:"foreground" pointer path uses.
- let raised = unsafe {
- let a = SetWindowPos(
- target,
- HWND_TOPMOST,
- 0,
- 0,
- 0,
- 0,
- SWP_NOACTIVATE | SWP_NOMOVE | SWP_NOSIZE,
- )
- .is_ok();
- let b = SetWindowPos(
- target,
- HWND_NOTOPMOST,
- 0,
- 0,
- 0,
- 0,
- SWP_NOACTIVATE | SWP_NOMOVE | SWP_NOSIZE,
- )
- .is_ok();
- a && b
- };
-
- // The caller explicitly requested a visible transition, so the
- // shared foreground helper may claim the most-recent-input token.
- let _ = unsafe { crate::input::force_foreground_assisted(target) };
- let now_fg = unsafe { GetForegroundWindow() };
-
- // A restored HWND can stop reporting iconic before its compositor
- // surface is painted. Flush DWM before returning, but do not make
- // the restore operation depend on any particular capture backend.
- // Capture has its own WGC fallback for freshly restored surfaces.
- if was_minimized {
- let _ = unsafe { DwmFlush() };
- }
- Ok((prev_fg_addr, now_fg.0 as u64, raised, was_minimized))
- })
- .await;
-
- match outcome {
- Ok(Ok((prev, now, raised, restored))) => {
- let focused = now == hwnd;
- let msg = if focused {
- format!("✅ bring_to_front: pid {pid} hwnd 0x{hwnd:x} is now foreground (was 0x{prev:x}).")
- } else if raised {
- format!(
- "bring_to_front: exact target hwnd 0x{hwnd:x} was raised in z-order, but \
- Windows kept foreground on hwnd 0x{now:x}; foreground activation failed."
- )
- } else {
- format!(
- "bring_to_front could neither focus nor raise target 0x{hwnd:x} (current \
- foreground 0x{now:x})."
- )
- };
- let structured = if focused {
- json!({
- "previous_fg_hwnd": format!("0x{prev:x}"),
- "now_fg_hwnd": format!("0x{now:x}"),
- "target_hwnd": format!("0x{hwnd:x}"),
- "landed_on_target": true,
- "raised": raised,
- "restored": restored,
- })
- } else {
- json!({
- "code": "foreground_unavailable",
- "effect": "refused",
- "pid": pid,
- "window_id": hwnd,
- "actual_foreground_window_id": now,
- "previous_fg_hwnd": format!("0x{prev:x}"),
- "now_fg_hwnd": format!("0x{now:x}"),
- "target_hwnd": format!("0x{hwnd:x}"),
- "landed_on_target": false,
- "raised": raised,
- "restored": restored,
- })
- };
- if focused {
- ToolResult::text(msg).with_structured(structured)
- } else {
- ToolResult::error(msg).with_structured(structured)
- }
- }
- Ok(Err(e)) => ToolResult::error(format!("bring_to_front: {e}")),
- Err(join) => {
- ToolResult::error(format!("bring_to_front: blocking-task join error: {join}"))
- }
+fn wait_for_cached_element_keyboard_focus(
+ admitted: &Option<crate::uia::snapshot::RetainedElement>,
+ timeout: std::time::Duration,
+) -> bool {
+ let deadline = std::time::Instant::now() + timeout;
+ loop {
+ if admitted
+ .as_ref()
+ .and_then(|element| element.element_has_keyboard_focus())
+ == Some(true)
+ {
+ return true;
}
- }
-}
-
-pub struct KillAppTool;
-static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-
-#[async_trait]
-impl Tool for KillAppTool {
- fn def(&self) -> &ToolDef {
- KILL_DEF.get_or_init(|| ToolDef {
- name: "kill_app".into(),
- description: "Force-terminate a process by pid. Use when the standard close path \
- (Alt+F4 / WM_CLOSE via `click` on the X button) fails to make the process \
- exit — typical for UWP / WinUI3 apps (Calculator, Photos, modern Notepad) \
- that route WM_CLOSE into a suspended-but-resident state. Equivalent to \
- `taskkill /F /PID <pid>`. Unsaved state is lost. Prefer the click-the-X path \
- first; only escalate to `kill_app` when polite close didn't terminate."
- .into(),
- input_schema: json!({"type":"object","required":["pid"],"properties":{
- "pid":{"type":"integer","description":"PID of the process to terminate."}
- },"additionalProperties":false}),
- read_only: false,
- destructive: true,
- idempotent: true,
- open_world: false,
- })
- }
-
- async fn protected_resource_scope(
- &self,
- adapter_id: &str,
- args: &Value,
- ) -> Result<Option<Value>, String> {
- if adapter_id != "process_control" {
- return Ok(None);
+ if std::time::Instant::now() >= deadline {
+ return false;
}
- use cua_driver_core::browser::platform::BrowserPlatform;
- let pid = args
- .get("pid")
- .and_then(Value::as_i64)
- .filter(|pid| *pid > 0)
- .ok_or_else(|| "kill_app requires a positive integer pid".to_owned())?;
- let fingerprint = crate::browser_platform::WindowsBrowserPlatform::default()
- .process_fingerprint(pid)
- .await
- .map_err(|error| error.message)?;
- Ok(Some(json!({
- "kind": "process_instance",
- "fingerprint": fingerprint,
- })))
+ std::thread::sleep(std::time::Duration::from_millis(10));
}
+}
- async fn invoke(&self, args: Value) -> ToolResult {
- if let Some(expected) = args.get("_protected_process_fingerprint") {
- let current = match self
- .protected_resource_scope("process_control", &args)
- .await
- {
- Ok(Some(scope)) => scope["fingerprint"].clone(),
- Ok(None) => Value::Null,
- Err(message) => return kill_app_stale_process_refusal(message),
- };
- if current != *expected {
- return kill_app_stale_process_refusal(
- "the process identity changed at the termination boundary".to_owned(),
- );
- }
- }
- let pid_v: u32 = match args.get("pid").and_then(|v| v.as_u64()) {
- Some(p) if p > 0 && p <= u32::MAX as u64 => p as u32,
- Some(_) => {
- return ToolResult::error("kill_app: `pid` must be a positive integer".to_string())
- }
- None => {
- return ToolResult::error(
- "kill_app: missing required integer field `pid`".to_string(),
- )
- }
- };
-
- // Run the syscalls on a blocking thread — TerminateProcess + the
- // WaitForSingleObject confirmation are both blocking, and we don't
- // want to stall the tokio reactor.
- let outcome = tokio::task::spawn_blocking(move || -> Result<(), String> {
- use windows::Win32::Foundation::{CloseHandle, WAIT_OBJECT_0};
- use windows::Win32::System::Threading::{
- OpenProcess, TerminateProcess, WaitForSingleObject, PROCESS_SYNCHRONIZE,
- PROCESS_TERMINATE,
- };
-
- // SAFETY: OpenProcess returns a Result<HANDLE> in windows-rs;
- // PROCESS_TERMINATE | PROCESS_SYNCHRONIZE lets us both kill the
- // target AND wait for the OS to fully reap it (so the caller's
- // follow-up list_apps reflects the change immediately).
- let h = unsafe { OpenProcess(PROCESS_TERMINATE | PROCESS_SYNCHRONIZE, false, pid_v) }
- .map_err(|e| {
- format!(
- "OpenProcess(pid={pid_v}, PROCESS_TERMINATE|PROCESS_SYNCHRONIZE) failed: {e}. \
- The process may not exist, or the daemon lacks the right to terminate it. \
- Try listing processes first to confirm the pid is correct."
- )
- })?;
-
- let term_result = unsafe { TerminateProcess(h, 1) };
-
- // WaitForSingleObject so the caller can assume the process is
- // really gone on return. 2-second cap — TerminateProcess is
- // synchronous in practice but the OS still needs a moment to
- // reap kernel structures. WAIT_TIMEOUT is treated as a soft
- // success (the kill request landed; the process may take a
- // little longer to disappear from list_apps).
- let wait_status = unsafe { WaitForSingleObject(h, 2000) };
- let _ = unsafe { CloseHandle(h) };
-
- match (term_result, wait_status) {
- (Ok(()), s) if s == WAIT_OBJECT_0 => Ok(()),
- (Ok(()), _) => Ok(()), // killed, wait timed out — caller can re-check
- (Err(e), _) => Err(format!(
- "TerminateProcess(pid={pid_v}) failed: {e}. The handle was opened \
- successfully but the kill itself was rejected; this is unusual for \
- PROCESS_TERMINATE rights and may indicate a protected-process target \
- (PPL, e.g. csrss / system services)."
- )),
- }
- })
- .await;
+/// Establish exact UIA child focus after the owning top-level HWND is already
+/// confirmed foreground. Chromium providers sometimes reject UIA `SetFocus`;
+/// in that case a real click at the accessibility-derived center is the
+/// bounded fallback. Both routes require `CurrentHasKeyboardFocus` read-back
+/// before any keyboard input is allowed to leave the driver.
+pub(crate) fn focus_cached_element_for_foreground(
+ admitted: &Option<crate::uia::snapshot::RetainedElement>,
+ hwnd: u64,
+ element_index: usize,
+ click_point: Option<(i32, i32)>,
+) -> anyhow::Result<()> {
+ // `fg_bypass` is a background-only shield: for Chromium it disables the
+ // top-level HWND while UIA runs so the provider cannot self-activate. A
+ // disabled foreground window immediately loses foreground on Windows,
+ // which would make the enclosing verify-before-SendInput transaction
+ // fail closed. The foreground route has already activated the exact HWND,
+ // so focus the cached element directly and verify it below.
+ let set_focus = admitted
+ .as_ref()
+ .ok_or_else(|| anyhow::anyhow!("missing admitted element"))
+ .and_then(|element| element.focus_element());
+ if set_focus.is_ok()
+ && wait_for_cached_element_keyboard_focus(admitted, std::time::Duration::from_millis(350))
+ {
+ return Ok(());
+ }
- match outcome {
- Ok(Ok(())) => ToolResult::text(format!("✅ Terminated pid {pid_v}.")),
- Ok(Err(e)) => ToolResult::error(format!("kill_app: {e}")),
- Err(join_err) => {
- ToolResult::error(format!("kill_app: blocking-task join error: {join_err}"))
- }
+ if let Some((x, y)) = click_point {
+ crate::input::send_click_synthesized_active_mods(hwnd, x, y, 1, "left", &[])?;
+ if wait_for_cached_element_keyboard_focus(admitted, std::time::Duration::from_millis(500)) {
+ return Ok(());
}
}
-}
-fn kill_app_stale_process_refusal(message: String) -> ToolResult {
- ToolResult::error(message.clone()).with_structured(json!({
- "status": "refused",
- "refusal": {
- "code": "protected_resource_scope_stale",
- "message": message,
- }
- }))
+ let detail = set_focus
+ .err()
+ .map(|error| format!("; UIA SetFocus failed: {error}"))
+ .unwrap_or_default();
+ anyhow::bail!(
+ "foreground_unavailable: Windows did not confirm keyboard focus on exact UIA element \
+ [{element_index}] in HWND 0x{hwnd:x}{detail}; no keyboard input was sent"
+ )
}
-// ── debug_window_info ─────────────────────────────────────────────────────────
-//
-// Diagnostic tool: dump everything the daemon (Session 2 when launched via
-// `cua-driver autostart kick`) sees about a given pid's top-level windows.
-// Includes Win32 class names, owning .exe basename + full path, and — when
-// CUIAutomation is available — the currently-focused UIA element with the
-// list of patterns it supports.
-//
-// Built for CUA-543 investigation: the routing between PostMessage-based
-// input and UIA-pattern-based input depends on what the OS actually reports
-// for modern XAML / UWP / WinUI3 targets. SSH-side PowerShell probes from
-// Session 0 see nothing (cross-session HWNDs return empty class names);
-// this tool gives the same view from inside the daemon process.
-
-pub struct DebugWindowInfoTool;
-static DEBUG_WI_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+// `ScreenshotTool` and `ScreenshotCompatTool` were removed in PR #1692 —
+// `get_window_state` (which now always returns a screenshot) is the single
+// canonical screenshot path. The underlying capture functions
+// (`crate::capture::screenshot_window_bytes_with_occlusion`,
+// `screenshot_display_bytes`, etc.) and the WGC backend
+// (`crate::wgc::screenshot_window_via_wgc`) are still in use by
+// `GetWindowStateTool` — that's where the actual screenshot machinery
+// lives now.
-#[async_trait]
-impl Tool for DebugWindowInfoTool {
- fn def(&self) -> &ToolDef {
- DEBUG_WI_DEF.get_or_init(|| ToolDef {
- name: "debug_window_info".into(),
- description: "Diagnostic: dump everything cua-driver sees about a pid's \
- top-level windows from the daemon's session perspective. Returns \
- window class names, owning .exe basename + path, and — when \
- CUIAutomation succeeds — the focused UIA element with the list of \
- patterns it supports (ValuePattern, InvokePattern, TextPattern, \
- TogglePattern, etc.). Used to design / debug input routing for \
- XAML / UWP / WinUI3 targets; see CUA-543."
- .into(),
- input_schema: json!({"type":"object","required":["pid"],"properties":{
- "pid":{"type":"integer","description":"PID of the process to inspect."}
- },"additionalProperties":false}),
- read_only: true,
- destructive: false,
- idempotent: true,
- open_world: false,
- })
+/// Chromium/Electron windows silently drop `PostMessage` mouse events — their
+/// input thread only honors SendInput-origin events (#1623). `ClickTool`
+/// auto-routes Chromium targets through SendInput, but `DoubleClickTool` /
+/// `RightClickTool` did not, so element/pixel gestures on Electron apps
+/// (Obsidian, VS Code, Slack, …) reached `post_click_screen` and no-op'd
+/// silently. This mirrors that short-circuit for those tools (#1984): when
+/// `hwnd` is a Chromium window, deliver `count` clicks of `button` at screen
+/// `(sx, sy)` via SendInput with async foreground restore and return
+/// `Some(result)`. Returns `None` for non-Chromium targets so the caller
+/// proceeds to its normal PostMessage path.
+pub(crate) async fn chromium_click_short_circuit(
+ hwnd: u64,
+ sx: i32,
+ sy: i32,
+ count: usize,
+ button: &str,
+ pid: u32,
+ gesture: &str,
+) -> Option<ToolResult> {
+ let is_chromium =
+ tokio::task::spawn_blocking(move || crate::input::is_chromium_target_window(hwnd))
+ .await
+ .unwrap_or(false);
+ if !is_chromium {
+ return None;
}
+ // Capture the pre-click foreground so the poller can restore it even if
+ // Chromium re-activates itself from a renderer-side handler (same pattern
+ // and rationale as the ClickTool Chromium branch).
+ let prev_fg_addr =
+ unsafe { windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize };
+ let button_owned = button.to_string();
+ let send_result = tokio::task::spawn_blocking(move || {
+ crate::input::send_click_synthesized(hwnd, sx, sy, count, &button_owned)
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ Some(match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent {gesture} via SendInput to pid {pid} at screen ({sx},{sy}) (Chromium target)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ })
+}
- async fn invoke(&self, args: Value) -> ToolResult {
- let pid_v: u32 = match args.get("pid").and_then(|v| v.as_u64()) {
- Some(p) if p > 0 && p <= u32::MAX as u64 => p as u32,
- _ => {
- return ToolResult::error("debug_window_info: missing or invalid `pid`".to_string())
- }
- };
-
- let outcome = tokio::task::spawn_blocking(move || -> serde_json::Value {
- use std::collections::HashSet;
- use windows::Win32::Foundation::{CloseHandle, HWND, LPARAM, BOOL, TRUE};
- use windows::Win32::System::Com::{
- CoCreateInstance, CoInitializeEx, CoUninitialize,
- CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED,
- };
- use windows::Win32::System::RemoteDesktop::ProcessIdToSessionId;
- use windows::Win32::System::Threading::{
- GetCurrentProcessId, OpenProcess,
- QueryFullProcessImageNameW, PROCESS_NAME_FORMAT,
- PROCESS_QUERY_LIMITED_INFORMATION,
- };
- use windows::Win32::UI::Accessibility::{
- CUIAutomation, IUIAutomation,
- UIA_InvokePatternId, UIA_TextPatternId, UIA_TogglePatternId,
- UIA_ValuePatternId, UIA_LegacyIAccessiblePatternId,
- UIA_SelectionItemPatternId, UIA_ExpandCollapsePatternId,
- };
- use windows::Win32::UI::WindowsAndMessaging::{
- EnumWindows, GetClassNameW, GetWindow, GetWindowTextW,
- GetWindowThreadProcessId, IsWindowVisible, GW_OWNER,
- };
-
- // Helper: read class name + title
- fn class_name(h: HWND) -> Option<String> {
- let mut buf = [0u16; 256];
- let n = unsafe { GetClassNameW(h, &mut buf) };
- if n <= 0 { None } else { Some(String::from_utf16_lossy(&buf[..n as usize])) }
- }
- fn window_title(h: HWND) -> Option<String> {
- let mut buf = [0u16; 512];
- let n = unsafe { GetWindowTextW(h, &mut buf) };
- if n <= 0 { None } else { Some(String::from_utf16_lossy(&buf[..n as usize])) }
+/// Fire UIA `InvokePattern.Invoke()` `count` times (min 2) in rapid succession
+/// on a cached WinUI3 element. Two rapid Invokes land within the system
+/// double-click time, so a Button's Click handler folds them into a
+/// double-click (the harness records `last_action=double_click`).
+///
+/// This is the **only** background actuator that lands on a WinUI3 element: the
+/// content island ignores posted `WM_*BUTTON` messages (measured: a posted
+/// secondary click never fires `RightTapped`), and the synthetic pen/touch
+/// injector click-activates the frame (8/8 foreground steals). UIA delivery
+/// goes through the accessibility channel, not the input queue.
+///
+/// Contract: a XAML `Invoke` handler calls `SetFocus()` → `SetForegroundWindow(self)`,
+/// which the `EnableWindow` UWP bypass does NOT gate (measured: a bare double
+/// Invoke stole foreground). We therefore hold the frame **non-activatable**
+/// (`WS_EX_NOACTIVATE`, via `NoActivateGuard`) for the whole burst, which is
+/// what makes Windows refuse the self-activation — the same guard ClickTool
+/// already relies on for single-click. Runs blocking COM, so call inside
+/// `spawn_blocking`. Returns `Some(Ok)` on success, `Some(Err)` if Invoke
+/// failed, `None` if the element isn't cached or has no InvokePattern.
+fn winui3_uia_multi_invoke(
+ admitted: &Option<crate::uia::snapshot::RetainedElement>,
+ hwnd: u64,
+ count: usize,
+) -> Option<anyhow::Result<()>> {
+ let guard = admitted.as_ref().filter(|element| element.is_uia())?;
+ let ptr = guard.as_ptr();
+ use windows::core::Interface;
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationInvokePattern, UIA_InvokePatternId,
+ };
+ // Hold the frame non-activatable so the XAML Invoke handler's
+ // SetFocus()→SetForegroundWindow(self) is refused for the whole burst.
+ let _noact =
+ crate::input::NoActivateGuard::arm(windows::Win32::Foundation::HWND(hwnd as *mut _));
+ let elem: IUIAutomationElement = unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
+ let outcome: Option<anyhow::Result<()>> = (|| {
+ let pattern = unsafe { elem.GetCurrentPattern(UIA_InvokePatternId) }.ok()?;
+ let inv = pattern.cast::<IUIAutomationInvokePattern>().ok()?;
+ let n = count.max(2);
+ let mut res = Ok(());
+ for i in 0..n {
+ let r = crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
+ inv.Invoke()
+ });
+ if let Err(e) = r {
+ res = Err(anyhow::anyhow!("UIA Invoke #{i}: {e}"));
+ break;
}
-
- // ── Process info: session id + exe path
- let mut session_id: u32 = 0;
- let _ = unsafe { ProcessIdToSessionId(pid_v, &mut session_id) };
-
- let (exe_path, exe_basename) = unsafe {
- match OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, false, pid_v) {
- Ok(h) => {
- let mut buf = [0u16; 1024];
- let mut len: u32 = buf.len() as u32;
- let ok = QueryFullProcessImageNameW(
- h, PROCESS_NAME_FORMAT(0),
- windows::core::PWSTR(buf.as_mut_ptr()),
- &mut len,
- );
- let _ = CloseHandle(h);
- if ok.is_ok() && len > 0 {
- let path = String::from_utf16_lossy(&buf[..len as usize]);
- let base = path
- .rsplit(|c: char| c == '\\' || c == '/')
- .next()
- .unwrap_or(&path)
- .to_ascii_lowercase();
- (Some(path), Some(base))
- } else {
- (None, None)
- }
- }
- Err(_) => (None, None),
- }
- };
-
- // ── Enumerate top-level visible windows owned by this pid
- struct Ctx { target_pid: u32, hwnds: Vec<isize> }
- extern "system" fn enum_cb(hwnd: HWND, lparam: LPARAM) -> BOOL {
- unsafe {
- let ctx = &mut *(lparam.0 as *mut Ctx);
- let mut p: u32 = 0;
- GetWindowThreadProcessId(hwnd, Some(&mut p));
- if p == ctx.target_pid && IsWindowVisible(hwnd).as_bool() {
- // Only true top-level (no owner) windows.
- let owner = GetWindow(hwnd, GW_OWNER);
- if owner.unwrap_or_default().is_invalid() {
- ctx.hwnds.push(hwnd.0 as isize);
- }
- }
- TRUE
- }
+ if i + 1 < n {
+ std::thread::sleep(std::time::Duration::from_millis(40));
}
- let mut ctx = Ctx { target_pid: pid_v, hwnds: vec![] };
- let _ = unsafe {
- EnumWindows(Some(enum_cb), LPARAM(&mut ctx as *mut Ctx as isize))
- };
+ }
+ Some(res)
+ })();
+ // The cache owns this element's ref (AddRef'd under the cache lock by
+ // get_element_retained); `from_raw` took ownership without AddRef, so forget
+ // it to avoid double-releasing the cache's ref.
+ std::mem::forget(elem);
+ outcome
+}
- // ── Init UIA (best-effort; non-fatal if it fails)
- let mut focused_info: Option<serde_json::Value> = None;
- let coinit_ok = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED).is_ok() };
- if let Ok(uia) = unsafe {
- CoCreateInstance::<_, IUIAutomation>(&CUIAutomation, None, CLSCTX_INPROC_SERVER)
- } {
- if let Ok(focused) = unsafe { uia.GetFocusedElement() } {
- // Patterns we care about for input routing
- let pattern_ids = [
- (UIA_ValuePatternId.0, "Value"),
- (UIA_InvokePatternId.0, "Invoke"),
- (UIA_TextPatternId.0, "Text"),
- (UIA_TogglePatternId.0, "Toggle"),
- (UIA_ExpandCollapsePatternId.0, "ExpandCollapse"),
- (UIA_SelectionItemPatternId.0, "SelectionItem"),
- (UIA_LegacyIAccessiblePatternId.0, "LegacyIAccessible"),
- ];
- let mut patterns: Vec<&str> = vec![];
- for (id, name) in &pattern_ids {
- let pid_struct = windows::Win32::UI::Accessibility::UIA_PATTERN_ID(*id);
- if unsafe { focused.GetCurrentPattern(pid_struct) }.is_ok() {
- patterns.push(name);
- }
- }
- let name = unsafe { focused.CurrentName() }
- .ok()
- .map(|b| b.to_string())
- .unwrap_or_default();
- let class = unsafe { focused.CurrentClassName() }
- .ok()
- .map(|b| b.to_string())
- .unwrap_or_default();
- let ctype = unsafe { focused.CurrentControlType() }
- .map(|t| t.0)
- .unwrap_or(0);
- let value_ro = if patterns.contains(&"Value") {
- unsafe {
- focused.GetCurrentPattern(UIA_ValuePatternId)
- .and_then(|p| p.cast::<windows::Win32::UI::Accessibility::IUIAutomationValuePattern>())
- .and_then(|vp| vp.CurrentIsReadOnly())
- .map(|b| b.as_bool())
- .ok()
- }
- } else { None };
- focused_info = Some(serde_json::json!({
- "name": name,
- "class_name": class,
- "control_type_id": ctype,
- "supported_patterns": patterns,
- "value_pattern_is_read_only": value_ro,
- }));
+/// WinUI3 background gesture resolution, shared by click / double_click /
+/// right_click / drag. For a WinUI3 (`WinUIDesktopWin32WindowClass`) target:
+///
+/// - **double-click (left)** on a cached InvokePattern element → a double UIA
+/// Invoke (lands + holds the no-foreground contract; see
+/// [`winui3_uia_multi_invoke`]).
+/// - **everything else** — right/middle click, double-click without an
+/// invokable cached element, any drag — cannot both land AND hold the
+/// contract on WinUI3: UIA has no right-click/drag analogue, the content
+/// island ignores posted `WM_*BUTTON`, and the pointer injector
+/// click-activates the frame. So we return the structured
+/// `background_unavailable` error (the honest result — the caller can retry
+/// that action with `delivery_mode:"foreground"`) instead of a silent
+/// no-op that reports false success.
+///
+/// Returns `None` for non-WinUI3 targets (caller falls through to its normal
+/// routing).
+pub(crate) async fn winui3_background_gesture(
+ admitted: &Option<crate::uia::snapshot::RetainedElement>,
+ pid: u32,
+ hwnd: u64,
+ idx: Option<usize>,
+ count: usize,
+ button: &str,
+) -> Option<ToolResult> {
+ let is_w = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::delivery::is_winui3_target_window(hwnd)
+ }
+ })
+ .await
+ .unwrap_or(false);
+ if !is_w {
+ return None;
+ }
+ if count >= 2 && button == "left" {
+ if let Some(idx) = idx {
+ let uia = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ winui3_uia_multi_invoke(&admitted, hwnd, count)
}
- }
- if coinit_ok {
- unsafe { CoUninitialize(); }
- }
-
- // ── Build per-window JSON
- let xaml_classes: HashSet<&str> = [
- "ApplicationFrameWindow",
- "WinUIDesktopWin32WindowClass",
- "Windows.UI.Core.CoreWindow",
- "Microsoft.UI.Content.DesktopChildSiteBridge",
- ].iter().copied().collect();
- let xaml_exes: HashSet<&str> = [
- "notepad.exe", "calculatorapp.exe", "calc.exe",
- "applicationframehost.exe", "photos.exe", "systemsettings.exe",
- ].iter().copied().collect();
-
- let windows_json: Vec<serde_json::Value> = ctx.hwnds.iter()
- .map(|&h| {
- let hwnd = HWND(h as *mut _);
- let cls = class_name(hwnd);
- let title = window_title(hwnd);
- let xaml_class_match = cls.as_deref()
- .map(|c| xaml_classes.contains(c))
- .unwrap_or(false);
- serde_json::json!({
- "hwnd": h,
- "class_name": cls,
- "title": title,
- "xaml_class_match": xaml_class_match,
- })
- })
- .collect();
-
- let exe_xaml_match = exe_basename.as_deref()
- .map(|e| xaml_exes.contains(e))
- .unwrap_or(false);
-
- let daemon_pid = unsafe { GetCurrentProcessId() };
- let mut daemon_session: u32 = 0;
- let _ = unsafe { ProcessIdToSessionId(daemon_pid, &mut daemon_session) };
-
- serde_json::json!({
- "pid": pid_v,
- "session_id": session_id,
- "daemon_pid": daemon_pid,
- "daemon_session_id": daemon_session,
- "exe_path": exe_path,
- "exe_basename": exe_basename,
- "exe_xaml_match": exe_xaml_match,
- "windows": windows_json,
- "focused_element": focused_info,
- "xaml_routing_recommended": exe_xaml_match
- || windows_json.iter()
- .any(|w| w.get("xaml_class_match").and_then(|v| v.as_bool()).unwrap_or(false)),
})
- }).await;
-
- match outcome {
- Ok(j) => {
- let pretty = serde_json::to_string_pretty(&j).unwrap_or_else(|_| j.to_string());
- ToolResult::text(pretty).with_structured(j)
+ .await
+ .ok()
+ .flatten();
+ if let Some(Ok(())) = uia {
+ return Some(ToolResult::text(format!(
+ "✅ Double-invoked WinUI3 element [{idx}] x{count} via UIA \
+ (pid {pid}, background, no foreground swap)."
+ )));
}
- Err(e) => ToolResult::error(format!("debug_window_info: join error: {e}")),
}
}
+ Some(crate::input::delivery::background_unavailable_error(
+ hwnd,
+ crate::input::delivery::EventKind::MouseClick,
+ ))
+}
+
+/// Read the primary display size in PHYSICAL pixels.
+///
+/// With permonitorv2 DPI awareness (set in cua-driver.manifest),
+/// `SM_CXSCREEN` / `SM_CYSCREEN` already return physical pixels — the same
+/// coordinate space screenshots and pixel clicks use on Windows. Shared by
+/// `get_screen_size` and `get_desktop_state`.
+pub(crate) fn physical_screen_size() -> (i32, i32) {
+ use windows::Win32::UI::WindowsAndMessaging::{GetSystemMetrics, SM_CXSCREEN, SM_CYSCREEN};
+ unsafe { (GetSystemMetrics(SM_CXSCREEN), GetSystemMetrics(SM_CYSCREEN)) }
}
// ── registry builder ──────────────────────────────────────────────────────────
@@ -10691,33 +1491,54 @@ pub fn build_registry_with_provider(
}
#[cfg(test)]
-mod cursor_key_resolution_tests;
-
-#[cfg(test)]
-mod launch_focus_restore_decision_tests;
-
-#[cfg(test)]
-mod chromium_flag_injection_tests;
-
-#[cfg(test)]
-mod click_capture_id_schema_tests;
-
-#[cfg(test)]
-mod snapshot_coordinate_tests;
-
-#[cfg(test)]
-mod set_window_frame_geometry_tests;
+mod cursor_key_resolution_tests {
+ use super::{resolve_cursor_key, NO_CURSOR};
+ use serde_json::json;
-#[cfg(test)]
-mod desktop_scope_tests;
+ /// Precedence is core `tool_args::session_key`'s contract; Windows only
+ /// owns the cursor-less fallback for a direct call without lifecycle
+ /// metadata.
+ #[test]
+ fn anonymous_direct_call_is_cursor_less() {
+ assert_eq!(resolve_cursor_key(&json!({ "pid": 1 })), NO_CURSOR);
+ assert_eq!(
+ resolve_cursor_key(&json!({ "session": "", "cursor_id": "" })),
+ NO_CURSOR
+ );
+ assert_eq!(
+ resolve_cursor_key(&json!({ "_session_id": "mcp-1-2" })),
+ "mcp-1-2"
+ );
+ }
+}
#[cfg(test)]
-mod drag_modifier_tests;
+mod pid_window_target_tests {
+ use super::*;
+ use cua_driver_core::window_target::{resolve_pid_window_target, PidWindowTargetResolution};
-#[cfg(test)]
-mod pid_window_target_tests;
+ fn window(hwnd: u64, pid: u32) -> crate::win32::WindowInfo {
+ crate::win32::WindowInfo {
+ hwnd,
+ pid,
+ title: format!("Document {hwnd}"),
+ x: 0,
+ y: 0,
+ width: 640,
+ height: 480,
+ is_on_screen: true,
+ minimized: false,
+ }
+ }
-#[cfg(test)]
-mod display_only_tests;
-#[cfg(test)]
-mod value_write_readback_tests;
+ #[test]
+ fn same_pid_sibling_windows_are_ambiguous() {
+ let candidates =
+ window_target_candidates_for_pid([window(7, 42), window(8, 42), window(9, 99)], 42);
+ assert!(matches!(
+ resolve_pid_window_target(candidates),
+ PidWindowTargetResolution::Ambiguous(windows)
+ if windows.iter().map(|window| window.window_id).collect::<Vec<_>>() == [7, 8]
+ ));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/background_element_click_record_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/background_element_click_record_tests.rs
deleted file mode 100644
index b0af8269af..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/background_element_click_record_tests.rs
+++ /dev/null
@@ -1,62 +0,0 @@
-use super::{background_element_click_result, ActionTransport};
-
-#[test]
-fn raw_fallback_reports_its_physical_transport() {
- for (transport, path) in [
- (ActionTransport::WindowsTargetedInjection, "pixel"),
- (ActionTransport::WindowsPostMessage, "post_message"),
- ] {
- let result = background_element_click_result(
- "physical fallback".into(),
- transport,
- vec![ActionTransport::WindowsUiaInvoke],
- );
- assert_eq!(result.structured_content.as_ref().unwrap()["path"], path);
- let record = result.action_record.unwrap();
- assert_eq!(record.transport, transport);
- let truth = record.debug_json();
- assert_ne!(truth["route"], "accessibility");
- assert_eq!(
- record.attempts[0].transport,
- ActionTransport::WindowsUiaInvoke
- );
- assert_eq!(record.fallbacks[0].to, transport);
- }
-}
-
-#[test]
-fn failed_provider_calls_disqualify_single_action_marker_exemption() {
- let result = background_element_click_result(
- "semantic success after provider failure".into(),
- ActionTransport::WindowsUiaToggle,
- vec![ActionTransport::WindowsUiaInvoke],
- );
- let record = result.action_record.unwrap();
- assert_eq!(record.transport, ActionTransport::WindowsUiaToggle);
- assert_eq!(record.attempts.len(), 1);
- assert_eq!(record.fallbacks.len(), 1);
- assert_eq!(record.fallbacks[0].from, ActionTransport::WindowsUiaInvoke);
- assert_eq!(record.fallbacks[0].to, ActionTransport::WindowsUiaToggle);
- // Point-free recording requires an empty fallback journal.
- assert!(!record.debug_json()["fallbacks"]
- .as_array()
- .unwrap()
- .is_empty());
-}
-
-#[test]
-fn single_provider_success_has_no_invented_attempts() {
- for transport in [
- ActionTransport::WindowsUiaInvoke,
- ActionTransport::WindowsUiaToggle,
- ActionTransport::WindowsUiaSelection,
- ActionTransport::WindowsUiaExpandCollapse,
- ] {
- let result = background_element_click_result("semantic success".into(), transport, vec![]);
- assert_eq!(result.structured_content.as_ref().unwrap()["path"], "ax");
- let record = result.action_record.unwrap();
- assert_eq!(record.transport, transport);
- assert!(record.attempts.is_empty());
- assert!(record.fallbacks.is_empty());
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/chromium_flag_injection_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/chromium_flag_injection_tests.rs
deleted file mode 100644
index c7235df25d..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/chromium_flag_injection_tests.rs
+++ /dev/null
@@ -1,108 +0,0 @@
-use super::{inject_chromium_anti_throttling_flags, is_chromium_browser_target};
-
-#[test]
-fn detects_bare_browser_names() {
- for name in [
- "msedge", "chrome", "brave", "opera", "vivaldi", "chromium", "thorium", "iridium",
- "browser", "arc",
- ] {
- assert!(is_chromium_browser_target(name), "{name} should match");
- assert!(
- is_chromium_browser_target(&format!("{name}.exe")),
- "{name}.exe should match"
- );
- // Case-insensitive.
- assert!(
- is_chromium_browser_target(&name.to_uppercase()),
- "uppercase {name} should match"
- );
- }
-}
-
-#[test]
-fn detects_full_paths() {
- assert!(is_chromium_browser_target(
- r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"
- ));
- assert!(is_chromium_browser_target(
- r"C:\Program Files\Google\Chrome\Application\chrome.exe"
- ));
- // Forward slashes too (some shells write paths that way).
- assert!(is_chromium_browser_target(
- r"C:/Program Files/Google/Chrome/Application/chrome.exe"
- ));
-}
-
-#[test]
-fn detects_launch_path_with_trailing_args() {
- // Round-tripped launch_path from list_apps with shortcut arguments.
- assert!(is_chromium_browser_target(
- r#""C:\Program Files\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 2""#
- ));
-}
-
-#[test]
-fn does_not_match_non_chromium_apps() {
- for name in ["firefox", "notepad", "explorer", "code", "soffice"] {
- assert!(!is_chromium_browser_target(name), "{name} should NOT match");
- assert!(
- !is_chromium_browser_target(&format!("{name}.exe")),
- "{name}.exe should NOT match"
- );
- }
- // Empty target.
- assert!(!is_chromium_browser_target(""));
-}
-
-#[test]
-fn injects_three_flags_into_empty_args() {
- let mut args: Vec<String> = vec![];
- inject_chromium_anti_throttling_flags(&mut args);
- assert!(args.contains(&"--disable-features=CalculateNativeWinOcclusion".to_string()));
- assert!(args.contains(&"--disable-backgrounding-occluded-windows".to_string()));
- assert!(args.contains(&"--disable-renderer-backgrounding".to_string()));
- assert_eq!(args.len(), 3);
-}
-
-#[test]
-fn merges_into_existing_disable_features_list() {
- let mut args = vec!["--disable-features=Foo,Bar".to_string()];
- inject_chromium_anti_throttling_flags(&mut args);
- // Should NOT have two --disable-features= entries.
- let dfe: Vec<_> = args
- .iter()
- .filter(|a| a.starts_with("--disable-features="))
- .collect();
- assert_eq!(dfe.len(), 1);
- assert!(dfe[0].contains("CalculateNativeWinOcclusion"));
- assert!(dfe[0].contains("Foo"));
- assert!(dfe[0].contains("Bar"));
-}
-
-#[test]
-fn idempotent_when_all_flags_already_present() {
- let mut args = vec![
- "--disable-features=CalculateNativeWinOcclusion".to_string(),
- "--disable-backgrounding-occluded-windows".to_string(),
- "--disable-renderer-backgrounding".to_string(),
- ];
- let before = args.clone();
- inject_chromium_anti_throttling_flags(&mut args);
- assert_eq!(args, before, "must not duplicate flags");
-}
-
-#[test]
-fn preserves_user_url_argument_after_flags() {
- let mut args = vec!["file:///C:/test_page.html".to_string()];
- inject_chromium_anti_throttling_flags(&mut args);
- // URL must still be present.
- assert!(args.iter().any(|a| a == "file:///C:/test_page.html"));
- // All three flags now in args.
- assert!(args
- .iter()
- .any(|a| a == "--disable-features=CalculateNativeWinOcclusion"));
- assert!(args
- .iter()
- .any(|a| a == "--disable-backgrounding-occluded-windows"));
- assert!(args.iter().any(|a| a == "--disable-renderer-backgrounding"));
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/click_capture_id_schema_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/click_capture_id_schema_tests.rs
deleted file mode 100644
index 709bdce421..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/click_capture_id_schema_tests.rs
+++ /dev/null
@@ -1,17 +0,0 @@
-use super::ClickTool;
-use cua_driver_core::tool::Tool;
-
-/// The portable click contract pins `button` and the other accepted
-/// fields. A live schema may broaden `capture_id`, which the subset gate
-/// allows, so the non-empty bound is pinned here.
-#[test]
-fn schema_requires_non_empty_capture_id() {
- let tool = ClickTool {
- state: super::ToolState::new(None),
- };
- let d = tool.def();
- let props = d.input_schema.get("properties").expect("properties");
- let capture_id = props.get("capture_id").expect("capture_id field present");
- assert_eq!(capture_id["type"], "string");
- assert_eq!(capture_id["minLength"], 1);
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/cursor_key_resolution_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/cursor_key_resolution_tests.rs
deleted file mode 100644
index c943b06a52..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/cursor_key_resolution_tests.rs
+++ /dev/null
@@ -1,18 +0,0 @@
-use super::{resolve_cursor_key, NO_CURSOR};
-use serde_json::json;
-
-/// Precedence is core `tool_args::session_key`'s contract; Windows only
-/// owns the cursor-less fallback for a direct call without lifecycle
-/// metadata.
-#[test]
-fn anonymous_direct_call_is_cursor_less() {
- assert_eq!(resolve_cursor_key(&json!({ "pid": 1 })), NO_CURSOR);
- assert_eq!(
- resolve_cursor_key(&json!({ "session": "", "cursor_id": "" })),
- NO_CURSOR
- );
- assert_eq!(
- resolve_cursor_key(&json!({ "_session_id": "mcp-1-2" })),
- "mcp-1-2"
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/desktop_scope_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/desktop_scope_tests.rs
deleted file mode 100644
index cc55418372..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/desktop_scope_tests.rs
+++ /dev/null
@@ -1,76 +0,0 @@
-use super::{is_windowless_desktop_action, GetDesktopStateTool};
-use cua_driver_core::tool::Tool;
-use serde_json::json;
-
-// ── is_windowless_desktop_action ──────────────────────────────────────────
-
-#[test]
-fn windowless_true_for_xy_under_desktop_scope_click_shape() {
- // Click arg shape: {x, y}.
- assert!(is_windowless_desktop_action(
- &json!({"x": 10, "y": 20, "scope": "desktop"})
- ));
-}
-
-#[test]
-fn windowless_true_for_xy_under_desktop_scope_scroll_shape() {
- // Scroll arg shape: {direction, x, y}.
- assert!(is_windowless_desktop_action(&json!({
- "direction": "down", "x": 10, "y": 20, "scope": "desktop"
- })));
-}
-
-#[test]
-fn windowless_false_when_pid_present() {
- assert!(!is_windowless_desktop_action(&json!({
- "x": 10, "y": 20, "pid": 5, "scope": "desktop"
- })));
-}
-
-#[test]
-fn windowless_false_when_window_id_present() {
- assert!(!is_windowless_desktop_action(&json!({
- "x": 10, "y": 20, "window_id": 99, "scope": "desktop"
- })));
-}
-
-#[test]
-fn windowless_false_under_window_scope() {
- assert!(!is_windowless_desktop_action(&json!({
- "x": 10, "y": 20, "scope": "window"
- })));
-}
-
-#[test]
-fn windowless_false_when_xy_missing() {
- assert!(!is_windowless_desktop_action(
- &json!({"x": 10, "scope": "desktop"})
- ));
- assert!(!is_windowless_desktop_action(
- &json!({"y": 20, "scope": "desktop"})
- ));
- assert!(!is_windowless_desktop_action(&json!({"scope": "desktop"})));
- // Non-numeric x/y must not qualify.
- assert!(!is_windowless_desktop_action(&json!({
- "x": "10", "y": "20", "scope": "desktop"
- })));
-}
-
-// ── get_desktop_state schema ──────────────────────────────────────────────
-
-#[test]
-fn get_desktop_state_schema_rejects_window_scoped_fields() {
- let tool = GetDesktopStateTool {
- state: super::ToolState::new(None),
- };
- // The portable get_desktop_state contract pins the accepted fields and
- // annotations; it cannot see a window-scoped field added to live.
- let d = tool.def();
- let props = d.input_schema["properties"].as_object().unwrap();
- assert!(!props.contains_key("pid"), "must not accept pid");
- assert!(
- !props.contains_key("window_id"),
- "must not accept window_id"
- );
- assert_eq!(d.input_schema["additionalProperties"], json!(false));
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs
deleted file mode 100644
index e39fda62c1..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/display_only_tests.rs
+++ /dev/null
@@ -1,35 +0,0 @@
-use super::{GetWindowStateTool, ToolState};
-use cua_driver_core::tool::Tool;
-
-// T3 Code feature-detects the preview mode from this schema property.
-#[test]
-fn display_only_is_advertised() {
- let tool = GetWindowStateTool {
- state: ToolState::new(None),
- };
- assert_eq!(
- tool.def().input_schema["properties"]["display_only"]["type"],
- "boolean"
- );
-}
-
-#[tokio::test]
-async fn display_only_refuses_accessibility_snapshot_requests() {
- let tool = GetWindowStateTool {
- state: ToolState::new(None),
- };
- for tree in [None, Some(true)] {
- let mut args = serde_json::json!({"pid": 42, "window_id": 7, "display_only": true});
- if let Some(tree) = tree {
- args["include_accessibility_tree"] = serde_json::json!(tree);
- }
- let result = tool.invoke(args).await;
- assert_eq!(
- serde_json::to_value(result).unwrap(),
- serde_json::to_value(cua_driver_core::protocol::ToolResult::error(
- "display_only requires include_accessibility_tree:false"
- ))
- .unwrap()
- );
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/drag_modifier_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/drag_modifier_tests.rs
deleted file mode 100644
index 3f133444d9..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/drag_modifier_tests.rs
+++ /dev/null
@@ -1,33 +0,0 @@
-use super::DragTool;
-use cua_driver_core::tool::Tool;
-use serde_json::json;
-
-// A background drag posts mouse messages or injects pen input, neither of
-// which carries live modifier-key state. The refusal comes before any window
-// lookup or input, so an unknown pid never reaches the OS.
-#[tokio::test]
-async fn background_drag_with_modifiers_is_refused_before_input() {
- let tool = DragTool {
- state: super::ToolState::new(None),
- };
- for delivery in [json!(null), json!("background")] {
- let mut args = json!({
- "pid": 0x7fff_fff0_i64,
- "from_x": 10, "from_y": 10, "to_x": 50, "to_y": 10,
- "modifier": ["ctrl"]
- });
- if !delivery.is_null() {
- args["delivery_mode"] = delivery;
- }
- let result = tool.invoke(args).await;
- assert_eq!(
- result.is_error,
- Some(true),
- "modified background drag was not refused"
- );
- let structured = result.structured_content.expect("structured refusal");
- assert_eq!(structured["code"], "background_unavailable");
- assert_eq!(structured["effect"], "refused");
- assert_eq!(structured["escalation"]["recommended"], "foreground");
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/get_window_state_actions_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/get_window_state_actions_tests.rs
deleted file mode 100644
index 1690dc3e34..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/get_window_state_actions_tests.rs
+++ /dev/null
@@ -1,38 +0,0 @@
-use super::*;
-use crate::uia::UiaNode;
-
-fn node(actions: Vec<String>) -> UiaNode {
- UiaNode {
- element_index: Some(1),
- control_type: "Button".to_owned(),
- name: Some("OK".to_owned()),
- value: None,
- automation_id: None,
- help_text: None,
- actions,
- enabled: Some(true),
- selected: None,
- element_ptr: 0,
- center_x: 0,
- center_y: 0,
- rect: None,
- msaa_role: None,
- depth: 0,
- parent_element_index: None,
- in_web_content: false,
- }
-}
-
-#[test]
-fn element_entry_includes_actions_when_present() {
- let n = node(vec!["invoke".to_owned(), "toggle".to_owned()]);
- let entry = build_element_entry(&n, None).unwrap();
- assert_eq!(entry["actions"], json!(["invoke", "toggle"]));
-}
-
-#[test]
-fn element_entry_omits_actions_when_empty() {
- let n = node(Vec::new());
- let entry = build_element_entry(&n, None).unwrap();
- assert!(entry.get("actions").is_none());
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/launch_focus_restore_decision_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/launch_focus_restore_decision_tests.rs
deleted file mode 100644
index 8c3da651ce..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/launch_focus_restore_decision_tests.rs
+++ /dev/null
@@ -1,50 +0,0 @@
-use super::{
- first_unopened_shell_url_index, should_restore_foreground_after_launch, LaunchTargetShape,
-};
-
-#[test]
-fn only_app_identifying_launches_restore_foreground() {
- for (fields, restore) in [
- // A urls-only launch explicitly asks for navigation in the default
- // browser; that browser is the legitimate foreground.
- (&["urls"][..], false),
- (&["name"], true),
- (&["path"], true),
- // The AUMID path restores synchronously in launch_uwp.rs; the
- // caller gates the polling restore on `aumid_for_uwp.is_none()`.
- (&["aumid"], true),
- (&["bundle_id"], true),
- (&["launch_path"], true),
- // An app-identifying field alongside urls opens them in that app
- // in the background, so the prior foreground is restored.
- (&["name", "urls"], true),
- (&["path", "urls"], true),
- (&["aumid", "urls"], true),
- // Nothing was launched.
- (&[], false),
- ] {
- let shape = LaunchTargetShape {
- has_aumid: fields.contains(&"aumid"),
- has_bundle_id: fields.contains(&"bundle_id"),
- has_name: fields.contains(&"name"),
- has_path: fields.contains(&"path"),
- has_launch_path: fields.contains(&"launch_path"),
- has_urls: fields.contains(&"urls"),
- };
- assert_eq!(
- should_restore_foreground_after_launch(shape),
- restore,
- "{fields:?}"
- );
- }
-}
-
-#[test]
-fn named_app_launch_forwards_every_url() {
- assert_eq!(first_unopened_shell_url_index(true), 0);
-}
-
-#[test]
-fn urls_only_launch_does_not_reopen_primary_url() {
- assert_eq!(first_unopened_shell_url_index(false), 1);
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/list_windows_z_index_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/list_windows_z_index_tests.rs
deleted file mode 100644
index 6a0bd94876..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/list_windows_z_index_tests.rs
+++ /dev/null
@@ -1,46 +0,0 @@
-use super::{drop_own_windows, exact_window_ownership_result, z_index_from_front_to_back};
-use crate::win32::WindowInfo;
-
-#[test]
-fn explicit_pid_hwnd_guard_refuses_wrong_or_stale_owners() {
- assert!(exact_window_ownership_result(42, 7, Some(42)).is_ok());
-
- let wrong = exact_window_ownership_result(42, 7, Some(99)).unwrap_err();
- assert_eq!(wrong.is_error, Some(true));
- assert_eq!(
- wrong.structured_content.as_ref().unwrap()["code"],
- "window_target_mismatch"
- );
- assert_eq!(wrong.structured_content.as_ref().unwrap()["owner_pid"], 99);
-
- let stale = exact_window_ownership_result(42, 7, None).unwrap_err();
- assert_eq!(
- stale.structured_content.as_ref().unwrap()["code"],
- "window_target_not_found"
- );
-}
-
-fn window(hwnd: u64, pid: u32) -> WindowInfo {
- WindowInfo {
- hwnd,
- pid,
- title: format!("window {hwnd}"),
- x: 0,
- y: 0,
- width: 1280,
- height: 800,
- is_on_screen: true,
- minimized: false,
- }
-}
-
-#[test]
-fn the_drivers_own_windows_are_dropped_and_the_rest_keep_contiguous_z() {
- let mut windows = vec![window(1, 7), window(2, 42), window(3, 7), window(4, 9)];
- drop_own_windows(&mut windows, 7);
- assert_eq!(windows.iter().map(|w| w.hwnd).collect::<Vec<_>>(), [2, 4]);
- let z: Vec<_> = (0..windows.len())
- .map(|i| z_index_from_front_to_back(windows.len(), i))
- .collect();
- assert_eq!(z, [1, 0]);
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pid_window_target_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pid_window_target_tests.rs
deleted file mode 100644
index f0631d067b..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pid_window_target_tests.rs
+++ /dev/null
@@ -1,27 +0,0 @@
-use super::*;
-use cua_driver_core::window_target::{resolve_pid_window_target, PidWindowTargetResolution};
-
-fn window(hwnd: u64, pid: u32) -> crate::win32::WindowInfo {
- crate::win32::WindowInfo {
- hwnd,
- pid,
- title: format!("Document {hwnd}"),
- x: 0,
- y: 0,
- width: 640,
- height: 480,
- is_on_screen: true,
- minimized: false,
- }
-}
-
-#[test]
-fn same_pid_sibling_windows_are_ambiguous() {
- let candidates =
- window_target_candidates_for_pid([window(7, 42), window(8, 42), window(9, 99)], 42);
- assert!(matches!(
- resolve_pid_window_target(candidates),
- PidWindowTargetResolution::Ambiguous(windows)
- if windows.iter().map(|window| window.window_id).collect::<Vec<_>>() == [7, 8]
- ));
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pixel_click_transport_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pixel_click_transport_tests.rs
deleted file mode 100644
index 6c7922bb07..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/pixel_click_transport_tests.rs
+++ /dev/null
@@ -1,131 +0,0 @@
-use super::{finish_pixel_uia_attempt, posted_pixel_click_result};
-use crate::uia::windows_enum::PointInvokeOutcome;
-use cua_driver_core::action_record::{
- ActionExecutionRecord, ActionTransport, ActualDelivery, RequestedDelivery,
-};
-
-#[test]
-fn uia_unavailable_errors_advertise_foreground_escalation() {
- use cua_driver_core::action_record::EscalationKind;
- use cua_driver_core::protocol::Content;
- for (outcome, status) in [
- (PointInvokeOutcome::Busy, "busy"),
- (PointInvokeOutcome::Timeout, "timeout"),
- (PointInvokeOutcome::Unavailable, "unavailable"),
- ] {
- let result = finish_pixel_uia_attempt(outcome, 7, 3, 4)
- .expect("unavailable UIA must stop the route without fallback");
- assert!(result.is_error.unwrap_or(false), "{outcome:?}");
- let data = result
- .structured_content
- .as_ref()
- .expect("structured error");
- assert_eq!(data["code"], "background_unavailable");
- assert_eq!(data["uia_status"], status);
- assert_eq!(data["path"], "ax");
- assert_eq!(data["effect"], "unverifiable");
- assert_eq!(
- data["suggestion"].as_str(),
- Some("Retry this action with delivery_mode:\"foreground\"."),
- "{outcome:?}"
- );
- assert_eq!(
- data["escalation"]["recommended"].as_str(),
- Some("foreground"),
- "{outcome:?}"
- );
- let reason = data["escalation"]["reason"]
- .as_str()
- .expect("escalation reason");
- assert!(
- reason.contains(status),
- "reason must name the UIA status: {reason}"
- );
- assert!(
- reason.contains("delivery_mode:\"foreground\""),
- "reason must name the next rung: {reason}"
- );
- let text = match &result.content[0] {
- Content::Text { text, .. } => text,
- _ => panic!("expected text content for {outcome:?}"),
- };
- assert!(
- text.contains("No fallback input was sent"),
- "text must keep the no-replay guarantee: {text}"
- );
- assert!(
- text.contains("delivery_mode:\"foreground\""),
- "text must surface the escalation: {text}"
- );
- let record = ActionExecutionRecord::from_legacy(
- "click",
- &serde_json::json!({ "delivery_mode": "background" }),
- data,
- )
- .expect("UIA refusal should normalize into the public action contract");
- let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
- .expect("serialize ActionResult");
- assert_eq!(public["effect"], "unverifiable", "{outcome:?}");
- }
- // The completed-miss boundary is unchanged: only Miss falls through,
- // and a delivered Invoke carries no escalation hint.
- assert!(finish_pixel_uia_attempt(PointInvokeOutcome::Miss, 7, 3, 4).is_none());
- let ok = finish_pixel_uia_attempt(PointInvokeOutcome::Invoked, 7, 3, 4)
- .expect("invoked click reports success");
- assert!(!ok.is_error.unwrap_or(false));
- let ok_data = ok.structured_content.as_ref().expect("structured success");
- assert_eq!(ok_data["path"], "ax");
- assert!(ok_data.get("escalation").is_none());
- assert!(ok_data.get("suggestion").is_none());
- let ok_record = ActionExecutionRecord::from_legacy(
- "click",
- &serde_json::json!({ "delivery_mode": "background" }),
- ok_data,
- )
- .expect("UIA invoke should normalize into the public action contract");
- let ok_public = serde_json::to_value(ok_record.public_result().expect("valid ActionResult"))
- .expect("serialize ActionResult");
- assert_eq!(ok_public["effect"], "unverifiable");
- // The hint must flow into the existing escalation pipeline, not sit
- // as inert metadata: Timeout (the #3621 case) normalizes to a
- // foreground-delivery escalation on the internal record.
- let timeout_data = finish_pixel_uia_attempt(PointInvokeOutcome::Timeout, 7, 3, 4)
- .expect("timeout stops the route")
- .structured_content
- .expect("structured error");
- let timeout_record = ActionExecutionRecord::from_legacy(
- "click",
- &serde_json::json!({ "delivery_mode": "background" }),
- &timeout_data,
- )
- .expect("UIA timeout should normalize into the public action contract");
- assert_eq!(
- timeout_record.escalation.map(|escalation| escalation.kind),
- Some(EscalationKind::RetryWithForegroundDelivery)
- );
-}
-
-#[test]
-fn post_message_pixel_click_reports_synthetic_transport() {
- let result = posted_pixel_click_result(42, "click");
- let structured = result
- .structured_content
- .as_ref()
- .expect("posted click should expose legacy transport metadata");
- assert_eq!(structured["path"], "post_message");
-
- let record = ActionExecutionRecord::from_legacy(
- "click",
- &serde_json::json!({ "delivery_mode": "background" }),
- structured,
- )
- .expect("posted click should normalize into the public action contract");
- assert_eq!(record.transport, ActionTransport::WindowsPostMessage);
- assert_eq!(record.requested_delivery, RequestedDelivery::Background);
- assert_eq!(record.actual_delivery, Some(ActualDelivery::Background));
-
- let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
- .expect("serialize ActionResult");
- assert_eq!(public["route"], "synthetic_events");
- assert_eq!(public["delivery"]["mode"], "background");
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/post_message_readback_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/post_message_readback_tests.rs
deleted file mode 100644
index b1a69139b9..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/post_message_readback_tests.rs
+++ /dev/null
@@ -1,47 +0,0 @@
-use super::{changed_contains_post_message_result, post_message_readback_observed};
-
-#[test]
-fn observes_only_a_changed_value_containing_the_requested_text() {
- assert!(post_message_readback_observed(
- Some("prefix"),
- Some("prefixhello"),
- "hello"
- ));
- assert!(!post_message_readback_observed(None, None, "hello"));
- assert!(!post_message_readback_observed(
- Some("hello"),
- Some("hello"),
- "hello"
- ));
- assert!(!post_message_readback_observed(
- Some(""),
- Some("h"),
- "hello"
- ));
- assert!(!post_message_readback_observed(
- Some("before"),
- Some("different"),
- "hello"
- ));
-}
-
-#[test]
-fn changed_contains_remains_unverifiable_without_retry_escalation() {
- let result = changed_contains_post_message_result(5);
- assert_eq!(result["effect"], "unverifiable");
- assert_eq!(result["verify"], "changed_contains");
- assert_eq!(result["verified"], false);
- assert!(result.get("escalation").is_none());
-
- let record = cua_driver_core::action_record::ActionExecutionRecord::from_legacy(
- "type_text",
- &serde_json::json!({"delivery_mode": "background"}),
- &result,
- )
- .expect("changed-and-contains PostMessage result should normalize");
- let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
- .expect("serialize ActionResult");
- assert_eq!(public["effect"], "unverifiable");
- assert!(public.get("escalation").is_none());
- assert!(public.get("evidence").is_none());
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/set_window_frame_geometry_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/set_window_frame_geometry_tests.rs
deleted file mode 100644
index 0f30e69c28..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/set_window_frame_geometry_tests.rs
+++ /dev/null
@@ -1,27 +0,0 @@
-use super::outer_frame_for_visible_request;
-
-#[test]
-fn compensates_for_invisible_resize_borders() {
- assert_eq!(
- outer_frame_for_visible_request(
- (65, 52, 872, 626),
- (40, 40, 886, 633),
- (47, 40, 872, 626),
- )
- .unwrap(),
- (58, 52, 886, 633)
- );
-}
-
-#[test]
-fn leaves_outer_request_unchanged_when_dwm_bounds_are_unavailable() {
- assert_eq!(
- outer_frame_for_visible_request(
- (-25, 10, 640, 480),
- (20, 30, 800, 600),
- (20, 30, 800, 600),
- )
- .unwrap(),
- (-25, 10, 640, 480)
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/snapshot_coordinate_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/snapshot_coordinate_tests.rs
deleted file mode 100644
index 820553509a..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/snapshot_coordinate_tests.rs
+++ /dev/null
@@ -1,77 +0,0 @@
-use super::{focus_by_pixel_click_args, ToolState, ZoomTool};
-use cua_driver_core::tool::Tool;
-
-#[test]
-fn schema_keeps_pid_optional_for_window_owned_lookup() {
- let tool = ZoomTool {
- state: ToolState::new(None),
- };
- let required = tool.def().input_schema["required"].as_array().unwrap();
- assert!(!required.iter().any(|field| field == "pid"));
- assert!(tool.def().input_schema["properties"].get("pid").is_some());
-}
-
-#[test]
-fn press_key_native_element_focus_skips_second_screenshot_scaling() {
- let native = focus_by_pixel_click_args(
- 42,
- Some(7),
- 120.0,
- 80.0,
- false,
- None,
- Some("client-a".to_owned()),
- false,
- true,
- );
- assert_eq!(native["_native_coordinates"], true);
-
- let screenshot = focus_by_pixel_click_args(
- 42,
- Some(7),
- 60.0,
- 40.0,
- false,
- None,
- Some("client-a".to_owned()),
- false,
- false,
- );
- assert!(screenshot.get("_native_coordinates").is_none());
-}
-
-#[test]
-fn resolver_refuses_cached_center_read_while_minimized() {
- use crate::uia::snapshot::{RetainedElement, SnapshotKind};
- use windows::Win32::UI::WindowsAndMessaging::{
- GetSystemMetrics, SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN,
- SM_YVIRTUALSCREEN,
- };
- let (x, y, w, h) = unsafe {
- (
- GetSystemMetrics(SM_XVIRTUALSCREEN),
- GetSystemMetrics(SM_YVIRTUALSCREEN),
- GetSystemMetrics(SM_CXVIRTUALSCREEN),
- GetSystemMetrics(SM_CYVIRTUALSCREEN),
- )
- };
- assert!(w > 0 && h > 0, "runner has no virtual-screen metrics");
- assert!(crate::input::point_on_virtual_desktop(x, y));
-
- let none = super::resolve_onscreen_point_with_scroll(&None, 0, 1, x, y, "clicking");
- assert_eq!(none.ok(), Some((x, y)));
-
- for kind in [SnapshotKind::Uia, SnapshotKind::Msaa] {
- let fresh = Some(RetainedElement::detached(kind, (x, y), false));
- let ok = super::resolve_onscreen_point_with_scroll(&fresh, 0, 1, x, y, "clicking");
- assert_eq!(ok.ok(), Some((x, y)));
-
- let stale = Some(RetainedElement::detached(kind, (x, y), true));
- let err = super::resolve_onscreen_point_with_scroll(&stale, 0, 1, x, y, "clicking")
- .expect_err("flagged element must be refused");
- let sc = err.structured_content.expect("structured refusal");
- assert_eq!(sc["status"], "refused");
- assert_eq!(sc["code"], "element_not_visible");
- assert_eq!(sc["refusal"]["code"], "element_not_visible");
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/value_write_readback_tests.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/value_write_readback_tests.rs
deleted file mode 100644
index 9b7c1867c1..0000000000
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_/value_write_readback_tests.rs
+++ /dev/null
@@ -1,58 +0,0 @@
-use super::{classify_value_write_readback, value_write_structured_result};
-
-#[test]
-fn confirms_when_the_expected_value_replaces_the_prior_value() {
- assert_eq!(
- classify_value_write_readback(Some("beforeinserted"), "before", "beforeinserted"),
- "confirmed"
- );
-}
-
-#[test]
-fn treats_stale_or_unreadable_value_as_pending() {
- assert_eq!(
- classify_value_write_readback(Some("old value"), "old value", "old valueinserted"),
- "pending"
- );
- assert_eq!(
- classify_value_write_readback(None, "old value", "old valueinserted"),
- "pending"
- );
-}
-
-#[test]
-fn deferred_publication_remains_unverifiable_without_retry_escalation() {
- let result = value_write_structured_result(8, "pending", false);
- assert_eq!(result["effect"], "unverifiable");
- assert_eq!(result["verify"], "pending");
- assert_eq!(result["verified"], false);
- assert!(result.get("escalation").is_none());
-
- let record = cua_driver_core::action_record::ActionExecutionRecord::from_legacy(
- "type_text",
- &serde_json::json!({"delivery_mode": "background"}),
- &result,
- )
- .expect("deferred ValuePattern result should normalize");
- let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
- .expect("serialize ActionResult");
- assert_eq!(public["effect"], "unverifiable");
- assert!(public.get("escalation").is_none());
- assert!(public.get("evidence").is_none());
-}
-
-#[test]
-fn does_not_confirm_when_stale_value_contains_the_typed_text() {
- assert_eq!(
- classify_value_write_readback(Some("10.00"), "10.00", "10.000"),
- "pending"
- );
-}
-
-#[test]
-fn does_not_confirm_an_empty_write() {
- assert_eq!(
- classify_value_write_readback(Some("unchanged"), "unchanged", "unchanged"),
- "pending"
- );
-}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/invoke_menu.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/invoke_menu.rs
new file mode 100644
index 0000000000..8b1297402f
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/invoke_menu.rs
@@ -0,0 +1,152 @@
+use async_trait::async_trait;
+use cua_driver_contract::InvokeMenuInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_input,
+};
+use serde_json::{json, Value};
+
+// ── invoke_menu ───────────────────────────────────────────────────────────
+
+pub struct InvokeMenuTool;
+static INVOKE_MENU_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+fn normalized_menu_path(path: Vec<String>) -> Result<Vec<String>, String> {
+ if path.is_empty() || path.len() > 16 {
+ return Err("invoke_menu: path must contain between 1 and 16 segments".into());
+ }
+ path.into_iter()
+ .enumerate()
+ .map(|(index, segment)| {
+ let segment = segment.trim();
+ if segment.is_empty() {
+ Err(format!("invoke_menu: path segment {index} is empty"))
+ } else {
+ Ok(segment.to_owned())
+ }
+ })
+ .collect()
+}
+
+fn menu_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": { "code": "menu_path_unavailable", "message": message }
+ }))
+}
+
+/// Activate a window for a visible native-menu operation. Windows may reject
+/// the first foreground request after the test/user session has been idle even
+/// when we attach to the current foreground thread. A reserved no-name key has
+/// no application meaning but makes this process the owner of the most recent
+/// input, allowing a bounded retry without sending a real shortcut or text to
+/// whichever application currently has focus.
+fn activate_window_for_menu(hwnd: windows::Win32::Foundation::HWND) -> Result<bool, String> {
+ let (activated, assisted) = unsafe { crate::input::force_foreground_assisted(hwnd) };
+ if activated {
+ Ok(assisted)
+ } else {
+ Err("invoke_menu: target window could not be activated".to_owned())
+ }
+}
+
+#[async_trait]
+impl Tool for InvokeMenuTool {
+ fn def(&self) -> &ToolDef {
+ INVOKE_MENU_DEF.get_or_init(|| {
+ let contract =
+ cua_driver_contract::tool_contract("invoke_menu").expect("invoke_menu contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecord, ActionTransport, ActualDelivery,
+ EvidenceKind, RequestedDelivery,
+ };
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetForegroundWindow, GetWindowThreadProcessId, IsWindow,
+ };
+
+ let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let path = match normalized_menu_path(input.path) {
+ Ok(path) => path,
+ Err(error) => return menu_refusal(error),
+ };
+ let hwnd_value = input.window_id;
+ let hwnd = HWND(hwnd_value as usize as *mut _);
+ if !unsafe { IsWindow(hwnd) }.as_bool() {
+ return menu_refusal("invoke_menu: window_id is closed, stale, or invalid".into());
+ }
+ let mut owner_pid = 0_u32;
+ unsafe { GetWindowThreadProcessId(hwnd, Some(&mut owner_pid)) };
+ if owner_pid != input.pid {
+ return menu_refusal("invoke_menu: window_id does not belong to pid".into());
+ }
+
+ let outcome = tokio::task::spawn_blocking(move || {
+ // `HWND` wraps a raw pointer and is intentionally not `Send`.
+ // Move the integer handle into the blocking worker and rebuild the
+ // process-local wrapper there instead of carrying it across threads.
+ let hwnd = HWND(hwnd_value as usize as *mut _);
+ let prior = unsafe { GetForegroundWindow() };
+ let needs_activation = prior != hwnd;
+ let used_activation_retry = if needs_activation {
+ activate_window_for_menu(hwnd)?
+ } else {
+ false
+ };
+ if needs_activation {
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ }
+ let result = crate::uia::invoke_menu_path(hwnd_value, &path);
+ if needs_activation && !prior.0.is_null() {
+ let _ = unsafe { crate::input::force_foreground_attached(prior) };
+ }
+ result.map(|()| used_activation_retry)
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(used_activation_retry)) => ToolResult::text(
+ "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",
+ )
+ .with_action_record(
+ ActionExecutionRecord::builder(
+ ActionEffect::Unverifiable,
+ ActionTransport::WindowsUiaInvoke,
+ RequestedDelivery::Foreground,
+ )
+ .actual_delivery(ActualDelivery::Foreground)
+ .evidence(ActionEvidence {
+ kind: EvidenceKind::NativeApiResult,
+ detail: "Every menu hop resolved uniquely and UI Automation accepted the final action"
+ .into(),
+ })
+ .detail(if used_activation_retry {
+ "target activation required a bounded reserved-key foreground-lock retry"
+ } else {
+ "target was already foreground or activated without a foreground-lock retry"
+ })
+ .build()
+ .expect("invoke_menu record is valid"),
+ ),
+ Ok(Err(error)) => menu_refusal(error),
+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/kill_app.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/kill_app.rs
new file mode 100644
index 0000000000..14a29b61f6
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/kill_app.rs
@@ -0,0 +1,166 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── kill_app ──────────────────────────────────────────────────────────────────
+//
+// Force-terminate a process by pid. Background:
+// * Win32 / GDI apps respond cleanly to `WM_CLOSE` (Alt+F4 via PostMessage),
+// so `click` on the X button or a hand-rolled hotkey gesture suffices.
+// * UWP / WinUI3 apps (Calculator, Photos, modern Settings, Win11 Notepad)
+// have backgrounding semantics — they accept `WM_CLOSE` but route it
+// into a "minimize to suspended" path instead of exiting. Cooperative
+// close-button automation produces orphan processes that survive every
+// polite eviction attempt. See CUA-541 for the live repro from
+// WINDOWS_CLAUDE_CODE_TEST_PLAN.md Test G (3 leftover CalculatorApp.exe).
+//
+// `kill_app` is the force-terminate escalation — `taskkill /F` equivalent —
+// keyed on pid. Marked `destructive: true` so MCP clients with permission
+// gating prompt before invoking.
+
+pub struct KillAppTool;
+static KILL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for KillAppTool {
+ fn def(&self) -> &ToolDef {
+ KILL_DEF.get_or_init(|| ToolDef {
+ name: "kill_app".into(),
+ description: "Force-terminate a process by pid. Use when the standard close path \
+ (Alt+F4 / WM_CLOSE via `click` on the X button) fails to make the process \
+ exit — typical for UWP / WinUI3 apps (Calculator, Photos, modern Notepad) \
+ that route WM_CLOSE into a suspended-but-resident state. Equivalent to \
+ `taskkill /F /PID <pid>`. Unsaved state is lost. Prefer the click-the-X path \
+ first; only escalate to `kill_app` when polite close didn't terminate."
+ .into(),
+ input_schema: json!({"type":"object","required":["pid"],"properties":{
+ "pid":{"type":"integer","description":"PID of the process to terminate."}
+ },"additionalProperties":false}),
+ read_only: false,
+ destructive: true,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn protected_resource_scope(
+ &self,
+ adapter_id: &str,
+ args: &Value,
+ ) -> Result<Option<Value>, String> {
+ if adapter_id != "process_control" {
+ return Ok(None);
+ }
+ use cua_driver_core::browser::platform::BrowserPlatform;
+ let pid = args
+ .get("pid")
+ .and_then(Value::as_i64)
+ .filter(|pid| *pid > 0)
+ .ok_or_else(|| "kill_app requires a positive integer pid".to_owned())?;
+ let fingerprint = crate::browser_platform::WindowsBrowserPlatform::default()
+ .process_fingerprint(pid)
+ .await
+ .map_err(|error| error.message)?;
+ Ok(Some(json!({
+ "kind": "process_instance",
+ "fingerprint": fingerprint,
+ })))
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if let Some(expected) = args.get("_protected_process_fingerprint") {
+ let current = match self
+ .protected_resource_scope("process_control", &args)
+ .await
+ {
+ Ok(Some(scope)) => scope["fingerprint"].clone(),
+ Ok(None) => Value::Null,
+ Err(message) => return kill_app_stale_process_refusal(message),
+ };
+ if current != *expected {
+ return kill_app_stale_process_refusal(
+ "the process identity changed at the termination boundary".to_owned(),
+ );
+ }
+ }
+ let pid_v: u32 = match args.get("pid").and_then(|v| v.as_u64()) {
+ Some(p) if p > 0 && p <= u32::MAX as u64 => p as u32,
+ Some(_) => {
+ return ToolResult::error("kill_app: `pid` must be a positive integer".to_string())
+ }
+ None => {
+ return ToolResult::error(
+ "kill_app: missing required integer field `pid`".to_string(),
+ )
+ }
+ };
+
+ // Run the syscalls on a blocking thread — TerminateProcess + the
+ // WaitForSingleObject confirmation are both blocking, and we don't
+ // want to stall the tokio reactor.
+ let outcome = tokio::task::spawn_blocking(move || -> Result<(), String> {
+ use windows::Win32::Foundation::{CloseHandle, WAIT_OBJECT_0};
+ use windows::Win32::System::Threading::{
+ OpenProcess, TerminateProcess, WaitForSingleObject, PROCESS_SYNCHRONIZE,
+ PROCESS_TERMINATE,
+ };
+
+ // SAFETY: OpenProcess returns a Result<HANDLE> in windows-rs;
+ // PROCESS_TERMINATE | PROCESS_SYNCHRONIZE lets us both kill the
+ // target AND wait for the OS to fully reap it (so the caller's
+ // follow-up list_apps reflects the change immediately).
+ let h = unsafe { OpenProcess(PROCESS_TERMINATE | PROCESS_SYNCHRONIZE, false, pid_v) }
+ .map_err(|e| {
+ format!(
+ "OpenProcess(pid={pid_v}, PROCESS_TERMINATE|PROCESS_SYNCHRONIZE) failed: {e}. \
+ The process may not exist, or the daemon lacks the right to terminate it. \
+ Try listing processes first to confirm the pid is correct."
+ )
+ })?;
+
+ let term_result = unsafe { TerminateProcess(h, 1) };
+
+ // WaitForSingleObject so the caller can assume the process is
+ // really gone on return. 2-second cap — TerminateProcess is
+ // synchronous in practice but the OS still needs a moment to
+ // reap kernel structures. WAIT_TIMEOUT is treated as a soft
+ // success (the kill request landed; the process may take a
+ // little longer to disappear from list_apps).
+ let wait_status = unsafe { WaitForSingleObject(h, 2000) };
+ let _ = unsafe { CloseHandle(h) };
+
+ match (term_result, wait_status) {
+ (Ok(()), s) if s == WAIT_OBJECT_0 => Ok(()),
+ (Ok(()), _) => Ok(()), // killed, wait timed out — caller can re-check
+ (Err(e), _) => Err(format!(
+ "TerminateProcess(pid={pid_v}) failed: {e}. The handle was opened \
+ successfully but the kill itself was rejected; this is unusual for \
+ PROCESS_TERMINATE rights and may indicate a protected-process target \
+ (PPL, e.g. csrss / system services)."
+ )),
+ }
+ })
+ .await;
+
+ match outcome {
+ Ok(Ok(())) => ToolResult::text(format!("✅ Terminated pid {pid_v}.")),
+ Ok(Err(e)) => ToolResult::error(format!("kill_app: {e}")),
+ Err(join_err) => {
+ ToolResult::error(format!("kill_app: blocking-task join error: {join_err}"))
+ }
+ }
+ }
+}
+
+fn kill_app_stale_process_refusal(message: String) -> ToolResult {
+ ToolResult::error(message.clone()).with_structured(json!({
+ "status": "refused",
+ "refusal": {
+ "code": "protected_resource_scope_stale",
+ "message": message,
+ }
+ }))
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/launch_app.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/launch_app.rs
new file mode 100644
index 0000000000..b908e618d9
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/launch_app.rs
@@ -0,0 +1,1266 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ window_target::z_index_from_front_to_back,
+};
+use serde_json::{json, Value};
+
+use super::impl_::{
+ is_chromium_browser_target, restore_foreground_polling_best_effort, split_launchable_target,
+};
+
+/// Index of the first URL not already consumed as the primary shell target.
+/// An explicit app target consumes no URL; a URLs-only launch consumes index 0.
+fn first_unopened_shell_url_index(has_app_target: bool) -> usize {
+ usize::from(!has_app_target)
+}
+
+/// Anti-throttling flags injected on hidden Chromium launches (#1620).
+///
+/// `launch_app` uses `SW_SHOWNOACTIVATE` so the launched window is
+/// non-foreground from birth. Chromium's `CalculateNativeWinOcclusion`
+/// feature treats that as occluded and suspends the renderer process for
+/// the tab's entire lifetime — UIA tree exposes only browser chrome, no
+/// page DOM, and `PrintWindow` returns a blank body.
+///
+/// `CalculateNativeWinOcclusion` is the root cause; the two
+/// `--disable-backgrounding-*` flags backstop the same effect through the
+/// process-priority and renderer-throttling layers (Chromium suspends
+/// renderers on multiple signals).
+const CHROMIUM_ANTI_THROTTLING_FLAGS: &[&str] = &[
+ "--disable-features=CalculateNativeWinOcclusion",
+ "--disable-backgrounding-occluded-windows",
+ "--disable-renderer-backgrounding",
+];
+
+/// Prepend the Chromium anti-throttling flags to `extra_args` unless the
+/// caller already has each one. Merges into an existing `--disable-features=`
+/// list rather than appending a second `--disable-features=` entry (Chromium
+/// has subtle merging rules across duplicate flags).
+fn inject_chromium_anti_throttling_flags(extra_args: &mut Vec<String>) {
+ const OCCLUSION_FEATURE: &str = "CalculateNativeWinOcclusion";
+
+ // Merge `CalculateNativeWinOcclusion` into any existing
+ // `--disable-features=` entry, or insert a fresh one at the front.
+ let has_occlusion = extra_args.iter().any(|a| {
+ a.strip_prefix("--disable-features=")
+ .map(|v| v.split(',').any(|f| f.trim() == OCCLUSION_FEATURE))
+ .unwrap_or(false)
+ });
+ if !has_occlusion {
+ if let Some(idx) = extra_args
+ .iter()
+ .position(|a| a.starts_with("--disable-features="))
+ {
+ extra_args[idx] = format!("{},{OCCLUSION_FEATURE}", extra_args[idx]);
+ } else {
+ extra_args.insert(0, format!("--disable-features={OCCLUSION_FEATURE}"));
+ }
+ }
+
+ // Boolean toggles — insert at front only when not already present.
+ for flag in &[
+ "--disable-backgrounding-occluded-windows",
+ "--disable-renderer-backgrounding",
+ ] {
+ if !extra_args.iter().any(|a| a == flag) {
+ extra_args.insert(0, (*flag).to_string());
+ }
+ }
+ let _ = CHROMIUM_ANTI_THROTTLING_FLAGS; // referenced for docs alignment
+}
+
+// ── launch_app ───────────────────────────────────────────────────────────────
+
+/// Shape of the resolved launch-target params, used to decide whether the
+/// best-effort foreground-restore guard should fire.
+///
+/// The decision is intentionally a pure function of the params (not of any
+/// HWND state) so it can be unit-tested. See
+/// [`should_restore_foreground_after_launch`].
+#[derive(Clone, Copy, Debug)]
+struct LaunchTargetShape {
+ has_aumid: bool,
+ has_bundle_id: bool,
+ has_name: bool,
+ has_path: bool,
+ has_launch_path: bool,
+ has_urls: bool,
+}
+
+/// True iff the launch is "open an app in the background" (so we want to
+/// restore the user's foreground after the spawn activates the new window)
+/// rather than "open a URL in the default browser" (where the user
+/// explicitly asked for that page to come up).
+///
+/// Rule, matching the audit spec:
+/// - URLs-only call (no app-identifying field set) → SKIP restore. The
+/// user asked for `https://…` to be openable in their default browser;
+/// a browser instance freshly launched for that URL is the legitimate
+/// foreground.
+/// - Any app-identifying field present (`aumid` / `bundle_id` / `name` /
+/// `path` / `launch_path`) → RESTORE. The intent is hidden-launch +
+/// background drive; the no-foreground contract applies.
+fn should_restore_foreground_after_launch(shape: LaunchTargetShape) -> bool {
+ let has_app_identifier = shape.has_aumid
+ || shape.has_bundle_id
+ || shape.has_name
+ || shape.has_path
+ || shape.has_launch_path;
+ // URLs-only carve-out: when the caller passed urls AND no
+ // app-identifying field, skip the restore.
+ if shape.has_urls && !has_app_identifier {
+ return false;
+ }
+ has_app_identifier
+}
+
+/// Bound on one standard-user shell launch helper. It stays inside the 15 s
+/// launch deadline so a helper stuck on a shell dialog is terminated rather
+/// than left behind.
+const STANDARD_USER_SHELL_LAUNCH_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(12);
+
+pub struct LaunchAppTool;
+static LAUNCH_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for LaunchAppTool {
+ fn def(&self) -> &ToolDef {
+ LAUNCH_DEF.get_or_init(|| ToolDef {
+ name: "launch_app".into(),
+ // Description ported from Swift `LaunchAppTool.swift` with
+ // Windows-specific notes. `bundle_id` is accepted as an alias
+ // for `name` on Windows, with one extra behavior: if the
+ // string looks like an AUMID (App User Model ID — contains
+ // `!`, e.g. `Microsoft.WindowsNotepad_8wekyb3d8bbwe!App`) we
+ // route through `IApplicationActivationManager` so packaged
+ // (Microsoft Store / UWP / MSIX) apps return the real
+ // process pid instead of a stub-redirect pid.
+ description: "Launch a Windows app hidden — the driver never brings the target to \
+ the foreground, the target's window is launched with SW_SHOWNOACTIVATE so it \
+ does not steal focus from whatever is currently frontmost.\n\n\
+ Provide either `bundle_id` / `name` / `aumid` (resolved as below) or `path` \
+ (full path to an executable). If both `name` and `path` are given, `path` wins. \
+ `urls` opens each URL in the default browser without activating it.\n\n\
+ Routing order: explicit `aumid` (or a `bundle_id` containing `!`) activates the \
+ packaged app via `IApplicationActivationManager::ActivateApplication`, returning \
+ the real packaged-process pid — required on Win11 for built-in apps like \
+ Notepad, Calculator, and Paint, which now ship as Microsoft Store packages \
+ where the legacy .exe in System32 is a ~7 KB stub that exits immediately. A \
+ plain `name` first attempts a `shell:AppsFolder` lookup; packaged matches use \
+ the activation manager while desktop registrations are launched through their \
+ shell parsing path. A miss falls back to ShellExecuteEx's PATH search.\n\n\
+ Returns the launched app's pid, name, active flag, AND a `windows` array — \
+ same per-window shape `list_windows` returns. When the launch settles but no \
+ window has materialized yet (transient; rare), `windows` comes back empty — \
+ call `list_windows(pid)` explicitly a moment later. `bundle_id` in the \
+ response is set to the AUMID actually used for packaged-app launches and \
+ `null` for ShellExecuteEx launches.\n\n\
+ Windows-only field: `path` (Swift uses `bundle_id` since macOS apps resolve via \
+ LaunchServices; Windows has no LaunchServices, so `path` is the canonical form). \
+ The macOS-specific `webkit_inspector_port`, \
+ `creates_new_application_instance`, and `additional_arguments` fields are \
+ accepted; `additional_arguments` is honored (forwarded as ShellExecuteEx \
+ parameters or as the AUMID activation arguments string). \
+ The remaining macOS-only fields currently no-op on Windows.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "bundle_id":{"type":"string","description":"App User Model ID (AUMID) for a packaged app — pattern `{PackageFamilyName}!{ApplicationId}`, e.g. `Microsoft.WindowsNotepad_8wekyb3d8bbwe!App`. Falls back to a `name` alias if no `!` is present. Either bundle_id, name, aumid, path, or launch_path must be provided."},
+ "aumid":{"type":"string","description":"Explicit AUMID for a packaged app. Cleaner alternative to overloading `bundle_id`. Takes precedence over `bundle_id` / `name` when present."},
+ "name":{"type":"string","description":"App display name. Tried against the `shell:AppsFolder` index first for packaged-app lookup; on a miss, passed to ShellExecuteEx's PATH search."},
+ "path":{"type":"string","description":"Full path to executable. Windows-only; takes precedence over name/bundle_id/aumid (but not over `launch_path`)."},
+ "launch_path":{"type":"string","description":"Round-trip the `launch_path` returned by `list_apps`. Highest precedence — when set, this exact string is handed to ShellExecuteEx unchanged. For Windows desktop apps it's the full `.exe` commandline (path + arguments preserved from the source shortcut); for UWP apps it's `shell:appsFolder\\{PackageFamilyName}!{AppId}`. For precise UWP pid capture, prefer `aumid` over `launch_path`."},
+ "urls":{"type":"array","items":{"type":"string"},"description":"URLs to open in the default browser via ShellExecuteEx (no activation)."},
+ "additional_arguments":{"type":"array","items":{"type":"string"},"description":"Extra command-line arguments passed to the launched process (or activation arguments for packaged apps)."},
+ "webkit_inspector_port":{"type":"integer","description":"Accepted for cross-platform parity; no-op on Windows."},
+ "creates_new_application_instance":{"type":"boolean","description":"Accepted for parity; no-op on Windows (ShellExecuteEx always creates a new process)."},
+ "start_minimized":{"type":"boolean","description":"When true, launch the app's window minimized to the taskbar instead of restored-but-not-activated. Use this when the agent wants to drive the app entirely in the background — the user's previously-frontmost window (e.g. terminal) stays visually on top. Desktop launches hold the foreground lock through startup and use SW_SHOWMINNOACTIVE; packaged-app activation remains broker-controlled and receives a best-effort SW_SHOWMINNOACTIVE post-pass. UIA / background dispatch still work on a minimized window; only `screenshot` and `delivery_mode:\"foreground\"` need it restored."}
+ },"additionalProperties":false}),
+ read_only: false, destructive: false, idempotent: true, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let launch_path_opt = args
+ .get("launch_path")
+ .and_then(|v| v.as_str())
+ .map(str::to_owned);
+ let path_opt = args.get("path").and_then(|v| v.as_str()).map(str::to_owned);
+ let name_opt = args.get("name").and_then(|v| v.as_str()).map(str::to_owned);
+ let bundle_id_opt = args
+ .get("bundle_id")
+ .and_then(|v| v.as_str())
+ .map(str::to_owned);
+ let aumid_opt = args
+ .get("aumid")
+ .and_then(|v| v.as_str())
+ .map(str::to_owned);
+ let urls: Vec<String> = args
+ .get("urls")
+ .and_then(|v| v.as_array())
+ .map(|a| {
+ a.iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect()
+ })
+ .unwrap_or_default();
+ let start_minimized = args
+ .get("start_minimized")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ // `nShow` for the ShellExecuteEx-based launches. SW_SHOWNOACTIVATE
+ // keeps the new window from grabbing focus but still renders it on
+ // top of the previous frontmost window. SW_SHOWMINNOACTIVE both
+ // skips activation AND drops the window to the taskbar, which is
+ // what an agent doing background work wants — the user's terminal
+ // stays visually on top. UIA / PostMessage / set_value all keep
+ // working against a minimized window; only screenshot and
+ // delivery_mode:"foreground" pay a cost.
+ let n_show: i32 = if start_minimized {
+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWMINNOACTIVE.0
+ } else {
+ windows::Win32::UI::WindowsAndMessaging::SW_SHOWNOACTIVATE.0
+ };
+ // Snapshot the set of currently-running pids BEFORE we launch. The
+ // detached start_minimized polling task uses this to detect "any
+ // process that came into existence as a result of our launch", so
+ // it can minimize their windows too — covers the launcher-stub
+ // case (LibreOffice swriter.exe → soffice.bin, GIMP gimp-3.exe →
+ // gimp-3.2.exe, etc.) where the launched pid exits and a child
+ // process with a different name owns the actual window. Captured
+ // here (before the launch) so the new soffice.bin pid won't be
+ // in the snapshot.
+ let pre_launch_pids: std::sync::Arc<std::collections::HashSet<u32>> = if start_minimized {
+ let pids: std::collections::HashSet<u32> = crate::win32::list_processes()
+ .into_iter()
+ .map(|p| p.pid)
+ .collect();
+ std::sync::Arc::new(pids)
+ } else {
+ std::sync::Arc::new(std::collections::HashSet::new())
+ };
+
+ // Capture the foreground window BEFORE any launch path runs so the
+ // post-spawn polling restore (below) has a target HWND to flip back
+ // to. Mirrors the macOS `FocusRestoreGuard` behavior. Best-effort —
+ // see `restore_foreground_polling_best_effort` for the actual
+ // restoration semantics.
+ //
+ // The UWP/AUMID path already restores foreground synchronously
+ // (see `launch_uwp::restore_foreground_best_effort`), so this only
+ // gates the legacy ShellExecuteExW path. We capture early so the
+ // value is consistent regardless of which sub-path runs.
+ //
+ // Stored as `usize` (not `HWND`) so this `invoke` future stays
+ // `Send` — `HWND` wraps `*mut c_void` and would taint the future
+ // for the multi-threaded tokio runtime if held across `.await`.
+ let foreground_before_addr: usize =
+ unsafe { windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize };
+
+ // Pure-function decision on whether to fire the post-launch
+ // restore. Shape mirrors the resolved params — see
+ // `should_restore_foreground_after_launch` for the rule.
+ let launch_shape = LaunchTargetShape {
+ has_aumid: aumid_opt.is_some(),
+ has_bundle_id: bundle_id_opt.is_some(),
+ has_name: name_opt.is_some(),
+ has_path: path_opt.is_some(),
+ has_launch_path: launch_path_opt.is_some(),
+ has_urls: !urls.is_empty(),
+ };
+ let restore_foreground = should_restore_foreground_after_launch(launch_shape);
+ let mut extra_args: Vec<String> = args
+ .get("additional_arguments")
+ .and_then(|v| v.as_array())
+ .map(|a| {
+ a.iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect()
+ })
+ .unwrap_or_default();
+ if args.get("cdp_debugging_port").is_some() {
+ return ToolResult::error(
+ "cdp_debugging_port moved to browser_prepare so DevTools is never enabled on an unproven user profile",
+ );
+ }
+ if launch_path_opt
+ .as_deref()
+ .into_iter()
+ .chain(path_opt.as_deref())
+ .chain(name_opt.as_deref())
+ .chain(extra_args.iter().map(String::as_str))
+ .any(cua_driver_core::launch_guard::contains_remote_debugging_flag)
+ {
+ return ToolResult::error(
+ cua_driver_core::launch_guard::REMOTE_DEBUGGING_LAUNCH_REFUSAL,
+ );
+ }
+
+ // Resolve target — launch_path > path > aumid > name > bundle_id
+ // (alias of name on Windows). launch_path is highest precedence for
+ // round-trip with list_apps; the value is handed to ShellExecuteEx
+ // unchanged (UWP routing below is bypassed when launch_path is set).
+ let target = launch_path_opt
+ .clone()
+ .or(path_opt.clone())
+ .or(aumid_opt.clone())
+ .or(name_opt.clone())
+ .or(bundle_id_opt.clone());
+
+ // #1620: auto-inject Chromium anti-throttling flags when the resolved
+ // target names a Chromium-based browser (Edge / Chrome / Brave /
+ // Vivaldi / Opera / Chromium / Arc / Thorium / Iridium / Yandex).
+ // launch_app hides the window via SW_SHOWNOACTIVATE; Chromium's
+ // occlusion logic suspends the renderer in that case, leaving the
+ // UIA tree empty and PrintWindow capturing a blank body. The three
+ // flags below defeat that for the lifetime of the launched process
+ // without affecting normal interactive use.
+ //
+ // Skipped when `aumid` or `bundle_id` route through the UWP path
+ // (packaged Edge is the exception; the desktop Edge channel is what
+ // ships now and that goes through the ShellExecuteEx path below).
+ if launch_path_opt.is_some() || path_opt.is_some() || name_opt.is_some() {
+ if let Some(t) = target.as_deref() {
+ if is_chromium_browser_target(t) {
+ inject_chromium_anti_throttling_flags(&mut extra_args);
+ }
+ }
+ }
+
+ if target.is_none() && urls.is_empty() {
+ // Error message lists every field the resolver actually accepts.
+ // The Swift-original "bundle_id or name" message predated the Windows
+ // additions (aumid, path, launch_path, urls) and made #1635 look like
+ // an aumid-specific bug. The actual cause of #1635 was the upstream
+ // PS argv quote-stripping bug fixed in #1637; this message just stops
+ // misleading anyone who hits the error for unrelated reasons.
+ return ToolResult::error(
+ "Provide one of: bundle_id, name, aumid, path, launch_path, or urls to identify the app to launch.",
+ );
+ }
+
+ // ── Packaged-app (UWP / MSIX) routing decision ──────────────────────
+ //
+ // Routing precedence — most explicit signal wins:
+ // 1. `launch_path` set: skip UWP routing entirely — round-trip
+ // semantics demand the value be handed to ShellExecuteEx
+ // unchanged. Callers wanting precise UWP pid capture should
+ // pass `aumid` instead.
+ // 2. `aumid` parameter (any value): packaged path, AUMID = value.
+ // 3. `bundle_id` containing `!`: packaged path, AUMID = value
+ // (Win32 PATH lookups never produce a `!` in the executable
+ // name, so this is a safe marker of caller intent).
+ // 4. `name` with no `path`: try `shell:AppsFolder` lookup. A
+ // packaged hit uses the resolved AUMID; a desktop registration
+ // uses `shell:AppsFolder\<AppUserModelID>` via ShellExecuteExW.
+ // On miss, fall through to the existing Win32 path.
+ // 5. Anything else: existing ShellExecuteExW path.
+ //
+ // `path` is intentionally excluded from packaged routing — when
+ // the caller has a concrete executable in mind they want
+ // CreateProcess semantics, not Start Menu activation.
+ let mut name_lookup_missed = false;
+ let mut shell_target_from_name: Option<String> = None;
+ let aumid_for_uwp: Option<String> = if launch_path_opt.is_some() || path_opt.is_some() {
+ None
+ } else if let Some(a) = aumid_opt.clone() {
+ Some(a)
+ } else if let Some(b) = bundle_id_opt
+ .clone()
+ .filter(|s| crate::launch_uwp::is_aumid(s))
+ {
+ Some(b)
+ } else if let Some(n) = name_opt.clone() {
+ // The cold AppsFolder COM walk can stop making progress when its
+ // interactive shell broker is unhealthy. Bound it before the
+ // separate ShellExecuteEx deadline below; the resolver keeps a
+ // timed-out worker single-flight until COM really returns.
+ match crate::launch_uwp::resolve_apps_folder_target_by_name_bounded(n.clone()).await {
+ Ok(Some(crate::launch_uwp::AppsFolderLaunchTarget::PackagedAumid(aumid))) => {
+ Some(aumid)
+ }
+ Ok(Some(crate::launch_uwp::AppsFolderLaunchTarget::ShellLaunchPath(
+ launch_path,
+ ))) => {
+ shell_target_from_name = Some(launch_path);
+ None
+ }
+ Ok(None) => {
+ name_lookup_missed = true;
+ None
+ }
+ Err(crate::launch_uwp::AppsFolderLookupError::Timeout) => {
+ return ToolResult::error(format!(
+ "App name lookup for {n:?} is temporarily unavailable: Windows did not respond to the shell:AppsFolder query within 4s. No app was launched; retry after the shell recovers, or pass an explicit path or aumid."
+ ))
+ }
+ Err(crate::launch_uwp::AppsFolderLookupError::Busy) => {
+ return ToolResult::error(format!(
+ "App name lookup for {n:?} is temporarily unavailable because a prior Windows shell lookup is still running or cooling down. No app was launched; retry later, or pass an explicit path or aumid."
+ ))
+ }
+ Err(crate::launch_uwp::AppsFolderLookupError::Unavailable) => {
+ return ToolResult::error(format!(
+ "App name lookup for {n:?} is unavailable because the Windows shell lookup worker failed. No app was launched; pass an explicit path or aumid."
+ ))
+ }
+ }
+ } else {
+ None
+ };
+ let resolved_target = shell_target_from_name.or(target);
+
+ // Single launch path. `display` is the human-readable identifier
+ // we report in the success header; for the packaged path it's
+ // the AUMID, which is more informative than the display name.
+ let display = aumid_for_uwp
+ .clone()
+ .or_else(|| resolved_target.clone())
+ .unwrap_or_else(|| urls.join(", "));
+
+ // When the resolved target came from `launch_path` it may carry
+ // shortcut arguments (e.g. `"C:\Path\chrome.exe" --profile-directory="Profile 2"`).
+ // Split it into a file token + arguments tail so ShellExecuteEx
+ // sees them separately — passing the whole string as lpFile would
+ // fail because the shell only resolves a single filesystem path
+ // there. No-op for AUMID tokens / plain paths without args / UWP
+ // `shell:appsFolder\..!..` tokens.
+ let (target_file_opt, extra_joined) = {
+ let base_extra = extra_args.join(" ");
+ if launch_path_opt.is_some() {
+ if let Some(t) = resolved_target.as_deref() {
+ let (file, args_tail) = split_launchable_target(t);
+ let combined = if args_tail.is_empty() {
+ base_extra
+ } else if base_extra.is_empty() {
+ args_tail
+ } else {
+ format!("{args_tail} {base_extra}")
+ };
+ let file_opt = if file.is_empty() { None } else { Some(file) };
+ (file_opt, combined)
+ } else {
+ (None, base_extra)
+ }
+ } else {
+ (resolved_target.clone(), base_extra)
+ }
+ };
+
+ // Strict no-activation is available only for the desktop launch path.
+ // UWP activation is broker-controlled and retains its existing
+ // restore-after-activation behavior.
+ let mut foreground_lock = if start_minimized && aumid_for_uwp.is_none() {
+ Some(crate::input::ForegroundLockGuard::acquire())
+ } else {
+ None
+ };
+ if foreground_lock
+ .as_ref()
+ .is_some_and(|guard| !guard.acquired())
+ {
+ return ToolResult::error(
+ "Background minimized launch is unavailable because Windows did not grant the \
+ foreground lock required to prevent the new process from activating. No process \
+ was started. Launch without start_minimized only when a foreground change is \
+ acceptable.",
+ )
+ .with_structured(json!({
+ "code": "background_unavailable",
+ "delivery_mode": "background",
+ "event_kind": "app_launch",
+ }));
+ }
+
+ // Branch: AUMID activation if we resolved one; else legacy
+ // ShellExecuteExW. Both branches still need to handle `urls`
+ // (additional URLs always go through ShellExecuteExW since the
+ // Start Menu doesn't activate by URL).
+ let pid = if let Some(aumid) = aumid_for_uwp.clone() {
+ let aumid_clone = aumid.clone();
+ let args_clone = extra_joined.clone();
+ let activation = tokio::task::spawn_blocking(move || {
+ crate::launch_uwp::launch_uwp(&aumid_clone, &args_clone)
+ })
+ .await;
+ match activation {
+ Ok(Ok(p)) => p,
+ Ok(Err(e)) => {
+ return ToolResult::error(format!(
+ "Failed to activate packaged app {aumid:?}: {e}"
+ ))
+ }
+ Err(e) => return ToolResult::error(format!("Task error: {e}")),
+ }
+ } else {
+ // Legacy ShellExecuteExW path for plain Win32 apps and for
+ // callers passing an explicit `path` / `launch_path`. An elevated
+ // Driver runs these launches with its standard-user token so the
+ // app does not inherit administrator rights (#3607).
+ let launch_token = match crate::standard_user_launch::app_launch_token() {
+ Ok(token) => token,
+ Err(message) => return ToolResult::error(message),
+ };
+ let urls_clone = urls.clone();
+ let target_for_shell = target_file_opt.clone();
+ let extra_for_shell = extra_joined.clone();
+ let n_show_for_shell = n_show;
+ let direct_minimized_exe = start_minimized
+ && target_for_shell.as_deref().is_some_and(|target| {
+ std::path::Path::new(target).is_file()
+ && std::path::Path::new(target)
+ .extension()
+ .is_some_and(|extension| extension.eq_ignore_ascii_case("exe"))
+ });
+ // Bound the shell launch with a timeout. An unregistered protocol
+ // or file association makes `ShellExecuteExW` block on a modal shell
+ // dialog ("you'll need a new app to open this …") on the *session*
+ // desktop — which a Session-0/headless daemon can neither see nor
+ // dismiss — wedging the launch (and that request) indefinitely.
+ // `SEE_MASK_FLAG_NO_UI` (below) suppresses the no-association error
+ // UI so that case fails fast with an error code; the timeout is the
+ // backstop for any *other* blocking broker dialog (SmartScreen, an
+ // elevation/consent surface) so a bad target can't hang the daemon.
+ let launch = tokio::task::spawn_blocking(move || -> anyhow::Result<u32> {
+ use windows::core::{PCWSTR, PWSTR};
+ use windows::Win32::Foundation::CloseHandle;
+ use windows::Win32::System::Threading::{
+ CreateProcessW, GetProcessId, PROCESS_CREATION_FLAGS, PROCESS_INFORMATION,
+ STARTF_USESHOWWINDOW, STARTUPINFOW,
+ };
+ use windows::Win32::UI::Shell::{
+ ShellExecuteExW, SEE_MASK_FLAG_NO_UI, SEE_MASK_NOCLOSEPROCESS,
+ SHELLEXECUTEINFOW,
+ };
+
+ fn to_wide(s: &str) -> Vec<u16> {
+ s.encode_utf16().chain(std::iter::once(0)).collect()
+ }
+ let op_w = to_wide("open");
+ let file_w = to_wide(if let Some(t) = target_for_shell.as_deref() {
+ t
+ } else {
+ urls_clone.first().map(|s| s.as_str()).unwrap_or("")
+ });
+ let args_w = to_wide(&extra_for_shell);
+
+ use crate::standard_user_launch::AppLaunchToken;
+ let pid = if direct_minimized_exe {
+ let target = target_for_shell
+ .as_deref()
+ .expect("checked executable path");
+ let command_line_text = if extra_for_shell.is_empty() {
+ format!(r#""{target}""#)
+ } else {
+ format!(r#""{target}" {extra_for_shell}"#)
+ };
+ if let AppLaunchToken::StandardUser(token) = &launch_token {
+ crate::standard_user_launch::create_process(
+ token,
+ target,
+ &command_line_text,
+ n_show_for_shell,
+ )
+ .map_err(anyhow::Error::msg)?
+ } else {
+ let mut command_line = to_wide(&command_line_text);
+ let startup = STARTUPINFOW {
+ cb: std::mem::size_of::<STARTUPINFOW>() as u32,
+ dwFlags: STARTF_USESHOWWINDOW,
+ wShowWindow: n_show_for_shell as u16,
+ ..Default::default()
+ };
+ let mut process = PROCESS_INFORMATION::default();
+ unsafe {
+ CreateProcessW(
+ PCWSTR(file_w.as_ptr()),
+ PWSTR(command_line.as_mut_ptr()),
+ None,
+ None,
+ false,
+ PROCESS_CREATION_FLAGS(0),
+ None,
+ PCWSTR::null(),
+ &startup,
+ &mut process,
+ )?;
+ let _ = CloseHandle(process.hThread);
+ let _ = CloseHandle(process.hProcess);
+ }
+ process.dwProcessId
+ }
+ } else if let AppLaunchToken::StandardUser(token) = &launch_token {
+ let file = target_for_shell
+ .as_deref()
+ .or_else(|| urls_clone.first().map(String::as_str))
+ .unwrap_or("");
+ crate::standard_user_launch::shell_execute(
+ token,
+ file,
+ &extra_for_shell,
+ n_show_for_shell,
+ STANDARD_USER_SHELL_LAUNCH_TIMEOUT,
+ )
+ .map_err(anyhow::Error::msg)?
+ } else {
+ let mut info = SHELLEXECUTEINFOW {
+ cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
+ fMask: SEE_MASK_NOCLOSEPROCESS | SEE_MASK_FLAG_NO_UI,
+ lpVerb: PCWSTR(op_w.as_ptr()),
+ lpFile: PCWSTR(file_w.as_ptr()),
+ lpParameters: if extra_for_shell.is_empty() {
+ PCWSTR::null()
+ } else {
+ PCWSTR(args_w.as_ptr())
+ },
+ nShow: n_show_for_shell,
+ ..Default::default()
+ };
+ unsafe {
+ ShellExecuteExW(&mut info)?;
+ }
+ if !info.hProcess.is_invalid() {
+ let p = unsafe { GetProcessId(info.hProcess) };
+ unsafe {
+ let _ = CloseHandle(info.hProcess);
+ }
+ p
+ } else {
+ 0
+ }
+ };
+
+ // Open any additional URLs in the default browser (no focus
+ // steal, no pid capture for these — Swift's NSWorkspace flow
+ // behaves the same way for secondary URLs).
+ // When an app target was launched above, none of the URLs has
+ // been consumed yet. URLs-only launches use the first URL as
+ // `lpFile`, so only their remaining URLs belong here.
+ let first_unopened_url = first_unopened_shell_url_index(target_for_shell.is_some());
+ for url in &urls_clone[first_unopened_url.min(urls_clone.len())..] {
+ if let AppLaunchToken::StandardUser(token) = &launch_token {
+ let _ = crate::standard_user_launch::shell_execute(
+ token,
+ url,
+ "",
+ n_show_for_shell,
+ STANDARD_USER_SHELL_LAUNCH_TIMEOUT,
+ );
+ continue;
+ }
+ let file = to_wide(url);
+ let mut url_info = SHELLEXECUTEINFOW {
+ cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
+ fMask: SEE_MASK_FLAG_NO_UI,
+ lpVerb: PCWSTR(op_w.as_ptr()),
+ lpFile: PCWSTR(file.as_ptr()),
+ nShow: n_show_for_shell,
+ ..Default::default()
+ };
+ unsafe {
+ let _ = ShellExecuteExW(&mut url_info);
+ }
+ }
+
+ Ok(pid)
+ });
+ // 15s is generous: ShellExecuteExW returns once activation starts
+ // (process created), not when the window appears, so a healthy
+ // launch resolves in well under a second. On timeout we abandon the
+ // blocking task (a spawn_blocking thread can't be cancelled — it
+ // unblocks if/when the modal is dismissed) and return an error so
+ // the daemon stays responsive instead of wedging on the request.
+ let result = tokio::time::timeout(std::time::Duration::from_secs(15), launch).await;
+
+ match result {
+ Ok(Ok(Ok(p))) => p,
+ Ok(Ok(Err(e))) if name_lookup_missed => {
+ return ToolResult::error(format!(
+ "App {:?} was not found in shell:AppsFolder or by Windows PATH/association lookup: {e}",
+ name_opt.as_deref().unwrap_or("")
+ ))
+ }
+ Ok(Ok(Err(e))) => return ToolResult::error(format!("Failed to launch: {e}")),
+ Ok(Err(e)) => return ToolResult::error(format!("Task error: {e}")),
+ Err(_elapsed) => {
+ return ToolResult::error(format!(
+ "Launch of {:?} timed out after 15s — the target likely has no \
+ registered handler and a blocking shell dialog appeared on the \
+ session desktop; aborted to keep the daemon responsive.",
+ target_file_opt
+ .as_deref()
+ .or_else(|| urls.first().map(|s| s.as_str()))
+ .unwrap_or("")
+ ))
+ }
+ }
+ };
+
+ // Best-effort foreground-restore for the legacy ShellExecuteExW
+ // path. The UWP/AUMID branch already restores foreground
+ // synchronously inside `launch_uwp::launch_uwp`, so we only spawn
+ // the polling task when we went through the ShellExecuteExW
+ // branch. URLs-only invocations are excluded via `restore_foreground`
+ // (see `should_restore_foreground_after_launch`).
+ //
+ // `HWND` wraps a raw `*mut c_void` which is `!Send`, so we marshal
+ // the foreground handle across the spawn boundary as a `usize` and
+ // reconstruct the `HWND` only inside the synchronous probes (see
+ // `restore_foreground_polling_best_effort`). Mirrors how
+ // `launch_uwp.rs` carries its restore target.
+ if aumid_for_uwp.is_none() && restore_foreground {
+ let spawned_pid_for_restore = pid;
+ let target_foreground_addr = foreground_before_addr;
+ tokio::spawn(async move {
+ restore_foreground_polling_best_effort(
+ target_foreground_addr,
+ spawned_pid_for_restore,
+ )
+ .await;
+ });
+ }
+
+ // When the packaged path was taken we still need to fan out any
+ // extra URLs to the default browser (ShellExecuteEx — no pid
+ // capture, mirroring Swift's NSWorkspace secondary-URL flow).
+ if aumid_for_uwp.is_some() && !urls.is_empty() {
+ let urls_clone = urls.clone();
+ let n_show_for_urls = n_show;
+ let _ = tokio::task::spawn_blocking(move || {
+ use windows::core::PCWSTR;
+ use windows::Win32::UI::Shell::{ShellExecuteExW, SHELLEXECUTEINFOW};
+ fn to_wide(s: &str) -> Vec<u16> {
+ s.encode_utf16().chain(std::iter::once(0)).collect()
+ }
+ let op_w = to_wide("open");
+ for url in &urls_clone {
+ let file = to_wide(url);
+ let mut url_info = SHELLEXECUTEINFOW {
+ cbSize: std::mem::size_of::<SHELLEXECUTEINFOW>() as u32,
+ lpVerb: PCWSTR(op_w.as_ptr()),
+ lpFile: PCWSTR(file.as_ptr()),
+ nShow: n_show_for_urls,
+ ..Default::default()
+ };
+ unsafe {
+ let _ = ShellExecuteExW(&mut url_info);
+ }
+ }
+ })
+ .await;
+ }
+
+ // Resolve the pid's windows so the caller can skip a list_windows
+ // round-trip — same approach as Swift's `resolveWindows`. Retry
+ // 5×200ms; Win32 window registration can lag the launch.
+ //
+ // Launcher-stub fallback: when the launched binary is a wrapper that
+ // re-execs and exits (GIMP's `gimp-3.exe` → `gimp-3.2.exe`; LO's
+ // `swriter.exe` → `soffice.bin`), the launched pid never gets a
+ // window — list_windows(Some(pid)) stays empty forever. After the
+ // primary retry budget we fall back to scanning the launched pid's
+ // descendant + name-related processes and pick the first one with a
+ // window. The resolved pid is reflected in the response's `pid` field
+ // so callers can target it with subsequent calls. See #1615.
+ let mut windows_json: Vec<serde_json::Value> = Vec::new();
+ let mut resolved_pid: u32 = pid;
+
+ // UWP / packaged-app host-window resolution. `launch_uwp` returns the
+ // real packaged-app pid (e.g. CalculatorApp.exe), but a UWP app's
+ // top-level window — the HWND a caller must actually drive — is owned
+ // by `ApplicationFrameHost.exe`, not by that process. The app process
+ // only owns a `CoreWindow` reparented as a child of the AFH frame, so
+ // `list_windows(Some(app_pid))` is empty and the generic retry loop
+ // below would either spin out to a stub-fallback miss or return no
+ // windows at all (the original "launch_app of a UWP app returns empty
+ // windows[] / a pid you can't drive" bug). Map app_pid → its AFH host
+ // frame and report `(frame_hwnd, afh_pid)` so the returned handles
+ // resolve in `get_window_state` and element actions land. Retry: the
+ // frame can lag the activation by a few hundred ms.
+ if aumid_for_uwp.is_some() {
+ for _ in 0..10 {
+ let host =
+ tokio::task::spawn_blocking(move || crate::win32::resolve_uwp_host_window(pid))
+ .await
+ .unwrap_or(None);
+ if let Some(w) = host {
+ windows_json = vec![json!({
+ "window_id": w.hwnd, "title": w.title,
+ "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
+ "layer": 0,
+ "z_index": 0,
+ "is_on_screen": true,
+ })];
+ // Report the AFH pid: that's the pid that owns the frame
+ // HWND, so `get_window_state(resolved_pid, window_id)`
+ // validates and `list_windows(Some(resolved_pid))` lists it.
+ resolved_pid = w.pid;
+ break;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;
+ }
+ }
+
+ for _ in 0..5 {
+ if !windows_json.is_empty() {
+ break;
+ }
+ let wins =
+ tokio::task::spawn_blocking(move || crate::win32::list_windows_win32_first(pid))
+ .await
+ .unwrap_or_default();
+ if !wins.is_empty() {
+ let window_count = wins.len();
+ windows_json = wins.iter().enumerate().map(|(position, w)| json!({
+ "window_id": w.hwnd, "title": w.title,
+ "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
+ "layer": 0,
+ "z_index": z_index_from_front_to_back(window_count, position),
+ "is_on_screen": w.is_on_screen,
+ })).collect();
+ break;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;
+ }
+ if windows_json.is_empty() {
+ // Launcher-stub fallback. Compute the exe basename from the
+ // launchable target so name-based matching has something to work
+ // with (e.g. "gimp-3.exe" → prefix "gimp" matches "gimp-3.2.exe").
+ let basename_for_match = target_file_opt
+ .as_deref()
+ .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())
+ .unwrap_or("")
+ .to_owned();
+ // Known-slow launchers get an extended retry budget. GIMP 3.x in
+ // particular spends 10-20s on its first launch (font cache rebuild,
+ // plugin scan, etc.) before the main window registers. We don't
+ // want to wait 20s for every launcher — gate on basename prefix
+ // matching known-slow apps. Add to this list as encountered.
+ let bn_lower = basename_for_match.to_ascii_lowercase();
+ let is_slow_launcher = bn_lower.starts_with("gimp")
+ || bn_lower.starts_with("blender") // OpenGL init can stall
+ || bn_lower.starts_with("inkscape") // similar GTK pattern
+ || bn_lower.starts_with("krita")
+ || bn_lower.starts_with("freecad");
+ let max_candidate_attempts: usize = if is_slow_launcher { 30 } else { 3 };
+
+ let basename_clone = basename_for_match.clone();
+ let candidates_initial = tokio::task::spawn_blocking(move || {
+ crate::win32::related_processes(pid, &basename_clone)
+ })
+ .await
+ .unwrap_or_default();
+
+ // For slow launchers we may also need to RE-SCAN candidates over
+ // time, because the wrapper may not have spawned its child yet
+ // when we first scanned. Cap total wait at ~12s (slow) / 0.6s (fast).
+ let mut tried: std::collections::HashSet<u32> = std::collections::HashSet::new();
+ tried.insert(pid); // already tried in the primary loop
+ let mut candidate_queue: Vec<u32> = candidates_initial
+ .into_iter()
+ .filter(|p| tried.insert(*p))
+ .collect();
+ let mut total_attempts: usize = 0;
+ 'outer: loop {
+ while let Some(candidate_pid) = candidate_queue.pop() {
+ for _ in 0..max_candidate_attempts {
+ total_attempts += 1;
+ let wins = tokio::task::spawn_blocking(move || {
+ crate::win32::list_windows_win32_first(candidate_pid)
+ })
+ .await
+ .unwrap_or_default();
+ if !wins.is_empty() {
+ let window_count = wins.len();
+ windows_json = wins.iter().enumerate().map(|(position, w)| json!({
+ "window_id": w.hwnd, "title": w.title,
+ "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
+ "layer": 0,
+ "z_index": z_index_from_front_to_back(window_count, position),
+ "is_on_screen": w.is_on_screen,
+ })).collect();
+ resolved_pid = candidate_pid;
+ break 'outer;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;
+ }
+ }
+ // For slow launchers, keep re-scanning descendants — the
+ // wrapper may not have spawned its child yet. Cap total
+ // wait at ~12s (60 × 200ms) for the slow path.
+ if !is_slow_launcher || total_attempts > 60 {
+ break;
+ }
+ // Give the wrapper a moment to spawn before re-scanning.
+ tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+ total_attempts += 3; // count the 500ms wait as 3 attempts
+ let basename_rescan = basename_for_match.clone();
+ let fresh = tokio::task::spawn_blocking(move || {
+ crate::win32::related_processes(pid, &basename_rescan)
+ })
+ .await
+ .unwrap_or_default();
+ let new_ones: Vec<u32> = fresh.into_iter().filter(|p| tried.insert(*p)).collect();
+ candidate_queue = new_ones;
+ // Continue the outer loop regardless — even with empty
+ // new_ones we want another iteration that hits the
+ // total_attempts cap. The loop body handles the empty queue
+ // by falling through to the re-scan again.
+ }
+ }
+ let pid = resolved_pid;
+
+ // start_minimized post-pass: drop every resolved window to the
+ // taskbar. The desktop ShellExecuteEx path already minimizes via
+ // nShow=SW_SHOWMINNOACTIVE, but three cases still need this
+ // fallback:
+ // 1. AUMID / UWP activations come up un-minimized —
+ // ActivateApplication has no nShow.
+ // 2. Apps that ignore the nShow hint and decide their own geometry
+ // (notable offenders: LibreOffice, some Electron apps).
+ // 3. Launcher-stub flows (LO swriter.exe → soffice.bin, GIMP, etc.)
+ // where the launched pid exits before the real window appears,
+ // so `windows_json` is empty at this point.
+ //
+ // To cover (3) we spawn a background polling task that keeps
+ // looking for windows under the launched pid family for up to 5 s
+ // and minimizes anything that materializes. The task runs detached
+ // so the launch_app response isn't held up by it.
+ //
+ // SW_SHOWMINNOACTIVE preserves the foreground while minimizing. Using
+ // SW_MINIMIZE here would itself activate the next z-order window and
+ // force a visible restore-after-steal cycle.
+ if start_minimized {
+ // First, minimize anything already resolved (covers the common
+ // single-process path where windows_json was populated).
+ let immediate_hwnds: Vec<u64> = windows_json
+ .iter()
+ .filter_map(|w| w["window_id"].as_u64())
+ .collect();
+ // Capture pid + basename for the post-launch polling task to
+ // pick up launcher-stub child processes.
+ let stub_basename = target_file_opt
+ .as_deref()
+ .and_then(|t| t.rsplit(|c: char| c == '\\' || c == '/').next())
+ .unwrap_or("")
+ .to_owned();
+ // parent_pid is the launched pid family root — used below to
+ // constrain the post-launch minimize sweep to descendants /
+ // name-relatives of the launched app, so we don't accidentally
+ // minimize a user's unrelated app that started during the 5 s
+ // poll window.
+ let parent_pid = pid;
+ let immediate_hwnds_for_poll = immediate_hwnds.clone();
+ let _ = tokio::task::spawn_blocking(move || {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_SHOWMINNOACTIVE};
+ for h in immediate_hwnds {
+ unsafe {
+ let _ = ShowWindow(HWND(h as *mut _), SW_SHOWMINNOACTIVE);
+ }
+ }
+ })
+ .await;
+ // Poll launcher-stub late-window cases before returning. The
+ // start_minimized contract is observable at response time; a
+ // detached task allowed callers to see a transient restored
+ // window immediately after a successful launch.
+ // Strategy: every 200 ms for 5 s, find pids that
+ // (a) weren't in the pre-launch snapshot, AND
+ // (b) are part of the launched app's family — either a
+ // descendant of the launched root pid, OR a process
+ // sharing the launched binary's name prefix (covers
+ // renamed-binary chains like swriter.exe → soffice.bin
+ // where the child's name doesn't match but it's a
+ // direct descendant).
+ // Minimize THEIR visible windows only. Without the family
+ // constraint, anything the user opens during the 5 s window
+ // (Chrome, terminal, IDE) would also get minimized — that's
+ // the regression CodeRabbit flagged.
+ //
+ // Loop ends early once we've minimized the first set of
+ // windows AND they remain minimized for three ticks — the typical
+ // app has its main window up within ~2 s of launch.
+ let pre_pids = pre_launch_pids.clone();
+ let basename_for_poll = stub_basename.clone();
+ let launch_foreground_lock = foreground_lock.take();
+ (async move {
+ use std::collections::HashSet;
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ IsIconic, ShowWindow, SW_SHOWMINNOACTIVE,
+ };
+ let _foreground_lock = launch_foreground_lock;
+ let mut minimized: HashSet<u64> = immediate_hwnds_for_poll.into_iter().collect();
+ let mut idle_ticks_after_any_hit: u8 = 0;
+ let mut hit_count_total = minimized.len();
+ for _ in 0..25 {
+ let pre_pids_clone = pre_pids.clone();
+ let basename_clone = basename_for_poll.clone();
+ // Build the family pid set fresh each tick — both
+ // descendant graph and name-relatives can grow as
+ // launcher-stub chains spawn deeper children.
+ let family_new_pids: Vec<u32> = tokio::task::spawn_blocking(move || {
+ let related: std::collections::HashSet<u32> =
+ crate::win32::related_processes(parent_pid, &basename_clone)
+ .into_iter()
+ .chain(std::iter::once(parent_pid))
+ .collect();
+ crate::win32::list_processes()
+ .into_iter()
+ .filter(|p| {
+ !pre_pids_clone.contains(&p.pid) && related.contains(&p.pid)
+ })
+ .map(|p| p.pid)
+ .collect()
+ })
+ .await
+ .unwrap_or_default();
+ let mut tick_hits: usize = 0;
+ for cpid in family_new_pids {
+ let wins = tokio::task::spawn_blocking(move || {
+ crate::win32::list_windows(Some(cpid))
+ })
+ .await
+ .unwrap_or_default();
+ for w in wins {
+ let is_new = minimized.insert(w.hwnd);
+ if is_new {
+ hit_count_total += 1;
+ }
+ let hwnd_iso = w.hwnd as usize;
+ let restored = tokio::task::spawn_blocking(move || unsafe {
+ let hwnd = HWND(hwnd_iso as *mut _);
+ if IsIconic(hwnd).as_bool() {
+ false
+ } else {
+ let _ = ShowWindow(hwnd, SW_SHOWMINNOACTIVE);
+ true
+ }
+ })
+ .await
+ .unwrap_or(false);
+ if is_new || restored {
+ tick_hits += 1;
+ }
+ }
+ }
+ if tick_hits == 0 {
+ idle_ticks_after_any_hit += 1;
+ if hit_count_total > 0 && idle_ticks_after_any_hit >= 3 {
+ // Stable: known windows remained minimized and no
+ // new window appeared for 600 ms.
+ break;
+ }
+ } else {
+ idle_ticks_after_any_hit = 0;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;
+ }
+ })
+ .await;
+ }
+
+ // Match Swift text format 1:1.
+ let mut summary = format!("✅ Launched {display} (pid {pid}) in background.");
+ if !windows_json.is_empty() {
+ summary.push_str("\n\nWindows:");
+ for w in &windows_json {
+ let title = w["title"].as_str().unwrap_or("");
+ let title_disp = if title.is_empty() {
+ "(no title)".to_owned()
+ } else {
+ format!("\"{title}\"")
+ };
+ let wid = w["window_id"].as_u64().unwrap_or(0);
+ summary.push_str(&format!("\n- {title_disp} [window_id: {wid}]"));
+ }
+ summary.push_str(&format!(
+ "\n→ Call get_window_state(pid: {pid}, window_id) to inspect."
+ ));
+ }
+ // `bundle_id` is the AUMID when we went through the packaged-app
+ // path (so the caller can round-trip the same value to relaunch),
+ // and `null` for plain Win32 launches (which truly have no bundle
+ // identifier concept on Windows).
+ let bundle_id_response = match aumid_for_uwp.as_ref() {
+ Some(a) => serde_json::Value::String(a.clone()),
+ None => serde_json::Value::Null,
+ };
+ let structured = json!({
+ "pid": pid,
+ "bundle_id": bundle_id_response,
+ "name": display,
+ "running": true,
+ "active": false, // SW_SHOWNOACTIVATE — Swift's background-launch invariant
+ "windows": windows_json,
+ });
+ ToolResult::text(summary).with_structured(structured)
+ }
+}
+
+#[cfg(test)]
+mod launch_focus_restore_decision_tests {
+ use super::{
+ first_unopened_shell_url_index, should_restore_foreground_after_launch, LaunchTargetShape,
+ };
+
+ #[test]
+ fn only_app_identifying_launches_restore_foreground() {
+ for (fields, restore) in [
+ // A urls-only launch explicitly asks for navigation in the default
+ // browser; that browser is the legitimate foreground.
+ (&["urls"][..], false),
+ (&["name"], true),
+ (&["path"], true),
+ // The AUMID path restores synchronously in launch_uwp.rs; the
+ // caller gates the polling restore on `aumid_for_uwp.is_none()`.
+ (&["aumid"], true),
+ (&["bundle_id"], true),
+ (&["launch_path"], true),
+ // An app-identifying field alongside urls opens them in that app
+ // in the background, so the prior foreground is restored.
+ (&["name", "urls"], true),
+ (&["path", "urls"], true),
+ (&["aumid", "urls"], true),
+ // Nothing was launched.
+ (&[], false),
+ ] {
+ let shape = LaunchTargetShape {
+ has_aumid: fields.contains(&"aumid"),
+ has_bundle_id: fields.contains(&"bundle_id"),
+ has_name: fields.contains(&"name"),
+ has_path: fields.contains(&"path"),
+ has_launch_path: fields.contains(&"launch_path"),
+ has_urls: fields.contains(&"urls"),
+ };
+ assert_eq!(
+ should_restore_foreground_after_launch(shape),
+ restore,
+ "{fields:?}"
+ );
+ }
+ }
+
+ #[test]
+ fn named_app_launch_forwards_every_url() {
+ assert_eq!(first_unopened_shell_url_index(true), 0);
+ }
+
+ #[test]
+ fn urls_only_launch_does_not_reopen_primary_url() {
+ assert_eq!(first_unopened_shell_url_index(false), 1);
+ }
+}
+
+#[cfg(test)]
+mod chromium_flag_injection_tests {
+ use super::{inject_chromium_anti_throttling_flags, is_chromium_browser_target};
+
+ #[test]
+ fn detects_bare_browser_names() {
+ for name in [
+ "msedge", "chrome", "brave", "opera", "vivaldi", "chromium", "thorium", "iridium",
+ "browser", "arc",
+ ] {
+ assert!(is_chromium_browser_target(name), "{name} should match");
+ assert!(
+ is_chromium_browser_target(&format!("{name}.exe")),
+ "{name}.exe should match"
+ );
+ // Case-insensitive.
+ assert!(
+ is_chromium_browser_target(&name.to_uppercase()),
+ "uppercase {name} should match"
+ );
+ }
+ }
+
+ #[test]
+ fn detects_full_paths() {
+ assert!(is_chromium_browser_target(
+ r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"
+ ));
+ assert!(is_chromium_browser_target(
+ r"C:\Program Files\Google\Chrome\Application\chrome.exe"
+ ));
+ // Forward slashes too (some shells write paths that way).
+ assert!(is_chromium_browser_target(
+ r"C:/Program Files/Google/Chrome/Application/chrome.exe"
+ ));
+ }
+
+ #[test]
+ fn detects_launch_path_with_trailing_args() {
+ // Round-tripped launch_path from list_apps with shortcut arguments.
+ assert!(is_chromium_browser_target(
+ r#""C:\Program Files\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 2""#
+ ));
+ }
+
+ #[test]
+ fn does_not_match_non_chromium_apps() {
+ for name in ["firefox", "notepad", "explorer", "code", "soffice"] {
+ assert!(!is_chromium_browser_target(name), "{name} should NOT match");
+ assert!(
+ !is_chromium_browser_target(&format!("{name}.exe")),
+ "{name}.exe should NOT match"
+ );
+ }
+ // Empty target.
+ assert!(!is_chromium_browser_target(""));
+ }
+
+ #[test]
+ fn injects_three_flags_into_empty_args() {
+ let mut args: Vec<String> = vec![];
+ inject_chromium_anti_throttling_flags(&mut args);
+ assert!(args.contains(&"--disable-features=CalculateNativeWinOcclusion".to_string()));
+ assert!(args.contains(&"--disable-backgrounding-occluded-windows".to_string()));
+ assert!(args.contains(&"--disable-renderer-backgrounding".to_string()));
+ assert_eq!(args.len(), 3);
+ }
+
+ #[test]
+ fn merges_into_existing_disable_features_list() {
+ let mut args = vec!["--disable-features=Foo,Bar".to_string()];
+ inject_chromium_anti_throttling_flags(&mut args);
+ // Should NOT have two --disable-features= entries.
+ let dfe: Vec<_> = args
+ .iter()
+ .filter(|a| a.starts_with("--disable-features="))
+ .collect();
+ assert_eq!(dfe.len(), 1);
+ assert!(dfe[0].contains("CalculateNativeWinOcclusion"));
+ assert!(dfe[0].contains("Foo"));
+ assert!(dfe[0].contains("Bar"));
+ }
+
+ #[test]
+ fn idempotent_when_all_flags_already_present() {
+ let mut args = vec![
+ "--disable-features=CalculateNativeWinOcclusion".to_string(),
+ "--disable-backgrounding-occluded-windows".to_string(),
+ "--disable-renderer-backgrounding".to_string(),
+ ];
+ let before = args.clone();
+ inject_chromium_anti_throttling_flags(&mut args);
+ assert_eq!(args, before, "must not duplicate flags");
+ }
+
+ #[test]
+ fn preserves_user_url_argument_after_flags() {
+ let mut args = vec!["file:///C:/test_page.html".to_string()];
+ inject_chromium_anti_throttling_flags(&mut args);
+ // URL must still be present.
+ assert!(args.iter().any(|a| a == "file:///C:/test_page.html"));
+ // All three flags now in args.
+ assert!(args
+ .iter()
+ .any(|a| a == "--disable-features=CalculateNativeWinOcclusion"));
+ assert!(args
+ .iter()
+ .any(|a| a == "--disable-backgrounding-occluded-windows"));
+ assert!(args.iter().any(|a| a == "--disable-renderer-backgrounding"));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/list_apps.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/list_apps.rs
new file mode 100644
index 0000000000..30e3b2d3e6
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/list_apps.rs
@@ -0,0 +1,237 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+
+// ── list_apps ────────────────────────────────────────────────────────────────
+
+pub struct ListAppsTool;
+
+static LIST_APPS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ListAppsTool {
+ fn def(&self) -> &ToolDef {
+ LIST_APPS_DEF.get_or_init(|| ToolDef {
+ name: "list_apps".into(),
+ description:
+ "List Windows apps — both currently running and installed-but-not-running — \
+ with per-app state flags:\n\n\
+ - running: is a process for this app live? (pid is 0 when false)\n\
+ - active: is it the system-frontmost app? (implies running)\n\
+ - kind: `\"desktop\"` for `.exe`-backed apps (resolved from Start-Menu \
+ shortcuts) or `\"uwp\"` for packaged store apps.\n\
+ - launch_path: what `launch_app` would consume.\n\
+ * desktop: full `.exe` commandline (path + any arguments preserved from \
+ the source `.lnk`; the exe is quoted when it contains whitespace).\n\
+ * uwp: `shell:appsFolder\\{PackageFamilyName}!{AppId}` where `{AppId}` \
+ falls back to `App` when the package manifest does not expose a \
+ specific Application.Id.\n\
+ - last_used: RFC3339 mtime of the launcher (`.lnk` for desktop, \
+ package install location for UWP), when readable.\n\n\
+ Running apps are derived from visible top-level windows + the foreground \
+ pid (mirrors NSApplicationActivationPolicyRegular's intent: background \
+ services and console processes are excluded). Installed apps are merged \
+ from Start-Menu `.lnk` enumeration and the WinRT PackageManager — running \
+ processes whose executable matches a Start-Menu target are folded into a \
+ single entry with `running: true`.\n\n\
+ Use this for \"is X installed?\" as well as \"is X running?\". For per-window \
+ state — on-screen, minimized, window titles — call list_windows instead. \
+ For just opening an app — running or not — call launch_app({path: ...}) \
+ directly; list_apps is not a prerequisite."
+ .into(),
+ input_schema: json!({"type":"object","properties":{},"additionalProperties":false}),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ // Strategy:
+ // 1. Enumerate top-level visible windows → derive the set of running pids
+ // that have a user-facing surface.
+ // 2. Enumerate the process table to map pid → executable name and full
+ // path (used to merge against Start-Menu / UWP launch_paths).
+ // 3. Enumerate installed apps from Start-Menu shortcuts + WinRT
+ // PackageManager. Each installed entry has a launch_path.
+ // 4. Merge: a running pid whose executable path matches an installed
+ // desktop launch_path becomes a single entry with running=true and
+ // that launch_path. Remaining installed entries are emitted with
+ // running=false, pid=0. Remaining running pids (no installed-app
+ // match) are emitted as running=true with launch_path=null.
+ let apps = tokio::task::spawn_blocking(|| -> Vec<serde_json::Value> {
+ use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
+ use windows::Win32::UI::WindowsAndMessaging::GetWindowThreadProcessId;
+
+ // Per-step timings logged via `tracing::debug!` (silent at default
+ // log level — enable with `RUST_LOG=cua_driver=debug` when debugging
+ // a slow list_apps invocation).
+ let t0 = std::time::Instant::now();
+ let wins = crate::win32::list_windows_via_win32(None);
+ tracing::debug!(target: "list_apps", "step 1 list_windows: {} wins ({}ms)", wins.len(), t0.elapsed().as_millis());
+ let mut seen_pids = std::collections::HashSet::new();
+ let mut running_pids: Vec<u32> = Vec::new();
+ for w in &wins {
+ if seen_pids.insert(w.pid) { running_pids.push(w.pid); }
+ }
+
+ let t1 = std::time::Instant::now();
+ let foreground_pid = unsafe {
+ let fg = GetForegroundWindow();
+ let mut pid = 0u32;
+ let _ = GetWindowThreadProcessId(fg, Some(&mut pid));
+ pid
+ };
+ tracing::debug!(target: "list_apps", "step 2 fg-window: pid={} ({}ms)", foreground_pid, t1.elapsed().as_millis());
+
+ let t2 = std::time::Instant::now();
+ let procs = crate::win32::list_processes();
+ tracing::debug!(target: "list_apps", "step 3 list_processes: {} procs ({}ms)", procs.len(), t2.elapsed().as_millis());
+ let pid_to_name: std::collections::HashMap<u32, String> =
+ procs.iter().map(|p| (p.pid, p.name.clone())).collect();
+
+ let t3 = std::time::Instant::now();
+ let installed = crate::win32::list_installed_apps();
+ tracing::debug!(target: "list_apps", "step 4 list_installed_apps: {} installed ({}ms)", installed.len(), t3.elapsed().as_millis());
+ // Lowercase-exe-basename → installed-app indices. We match running
+ // processes by basename rather than full path because the process
+ // table's executable name is `notepad.exe` while the Start-Menu
+ // shortcut target is `C:\Windows\System32\notepad.exe` — both
+ // sides need to be normalised to compare. Multiple Start-Menu
+ // shortcuts often target the same basename (different `chrome.exe`
+ // launchers, multiple `code.exe` profiles, …), so we bucket as
+ // `Vec<usize>` and let `disambiguate_installed_match` pick a
+ // winner per running pid rather than letting the last write win.
+ let mut by_exe: std::collections::HashMap<String, Vec<usize>> =
+ std::collections::HashMap::new();
+ for (i, app) in installed.iter().enumerate() {
+ if app.kind == "desktop" {
+ let basename = std::path::Path::new(&app.launch_path)
+ .file_name()
+ .and_then(|s| s.to_str())
+ .unwrap_or("")
+ .to_ascii_lowercase();
+ if !basename.is_empty() {
+ by_exe.entry(basename).or_default().push(i);
+ }
+ }
+ }
+
+ let mut consumed_installed: std::collections::HashSet<usize> =
+ std::collections::HashSet::new();
+ let mut out: Vec<serde_json::Value> = Vec::new();
+
+ for &pid in &running_pids {
+ let name = pid_to_name.get(&pid).cloned().unwrap_or_else(|| "<unknown>".into());
+ let key = name.to_ascii_lowercase();
+ let candidates = by_exe.get(&key).map(|v| v.as_slice()).unwrap_or(&[]);
+ let merged = disambiguate_installed_match(candidates, &installed, &name);
+ if let Some(idx) = merged { consumed_installed.insert(idx); }
+ let (display_name, bundle_id, launch_path, kind, last_used) = match merged {
+ Some(idx) => {
+ let a = &installed[idx];
+ (
+ a.name.clone(),
+ Some(a.bundle_id.clone()),
+ Some(a.launch_path.clone()),
+ Some(a.kind.clone()),
+ a.last_used.clone(),
+ )
+ }
+ None => (name, None, None, Some("desktop".to_owned()), None),
+ };
+ out.push(json!({
+ "pid": pid,
+ "bundle_id": bundle_id,
+ "name": display_name,
+ "running": true,
+ "active": pid == foreground_pid,
+ "kind": kind,
+ "launch_path": launch_path,
+ "last_used": last_used,
+ "windows": Vec::<serde_json::Value>::new(),
+ }));
+ }
+ for (i, app) in installed.iter().enumerate() {
+ if consumed_installed.contains(&i) { continue }
+ out.push(json!({
+ "pid": 0,
+ "bundle_id": app.bundle_id.clone(),
+ "name": app.name.clone(),
+ "running": false,
+ "active": false,
+ "kind": app.kind.clone(),
+ "launch_path": app.launch_path.clone(),
+ "last_used": app.last_used.clone(),
+ "windows": Vec::<serde_json::Value>::new(),
+ }));
+ }
+ out
+ }).await.unwrap_or_default();
+
+ let running_count = apps
+ .iter()
+ .filter(|a| a["running"].as_bool().unwrap_or(false))
+ .count();
+ let total = apps.len();
+ let installed_only = total - running_count;
+ let mut lines = vec![format!(
+ "✅ Found {total} app(s): {running_count} running, {installed_only} installed-not-running."
+ )];
+ for app in apps
+ .iter()
+ .filter(|a| a["running"].as_bool().unwrap_or(false))
+ {
+ let name = app["name"].as_str().unwrap_or("?");
+ let pid = app["pid"].as_u64().unwrap_or(0);
+ lines.push(format!("- {name} (pid {pid})"));
+ }
+ // `apps` is the unified shape; `processes` retained for any pre-existing
+ // callers that consumed the old running-only shape.
+ let structured = json!({
+ "apps": apps,
+ "processes": apps.iter().filter(|a| a["running"].as_bool().unwrap_or(false))
+ .map(|a| json!({
+ "pid": a["pid"], "name": a["name"]
+ })).collect::<Vec<_>>(),
+ });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
+
+/// Pick which of several Start-Menu / UWP-derived installed apps best matches
+/// a running pid whose exe basename collided. Precedence:
+/// 1. exact case-insensitive equality between `proc_exe_basename` and the
+/// installed app's launch_path basename (always true here; left for
+/// future per-path matches if we ever bucket on something coarser)
+/// 2. most recently modified launcher (`last_used` desc — RFC3339 strings
+/// are lexicographically ordered the same as chronologically)
+/// 3. first candidate (deterministic by source order)
+fn disambiguate_installed_match(
+ candidates: &[usize],
+ installed: &[crate::win32::InstalledApp],
+ _proc_exe_basename: &str,
+) -> Option<usize> {
+ if candidates.is_empty() {
+ return None;
+ }
+ if candidates.len() == 1 {
+ return Some(candidates[0]);
+ }
+
+ candidates
+ .iter()
+ .copied()
+ .max_by(|&a, &b| {
+ installed[a]
+ .last_used
+ .as_deref()
+ .unwrap_or("")
+ .cmp(installed[b].last_used.as_deref().unwrap_or(""))
+ })
+ .or_else(|| candidates.first().copied())
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/list_windows.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/list_windows.rs
new file mode 100644
index 0000000000..cde9fc1db2
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/list_windows.rs
@@ -0,0 +1,242 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ window_target::z_index_from_front_to_back,
+};
+use serde_json::{json, Value};
+
+#[cfg(test)]
+use super::impl_::exact_window_ownership_result;
+
+// ── list_windows ─────────────────────────────────────────────────────────────
+
+pub struct ListWindowsTool;
+
+static LIST_WINDOWS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ListWindowsTool {
+ fn def(&self) -> &ToolDef {
+ LIST_WINDOWS_DEF.get_or_init(|| ToolDef {
+ name: "list_windows".into(),
+ // Description matches Swift `ListWindowsTool.swift` semantics —
+ // Windows-specific caveat: no Spaces concept, so on_current_space
+ // / space_ids are always omitted and current_space_id is null.
+ description: "List every top-level window currently known to the window manager. \
+ Each record self-contains its owning app identity so the caller never has to \
+ join back against list_apps.\n\n\
+ Use this — not list_apps — for any window-level reasoning: \"does this app \
+ have a visible window right now?\", \"which of this pid's windows is the main \
+ one?\".\n\n\
+ Per-record fields: window_id (HWND), pid + app_name, title, \
+ bounds {x, y, width, height}, layer (always 0), z_index (integer or null; higher \
+ values are closer to the front; null means stacking order is unavailable and \
+ callers must not infer one), is_on_screen, minimized. To select a frontmost \
+ candidate, take the maximum integer z_index; if every value is null, use an \
+ explicit fallback instead of relying on array order. The macOS-specific \
+ on_current_space / space_ids fields are \
+ omitted on Windows; current_space_id is null.\n\n\
+ Inputs: pid (optional pid filter), app (app-name filter), on_screen_only (bool, \
+ default false). Filter with app or pid: an unfiltered call lists every window. \
+ The driver's own overlay is omitted.".into(),
+ input_schema: json!({"type":"object","properties":{
+ "pid":{"type":"integer","description":"Optional pid filter. When set, only this pid's windows are returned."},
+ "app":{"type":"string","description":cua_driver_core::tool_args::LIST_WINDOWS_APP_FILTER_DESCRIPTION},
+ "on_screen_only":{"type":"boolean","description":"When true, drop windows that aren't currently on-screen. Default false."}
+ },"additionalProperties":false}),
+ read_only: true, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let filter_pid = args.opt_u64("pid").map(|v| v as u32);
+ let on_screen_only = args.bool_or("on_screen_only", false);
+ let (mut windows, pid_to_name) = tokio::task::spawn_blocking(move || {
+ let wins = crate::win32::list_windows(filter_pid);
+ let procs = crate::win32::list_processes();
+ let map: std::collections::HashMap<u32, String> =
+ procs.into_iter().map(|p| (p.pid, p.name)).collect();
+ (wins, map)
+ })
+ .await
+ .unwrap_or_default();
+ if on_screen_only {
+ windows.retain(|w| w.is_on_screen);
+ }
+ drop_own_windows(&mut windows, std::process::id());
+ if let Some(filter) = args.opt_str("app") {
+ windows.retain(|w| {
+ let app_name = pid_to_name.get(&w.pid).map(String::as_str).unwrap_or("");
+ cua_driver_core::tool_args::app_filter_matches(&filter, app_name, None)
+ });
+ }
+
+ // Swift surfaces a warning when a pid filter matches nothing.
+ let missing_pid_warning = if let Some(fp) = filter_pid {
+ if windows.is_empty() {
+ use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
+ use windows::Win32::UI::WindowsAndMessaging::GetWindowThreadProcessId;
+ let fg_pid = unsafe {
+ let fg = GetForegroundWindow();
+ let mut p = 0u32;
+ let _ = GetWindowThreadProcessId(fg, Some(&mut p));
+ p
+ };
+ let fg_name = pid_to_name.get(&fg_pid).map(|s| s.as_str()).unwrap_or("?");
+ Some(format!(
+ "⚠️ No windows found for pid {fp}. The pid may be wrong or the app may not \
+ have created a window yet. The current frontmost app appears to be \
+ \"{fg_name}\" (pid {fg_pid})."
+ ))
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
+ // z_index: list_windows merges EnumWindows first (which the Win32
+ // window manager returns in canonical top-to-bottom z-order), then
+ // appends any UIA-only HWNDs UIA found that EnumWindows missed
+ // (UIA-only entries land at the bottom of the z-stack — they have
+ // no canonical Win32 ordering). Higher list index = farther from
+ // front. Swift convention: higher z_index = closer to front.
+ // Invert via `(len - 1 - i)` so the front-most window gets the
+ // largest z.
+ let n = windows.len();
+ let records: Vec<serde_json::Value> = windows
+ .iter()
+ .enumerate()
+ .map(|(i, w)| {
+ let app_name = pid_to_name.get(&w.pid).cloned().unwrap_or_default();
+ let z_index = z_index_from_front_to_back(n, i) as i64;
+ json!({
+ "window_id": w.hwnd,
+ "pid": w.pid,
+ "app_name": app_name,
+ "title": w.title,
+ "bounds": { "x": w.x, "y": w.y, "width": w.width, "height": w.height },
+ "layer": 0,
+ "z_index": z_index,
+ "is_on_screen": w.is_on_screen,
+ "minimized": w.minimized,
+ })
+ })
+ .collect();
+
+ let total = records.len();
+ let pid_count = windows
+ .iter()
+ .map(|w| w.pid)
+ .collect::<std::collections::HashSet<_>>()
+ .len();
+ let on_screen = records
+ .iter()
+ .filter(|r| r["is_on_screen"].as_bool().unwrap_or(false))
+ .count();
+ // Swift text format 1:1 — append the SPI-unavailable note since
+ // Windows has no equivalent of macOS Spaces.
+ let header = format!(
+ "✅ Found {total} window(s) across {pid_count} app(s); {on_screen} on-screen. \
+ (SkyLight Space SPIs unavailable — on_current_space / space_ids omitted.)"
+ );
+ let mut lines = vec![header];
+ if let Some(warning) = missing_pid_warning {
+ lines.push(warning);
+ }
+ for r in &records {
+ let app = r["app_name"].as_str().unwrap_or("?");
+ let pid = r["pid"].as_u64().unwrap_or(0);
+ let tt = r["title"].as_str().unwrap_or("");
+ let wid = r["window_id"].as_u64().unwrap_or(0);
+ let title_disp = if tt.is_empty() {
+ "(no title)".to_owned()
+ } else {
+ format!("\"{tt}\"")
+ };
+ let tag = if r["is_on_screen"].as_bool().unwrap_or(false) {
+ ""
+ } else {
+ " [off-screen]"
+ };
+ lines.push(format!(
+ "- {app} (pid {pid}) {title_disp} [window_id: {wid}]{tag}"
+ ));
+ }
+
+ // Legacy alias: keep the old flat fields under each record for any
+ // pre-existing Rust callers, but they read from the same source.
+ let legacy_windows: Vec<serde_json::Value> = windows
+ .iter()
+ .map(|w| {
+ json!({
+ "window_id": w.hwnd, "pid": w.pid, "title": w.title,
+ "x": w.x, "y": w.y, "width": w.width, "height": w.height,
+ "is_on_screen": w.is_on_screen, "minimized": w.minimized,
+ })
+ })
+ .collect();
+ let structured = json!({
+ "windows": records,
+ "current_space_id": serde_json::Value::Null, // Windows has no Spaces
+ "_legacy_windows": legacy_windows,
+ });
+ ToolResult::text(lines.join("\n")).with_structured(structured)
+ }
+}
+
+fn drop_own_windows(windows: &mut Vec<crate::win32::WindowInfo>, own_pid: u32) {
+ windows.retain(|w| w.pid != own_pid);
+}
+
+#[cfg(test)]
+mod list_windows_z_index_tests {
+ use super::{drop_own_windows, exact_window_ownership_result, z_index_from_front_to_back};
+ use crate::win32::WindowInfo;
+
+ #[test]
+ fn explicit_pid_hwnd_guard_refuses_wrong_or_stale_owners() {
+ assert!(exact_window_ownership_result(42, 7, Some(42)).is_ok());
+
+ let wrong = exact_window_ownership_result(42, 7, Some(99)).unwrap_err();
+ assert_eq!(wrong.is_error, Some(true));
+ assert_eq!(
+ wrong.structured_content.as_ref().unwrap()["code"],
+ "window_target_mismatch"
+ );
+ assert_eq!(wrong.structured_content.as_ref().unwrap()["owner_pid"], 99);
+
+ let stale = exact_window_ownership_result(42, 7, None).unwrap_err();
+ assert_eq!(
+ stale.structured_content.as_ref().unwrap()["code"],
+ "window_target_not_found"
+ );
+ }
+
+ fn window(hwnd: u64, pid: u32) -> WindowInfo {
+ WindowInfo {
+ hwnd,
+ pid,
+ title: format!("window {hwnd}"),
+ x: 0,
+ y: 0,
+ width: 1280,
+ height: 800,
+ is_on_screen: true,
+ minimized: false,
+ }
+ }
+
+ #[test]
+ fn the_drivers_own_windows_are_dropped_and_the_rest_keep_contiguous_z() {
+ let mut windows = vec![window(1, 7), window(2, 42), window(3, 7), window(4, 9)];
+ drop_own_windows(&mut windows, 7);
+ assert_eq!(windows.iter().map(|w| w.hwnd).collect::<Vec<_>>(), [2, 4]);
+ let z: Vec<_> = (0..windows.len())
+ .map(|i| z_index_from_front_to_back(windows.len(), i))
+ .collect();
+ assert_eq!(z, [1, 0]);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/mod.rs
index 575f2ed2e2..b4f8877215 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/mod.rs
@@ -5,15 +5,73 @@
use cua_driver_core::tool::ToolRegistry;
+#[cfg(target_os = "windows")]
+mod bring_to_front;
+#[cfg(target_os = "windows")]
+mod check_permissions;
+#[cfg(target_os = "windows")]
+mod click;
+#[cfg(target_os = "windows")]
+mod cursor_tools;
+#[cfg(target_os = "windows")]
+mod debug_window_info;
+#[cfg(target_os = "windows")]
+mod double_click;
+#[cfg(target_os = "windows")]
+mod drag;
+#[cfg(target_os = "windows")]
+mod get_accessibility_tree;
+#[cfg(target_os = "windows")]
+mod get_config;
+#[cfg(target_os = "windows")]
+mod get_cursor_position;
+#[cfg(target_os = "windows")]
+mod get_desktop_state;
+#[cfg(target_os = "windows")]
+mod get_screen_size;
+#[cfg(target_os = "windows")]
+mod get_window_state;
+#[cfg(target_os = "windows")]
+mod hotkey;
#[cfg(target_os = "windows")]
mod impl_;
#[cfg(target_os = "windows")]
+mod invoke_menu;
+#[cfg(target_os = "windows")]
+mod kill_app;
+#[cfg(target_os = "windows")]
+mod launch_app;
+#[cfg(target_os = "windows")]
+mod list_apps;
+#[cfg(target_os = "windows")]
+mod list_windows;
+#[cfg(target_os = "windows")]
+mod move_cursor;
+#[cfg(target_os = "windows")]
pub(crate) mod page;
#[cfg(target_os = "windows")]
pub(crate) mod page_bookmark;
// Pure title parsing for `page_bookmark`; ungated so its tests run on any host.
pub(crate) mod page_title_marker;
+#[cfg(target_os = "windows")]
+mod press_key;
+#[cfg(target_os = "windows")]
+mod right_click;
+#[cfg(target_os = "windows")]
+mod scroll;
mod scroll_coordinates;
+#[cfg(target_os = "windows")]
+mod set_config;
+#[cfg(target_os = "windows")]
+mod set_value;
+#[cfg(target_os = "windows")]
+mod set_window_frame;
+#[cfg(target_os = "windows")]
+mod type_text;
+#[cfg(target_os = "windows")]
+mod type_text_chars;
+#[cfg(target_os = "windows")]
+mod zoom;
#[cfg(not(target_os = "windows"))]
mod stubs;
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/move_cursor.rs
new file mode 100644
index 0000000000..b7c9a89d35
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/move_cursor.rs
@@ -0,0 +1,84 @@
+use async_trait::async_trait;
+use cua_driver_contract::MoveCursorInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{resolve_cursor_key, ToolState};
+
+// ── move_cursor ───────────────────────────────────────────────────────────────
+
+pub struct MoveCursorTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static MCURSOR_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for MoveCursorTool {
+ fn def(&self) -> &ToolDef {
+ MCURSOR_DEF.get_or_init(|| ToolDef {
+ name: "move_cursor".into(),
+ description:
+ "Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor."
+ .into(),
+ input_schema: json!({"type":"object","required":["x","y"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"desktop moves the real OS pointer; window moves only the agent overlay."},
+ "x":{"type":"number","description":"Destination X. Window scope: screen coordinates of the agent cursor overlay. Desktop scope: get_desktop_state screenshot pixels."},
+ "y":{"type":"number","description":"Destination Y, in the same space as x."},
+ "cursor_id":{"type":"string","description":"Cursor instance to move. Default: 'default'."}
+ },"additionalProperties":false}),
+ read_only: false,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.get("scope").and_then(Value::as_str) == Some("desktop") {
+ let input = match parse_typed_projection::<MoveCursorInput>("move_cursor", &args) {
+ Ok(input) => input,
+ Err(error) => return error,
+ };
+ let x = input.x;
+ let y = input.y;
+ return match crate::input::mouse::move_cursor_desktop(x as i32, y as i32) {
+ Ok(()) => ToolResult::text(format!(
+ "Moved the OS pointer to ({x:.1}, {y:.1}) (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SetCursorPos", "x": x, "y": y
+ })),
+ Err(error) => ToolResult::error(error.to_string()),
+ };
+ }
+ let x = args.f64_or("x", 0.0);
+ let y = args.f64_or("y", 0.0);
+ // The trusted dispatch boundary always supplies a lifecycle key,
+ // including for an unnamed implicit session.
+ let cursor_key = resolve_cursor_key(&args);
+ if !cursor_key.is_empty() {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, x, y);
+ }
+ // End pointing upper-left (45°) — matches Swift's
+ // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so
+ // the cursor settles to the natural macOS-style pose.
+ // Use the acknowledged animation path so a first-ever move seeds and
+ // displays the session cursor just as reliably as a coordinate click.
+ crate::overlay::animate_cursor_to(cursor_key.clone(), x, y).await;
+ let shown = if cursor_key.is_empty() {
+ "default"
+ } else {
+ cursor_key.as_str()
+ };
+ ToolResult::text(format!("Agent cursor '{shown}' moved to ({x:.1}, {y:.1})."))
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/press_key.rs
new file mode 100644
index 0000000000..ddd1f2b9cf
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/press_key.rs
@@ -0,0 +1,346 @@
+use async_trait::async_trait;
+use cua_driver_contract::PressKeyInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ bitmap_to_screen, focus_by_pixel, focus_cached_element_for_foreground,
+ position_keyboard_cursor, resolve_onscreen_point_with_scroll, ToolState,
+};
+
+fn screen_to_bitmap(hwnd: u64, sx: i32, sy: i32) -> (i32, i32) {
+ let (origin_x, origin_y) = bitmap_to_screen(hwnd, 0, 0);
+ (sx - origin_x, sy - origin_y)
+}
+
+// ── press_key ─────────────────────────────────────────────────────────────────
+
+pub struct PressKeyTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static PRESS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for PressKeyTool {
+ fn def(&self) -> &ToolDef {
+ PRESS_DEF.get_or_init(|| ToolDef {
+ name: "press_key".into(),
+ // Description ported from Swift `PressKeyTool.swift` with
+ // Windows-specific transport note (PostMessage WM_KEYDOWN/UP).
+ description: "Press and release a single key, delivered directly to the target pid's \
+ top-level window via PostMessage(WM_KEYDOWN/WM_KEYUP). The target does NOT need \
+ to be frontmost — no focus steal.\n\n\
+ Optional `window_id` selects a specific HWND when the pid owns more than one; \
+ without it the first visible top-level window for the pid is used.\n\n\
+ Key vocabulary: return, tab, escape, up/down/left/right, space, delete, home, \
+ end, pageup, pagedown, f1-f12, plus any letter or digit. Optional `modifiers` \
+ array takes ctrl/shift/alt/win. For true combinations (ctrl+c), `hotkey` is a \
+ cleaner surface.\n\n\
+ `element_token` focuses the cached UIA element before sending the key; the \
+ top-level window remains backgrounded when delivery_mode is background.".into(),
+ input_schema: json!({
+ "type":"object","required":["key"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the key to the current foreground application."},
+ "pid":{"type":"integer","description":"Target process ID."},
+ "key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},
+ "modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"Window-local screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the UIA path can't focus. Pass with y, no element_token."},
+ "y":{"type":"number","description":"Window-local screenshot-pixel Y (see x)."},
+ "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<PressKeyInput>("press_key", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let key = input.key;
+ let mods = input.modifiers.unwrap_or_default();
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let key_display = key.clone();
+ return match tokio::task::spawn_blocking(move || {
+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Sent {key_display} to the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "key": key_display,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop key task failed: {error}")),
+ };
+ }
+ let key = match args.require_str("key") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let mods: Vec<String> = args.str_array("modifiers");
+ let raw_pid = match args.require_i64("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let pid = raw_pid as u32;
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let delivery = DeliveryMode::from_args(&args);
+ // Swift requires window_id when element_index is supplied — ports the
+ // same validation.
+
+ // px form: pixel-click to focus, then the key goes to the focused element.
+ // Reuses click's translation + delivery_mode; after it, deliver via the
+ // plain background post path (the focus-click already handled fronting if
+ // foreground), so skip the background_unavailable check + foreground swap
+ // below. Mirrors macOS press_key.
+ let px_focus = {
+ let px = args.get("x").and_then(|v| v.as_f64());
+ let py = args.get("y").and_then(|v| v.as_f64());
+ if let (Some(cx), Some(cy)) = (px, py) {
+ if elem_idx.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to press_key, not both.",
+ );
+ }
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ hwnd_opt,
+ cx,
+ cy,
+ delivery.is_foreground(),
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ false,
+ )
+ .await
+ {
+ return e;
+ }
+ true
+ } else {
+ false
+ }
+ };
+
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // Classify known background drops before touching UIA focus. Focusing
+ // first made an honest Chromium refusal transiently activate the target.
+ let event_kind = if mods.is_empty() {
+ EventKind::Keystroke
+ } else {
+ EventKind::KeyCombo
+ };
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, event_kind)
+ {
+ return crate::input::delivery::background_unavailable_error(hwnd, event_kind);
+ }
+
+ // The px form's focus click already moved the cursor. Otherwise place
+ // a named session's cursor so a keyboard-first session stays visible.
+ if !px_focus {
+ let element_center = admitted
+ .as_ref()
+ .map(|element| (element.center.0 as f64, element.center.1 as f64));
+ position_keyboard_cursor(&self.state, &args, hwnd, element_center).await;
+ }
+
+ // W1: an element-addressed key needs the control's actual focus
+ // target, not merely its owning top-level HWND. Embedded WebView hosts
+ // can activate their frame from UIA SetFocus even under
+ // WS_EX_NOACTIVATE. Their proven-safe pixel route establishes renderer
+ // focus with a posted click, so reuse that route at the AX element's
+ // cached center.
+ let background_webview_focus = elem_idx.is_some()
+ && delivery == DeliveryMode::Background
+ && crate::input::has_chromium_descendant(hwnd);
+ let mut noact = if elem_idx.is_some() && delivery == DeliveryMode::Background {
+ Some(crate::input::NoActivateGuard::arm(
+ windows::Win32::Foundation::HWND(hwnd as *mut _),
+ ))
+ } else {
+ None
+ };
+ if let Some(idx) = elem_idx.filter(|_| background_webview_focus) {
+ let Some((cx, cy)) = admitted.as_ref().map(|element| element.center) else {
+ return ToolResult::error(format!(
+ "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
+ ));
+ };
+ let (cx, cy) = match resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx as usize,
+ cx,
+ cy,
+ "focusing for key delivery",
+ ) {
+ Ok(point) => point,
+ Err(result) => return result,
+ };
+ let (px, py) = screen_to_bitmap(hwnd, cx, cy);
+ // Release the outer guard before the shared pixel helper. ClickTool
+ // owns a guard around the click itself, then releases it before its
+ // renderer settle period. This is the exact route already proven by
+ // the PX background cell, including targeted injection fallback.
+ drop(noact.take());
+ if let Err(error) = focus_by_pixel(
+ &self.state,
+ pid,
+ Some(hwnd),
+ px as f64,
+ py as f64,
+ false,
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ false,
+ true,
+ )
+ .await
+ {
+ return error;
+ }
+ } else if elem_idx.is_some() && delivery != DeliveryMode::Foreground {
+ let focused = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || {
+ admitted
+ .as_ref()
+ .ok_or_else(|| anyhow::anyhow!("missing admitted element"))
+ .and_then(|element| element.focus_element())
+ })
+ }
+ })
+ .await;
+ match focused {
+ Ok(Ok(())) => {
+ if delivery == DeliveryMode::Background {
+ let _ = crate::input::wait_for_focused_descendant(
+ hwnd,
+ std::time::Duration::from_millis(500),
+ );
+ }
+ }
+ Ok(Err(e)) => return ToolResult::error(e.to_string()),
+ Err(e) => return ToolResult::error(format!("UIA focus task failed: {e}")),
+ }
+ }
+ let key_display = key.clone();
+ // Foreground: send_key_synthesized takes the SetForegroundWindow path.
+ // Skipped when px-focus already fronted/clicked the target — the key then
+ // goes via the plain background post path below.
+ if !px_focus && delivery == DeliveryMode::Foreground {
+ let focus_target = elem_idx.map(|idx| {
+ let point = admitted.as_ref().map(|element| element.center);
+ (idx as usize, point)
+ });
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized_after_focus(hwnd, &key, &m, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&admitted, hwnd, idx, point)?;
+ }
+ Ok(())
+ })
+ }
+ })
+ .await;
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent {key_display} via SendInput on pid {raw_pid} (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::post_key(hwnd, &key, &m)
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "📨 Sent {key_display} to pid {raw_pid} via PostMessage (not verified). \
+ If it didn't land, retry with delivery_mode:\"foreground\"."
+ ))
+ .with_structured(serde_json::json!({
+ "path": "key_events",
+ "key": key_display,
+ "verified": false,
+ "verify": "unreadable",
+ "effect": "unverifiable",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "background key delivery could not be confirmed — re-call \
+ with delivery_mode:\"foreground\" if a screenshot shows the key didn't land."
+ },
+ })),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/right_click.rs
new file mode 100644
index 0000000000..ab87b3365e
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/right_click.rs
@@ -0,0 +1,372 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ bitmap_to_screen, chromium_click_short_circuit, element_target_rect, overlay_glide_to,
+ overlay_glide_to_target, pin_overlay_above, resolve_cursor_key,
+ resolve_onscreen_point_with_scroll, restore_foreground_polling_best_effort, screenshot_scale,
+ winui3_background_gesture, ToolState,
+};
+
+// ── right_click ───────────────────────────────────────────────────────────────
+
+pub struct RightClickTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static RCLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for RightClickTool {
+ fn def(&self) -> &ToolDef {
+ RCLICK_DEF.get_or_init(|| ToolDef {
+ name: "right_click".into(),
+ // Description ported from Swift `RightClickTool.swift` semantics.
+ // Windows uses PostMessage(WM_RBUTTONDOWN/UP); the AXShowMenu
+ // analogue (UIA ShowContextMenu) is not yet wired up, so element
+ // path falls through to the pixel recipe at the cached center.
+ description: "Right-click against a target pid. Two addressing modes:\n\n\
+ - `element_token` (from the last `get_window_state` snapshot \
+ of that window) — posts WM_RBUTTONDOWN/UP at the element's cached on-screen \
+ center via PostMessage. (Windows has no AXShowMenu analogue wired up yet; \
+ Swift's AX-action path falls through to the same pixel recipe on non-\
+ advertising elements, so user-visible behavior matches.)\n\n\
+ - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG \
+ returned by `get_window_state`) — posts WM_RBUTTONDOWN/UP to the deepest \
+ child window at that point.\n\n\
+ Exactly one of `element_token` or (`x` AND `y`) must be provided. `pid` is \
+ required with `x`, `y`; an `element_token` carries its own pid. \
+ `modifier` is accepted for parity (no-op on Windows — PostMessage doesn't \
+ propagate modifier-key state).".into(),
+ input_schema: json!({"type":"object","required":[],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"HWND for the target window. Omit when element_token is supplied (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "x":{"type":"number","description":"X in window-local screenshot pixels. Must be provided together with y."},
+ "y":{"type":"number","description":"Y in window-local screenshot pixels. Must be provided together with x."},
+ "modifier": cua_driver_core::tool_schema::modifier_schema(),
+ "from_zoom":{"type":"boolean","description":"After a zoom call, pass true to translate zoom-image coords back to window space."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false}),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ // Swift error wording 1:1.
+ let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(p) => p,
+ None => return ToolResult::error("Missing required integer field pid."),
+ };
+ let pid = raw_pid as u32;
+ use cua_driver_core::tool_args::ArgsExt;
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let x = args.opt_f64("x");
+ let y = args.opt_f64("y");
+ let delivery = DeliveryMode::from_args(&args);
+ let cursor_key = resolve_cursor_key(&args);
+ // Port Swift's full validation set.
+ let has_xy = x.is_some() && y.is_some();
+ let partial_xy = x.is_some() != y.is_some();
+ if partial_xy {
+ return ToolResult::error("Provide both x and y together, not just one.");
+ }
+ if elem_idx.is_some() && has_xy {
+ return ToolResult::error("Provide either element_token or (x, y), not both.");
+ }
+ if elem_idx.is_none() && !has_xy {
+ return ToolResult::error(
+ "Provide element_token or (x, y) to address the right-click target.",
+ );
+ }
+
+ // Resolve HWND: explicit, or auto from pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ if let Some(idx) = elem_idx {
+ let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
+ Some(v) => v,
+ None => {
+ return ToolResult::error(format!(
+ "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
+ ))
+ }
+ };
+ let recorded_center = (cx, cy);
+ let (cx, cy) = match resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx,
+ cx,
+ cy,
+ "right-clicking",
+ ) {
+ Ok(p) => p,
+ Err(result) => return result,
+ };
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(
+ admitted.as_ref(),
+ (cx - recorded_center.0, cy - recorded_center.1),
+ ),
+ )
+ .await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: cx as f64,
+ y: cy as f64,
+ },
+ );
+ // delivery_mode:"background" on WinUI3: a right-click can't both land and
+ // hold the contract — UIA has no right-click pattern, the content
+ // island ignores posted WM_RBUTTON (measured: RightTapped never
+ // fired), and the pen-barrel injector click-activates the frame.
+ // Return the structured error (retry with delivery_mode:foreground).
+ if delivery == DeliveryMode::Background {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, Some(idx), 1, "right").await
+ {
+ return r;
+ }
+ }
+ // delivery_mode:"background" (default): try coordinate injection (no
+ // foreground swap) for drop-prone targets (#1984); a right-click
+ // injection that the actuator can't express falls back to the error.
+ if (cx, cy) != recorded_center {
+ crate::recording_hooks::capture_dispatch_click_target(hwnd, pid, cx, cy);
+ }
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, cx, cy, 1, "right")
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected right-click to [{idx}] at screen ({cx},{cy}) (background, no foreground swap)."
+ )),
+ Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ e.to_string(),
+ ),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ if delivery == DeliveryMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_click_synthesized_active_mods(
+ hwnd,
+ cx,
+ cy,
+ 1,
+ "right",
+ &[],
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent right-click via SendInput on [{idx}] at screen ({cx},{cy}) (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
+ if let Some(r) =
+ chromium_click_short_circuit(hwnd, cx, cy, 1, "right", pid, "right-click").await
+ {
+ return r;
+ }
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || -> anyhow::Result<String> {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, cx, cy, 1, "right")?;
+ // Match Swift's element-path text 1:1
+ // (`"✅ Shown menu for [N] role \"title\"."`). UIA role/title
+ // placeholder pending element-cache enrichment.
+ Ok(format!("✅ Shown menu for [{idx}] (screen ({cx}, {cy}))."))
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(msg)) => ToolResult::text(msg),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ } else if let (Some(mut px), Some(mut py)) = (x, y) {
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if from_zoom {
+ match self.state.zoom_context(&args, pid, Some(hwnd)) {
+ Ok(ctx) => {
+ let (wx, wy) = ctx.zoom_to_window(px, py);
+ px = wx;
+ py = wy;
+ }
+ Err(refusal) => return refusal,
+ }
+ } else {
+ let ratio = match screenshot_scale(&self.state, &args, pid, Some(hwnd)) {
+ Ok(ratio) => ratio,
+ Err(refusal) => return refusal,
+ };
+ px *= ratio;
+ py *= ratio;
+ }
+ // bitmap pixels -> screen via DWM-frame origin (see
+ // `bitmap_to_screen` doc).
+ let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32);
+ let (sx, sy) = (sx_i as f64, sy_i as f64);
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to(&cursor_key, sx, sy).await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ // delivery_mode:"background" on WinUI3: right-click can't land while
+ // holding the contract (no UIA right-click, posted WM_RBUTTON
+ // ignored, pen-barrel injector steals foreground) → structured error.
+ if delivery == DeliveryMode::Background {
+ if let Some(r) =
+ winui3_background_gesture(&admitted, pid, hwnd, None, 1, "right").await
+ {
+ return r;
+ }
+ }
+ // delivery_mode:"background" (default): try coordinate injection (no
+ // foreground swap) for drop-prone targets (#1984).
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseClick)
+ {
+ let inj = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::inject_click_screen(hwnd, sx_i, sy_i, 1, "right")
+ }
+ })
+ .await;
+ return match inj {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Injected right-click to pid {pid} at screen ({sx_i},{sy_i}) (background, no foreground swap)."
+ )),
+ Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
+ hwnd,
+ EventKind::MouseClick,
+ e.to_string(),
+ ),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ if delivery == DeliveryMode::Foreground {
+ let prev_fg_addr = unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
+ };
+ let send_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_click_synthesized_active_mods(
+ hwnd,
+ sx_i,
+ sy_i,
+ 1,
+ "right",
+ &[],
+ )
+ }
+ })
+ .await;
+ tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ return match send_result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent right-click via SendInput to pid {pid} at screen ({sx_i},{sy_i}) (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+ let (xi, yi) = (px as i32, py as i32);
+ // Chromium/Electron silently drops PostMessage clicks (#1984) — route via SendInput.
+ if let Some(r) =
+ chromium_click_short_circuit(hwnd, sx_i, sy_i, 1, "right", pid, "right-click").await
+ {
+ return r;
+ }
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::post_click_screen(hwnd, sx_i, sy_i, 1, "right")
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ // Swift pixel-path text 1:1.
+ ToolResult::text(format!(
+ "✅ Posted right-click to pid {pid} at window-pixel ({xi}, {yi}) → screen-point ({sx_i}, {sy_i})."))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ } else {
+ // Guarded above by the early validation block — unreachable.
+ unreachable!()
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/scroll.rs
new file mode 100644
index 0000000000..13e5625fd8
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/scroll.rs
@@ -0,0 +1,408 @@
+use async_trait::async_trait;
+use cua_driver_contract::{ScrollDirection, ScrollInput};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{bitmap_to_screen, is_windowless_desktop_action, ToolState};
+
+// ── scroll ────────────────────────────────────────────────────────────────────
+
+pub struct ScrollTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static SCROLL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ScrollTool {
+ fn def(&self) -> &ToolDef {
+ SCROLL_DEF.get_or_init(|| ToolDef {
+ name: "scroll".into(),
+ // Description ported from Swift `ScrollTool.swift` with the
+ // Windows-specific transport note (WM_VSCROLL/WM_HSCROLL).
+ description: "Scroll the target pid's focused region.\n\n\
+ Windows transport: WM_VSCROLL / WM_HSCROLL posted to the window — the same \
+ events the OS sends when scrollbars or trackpads scroll, so any window that \
+ handles scrollbars correctly responds to this. Swift uses synthesized \
+ keystrokes (PageDown / arrow keys) via auth-signed SLEventPostToPid; both \
+ approaches reach backgrounded windows.\n\n\
+ Mapping: `by: \"page\"` → SB_PAGEDOWN/UP/LEFT/RIGHT × amount; \
+ `by: \"line\"` → SB_LINEDOWN/UP/LEFT/RIGHT × amount.\n\n\
+ Note: `element_token` is accepted for cross-platform parity but currently \
+ no-op on Windows (UIA SetFocus not wired up yet — same caveat as `press_key`).".into(),
+ input_schema: json!({
+ "type":"object","required":["direction"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID for window scope. Optional with element_token (the token carries the pid). Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
+ "direction":{"type":"string","enum":["up","down","left","right"],"description":"Scroll direction."},
+ "by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},
+ "amount":{"type":"integer","minimum":1,"maximum":50,
+ "description":"Number of scroll ticks. Default 3."},
+ "x":{"type":"number","description":"With pid/window_id: window-local screenshot X used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute X for desktop scope. Must be paired with y."},
+ "y":{"type":"number","description":"With pid/window_id: window-local screenshot Y used to target a nested scroll surface in foreground mode. Without pid/window_id: screen-absolute Y for desktop scope. Must be paired with x."},
+ "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied; otherwise auto-resolves the pid's first visible window."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window","description":"Use \"desktop\" with x,y and no pid/window_id for screen-absolute coordinates from get_desktop_state. Default \"window\"."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: false, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ // ── Window-less screen-absolute branch (desktop target) ───────────────
+ // No pid/window_id + numeric x,y + desktop scope → synthesize a wheel
+ // event at the screen point via SendInput. The wheel routes to whatever
+ // window is under (x,y). up/down map to a vertical wheel (sign), and
+ // left/right to a horizontal wheel; `amount` is the tick count.
+ let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
+ let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
+ let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
+ && args.get("y").map(|v| v.is_number()).unwrap_or(false);
+ if !has_pid && !has_window_id && has_xy {
+ if !is_windowless_desktop_action(&args) {
+ return ToolResult::error(
+ "scroll: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",
+ )
+ .with_structured(serde_json::json!({
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
+ }));
+ }
+ let input = match parse_typed_projection::<ScrollInput>("scroll", &args) {
+ Ok(input) => input,
+ Err(error) => return error,
+ };
+ let sx = input.x as i32;
+ let sy = input.y as i32;
+ let direction = input.direction;
+ let amount = input.amount.unwrap_or(3) as u32;
+ // Direction → (horizontal?, sign). Positive ticks = up / right.
+ let (horizontal, sign) = match direction {
+ ScrollDirection::Up => (false, 1),
+ ScrollDirection::Down => (false, -1),
+ ScrollDirection::Right => (true, 1),
+ ScrollDirection::Left => (true, -1),
+ };
+ let ticks = sign * amount as i32;
+ let dir_disp = direction.as_str();
+ let result = tokio::task::spawn_blocking(move || {
+ crate::input::send_wheel_synthesized(sx, sy, ticks, horizontal)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Scrolled {dir_disp} via SendInput wheel ({amount} tick(s)) at screen ({sx},{sy}) (desktop scope)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Window-scoped arguments retain the richer platform contract.
+ let direction = match args.get("direction").and_then(|v| v.as_str()) {
+ Some(d) => d.to_owned(),
+ None => return ToolResult::error("Missing required string field direction."),
+ };
+ let amount = args.u64_or("amount", 3).clamp(1, 50) as u32;
+
+ // Swift error wording 1:1.
+ let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
+ Some(p) => p,
+ None => return ToolResult::error("Missing required integer field pid."),
+ };
+ let pid = raw_pid as u32;
+ let delivery = DeliveryMode::from_args(&args);
+ let by = args.str_or("by", "line");
+ let direction_display = direction.clone();
+ let by_display = by.clone();
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+
+ // Resolve HWND: use window_id if given, else first window for pid.
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+
+ // WebView2/Tauri hosts can expose the indexed scroll container through
+ // UIA while their top-level HWND ignores WM_VSCROLL. Prefer the
+ // accessibility channel for an indexed target; the message path below
+ // remains the fallback for native Win32 scrollbars.
+ if delivery == DeliveryMode::Background && crate::input::is_chromium_target_window(hwnd) {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::MouseScroll,
+ );
+ }
+ if let Some(idx) = elem_idx {
+ let prev_fg_addr = if delivery == DeliveryMode::Background {
+ Some(unsafe {
+ windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as isize
+ })
+ } else {
+ None
+ };
+ let _noact = if delivery == DeliveryMode::Background {
+ Some(crate::input::NoActivateGuard::arm(
+ windows::Win32::Foundation::HWND(hwnd as *mut _),
+ ))
+ } else {
+ None
+ };
+ let direction_for_uia = direction.clone();
+ let uia_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let retained = admitted.as_ref().ok_or_else(|| {
+ anyhow::anyhow!("element [{idx}] is not in the UIA cache")
+ })?;
+ if !retained.is_uia() {
+ anyhow::bail!("element [{idx}] is not a UIA scroll element");
+ }
+ unsafe {
+ crate::uia::scroll::scroll_element(
+ retained.as_ptr(),
+ &direction_for_uia,
+ amount,
+ )
+ }
+ }
+ })
+ .await;
+ if matches!(uia_result, Ok(Ok(()))) {
+ if delivery == DeliveryMode::Background {
+ // Keep WS_EX_NOACTIVATE armed through any WebView handler
+ // queued by the UIA operation.
+ tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+ if let Some(previous_addr) = prev_fg_addr {
+ use windows::Win32::Foundation::HWND;
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetAncestor, GetForegroundWindow, GA_ROOT,
+ };
+ let previous = HWND(previous_addr as *mut _);
+ let current = unsafe { GetForegroundWindow() };
+ let current_root = unsafe { GetAncestor(current, GA_ROOT) };
+ let target_root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) };
+ if !previous.0.is_null()
+ && current != previous
+ && !target_root.0.is_null()
+ && current_root == target_root
+ {
+ unsafe {
+ crate::input::force_foreground_attached(previous);
+ std::thread::sleep(std::time::Duration::from_millis(12));
+ crate::input::force_foreground_attached(previous);
+ }
+ }
+ }
+ }
+ return ToolResult::text(format!(
+ "Scrolled {direction} {amount} ticks via UIA (delivery_mode:background)."
+ ))
+ .with_structured(serde_json::json!({
+ "path": "uia",
+ "verified": false,
+ "delivery_mode": "background"
+ }));
+ }
+ }
+
+ if delivery == DeliveryMode::Background
+ && args.get("x").is_some_and(serde_json::Value::is_number)
+ && args.get("y").is_some_and(serde_json::Value::is_number)
+ {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::MouseScroll,
+ );
+ }
+
+ // delivery_mode:"background" — WM_VSCROLL/HSCROLL is silently dropped by
+ // Chromium and may be by GTK. Surface the standard structured
+ // background_unavailable error: its remediation (bring_to_front +
+ // delivery_mode:"foreground") is now a real recovery because scroll's
+ // foreground path is implemented below (SendInput wheel reaches the
+ // Chromium/GTK targets PostMessage can't).
+ if delivery == DeliveryMode::Background
+ && crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::MouseScroll)
+ {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::MouseScroll,
+ );
+ }
+ // delivery_mode:"foreground" — SendInput MOUSEEVENTF_WHEEL at the target
+ // window's screen center (send_wheel_synthesized). The wheel routes to
+ // the window under the cursor — queue-origin input Chromium/GTK accept —
+ // so it lands where PostMessage WM_*SCROLL is dropped. Mirrors the
+ // desktop-scope wheel branch above; `by` (line/page) folds into the tick
+ // count (page ≈ 3 detents).
+ if delivery == DeliveryMode::Foreground {
+ let (horizontal, sign) = match direction.as_str() {
+ "up" => (false, 1),
+ "down" => (false, -1),
+ "right" => (true, 1),
+ "left" => (true, -1),
+ other => {
+ return ToolResult::error(format!(
+ "scroll: unknown direction \"{other}\" — expected up, down, left, right."
+ ))
+ }
+ };
+ let per: i32 = if by == "page" { 3 } else { 1 };
+ let ticks = sign * (amount as i32) * per;
+ // A supplied PX target is window-local in the get_window_state
+ // bitmap. Route the wheel there so nested web scrollers receive it;
+ // otherwise retain the whole-window center fallback.
+ let px = args.get("x").and_then(|value| value.as_f64());
+ let py = args.get("y").and_then(|value| value.as_f64());
+ if px.is_some() != py.is_some() {
+ return ToolResult::error("scroll requires x and y together.");
+ }
+ let center = if let (Some(x), Some(y)) = (px, py) {
+ let (x, y) = match super::scroll_coordinates::window_bitmap_point(
+ &self.state.snapshots,
+ &args,
+ pid,
+ hwnd,
+ x,
+ y,
+ ) {
+ Ok(point) => point,
+ Err(refusal) => return refusal,
+ };
+ Some(bitmap_to_screen(hwnd, x, y))
+ } else {
+ tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ .into_iter()
+ .find(|w| w.hwnd == hwnd)
+ .map(|w| (w.x + w.width / 2, w.y + w.height / 2))
+ }
+ })
+ .await
+ .ok()
+ .flatten()
+ };
+ let (cx, cy) = match center {
+ Some(c) => c,
+ None => {
+ return ToolResult::error(format!(
+ "scroll: could not resolve on-screen bounds for window {hwnd:#x} \
+ (pid {pid}) to target the wheel. The window may be minimized or \
+ off-screen — call bring_to_front first."
+ ))
+ }
+ };
+ let dir_disp = direction.clone();
+ let tick_disp = ticks.abs();
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_wheel_synthesized(cx, cy, ticks, horizontal)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Scrolled {dir_disp} via SendInput wheel ({tick_disp} tick(s)) at \
+ screen ({cx},{cy}) (delivery_mode:foreground)."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || -> anyhow::Result<()> {
+ let _admission = &admitted;
+ use windows::Win32::Foundation::{HWND, LPARAM, WPARAM};
+ use windows::Win32::UI::WindowsAndMessaging::{
+ PostMessageW, SB_LINEDOWN, SB_LINELEFT, SB_LINERIGHT, SB_LINEUP, SB_PAGEDOWN,
+ SB_PAGELEFT, SB_PAGERIGHT, SB_PAGEUP, WM_HSCROLL, WM_VSCROLL,
+ };
+
+ let hwnd_win = HWND(hwnd as *mut _);
+ let use_page = by == "page";
+ let (msg, code) = match (direction.as_str(), use_page) {
+ ("up", false) => (WM_VSCROLL, SB_LINEUP),
+ ("up", true) => (WM_VSCROLL, SB_PAGEUP),
+ ("down", false) => (WM_VSCROLL, SB_LINEDOWN),
+ ("down", true) => (WM_VSCROLL, SB_PAGEDOWN),
+ ("left", false) => (WM_HSCROLL, SB_LINELEFT),
+ ("left", true) => (WM_HSCROLL, SB_PAGELEFT),
+ ("right", false) => (WM_HSCROLL, SB_LINERIGHT),
+ ("right", true) => (WM_HSCROLL, SB_PAGERIGHT),
+ _ => (WM_VSCROLL, SB_LINEDOWN),
+ };
+ for _ in 0..amount {
+ unsafe {
+ PostMessageW(hwnd_win, msg, WPARAM(code.0 as usize), LPARAM(0))?;
+ }
+ }
+ Ok(())
+ }
+ })
+ .await;
+
+ match result {
+ Ok(Ok(())) => {
+ // Match Swift's text format shape 1:1 (Swift uses key names
+ // like "pagedown"; Windows uses Win32 SB_* constants — show
+ // the actual mechanism for traceability).
+ let mech_name = match (direction_display.as_str(), by_display.as_str()) {
+ ("up", "page") => "SB_PAGEUP",
+ ("down", "page") => "SB_PAGEDOWN",
+ ("left", "page") => "SB_PAGELEFT",
+ ("right", "page") => "SB_PAGERIGHT",
+ ("up", _) => "SB_LINEUP",
+ ("down", _) => "SB_LINEDOWN",
+ ("left", _) => "SB_LINELEFT",
+ ("right", _) => "SB_LINERIGHT",
+ _ => "SB_LINEDOWN",
+ };
+ ToolResult::text(format!(
+ "✅ Scrolled pid {raw_pid} {direction_display} via {amount}× {mech_name} message(s)."
+ ))
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/set_config.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_config.rs
new file mode 100644
index 0000000000..78b0864df4
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_config.rs
@@ -0,0 +1,148 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+// ── set_config ────────────────────────────────────────────────────────────────
+
+fn with_cursor_motion_config_properties(mut schema: Value) -> Value {
+ if let Some(properties) = schema.get_mut("properties").and_then(Value::as_object_mut) {
+ properties.extend(cursor_overlay::motion_defaults::config_schema_properties());
+ properties.extend(cursor_overlay::badge_mode::config_schema_properties());
+ properties.extend(cursor_overlay::cursor_options::config_schema_properties());
+ }
+ schema
+}
+
+pub struct SetConfigTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static SCFG_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetConfigTool {
+ fn def(&self) -> &ToolDef {
+ SCFG_DEF.get_or_init(|| ToolDef {
+ name: "set_config".into(),
+ description: "Write a setting into the persistent driver config. Values take \
+ effect immediately. Pass each setting as its own field.\n\n\
+ Known keys:\n\
+ - `capture_mode` (string: `vision` | `ax` | `som`) — DEPRECATED and ignored; \
+ `get_window_state` always returns both the UIA tree and a screenshot. Still \
+ accepted/persisted for back-compat but has no effect.\n\
+ - `max_image_dimension` (integer)\n\
+ - `experimental_pip` (boolean; persisted to config.json, applies on next daemon restart — Windows backend stubbed today, see issue #1729)\n\
+ - `experimental_pip_geometry` (string `WxH` or `WxH+X+Y`; persisted; applies on next daemon restart)\n\n\
+ - `cursor.motion.style`, `cursor.motion.timing`, `cursor.motion.effects.<name>` (`on`, `off`, or `default`) — saved default cursor motion for sessions started afterwards. `cursor.motion: \"default\"` clears every saved cursor motion default.\n\
+ - `cursor.badges` (`minimal` default, `verbose`, or `off`) — agent cursor badge; applies to every cursor at once.\n\
+ - `cursor.float` (resting float): `on` (default), `off`, or `default`. `cursor.marks` (action marks), `cursor.rotate` (turn toward travel): `on`, `off` (default), or `default`. All apply to every cursor at once.\n\n\
+ Returns the full updated config in the same shape as `get_config`.".into(),
+ input_schema: with_cursor_motion_config_properties(json!({"type":"object","properties":{
+ "capture_mode":{"type":"string","enum":["ax","vision"],"description":"DEPRECATED and ignored — get_window_state always returns both the UIA tree and a screenshot. Still accepted/persisted for back-compat but has no effect. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
+ "max_image_dimension":{"type":"integer","description":"Max dimension for screenshot resizing (0 = no limit)."},
+ "experimental_pip":{"type":"boolean","description":"Enables PiP preview (applies next restart)."},
+ "experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position."}
+ },"additionalProperties":false})),
+ read_only: false, destructive: false, idempotent: true, open_world: false,
+ })
+ }
+ async fn invoke(&self, args: Value) -> ToolResult {
+ if args.get("capture_scope").is_some() {
+ return ToolResult::error(
+ "config key 'capture_scope' is retired; select a window or desktop target on each action",
+ )
+ .with_structured(json!({
+ "code": "config_key_retired",
+ "key": "capture_scope",
+ "replacement": "action.target",
+ }));
+ }
+ let motion_keys = match cursor_overlay::motion_defaults::apply_config_args(&args) {
+ Ok(keys) => keys,
+ Err(message) => return ToolResult::error(message),
+ };
+ let mut cfg = self.state.config.write().unwrap();
+ let badge_key = match cursor_overlay::badge_mode::apply_config_args(&args) {
+ Ok(key) => key,
+ Err(message) => return ToolResult::error(message),
+ };
+ let option_keys = match cursor_overlay::cursor_options::apply_config_args(&args) {
+ Ok(keys) => keys,
+ Err(message) => return ToolResult::error(message),
+ };
+ let mut applied = !motion_keys.is_empty() || badge_key.is_some() || !option_keys.is_empty();
+ if let Some(mode) = args.get("capture_mode").and_then(|v| v.as_str()) {
+ cfg.capture_mode = mode.to_owned();
+ if let Err(e) =
+ pip_preview::write_config_key("capture_mode", Value::String(mode.to_owned()))
+ {
+ tracing::warn!("set_config: failed to persist capture_mode: {e}");
+ }
+ applied = true;
+ }
+ if let Some(dim) = args.get("max_image_dimension").and_then(|v| v.as_u64()) {
+ cfg.max_image_dimension = dim as u32;
+ if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
+ tracing::warn!("set_config: failed to persist max_image_dimension: {e}");
+ }
+ applied = true;
+ }
+ if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
+ {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
+ }
+ applied = true;
+ }
+ if let Some(geom) = args
+ .get("experimental_pip_geometry")
+ .and_then(|v| v.as_str())
+ {
+ if pip_preview::PipGeometry::parse(geom).is_none() {
+ return ToolResult::error(format!(
+ "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
+ ));
+ }
+ if let Err(e) = pip_preview::write_config_key(
+ "experimental_pip_geometry",
+ Value::String(geom.to_owned()),
+ ) {
+ return ToolResult::error(format!(
+ "failed to persist experimental_pip_geometry: {e}"
+ ));
+ }
+ applied = true;
+ }
+ if !applied {
+ return ToolResult::error("set_config did not contain a known field.");
+ }
+ // Emit the same pretty-JSON payload as `get_config` (matches Swift's
+ // `set_config` return shape — both tools echo the full config after).
+ let cursor_enabled = self
+ .state
+ .cursor_registry
+ .all_states()
+ .first()
+ .map(|s| s.config.enabled)
+ .unwrap_or(true);
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
+ let payload = json!({
+ "schema_version": 1,
+ "version": env!("CARGO_PKG_VERSION"),
+ "source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
+ "platform": "windows",
+ "capture_mode": cfg.capture_mode,
+ "max_image_dimension": cfg.max_image_dimension,
+ "agent_cursor": { "enabled": cursor_enabled },
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry,
+ });
+ let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
+ ToolResult::text(format!("✅ {pretty}")).with_structured(payload)
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_value.rs
new file mode 100644
index 0000000000..17bce1f961
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_value.rs
@@ -0,0 +1,317 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ element_target_rect, overlay_glide_to_target, pin_overlay_above, resolve_cursor_key, ToolState,
+};
+
+// ── set_value ─────────────────────────────────────────────────────────────────
+
+pub struct SetValueTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static SET_VALUE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetValueTool {
+ fn def(&self) -> &ToolDef {
+ SET_VALUE_DEF.get_or_init(|| ToolDef {
+ name: "set_value".into(),
+ // Description ported from Swift `SetValueTool.swift` with the
+ // Windows transport (UIA ValuePattern instead of AXValue).
+ // Native Win32 combo boxes are handled by crate::win32::combo
+ // (the analogue of Swift's AXPopUpButton special case).
+ description: "Set a value on a UIA element via the ValuePattern interface.\n\n\
+ Two semantic modes (matching Swift's split):\n\
+ - **Standard input** (text fields, sliders, combo box edit): writes the value \
+ directly through UIA `IUIAutomationValuePattern::SetValue`. This is the \
+ canonical Windows write path, equivalent to Swift's `AXValue` write.\n\
+ - **ComboBox**: on a native Win32 combo box (WinForms, MFC, dialogs) selects \
+ the option whose text matches `value` and sends the app the notifications a \
+ user's choice sends (CBN_SELENDOK, CBN_SELCHANGE), without focusing the window; \
+ an editable combo gets the text typed into its edit field. A drop-down list \
+ without that option is refused with the list of options. Other combo boxes get \
+ a UIA value write, which the app may not treat as a selection.\n\n\
+ For free-form text entry that the target accepts via keystrokes only, prefer \
+ `type_text` — UIA ValuePattern writes are ignored by some web inputs (same \
+ caveat as Swift's `AXValue`-vs-WebKit).".into(),
+ input_schema: json!({
+ "type":"object","required":["value"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "pid":{"type":"integer","description":"Target process ID. Optional with element_token (the token carries the pid)."},
+ "window_id":{"type":"integer","description":"HWND of the window. Omit when element_token is supplied (the token carries it)."},
+ "element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "value":{"type":"string","description":"New value. UIA will coerce to the element's native type."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ // Swift: idempotent: true (setting same value twice is idempotent).
+ read_only: false, destructive: true, idempotent: true, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let cursor_key = resolve_cursor_key(&args);
+ let raw_pid = args.get("pid").and_then(|v| v.as_i64());
+ if raw_pid.is_none() {
+ return ToolResult::error("Missing required integer field pid.");
+ }
+ let pid = raw_pid.unwrap() as u32;
+ let value = match args.get("value").and_then(|v| v.as_str()) {
+ Some(v) => v.to_owned(),
+ None => return ToolResult::error("Missing required string field value."),
+ };
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (index, window_id, admitted) = resolved.into_parts(None);
+ let (Some(hwnd), Some(idx), Some(admitted)) = (window_id, index, admitted) else {
+ return ToolResult::error("set_value requires an admitted element target.");
+ };
+ if !admitted.is_uia() {
+ return ToolResult::error(
+ "set_value requires a UIA element; MSAA value writes are unsupported.",
+ );
+ }
+
+ // No-raise guard: a WPF/XAML automation peer calls UIElement.Focus() →
+ // SetForegroundWindow during ValuePattern.SetValue. WS_EX_NOACTIVATE on
+ // the target makes that a no-op while the value is still set, so a
+ // background SetValue can't steal the user's foreground. Held across the
+ // whole write. (No-op for delivery_mode:"foreground".)
+ let _noact = if crate::input::delivery::DeliveryMode::from_args(&args)
+ != crate::input::delivery::DeliveryMode::Foreground
+ {
+ Some(crate::input::NoActivateGuard::arm(
+ windows::Win32::Foundation::HWND(hwnd as *mut _),
+ ))
+ } else {
+ None
+ };
+
+ // Glide the agent cursor onto the target element before writing its
+ // value, so a value write gets the same visual feedback as a click —
+ // the viewer can see *where* the agent is acting. No-op when the
+ // overlay is disabled or the element has no cached center.
+ {
+ let (cx, cy) = admitted.center;
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(Some(&admitted), (0, 0)),
+ )
+ .await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: cx as f64,
+ y: cy as f64,
+ },
+ );
+ }
+
+ let value_for_task = value.clone();
+ let result = tokio::task::spawn_blocking({
+ move || -> anyhow::Result<SetValueRoute> {
+ let value = value_for_task;
+ let _noact = _noact;
+ let ptr = admitted.as_ptr();
+ use windows::core::{Interface, BSTR};
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationValuePattern, UIA_ComboBoxControlTypeId,
+ UIA_ValuePatternId,
+ };
+ let elem = std::mem::ManuallyDrop::new(unsafe {
+ IUIAutomationElement::from_raw(ptr as *mut _)
+ });
+ // A native Win32 combo box (or its edit field): select the
+ // option and notify the app the way the control does. A UIA
+ // value write would only change the displayed text.
+ let native = unsafe { elem.CurrentNativeWindowHandle() }
+ .ok()
+ .and_then(|h| crate::win32::combo::combo_for_window(h.0 as isize));
+ if let Some(combo) = native {
+ let write = crate::win32::combo::set_value(combo, &value)?;
+ return Ok(SetValueRoute::Win32Combo { combo, write });
+ }
+ let is_combo = unsafe { elem.CurrentControlType() }
+ .is_ok_and(|control_type| control_type == UIA_ComboBoxControlTypeId);
+ // Try ValuePattern first (text inputs, editable combos, etc).
+ // The SetValue is shielded by the EnableWindow bypass: a
+ // Chromium/Electron (or XAML) SetValue handler self-foregrounds via
+ // SetForegroundWindow, which WS_EX_NOACTIVATE alone does not stop;
+ // disabling the host for the duration does, while the a11y-channel
+ // write still lands. (No-op shield for non-XAML/non-Chromium hosts.)
+ if let Ok(pattern) = unsafe { elem.GetCurrentPattern(UIA_ValuePatternId) } {
+ if let Ok(vp) = pattern.cast::<IUIAutomationValuePattern>() {
+ let set =
+ crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
+ vp.SetValue(&BSTR::from(value.as_str()))
+ });
+ if set.is_ok() {
+ return Ok(if is_combo {
+ SetValueRoute::ComboValuePattern
+ } else {
+ SetValueRoute::Pattern("ValuePattern")
+ });
+ }
+ }
+ }
+ // Fall through to RangeValuePattern for Sliders / ProgressBars /
+ // numeric ranges. RangeValue.SetValue takes a double, so coerce
+ // the string. Documented gap-closer (PR #1699 harness exposed).
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationRangeValuePattern, UIA_RangeValuePatternId,
+ };
+ if let Ok(pattern) = unsafe { elem.GetCurrentPattern(UIA_RangeValuePatternId) } {
+ if let Ok(rv) = pattern.cast::<IUIAutomationRangeValuePattern>() {
+ let parsed: f64 = value.parse().map_err(|_| {
+ anyhow::anyhow!(
+ "set_value: target element exposes RangeValuePattern (Slider / \
+ ProgressBar / numeric range). `value` must be a parseable f64; \
+ got {value:?}."
+ )
+ })?;
+ crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
+ rv.SetValue(parsed)
+ })?;
+ return Ok(SetValueRoute::Pattern("RangeValuePattern"));
+ }
+ }
+ anyhow::bail!(
+ "set_value: element [{idx}] does not implement ValuePattern or \
+ RangeValuePattern. For controls with TogglePattern (CheckBox) or \
+ SelectionItemPattern (RadioButton / ComboBoxItem), use the `click` \
+ tool instead."
+ );
+ }
+ })
+ .await;
+ match result {
+ Ok(Ok(SetValueRoute::Pattern(pattern_name))) => {
+ ToolResult::text(format!("✅ Set AXValue on [{idx}] (UIA {pattern_name})."))
+ }
+ Ok(Ok(SetValueRoute::ComboValuePattern)) => ToolResult::text(format!(
+ "Set the UIA value of combo box [{idx}] to '{value}'. This combo box is not a \
+ native Win32 one, and a UIA value write may change the shown text without \
+ the app treating it as a selection. Check with get_window_state; if the app \
+ did not react, click the combo box and then the option."
+ ))
+ .with_structured(json!({ "path": "ax", "verified": false, "effect": "unverifiable" })),
+ Ok(Ok(SetValueRoute::Win32Combo { combo, write })) => {
+ win32_combo_set_value_result(idx, &value, combo, write, &args)
+ }
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+/// How `set_value` wrote the value.
+enum SetValueRoute {
+ /// A UIA pattern on a non-combo element (`ValuePattern`, `RangeValuePattern`).
+ Pattern(&'static str),
+ /// UIA `ValuePattern` on a combo box that has no native Win32 window.
+ ComboValuePattern,
+ /// Win32 combo messages plus the app's change notifications.
+ Win32Combo {
+ combo: crate::win32::combo::Combo,
+ write: crate::win32::combo::ComboWrite,
+ },
+}
+
+fn win32_combo_set_value_result(
+ idx: usize,
+ value: &str,
+ combo: crate::win32::combo::Combo,
+ write: crate::win32::combo::ComboWrite,
+ args: &Value,
+) -> ToolResult {
+ use crate::win32::combo::ComboWrite;
+ use cua_driver_core::action_record::{
+ ActionEffect, ActionEvidence, ActionExecutionRecordBuilder, ActionTransport,
+ ActualDelivery, EvidenceKind, RequestedDelivery,
+ };
+ let kind = if combo.editable {
+ "editable combo box"
+ } else {
+ "drop-down combo box"
+ };
+ let (selected, unchanged, readback_text, readback_index) = match &write {
+ ComboWrite::NoSuchOption { options } => {
+ let message = format!(
+ "set_value: the {kind} [{idx}] has no option '{value}'; nothing was changed. \
+ Options: {options:?}."
+ );
+ return ToolResult::error(message).with_structured(json!({
+ "code": "option_not_found",
+ "effect": "none",
+ "options": options,
+ }));
+ }
+ ComboWrite::Applied {
+ selected,
+ unchanged,
+ readback_text,
+ readback_index,
+ } => (
+ *selected,
+ *unchanged,
+ readback_text.clone(),
+ *readback_index,
+ ),
+ };
+ let verified = crate::win32::combo::write_verified(&write, value);
+ let seen = readback_text.clone().unwrap_or_default();
+ let how = match (combo.editable, selected) {
+ (false, _) => "CB_SETCURSEL, then CBN_SELENDOK and CBN_SELCHANGE to the app",
+ (true, Some(_)) => {
+ "typed into its edit field, then CB_SETCURSEL with CBN_SELENDOK and CBN_SELCHANGE \
+ to the app"
+ }
+ (true, None) => "typed into its edit field (the app gets CBN_EDITCHANGE)",
+ };
+ let text = if unchanged {
+ format!("✅ The {kind} [{idx}] already reads '{seen}'; nothing was sent.")
+ } else if verified {
+ format!("✅ Set the {kind} [{idx}] to '{seen}' ({how}); read back '{seen}'.")
+ } else {
+ format!(
+ "Sent '{value}' to the {kind} [{idx}] ({how}), but it reads back '{seen}' \
+ (selected index {readback_index:?}). Check with get_window_state."
+ )
+ };
+ let requested = match args.get("delivery_mode").and_then(Value::as_str) {
+ Some("foreground") => RequestedDelivery::Foreground,
+ _ => RequestedDelivery::Background,
+ };
+ let effect = if verified {
+ ActionEffect::Confirmed
+ } else {
+ ActionEffect::Unverifiable
+ };
+ let mut record =
+ ActionExecutionRecordBuilder::new(effect, ActionTransport::WindowsPostMessage, requested)
+ // Window messages reach the control without focusing or raising it.
+ .actual_delivery(ActualDelivery::Background)
+ .detail(text.clone());
+ if verified {
+ record = record.evidence(ActionEvidence {
+ kind: EvidenceKind::ValueReadback,
+ detail: format!("combo box reads back '{seen}' (index {readback_index:?})"),
+ });
+ }
+ match record.build() {
+ Ok(record) => ToolResult::text(text).with_action_record(record),
+ Err(error) => ToolResult::error(format!("set_value: invalid action record: {error:?}")),
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/set_window_frame.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_window_frame.rs
new file mode 100644
index 0000000000..1bea1caf6c
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/set_window_frame.rs
@@ -0,0 +1,282 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::Value;
+
+// ── set_window_frame ──────────────────────────────────────────────────────
+
+fn outer_frame_for_visible_request(
+ requested: (i32, i32, i32, i32),
+ outer_before: (i32, i32, i32, i32),
+ visible_before: (i32, i32, i32, i32),
+) -> Result<(i32, i32, i32, i32), String> {
+ let (requested_x, requested_y, requested_width, requested_height) = requested;
+ let (outer_x, outer_y, outer_width, outer_height) = outer_before;
+ let (visible_x, visible_y, visible_width, visible_height) = visible_before;
+ let left = i64::from(visible_x) - i64::from(outer_x);
+ let top = i64::from(visible_y) - i64::from(outer_y);
+ let right = i64::from(outer_x) + i64::from(outer_width)
+ - (i64::from(visible_x) + i64::from(visible_width));
+ let bottom = i64::from(outer_y) + i64::from(outer_height)
+ - (i64::from(visible_y) + i64::from(visible_height));
+ let adjusted = (
+ i64::from(requested_x) - left,
+ i64::from(requested_y) - top,
+ i64::from(requested_width) + left + right,
+ i64::from(requested_height) + top + bottom,
+ );
+ if adjusted.2 <= 0 || adjusted.3 <= 0 {
+ return Err("visible-frame adjustment produced a non-positive outer size".to_owned());
+ }
+ let to_i32 = |value: i64| {
+ i32::try_from(value)
+ .map_err(|_| "visible-frame adjustment exceeded Windows coordinate range".to_owned())
+ };
+ Ok((
+ to_i32(adjusted.0)?,
+ to_i32(adjusted.1)?,
+ to_i32(adjusted.2)?,
+ to_i32(adjusted.3)?,
+ ))
+}
+
+pub struct SetWindowFrameTool;
+
+static SET_WINDOW_FRAME_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for SetWindowFrameTool {
+ fn def(&self) -> &ToolDef {
+ SET_WINDOW_FRAME_DEF.get_or_init(|| {
+ let contract = cua_driver_contract::tool_contract("set_window_frame")
+ .expect("set_window_frame contract");
+ ToolDef {
+ name: contract.name,
+ description: contract.description,
+ input_schema: contract.input_schema,
+ read_only: contract.annotations.read_only,
+ destructive: contract.annotations.destructive,
+ idempotent: contract.annotations.idempotent,
+ open_world: contract.annotations.open_world,
+ }
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_contract::SetWindowFrameInput;
+ use cua_driver_core::action_record::{
+ effect_from_value_readback, ActionEvidence, ActionExecutionRecord, ActionTransport,
+ ActualDelivery, EvidenceKind, RequestedDelivery,
+ };
+
+ let input: SetWindowFrameInput =
+ match cua_driver_core::tool_args::parse_typed_input("set_window_frame", args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let outcome = tokio::task::spawn_blocking(move || {
+ use windows::Win32::{
+ Foundation::{HWND, RECT},
+ Graphics::Dwm::{DwmGetWindowAttribute, DWMWA_EXTENDED_FRAME_BOUNDS},
+ UI::WindowsAndMessaging::{
+ GetWindowRect, GetWindowThreadProcessId, IsIconic, IsWindow, IsZoomed,
+ SetWindowPos, SWP_NOACTIVATE, SWP_NOZORDER,
+ },
+ };
+
+ let values = [input.x, input.y, input.width, input.height];
+ if values.iter().any(|value| !value.is_finite())
+ || input.width <= 0.0
+ || input.height <= 0.0
+ {
+ return Err(
+ "x/y must be finite and width/height must be finite positive numbers"
+ .to_owned(),
+ );
+ }
+ let to_i32 = |name: &str, value: f64| -> Result<i32, String> {
+ let rounded = value.round();
+ if rounded < i32::MIN as f64 || rounded > i32::MAX as f64 {
+ Err(format!("{name} is out of range for a Windows window coordinate"))
+ } else {
+ Ok(rounded as i32)
+ }
+ };
+ let x = to_i32("x", input.x)?;
+ let y = to_i32("y", input.y)?;
+ let width = to_i32("width", input.width)?;
+ let height = to_i32("height", input.height)?;
+ if width <= 0 || height <= 0 {
+ return Err("width/height must round to positive Windows window sizes".to_owned());
+ }
+ let hwnd = HWND(input.window_id as usize as *mut std::ffi::c_void);
+ if !unsafe { IsWindow(hwnd).as_bool() } {
+ return Err(format!("window_id {} is closed, stale, or invalid", input.window_id));
+ }
+ let mut owner_pid = 0_u32;
+ unsafe { GetWindowThreadProcessId(hwnd, Some(&mut owner_pid)) };
+ if owner_pid != input.pid {
+ return Err(format!(
+ "window_id {} belongs to pid {owner_pid}, not pid {}",
+ input.window_id, input.pid
+ ));
+ }
+ if unsafe { IsIconic(hwnd).as_bool() } {
+ return Err(format!(
+ "window_id {} is minimized; restore it with bring_to_front before setting its frame",
+ input.window_id
+ ));
+ }
+ if unsafe { IsZoomed(hwnd).as_bool() } {
+ return Err(format!(
+ "window_id {} is maximized; restore it before setting its frame",
+ input.window_id
+ ));
+ }
+ let read_frames = |hwnd: HWND| -> Result<
+ ((i32, i32, i32, i32), (i32, i32, i32, i32)),
+ String,
+ > {
+ let mut outer = RECT::default();
+ unsafe { GetWindowRect(hwnd, &mut outer) }
+ .map_err(|error| format!("could not read the current window frame: {error}"))?;
+ let outer = (
+ outer.left,
+ outer.top,
+ outer.right - outer.left,
+ outer.bottom - outer.top,
+ );
+ let mut visible = RECT::default();
+ let visible = unsafe {
+ DwmGetWindowAttribute(
+ hwnd,
+ DWMWA_EXTENDED_FRAME_BOUNDS,
+ &mut visible as *mut RECT as *mut _,
+ std::mem::size_of::<RECT>() as u32,
+ )
+ }
+ .ok()
+ .map(|()| {
+ (
+ visible.left,
+ visible.top,
+ visible.right - visible.left,
+ visible.bottom - visible.top,
+ )
+ })
+ .filter(|(_, _, width, height)| *width > 0 && *height > 0)
+ .unwrap_or(outer);
+ Ok((outer, visible))
+ };
+ let (outer_before, before) = read_frames(hwnd)?;
+ let requested = (x, y, width, height);
+ let outer_requested =
+ outer_frame_for_visible_request(requested, outer_before, before)?;
+ let mutation_error = unsafe {
+ SetWindowPos(
+ hwnd,
+ HWND::default(),
+ outer_requested.0,
+ outer_requested.1,
+ outer_requested.2,
+ outer_requested.3,
+ SWP_NOACTIVATE | SWP_NOZORDER,
+ )
+ }
+ .err()
+ .map(|error| format!("SetWindowPos failed: {error}"));
+
+ let mut observed = None;
+ for _ in 0..6 {
+ if let Ok((_, visible)) = read_frames(hwnd) {
+ observed = Some(visible);
+ if observed == Some(requested) {
+ break;
+ }
+ }
+ std::thread::sleep(std::time::Duration::from_millis(40));
+ }
+ let changed = observed.is_some_and(|observed| before != observed);
+ Ok((
+ requested,
+ outer_requested,
+ observed,
+ changed,
+ mutation_error,
+ ))
+ })
+ .await;
+
+ let (requested, outer_requested, observed, changed, mutation_error) = match outcome {
+ Ok(Ok(outcome)) => outcome,
+ Ok(Err(error)) => return ToolResult::error(format!("set_window_frame: {error}")),
+ Err(error) => {
+ return ToolResult::error(format!(
+ "set_window_frame: blocking task failed: {error}"
+ ));
+ }
+ };
+ let confirmed = observed == Some(requested);
+ let effect = effect_from_value_readback(confirmed, changed, observed.is_some());
+ let mut record = ActionExecutionRecord::builder(
+ effect,
+ ActionTransport::WindowsSetWindowPos,
+ RequestedDelivery::NotApplicable,
+ )
+ .actual_delivery(ActualDelivery::NotApplicable)
+ .detail(format!(
+ "requested_visible={requested:?} requested_outer={outer_requested:?} observed_visible={observed:?} mutation_error={mutation_error:?}"
+ ));
+ if observed.is_some() {
+ record = record.evidence(ActionEvidence {
+ kind: EvidenceKind::ValueReadback,
+ detail: if confirmed {
+ "DWM visible-frame bounds matched the requested frame".into()
+ } else {
+ "DWM visible-frame bounds did not match the requested frame".into()
+ },
+ });
+ }
+ ToolResult::text(if confirmed {
+ "Set and verified the requested window frame."
+ } else if observed.is_none() {
+ "The native window mutation was attempted, but its resulting frame could not be read back."
+ } else {
+ "The window frame did not settle at the requested geometry."
+ })
+ .with_action_record(record.build().expect("set_window_frame record is valid"))
+ }
+}
+
+#[cfg(test)]
+mod set_window_frame_geometry_tests {
+ use super::outer_frame_for_visible_request;
+
+ #[test]
+ fn compensates_for_invisible_resize_borders() {
+ assert_eq!(
+ outer_frame_for_visible_request(
+ (65, 52, 872, 626),
+ (40, 40, 886, 633),
+ (47, 40, 872, 626),
+ )
+ .unwrap(),
+ (58, 52, 886, 633)
+ );
+ }
+
+ #[test]
+ fn leaves_outer_request_unchanged_when_dwm_bounds_are_unavailable() {
+ assert_eq!(
+ outer_frame_for_visible_request(
+ (-25, 10, 640, 480),
+ (20, 30, 800, 600),
+ (20, 30, 800, 600),
+ )
+ .unwrap(),
+ (-25, 10, 640, 480)
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text.rs
new file mode 100644
index 0000000000..35b8117219
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text.rs
@@ -0,0 +1,982 @@
+use async_trait::async_trait;
+use cua_driver_contract::TypeTextInput;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+ tool_args::parse_typed_projection,
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::{
+ element_target_rect, focus_by_pixel, focus_cached_element_for_foreground,
+ overlay_glide_to_target, pin_overlay_above, position_keyboard_cursor, resolve_cursor_key,
+ resolve_onscreen_point_with_scroll, ToolState,
+};
+
+// ── type_text ─────────────────────────────────────────────────────────────────
+
+pub struct TypeTextTool {
+ pub(crate) state: Arc<ToolState>,
+}
+
+static TYPE_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for TypeTextTool {
+ fn def(&self) -> &ToolDef {
+ TYPE_DEF.get_or_init(|| ToolDef {
+ name: "type_text".into(),
+ // Description ported from Swift `TypeTextTool.swift` with the
+ // Windows transport note. Swift tries AXSetAttribute(kAXSelectedText)
+ // first, falls back to character-by-character CGEvent synthesis.
+ // Windows uses character-by-character PostMessage(WM_CHAR) to the
+ // window's focused child — there's no UIA equivalent of
+ // AXSelectedText wired up yet, so it's always the synthesis path.
+ description: "Insert text into the target pid via character-by-character \
+ PostMessage(WM_CHAR) to the focused window. No focus steal.\n\n\
+ Special keys (Return, Escape, arrows, Tab) go through `press_key` / \
+ `hotkey` — they are not text.\n\n\
+ **Routing on Windows.** `element_token` selects a specific UIA element \
+ and writes its ValuePattern value. For no-element foreground typing, a \
+ separate capability-gated path can use the exact focused UIA Document \
+ only when it is writable and empty. \
+ Eligibility depends on control capabilities and live focus, not app names. \
+ Chromium/Electron windows keep their existing boundary because a UIA \
+ value echo does not prove that the renderer accepted the text. The \
+ path reads the value and caret back; after SetValue starts, an \
+ unverifiable result requires a fresh observation before any retry and \
+ is never replayed through native input. This replaces the full \
+ ValuePattern value, so rich-text formatting and native undo behavior \
+ are not guaranteed; single-line Edit controls \
+ keep the existing input route. For other XAML / WinUI3 / UWP cases, \
+ use `element_token` from `get_window_state`; background WM_CHAR does \
+ not reach their CoreInput dispatcher. Native ConsoleHost on Windows ARM64 is \
+ hard-refused because it can \
+ accept synthesized Unicode events without delivering them; use a process \
+ or PTY setup channel instead. Legacy Win32 apps still use the PostMessage \
+ path, preserving the no-focus-steal property.\n\n\
+ `delay_ms` (0–200, default 30) spaces successive characters on the \
+ PostMessage path so autocomplete and IME can keep up. Ignored on the \
+ UIA path (SetValue is atomic).".into(),
+ input_schema: json!({
+ "type":"object","required":["text"],"properties":{
+ "session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to type into the current foreground application."},
+ "pid":{"type":"integer","description":"Target process ID."},
+ "text":{"type":"string","description":"Text to insert at the focused element's cursor."},
+ "window_id":{"type":"integer","description":"HWND of the target window. Omit when element_token is supplied (the token carries it)."},
+ "element_token":{"type":"string","pattern":"^s[0-9a-f]{8}:[0-9]+$","description":"Opaque element handle from get_window_state. When supplied, type_text writes through UIA ValuePattern and reads that exact element back by handle. The shared ActionResult is confirmed only when the complete expected value is synchronously visible. If SetValue succeeds but read-back is stale or unavailable, the result is unverifiable with no escalation; take a fresh snapshot before retrying because deferred providers may publish only after this call returns."},
+ "x":{"type":"number","description":"Window-local screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the UIA/WM_CHAR path can't reach. Read straight off the get_window_state PNG, same convention as click."},
+ "y":{"type":"number","description":"Window-local screenshot-pixel Y of the field (see x)."},
+ "delay_ms":{"type":"integer","minimum":0,"maximum":200,"description":"Milliseconds between characters. Default 30."},
+ "delivery_mode": crate::input::delivery::delivery_mode_schema()
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ let cursor_key = resolve_cursor_key(&args);
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let input = match parse_typed_projection::<TypeTextInput>("type_text", &args) {
+ Ok(input) => input,
+ Err(result) => return result,
+ };
+ let text =
+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&input.text)
+ .into_owned();
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let text_len = text.chars().count();
+ if args.get("element_token").is_none()
+ && args.get("x").is_none()
+ && args.get("y").is_none()
+ {
+ if let Some(result) = try_focused_document_route(hwnd, None, &text, text_len).await
+ {
+ return result;
+ }
+ }
+ return match tokio::task::spawn_blocking(move || {
+ crate::input::send_text_synthesized(hwnd, &text)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) into the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "characters": text_len,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop type task failed: {error}")),
+ };
+ }
+ let text_raw = match args.require_str("text") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ // Strip trailing agent-protocol closing tags before delivery —
+ // catches the case where an LLM hallucinated its own tool-
+ // invocation tags into the text param (see text_sanitize docs).
+ // Returns Cow::Borrowed on the no-match fast path so the common
+ // case is allocation-free.
+ let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
+ .into_owned();
+ let raw_pid = match args.require_i64("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let pid = raw_pid as u32;
+ let resolved = match self.state.snapshots.resolve(pid as i32, &args) {
+ Ok(r) => r,
+ Err(e) => return e,
+ };
+ let (elem_idx, hwnd_opt, admitted) = resolved.into_parts(args.opt_u64("window_id"));
+ let delivery = DeliveryMode::from_args(&args);
+
+ // ── px form: focus by pixel-click, then type into the focused element ──
+ // Pass x,y (no element_token) for an *element px action*: pixel-click the
+ // field to give the Chromium/Electron renderer the real keyboard focus the
+ // UIA/WM_CHAR path can't establish on its own, then fall through to the
+ // focused-window WM_CHAR path below (which lands once focused). Reuses
+ // ClickTool's exact coordinate translation + delivery_mode. Mirrors macOS.
+ {
+ let px = args.get("x").and_then(|v| v.as_f64());
+ let py = args.get("y").and_then(|v| v.as_f64());
+ if let (Some(cx), Some(cy)) = (px, py) {
+ if elem_idx.is_some() {
+ return ToolResult::error(
+ "Pass either element_token (ax) or x,y (px) to type_text, not both.",
+ );
+ }
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if let Err(e) = focus_by_pixel(
+ &self.state,
+ pid,
+ hwnd_opt,
+ cx,
+ cy,
+ delivery.is_foreground(),
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ false,
+ )
+ .await
+ {
+ return e;
+ }
+ // elem_idx stays None → the type path below writes to the now-
+ // focused element via the WM_CHAR (key_events) rung.
+ }
+ }
+
+ // Same no-raise guard as click: a XAML/WPF ValuePattern.SetValue handler
+ // calls UIElement.Focus()→SetForegroundWindow; WS_EX_NOACTIVATE on the
+ // target makes that a no-op while the value still gets set. Safe because
+ // type_text never uses the SendInput foreground-swap path. No-op for
+ // delivery_mode:"foreground" and when no window_id was given.
+ let _noact = match (delivery != DeliveryMode::Foreground, hwnd_opt) {
+ (true, Some(h)) => Some(crate::input::NoActivateGuard::arm(
+ windows::Win32::Foundation::HWND(h as *mut _),
+ )),
+ _ => None,
+ };
+ let _delay_ms = args.u64_or("delay_ms", 30);
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::win32::list_windows(Some(pid))
+ }
+ })
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ if let Some(element) = admitted.as_ref() {
+ let (cx, cy) = element.center;
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(Some(element), (0, 0)),
+ )
+ .await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, cx as f64, cy as f64);
+ } else if args.get("x").and_then(Value::as_f64).is_none()
+ || args.get("y").and_then(Value::as_f64).is_none()
+ {
+ // Untargeted text still gives a named session visible feedback:
+ // its remembered position, or the window centre on the first action.
+ position_keyboard_cursor(&self.state, &args, hwnd, None).await;
+ }
+ let text_len = text.chars().count();
+
+ // Native ConsoleHost can accept every synthesized Unicode event while
+ // dropping the text at the prompt (observed on Windows 11 ARM64).
+ // Classify terminal delivery before the foreground SendInput branch so
+ // an explicit foreground retry cannot bypass the hard refusal.
+ match crate::terminal::type_text_policy(hwnd, delivery.is_foreground()) {
+ crate::terminal::TypeTextPolicy::Unsupported => {
+ return crate::terminal::native_consolehost_type_text_error(hwnd);
+ }
+ crate::terminal::TypeTextPolicy::ForegroundRequired => {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::TextInput,
+ );
+ }
+ crate::terminal::TypeTextPolicy::Supported => {}
+ }
+
+ // Refuse known background drops before the final WM_CHAR path. WPF is
+ // conditional: indexed text still has a working UIA ValuePattern route,
+ // while unindexed text would be posted to the top-level and disappear.
+ if delivery == DeliveryMode::Background
+ && (crate::input::delivery::would_be_silently_dropped(hwnd, EventKind::TextInput)
+ || (elem_idx.is_none() && crate::input::delivery::is_wpf_target_window(hwnd)))
+ {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::TextInput,
+ );
+ }
+ // delivery_mode:"foreground" — explicit SendInput Unicode with a brief
+ // foreground swap (send_text_synthesized), symmetric with press_key's
+ // foreground path. This is the rung that lands on VCL/LibreOffice
+ // document grids and other targets where PostMessage WM_CHAR is
+ // silently dropped. Honest by construction: if the foreground swap is
+ // rejected (daemon not at UIAccess integrity), it returns an error
+ // rather than a false success.
+ if delivery == DeliveryMode::Foreground {
+ if elem_idx.is_none()
+ && args.get("element_token").is_none()
+ && args.get("x").is_none()
+ && args.get("y").is_none()
+ {
+ if let Some(result) =
+ try_focused_document_route(hwnd, Some(pid), &text, text_len).await
+ {
+ return result;
+ }
+ }
+ // Resolve the optional click fallback before entering the atomic
+ // activation/focus/input transaction. The focus itself happens
+ // only after exact top-level foreground is confirmed.
+ let focus_target = if let Some(idx) = elem_idx {
+ let (cx, cy) = match admitted.as_ref().map(|element| element.center) {
+ Some(center) => center,
+ None => {
+ return ToolResult::error(format!(
+ "Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first."
+ ))
+ }
+ };
+ let (cx, cy) = match resolve_onscreen_point_with_scroll(
+ &admitted,
+ hwnd,
+ idx as usize,
+ cx,
+ cy,
+ "foreground typing",
+ ) {
+ Ok(point) => point,
+ Err(result) => return result,
+ };
+ Some((idx as usize, (cx, cy)))
+ } else {
+ None
+ };
+ let text_fg = text.clone();
+ let r = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ crate::input::send_text_synthesized_after_focus(hwnd, &text_fg, || {
+ if let Some((idx, point)) = focus_target {
+ focus_cached_element_for_foreground(&admitted, hwnd, idx, Some(point))?;
+ }
+ Ok(())
+ })
+ }
+ })
+ .await;
+ return match r {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Typed {text_len} char(s) on pid {raw_pid} via SendInput (delivery_mode:foreground)."
+ ))
+ .with_structured(serde_json::json!({
+ "path": "key_events",
+ "characters": text_len,
+ // Foreground SendInput delivered, but with no read-back the
+ // driver can't confirm the effect → unverifiable. No escalation:
+ // foreground IS the escalated rung (mirrors macOS PATH_KEY_EVENTS_FG).
+ "effect": "unverifiable",
+ })),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ };
+ }
+
+ // Pin the agent-cursor overlay above the target window so the synthetic
+ // cursor stays sandwiched at z+1 of the type target for the full
+ // duration of the keystrokes (both XAML/UIA and PostMessage paths).
+ pin_overlay_above(&cursor_key, hwnd);
+
+ // Glide the agent cursor onto the field being typed into, so the viewer
+ // can see *where* the agent is typing — same visual feedback as a click.
+ // Only when an element_index is supplied (we have its cached center);
+ // the focused-element path has no resolvable position to point at.
+ if let Some(element) = admitted.as_ref() {
+ let (cx, cy) = element.center;
+ overlay_glide_to_target(
+ &cursor_key,
+ cx as f64,
+ cy as f64,
+ element_target_rect(Some(element), (0, 0)),
+ )
+ .await;
+ crate::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::ClickPulse {
+ x: cx as f64,
+ y: cy as f64,
+ },
+ );
+ }
+
+ // CUA-543 routing: PostMessage WM_CHAR doesn't reach modern
+ // XAML / WinUI3 hosts. When the target is one of those AND the
+ // caller has supplied an element_index, route through UIA
+ // `ValuePattern.SetValue` — same code path the `set_value` tool
+ // uses, which we've verified works on modern Notepad / WinUI3.
+ // Legacy Win32 stays on the PostMessage path so the no-focus-
+ // steal property is preserved.
+ // ── Automatic routing — the caller need not know the framework. ──
+ // 1. With an element_index, try UIA ValuePattern.SetValue first: it works
+ // for WPF / WinForms / UWP / XAML and many web inputs, sets the value
+ // through the accessibility channel (no keystrokes), and the `_noact`
+ // guard blocks any self-foreground — so it never raises. Auto-falls-
+ // back to the WM_CHAR path below if the element has no ValuePattern
+ // (most legacy Win32 EDITs consume WM_CHAR fine without focus steal).
+ if let Some(idx) = elem_idx {
+ let text_for_uia = text.clone();
+ let set_result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ // Retain the element under the cache lock so a concurrent
+ // get_window_state snapshot-replace on the same (pid, hwnd)
+ // can't Release it to zero while this SetValue is in flight.
+ // The guard is held for the whole closure.
+ let element_guard = admitted.as_ref().filter(|element| element.is_uia())?;
+ let ptr = element_guard.as_ptr();
+ use windows::core::{Interface, BSTR};
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationValuePattern, UIA_ValuePatternId,
+ };
+ let elem: IUIAutomationElement =
+ unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
+ let result = (|| -> anyhow::Result<(String, String)> {
+ let pattern = unsafe { elem.GetCurrentPattern(UIA_ValuePatternId) }?;
+ let vp: IUIAutomationValuePattern = pattern.cast()?;
+ // Shield the SetValue against host self-foreground. A
+ // Chromium/Electron (or XAML) ValuePattern.SetValue handler
+ // calls SetForegroundWindow(self), which WS_EX_NOACTIVATE
+ // does NOT stop; the EnableWindow shield does (disabled
+ // top-level can't be foregrounded) while the a11y-channel
+ // SetValue still lands. Same fix as the UIA Invoke path.
+ crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
+ // ValuePattern has no caret/selection insertion API. Preserve
+ // the current document and append as the deterministic
+ // background fallback; replacing the whole value violates
+ // type_text's insertion contract for editors.
+ let current = vp
+ .CurrentValue()
+ .map(|value| value.to_string())
+ .unwrap_or_default();
+ let inserted = format!("{current}{text_for_uia}");
+ vp.SetValue(&BSTR::from(inserted.as_str()))?;
+ Ok((current, inserted))
+ })
+ })()
+ .ok();
+ std::mem::forget(elem);
+ result
+ }
+ })
+ .await
+ .unwrap_or(None);
+ if let Some((before, expected)) = set_result {
+ // Read the value back by element handle — focus-independent
+ // (works regardless of which window is foreground), proving the
+ // a11y write actually took rather than trusting SetValue's
+ // return alone.
+ let verify = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ let value = read_cached_element_value(&admitted);
+ classify_value_write_readback(value.as_deref(), &before, &expected)
+ }
+ })
+ .await
+ .unwrap_or("pending");
+ // SURFACE-AWARE VERIFICATION (mirrors macOS type_text). On
+ // Electron/Chromium web inputs the UIA layer accepts a
+ // `ValuePattern.SetValue` and echoes it straight back through
+ // `ValuePattern.CurrentValue` while the renderer/DOM never observes
+ // it — so a read-back "confirm" is a shim echo, not ground truth
+ // (the Windows analogue of the Slack-search AX false-confirm).
+ // Refuse to claim verified there: the screenshot is the only truth
+ // and the next rung is the element px action. Probe ONLY when it
+ // would otherwise confirm via the UIA value path, so native UIA
+ // types (WPF / WinForms / UWP / XAML) pay nothing. The signal is
+ // the window class — Chromium and every Electron host expose
+ // `Chrome_WidgetWin_*` (see `is_chromium_target_window`).
+ let ax_echo_surface =
+ verify == "confirmed" && crate::input::is_chromium_target_window(hwnd);
+ let verified = verify == "confirmed" && !ax_echo_surface;
+ let (mark, note) = if verified {
+ ("✅ Wrote", String::new())
+ } else if ax_echo_surface {
+ (
+ "📨 Sent (unverified)",
+ " — Electron/web surface: the UIA layer accepts and echoes the \
+ write but the renderer may not have observed it, so the driver \
+ cannot confirm via UIA. Verify via the screenshot; if it didn't \
+ land, re-type with the px form (pass x,y to pixel-focus the \
+ field)."
+ .to_string(),
+ )
+ } else {
+ (
+ "📨 Accepted (verification pending)",
+ " — the provider may publish its new value only after this call returns. Take a fresh snapshot before retrying."
+ .to_string(),
+ )
+ };
+ return ToolResult::text(format!(
+ "{mark} {text_len} char(s) on pid {raw_pid} via UIA ValuePattern \
+ (element_index=[{idx}]; verify: {verify}).{note}"
+ ))
+ .with_structured({
+ // `effect` mirrors the UIA read-back: a TRUSTED positive
+ // read-back is "confirmed"; a deferred provider or an Electron UIA echo
+ // that we refuse to trust is "unverifiable". Only the echo
+ // recommends pixel escalation.
+ let mut s = value_write_structured_result(text_len, verify, verified);
+ if ax_echo_surface {
+ // Electron UIA echo → the element px action is the next rung,
+ // not foreground (it's a renderer-focus problem, not an
+ // activation one).
+ s["escalation"] = serde_json::json!({
+ "recommended": "px",
+ "reason": "Electron/web surface — the UIA write was echoed \
+ but the renderer may not have observed it. \
+ Confirm via the screenshot; if it didn't land, \
+ re-type with the element px action (pass x,y to \
+ pixel-focus the field, then type)."
+ });
+ }
+ s
+ });
+ }
+ // ValuePattern unavailable → fall through to the WM_CHAR path.
+ }
+
+ // 2. No element_index on a WPF target: WM_CHAR is dropped and there's no
+ // element to SetValue. The only background path was a cloaked focus
+ // grab (SendInput), which the macOS-aligned contract forbids in
+ // background. Prefer supplying an element_index (path 1, UIA SetValue —
+ // never raises); otherwise surface background_unavailable so the agent
+ // escalates to delivery_mode:"foreground".
+ if elem_idx.is_none() && crate::input::delivery::is_wpf_target_window(hwnd) {
+ return crate::input::delivery::background_unavailable_error(
+ hwnd,
+ EventKind::TextInput,
+ );
+ }
+
+ // 3. Legacy Win32 / GDI / Chromium-IME — PostMessage WM_CHAR, no focus steal.
+ //
+ // Read-back verification (mirrors macOS type_text): WM_CHAR is silently
+ // dropped by some focused targets (notably a VCL/LibreOffice Calc grid
+ // not in cell-edit mode), and PostMessage reports success regardless. So
+ // snapshot the focused element's UIA value before and after; if it did
+ // not change we can't confirm the text landed and report "Sent
+ // (unverified)" with the foreground-escalation hint, instead of a blind
+ // success. If the focused element exposes no ValuePattern we likewise
+ // can't confirm (conservative: under-claim rather than false-succeed).
+ let text_for_post = text.clone();
+ let verify_pid = pid;
+ let verify_idx = elem_idx.map(|i| i as usize);
+ let result = tokio::task::spawn_blocking({
+ let admitted = admitted.clone();
+ move || {
+ let _admission = &admitted;
+ // Prefer a focus-independent read of the *specific* cached element
+ // when we have its index; only fall back to the (flaky, focus-
+ // dependent) system focused element when typing into "whatever is
+ // focused" with no element_token.
+ let read = |idx: Option<usize>| match idx {
+ Some(_) => read_cached_element_value(&admitted),
+ None => read_focused_value_uia(verify_pid),
+ };
+ let before = read(verify_idx);
+ let post_res = crate::input::post_type_text(hwnd, &text_for_post);
+ std::thread::sleep(std::time::Duration::from_millis(40));
+ let after = read(verify_idx);
+ let observed = post_message_readback_observed(
+ before.as_deref(),
+ after.as_deref(),
+ &text_for_post,
+ );
+ (post_res, before, after, observed)
+ }
+ })
+ .await;
+ match result {
+ Ok((Ok(()), before, after, observed)) => {
+ // Three-way verdict. Crucially, "couldn't read the field" is NOT
+ // the same as "the text didn't land": on Windows, UIA's
+ // GetFocusedElement is system-wide (unlike macOS's per-app
+ // AXFocusedUIElement), so when the target isn't the foreground
+ // app we cannot read it back even though a PostMessage WM_CHAR
+ // may have landed fine. Treating unreadable as failure would
+ // spuriously push agents to foreground and break the
+ // background-first contract. A changed value containing the
+ // requested text is useful evidence, but the insertion point is
+ // unknowable on WM_CHAR. Keep the action unverifiable so callers
+ // prove final state from a fresh snapshot before any retry.
+ match (&before, &after) {
+ (Some(_), Some(_)) if observed => ToolResult::text(format!(
+ "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage \
+ ({_delay_ms}ms delay; the field changed and now contains the \
+ requested text, but the insertion point is not known). Take a \
+ fresh snapshot to verify the intended final value before retrying."
+ ))
+ .with_structured(changed_contains_post_message_result(text_len)),
+ (Some(_), Some(_)) => ToolResult::text(format!(
+ "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage, but \
+ the focused field's value did not contain the requested text \
+ after dispatch — delivery was incomplete or dropped (e.g. a \
+ VCL/LibreOffice grid not in cell-edit mode). \
+ Retry with delivery_mode:\"foreground\"."
+ ))
+ .with_structured(serde_json::json!({
+ "path": "key_events", "characters": text_len,
+ "verified": false, "verify": "unchanged",
+ "effect": "unverifiable",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "background insert could not be confirmed — re-call \
+ with delivery_mode:\"foreground\" if a screenshot \
+ shows the text didn't land."
+ },
+ })),
+ _ => ToolResult::text(format!(
+ "📨 Sent {text_len} char(s) to pid {raw_pid} via PostMessage \
+ ({_delay_ms}ms delay; not verified — could not read \
+ the focused field back, e.g. the target isn't foreground or \
+ exposes no ValuePattern). If it didn't land, retry with \
+ delivery_mode:\"foreground\"."
+ ))
+ .with_structured(serde_json::json!({
+ "path": "key_events", "characters": text_len,
+ "verified": false, "verify": "unreadable",
+ "effect": "unverifiable",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "background insert could not be confirmed — re-call \
+ with delivery_mode:\"foreground\" if a screenshot \
+ shows the text didn't land."
+ },
+ })),
+ }
+ }
+ Ok((Err(e), _, _, _)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+async fn try_focused_document_route(
+ hwnd: u64,
+ expected_pid: Option<u32>,
+ text: &str,
+ text_len: usize,
+) -> Option<ToolResult> {
+ if crate::input::is_chromium_target_window(hwnd) {
+ return None;
+ }
+ let text = text.to_owned();
+ let outcome = tokio::task::spawn_blocking(move || {
+ crate::uia::focused_document::try_type_focused_document_append(hwnd, expected_pid, &text)
+ })
+ .await
+ .unwrap_or(
+ crate::uia::focused_document::FocusedDocumentTypeOutcome::Unknown(
+ "focused UIA task failed",
+ ),
+ );
+ focused_document_tool_result(outcome, text_len)
+}
+
+fn focused_document_tool_result(
+ outcome: crate::uia::focused_document::FocusedDocumentTypeOutcome,
+ text_len: usize,
+) -> Option<ToolResult> {
+ use crate::uia::focused_document::FocusedDocumentTypeOutcome;
+
+ match outcome {
+ FocusedDocumentTypeOutcome::NotApplicable => None,
+ FocusedDocumentTypeOutcome::Confirmed => Some(
+ ToolResult::text(format!(
+ "Wrote and verified {text_len} character(s) in the focused Document; caret is at the document end."
+ ))
+ .with_structured(json!({
+ "path": "ax",
+ "characters": text_len,
+ "verified": true,
+ "verify": "confirmed",
+ "effect": "confirmed",
+ })),
+ ),
+ FocusedDocumentTypeOutcome::Unknown(stage) => Some(
+ ToolResult::text(format!(
+ "Focused Document text may have been written but could not be verified ({stage}). Inspect the current document before retrying."
+ ))
+ .with_structured(json!({
+ "path": "ax",
+ "characters": text_len,
+ "verified": false,
+ "verify": "unverifiable",
+ "effect": "unverifiable",
+ "retryable": false,
+ })),
+ ),
+ }
+}
+
+#[cfg(test)]
+mod focused_document_result_tests {
+ use super::focused_document_tool_result;
+ use crate::uia::focused_document::FocusedDocumentTypeOutcome;
+ use cua_driver_contract::{ActionEffect, ActionRoute};
+ use cua_driver_core::action_record::{ActionExecutionRecord, ActionTransport};
+ use cua_driver_core::protocol::ToolResult;
+ use serde_json::json;
+
+ fn normalize(result: ToolResult) -> cua_driver_contract::ActionResult {
+ let structured = result
+ .structured_content
+ .expect("focused-document result includes structured evidence");
+ let record = ActionExecutionRecord::from_legacy(
+ "type_text",
+ &json!({"scope": "desktop"}),
+ &structured,
+ )
+ .expect("the action dispatcher recognizes the UIA value transport");
+ assert_eq!(record.transport, ActionTransport::WindowsUiaValue);
+ record
+ .public_result()
+ .expect("focused-document action result normalizes")
+ }
+
+ #[test]
+ fn focused_document_results_normalize_to_accessibility_with_truthful_effects() {
+ let confirmed = focused_document_tool_result(FocusedDocumentTypeOutcome::Confirmed, 12)
+ .expect("confirmed focused-document output");
+ let confirmed = normalize(confirmed);
+ assert_eq!(confirmed.route, ActionRoute::Accessibility);
+ assert_eq!(confirmed.effect, ActionEffect::Confirmed);
+
+ let unknown = focused_document_tool_result(
+ FocusedDocumentTypeOutcome::Unknown("value read-back failed"),
+ 12,
+ )
+ .expect("unknown focused-document output");
+ assert_eq!(
+ unknown.structured_content.as_ref().unwrap()["retryable"],
+ false
+ );
+ let unknown = normalize(unknown);
+ assert_eq!(unknown.route, ActionRoute::Accessibility);
+ assert_eq!(unknown.effect, ActionEffect::Unverifiable);
+ }
+}
+
+fn post_message_readback_observed(
+ before: Option<&str>,
+ after: Option<&str>,
+ requested_text: &str,
+) -> bool {
+ matches!(
+ (before, after),
+ (Some(before), Some(after))
+ if before != after && after.contains(requested_text)
+ )
+}
+
+fn value_write_structured_result(
+ text_len: usize,
+ verify: &str,
+ verified: bool,
+) -> serde_json::Value {
+ serde_json::json!({
+ "path": "ax",
+ "characters": text_len,
+ "verified": verified,
+ "verify": verify,
+ "effect": if verified { "confirmed" } else { "unverifiable" },
+ })
+}
+
+fn changed_contains_post_message_result(text_len: usize) -> serde_json::Value {
+ serde_json::json!({
+ "path": "key_events",
+ "characters": text_len,
+ "verified": false,
+ "verify": "changed_contains",
+ "effect": "unverifiable",
+ })
+}
+
+#[cfg(test)]
+mod post_message_readback_tests {
+ use super::{changed_contains_post_message_result, post_message_readback_observed};
+
+ #[test]
+ fn observes_only_a_changed_value_containing_the_requested_text() {
+ assert!(post_message_readback_observed(
+ Some("prefix"),
+ Some("prefixhello"),
+ "hello"
+ ));
+ assert!(!post_message_readback_observed(None, None, "hello"));
+ assert!(!post_message_readback_observed(
+ Some("hello"),
+ Some("hello"),
+ "hello"
+ ));
+ assert!(!post_message_readback_observed(
+ Some(""),
+ Some("h"),
+ "hello"
+ ));
+ assert!(!post_message_readback_observed(
+ Some("before"),
+ Some("different"),
+ "hello"
+ ));
+ }
+
+ #[test]
+ fn changed_contains_remains_unverifiable_without_retry_escalation() {
+ let result = changed_contains_post_message_result(5);
+ assert_eq!(result["effect"], "unverifiable");
+ assert_eq!(result["verify"], "changed_contains");
+ assert_eq!(result["verified"], false);
+ assert!(result.get("escalation").is_none());
+
+ let record = cua_driver_core::action_record::ActionExecutionRecord::from_legacy(
+ "type_text",
+ &serde_json::json!({"delivery_mode": "background"}),
+ &result,
+ )
+ .expect("changed-and-contains PostMessage result should normalize");
+ let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
+ .expect("serialize ActionResult");
+ assert_eq!(public["effect"], "unverifiable");
+ assert!(public.get("escalation").is_none());
+ assert!(public.get("evidence").is_none());
+ }
+}
+
+fn classify_value_write_readback(
+ value: Option<&str>,
+ before: &str,
+ expected: &str,
+) -> &'static str {
+ match value {
+ Some(value) if value == expected && value != before => "confirmed",
+ // SetValue returned success. Some providers publish their new Value only
+ // after this tool call unwinds, so an old or unreadable synchronous value
+ // is pending verification, not evidence that the write failed.
+ _ => "pending",
+ }
+}
+
+/// Read a **specific cached element's** text value by its `element_index`,
+/// for type_text read-back verification. Tries `ValuePattern.CurrentValue`
+/// then falls back to `TextPattern` document text. Unlike
+/// [`read_focused_value_uia`] this is **focus-independent**: it reads the exact
+/// element the caller typed into (by handle, from the per-(pid,window) cache),
+/// so it works whether or not the target is the foreground window. Returns
+/// `None` if the index isn't cached or the element exposes no readable text
+/// pattern. Mirrors the cache-retain + `mem::forget` discipline of the
+/// ValuePattern.SetValue path so the cached COM ref isn't released.
+fn read_cached_element_value(
+ admitted: &Option<crate::uia::snapshot::RetainedElement>,
+) -> Option<String> {
+ use windows::core::Interface;
+ use windows::Win32::UI::Accessibility::{
+ IUIAutomationElement, IUIAutomationTextPattern, IUIAutomationValuePattern,
+ UIA_TextPatternId, UIA_ValuePatternId,
+ };
+ let guard = admitted.as_ref().filter(|element| element.is_uia())?;
+ let ptr = guard.as_ptr();
+ let elem: IUIAutomationElement = unsafe { IUIAutomationElement::from_raw(ptr as *mut _) };
+ let val = unsafe {
+ elem.GetCurrentPattern(UIA_ValuePatternId)
+ .ok()
+ .and_then(|p| p.cast::<IUIAutomationValuePattern>().ok())
+ .and_then(|vp| vp.CurrentValue().ok())
+ .map(|b| b.to_string())
+ .or_else(|| {
+ elem.GetCurrentPattern(UIA_TextPatternId)
+ .ok()
+ .and_then(|p| p.cast::<IUIAutomationTextPattern>().ok())
+ .and_then(|tp| tp.DocumentRange().ok())
+ .and_then(|r| r.GetText(-1).ok())
+ .map(|b| b.to_string())
+ })
+ };
+ // The cache guard owns the ref; don't let `elem`'s Drop release it.
+ std::mem::forget(elem);
+ val
+}
+
+/// Read the currently-focused UIA element's `ValuePattern` value **iff it
+/// belongs to `expect_pid`**, for type_text read-back verification.
+/// Self-contained COM (STA) init/teardown so it can be called from a
+/// `spawn_blocking` worker. Returns `None` if UIA is unreachable, nothing is
+/// focused, the focused element is owned by a different process (e.g. the
+/// target is backgrounded and system focus is on the user's foreground app —
+/// we must NOT read that element's value), or the focused element exposes no
+/// `ValuePattern` (most rich-text/document surfaces). Callers treat `None` as
+/// "could not confirm" rather than failure. Mirrors the focused-element read in
+/// the `debug_window_info` tool.
+fn read_focused_value_uia(expect_pid: u32) -> Option<String> {
+ use windows::core::Interface;
+ use windows::Win32::System::Com::{
+ CoCreateInstance, CoInitializeEx, CoUninitialize, CLSCTX_INPROC_SERVER,
+ COINIT_APARTMENTTHREADED,
+ };
+ use windows::Win32::UI::Accessibility::{
+ CUIAutomation, IUIAutomation, IUIAutomationValuePattern, UIA_ValuePatternId,
+ };
+
+ let coinit_ok = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED).is_ok() };
+ let value = (|| -> Option<String> {
+ let uia: IUIAutomation =
+ unsafe { CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER) }.ok()?;
+ let focused = unsafe { uia.GetFocusedElement() }.ok()?;
+ // Scope to the target: system focus may be on a different app when the
+ // target is backgrounded; reading that element would give a bogus verdict.
+ if unsafe { focused.CurrentProcessId() }.ok()? as u32 != expect_pid {
+ return None;
+ }
+ let pattern = unsafe { focused.GetCurrentPattern(UIA_ValuePatternId) }.ok()?;
+ let vp: IUIAutomationValuePattern = pattern.cast().ok()?;
+ Some(unsafe { vp.CurrentValue() }.ok()?.to_string())
+ })();
+ if coinit_ok {
+ unsafe {
+ CoUninitialize();
+ }
+ }
+ value
+}
+
+#[cfg(test)]
+mod value_write_readback_tests {
+ use super::{classify_value_write_readback, value_write_structured_result};
+
+ #[test]
+ fn confirms_when_the_expected_value_replaces_the_prior_value() {
+ assert_eq!(
+ classify_value_write_readback(Some("beforeinserted"), "before", "beforeinserted"),
+ "confirmed"
+ );
+ }
+
+ #[test]
+ fn treats_stale_or_unreadable_value_as_pending() {
+ assert_eq!(
+ classify_value_write_readback(Some("old value"), "old value", "old valueinserted"),
+ "pending"
+ );
+ assert_eq!(
+ classify_value_write_readback(None, "old value", "old valueinserted"),
+ "pending"
+ );
+ }
+
+ #[test]
+ fn deferred_publication_remains_unverifiable_without_retry_escalation() {
+ let result = value_write_structured_result(8, "pending", false);
+ assert_eq!(result["effect"], "unverifiable");
+ assert_eq!(result["verify"], "pending");
+ assert_eq!(result["verified"], false);
+ assert!(result.get("escalation").is_none());
+
+ let record = cua_driver_core::action_record::ActionExecutionRecord::from_legacy(
+ "type_text",
+ &serde_json::json!({"delivery_mode": "background"}),
+ &result,
+ )
+ .expect("deferred ValuePattern result should normalize");
+ let public = serde_json::to_value(record.public_result().expect("valid ActionResult"))
+ .expect("serialize ActionResult");
+ assert_eq!(public["effect"], "unverifiable");
+ assert!(public.get("escalation").is_none());
+ assert!(public.get("evidence").is_none());
+ }
+
+ #[test]
+ fn does_not_confirm_when_stale_value_contains_the_typed_text() {
+ assert_eq!(
+ classify_value_write_readback(Some("10.00"), "10.00", "10.000"),
+ "pending"
+ );
+ }
+
+ #[test]
+ fn does_not_confirm_an_empty_write() {
+ assert_eq!(
+ classify_value_write_readback(Some("unchanged"), "unchanged", "unchanged"),
+ "pending"
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text_chars.rs
new file mode 100644
index 0000000000..821316ebd8
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/type_text_chars.rs
@@ -0,0 +1,82 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+// ── type_text_chars ───────────────────────────────────────────────────────────
+
+pub struct TypeTextCharsTool {
+ pub(crate) _state: Arc<ToolState>,
+}
+static TYPE_CHARS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for TypeTextCharsTool {
+ fn def(&self) -> &ToolDef {
+ TYPE_CHARS_DEF.get_or_init(|| ToolDef {
+ name: "type_text_chars".into(),
+ description: "Type text character-by-character with a configurable inter-character \
+ delay (default 30 ms). Useful for apps that miss keystrokes. \
+ Otherwise identical to type_text (WM_CHAR, no focus steal).".into(),
+ input_schema: json!({
+ "type":"object","required":["pid","text"],"properties":{
+ "pid":{"type":"integer","description":"Target process ID."},
+ "window_id":{"type":"integer","description":"HWND of the target window. Required with element_index."},
+ "text":{"type":"string","description":"Text to type, one character at a time."},
+ "delay_ms":{"type":"integer","description":"Milliseconds between chars (default 30)."},
+ "type_chars_only":{"type":"boolean","description":"Skip element focus, type directly. Default false."}
+ },"additionalProperties":false
+ }),
+ read_only: false, destructive: true, idempotent: false, open_world: true,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ use cua_driver_core::tool_args::ArgsExt;
+ let pid = args.u64_or("pid", 0) as u32;
+ let text_raw = match args.require_str("text") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ // Same trailing-protocol-tag scrub as the main TypeTextTool — see
+ // cua_driver_core::text_sanitize for rationale.
+ let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
+ .into_owned();
+ let delay_ms = args.u64_or("delay_ms", 30);
+ let hwnd_opt = args.opt_u64("window_id");
+ let hwnd = match hwnd_opt {
+ Some(h) => h,
+ None => {
+ let windows =
+ tokio::task::spawn_blocking(move || crate::win32::list_windows(Some(pid)))
+ .await
+ .unwrap_or_default();
+ match windows.first() {
+ Some(w) => w.hwnd,
+ None => {
+ return ToolResult::error(format!(
+ "No windows found for pid {pid}. Provide window_id."
+ ))
+ }
+ }
+ }
+ };
+ let text_len = text.chars().count();
+ let result = tokio::task::spawn_blocking(move || {
+ crate::input::post_type_text_with_delay(hwnd, &text, delay_ms)
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) with {delay_ms}ms delay."
+ )),
+ Ok(Err(e)) => ToolResult::error(e.to_string()),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/zoom.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/zoom.rs
new file mode 100644
index 0000000000..c40792ec8a
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/zoom.rs
@@ -0,0 +1,242 @@
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ snapshot_store::ZoomContext,
+ tool::{Tool, ToolDef},
+};
+use serde_json::{json, Value};
+use std::sync::Arc;
+
+use super::impl_::ToolState;
+
+#[cfg(test)]
+use super::impl_::{focus_by_pixel_click_args, resolve_onscreen_point_with_scroll};
+
+// ── zoom ──────────────────────────────────────────────────────────────────────
+
+pub struct ZoomTool {
+ pub(crate) state: Arc<ToolState>,
+}
+static ZOOM_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for ZoomTool {
+ fn def(&self) -> &ToolDef {
+ ZOOM_DEF.get_or_init(|| ToolDef {
+ name: "zoom".into(),
+ // Description ported from Swift `ZoomTool.swift`.
+ description: "Zoom into a rectangular region of a window screenshot at full (native) \
+ resolution. Use this when `get_window_state` returned a resized image and you \
+ need to read small text, identify icons, or verify UI details.\n\n\
+ Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted). \
+ Coordinates `x1, y1, x2, y2` are in the same pixel space as the screenshot \
+ returned by `get_window_state` (i.e. the resized image if `max_image_dimension` \
+ is active). The maximum zoom region width is 500 px in scaled-image coordinates.\n\n\
+ A 20% padding is automatically added on every side of the requested region so \
+ the target remains visible even if the caller's coordinates are slightly off.\n\n\
+ After a zoom, pass `from_zoom=true` to click/type_text to auto-translate \
+ coordinates back to full-window space. Coordinate actions return \
+ `screenshot_context_missing` when no current snapshot contains a \
+ screenshot owned by this session. `from_zoom` actions return \
+ `zoom_context_missing` when the zoom was never created or was replaced; call \
+ `get_window_state`, then `zoom`, again on the same connection.\n\n\
+ Windows-specific: `window_id` is the canonical addressing field (HWND). \
+ `pid` is preferred when available; otherwise the driver resolves the unique \
+ current snapshot owned by this session and window.".into(),
+ input_schema: json!({
+ "type":"object","required":["window_id","x1","y1","x2","y2"],"properties":{
+ "pid":{"type":"integer","description":"Optional target process ID. When omitted, the unique current snapshot for this session and HWND is used."},
+ "window_id":{"type":"integer","description":"HWND of the target window."},
+ "x1":{"type":"number","description":"Left edge of the region (resized-image pixels)."},
+ "y1":{"type":"number","description":"Top edge of the region (resized-image pixels)."},
+ "x2":{"type":"number","description":"Right edge of the region (resized-image pixels)."},
+ "y2":{"type":"number","description":"Bottom edge of the region (resized-image pixels)."}
+ },"additionalProperties":false
+ }),
+ // Swift annotation: idempotent: false.
+ read_only: true, destructive: false, idempotent: false, open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let requested_pid = match args.get("pid") {
+ None => None,
+ Some(value) => match value.as_i64().and_then(|pid| i32::try_from(pid).ok()) {
+ Some(pid) => Some(pid),
+ None => return ToolResult::error("pid must be a 32-bit integer"),
+ },
+ };
+ let hwnd = match args.get("window_id").and_then(|v| v.as_u64()) {
+ Some(v) => v,
+ None => return ToolResult::error("Missing required integer field window_id."),
+ };
+ use cua_driver_core::tool_args::ArgsExt;
+ let coerce = |k: &str| {
+ args.opt_f64(k)
+ .or_else(|| args.opt_i64(k).map(|i| i as f64))
+ };
+ let (x1, y1, x2, y2) = match (coerce("x1"), coerce("y1"), coerce("x2"), coerce("y2")) {
+ (Some(a), Some(b), Some(c), Some(d)) => (a, b, c, d),
+ _ => return ToolResult::error("Missing required region coordinates (x1, y1, x2, y2)."),
+ };
+ if x2 <= x1 || y2 <= y1 {
+ return ToolResult::error("Invalid region: x2 must be > x1 and y2 must be > y1.");
+ }
+ if x2 - x1 > 500.0 {
+ return ToolResult::error(format!(
+ "Zoom region too wide: {} px > 500 px max. Use a narrower region (max 500 px in scaled-image coordinates).",
+ (x2 - x1) as i64
+ ));
+ }
+
+ // Issue #1880: the caller's coordinates are in the (possibly downscaled)
+ // screenshot space of the last `get_window_state` call, but the crop below
+ // runs on a fresh native-resolution capture. Scale by the stored resize
+ // ratio so the crop lands on the intended region; the zoom context then
+ // holds native-pixel values, which is what `from_zoom` clicks expect.
+ let session_id = args.opt_str("_session_id");
+ let (pid, screenshot) = match self.state.snapshots.screenshot_context_for_zoom(
+ requested_pid,
+ hwnd,
+ session_id.as_deref(),
+ ) {
+ Ok(context) => context,
+ Err(refusal) => return refusal,
+ };
+ let (nx1, ny1, nx2, ny2) = (
+ x1 * screenshot.scale,
+ y1 * screenshot.scale,
+ x2 * screenshot.scale,
+ y2 * screenshot.scale,
+ );
+
+ let state = self.state.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ let png = crate::capture::screenshot_window_bytes(hwnd)?;
+ cursor_overlay::capture_utils::crop_png_to_jpeg(&png, nx1, ny1, nx2, ny2, 500)
+ })
+ .await;
+
+ match result {
+ Ok(Ok(crop)) => {
+ if let Err(refusal) = state.snapshots.set_zoom(
+ pid,
+ session_id.as_deref(),
+ ZoomContext {
+ screenshot,
+ origin_x: crop.origin_x,
+ origin_y: crop.origin_y,
+ scale_inv: crop.scale_inv,
+ },
+ ) {
+ return refusal;
+ }
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ let b64 = B64.encode(&crop.jpeg_bytes);
+ let (w, h) = (crop.out_w, crop.out_h);
+ use cua_driver_core::protocol::Content;
+ // Match Swift's text format 1:1.
+ let summary = "✅ Zoomed region captured at native resolution. \
+ To click a target in this image, use \
+ `click(pid, x, y, from_zoom=true)` where x,y are pixel \
+ coordinates in THIS zoomed image — the driver maps them \
+ back automatically."
+ .to_owned();
+ ToolResult {
+ content: vec![Content::image_jpeg(b64), Content::text(summary)],
+ is_error: None,
+ // Surface 7: `mime_type` mirrors the MCP image part's `mimeType`
+ // onto the structured payload (additive — `format` stays).
+ structured_content: Some(json!({
+ "width": w, "height": h, "format": "jpeg",
+ "mime_type": "image/jpeg"
+ })),
+ action_record: None,
+ }
+ }
+ Ok(Err(e)) => ToolResult::error(format!("Zoom failed: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
+ }
+ }
+}
+
+#[cfg(test)]
+mod snapshot_coordinate_tests {
+ use super::{focus_by_pixel_click_args, ToolState, ZoomTool};
+ use cua_driver_core::tool::Tool;
+
+ #[test]
+ fn schema_keeps_pid_optional_for_window_owned_lookup() {
+ let tool = ZoomTool {
+ state: ToolState::new(None),
+ };
+ let required = tool.def().input_schema["required"].as_array().unwrap();
+ assert!(!required.iter().any(|field| field == "pid"));
+ assert!(tool.def().input_schema["properties"].get("pid").is_some());
+ }
+
+ #[test]
+ fn press_key_native_element_focus_skips_second_screenshot_scaling() {
+ let native = focus_by_pixel_click_args(
+ 42,
+ Some(7),
+ 120.0,
+ 80.0,
+ false,
+ None,
+ Some("client-a".to_owned()),
+ false,
+ true,
+ );
+ assert_eq!(native["_native_coordinates"], true);
+
+ let screenshot = focus_by_pixel_click_args(
+ 42,
+ Some(7),
+ 60.0,
+ 40.0,
+ false,
+ None,
+ Some("client-a".to_owned()),
+ false,
+ false,
+ );
+ assert!(screenshot.get("_native_coordinates").is_none());
+ }
+
+ #[test]
+ fn resolver_refuses_cached_center_read_while_minimized() {
+ use crate::uia::snapshot::{RetainedElement, SnapshotKind};
+ use windows::Win32::UI::WindowsAndMessaging::{
+ GetSystemMetrics, SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN,
+ SM_YVIRTUALSCREEN,
+ };
+ let (x, y, w, h) = unsafe {
+ (
+ GetSystemMetrics(SM_XVIRTUALSCREEN),
+ GetSystemMetrics(SM_YVIRTUALSCREEN),
+ GetSystemMetrics(SM_CXVIRTUALSCREEN),
+ GetSystemMetrics(SM_CYVIRTUALSCREEN),
+ )
+ };
+ assert!(w > 0 && h > 0, "runner has no virtual-screen metrics");
+ assert!(crate::input::point_on_virtual_desktop(x, y));
+
+ let none = super::resolve_onscreen_point_with_scroll(&None, 0, 1, x, y, "clicking");
+ assert_eq!(none.ok(), Some((x, y)));
+
+ for kind in [SnapshotKind::Uia, SnapshotKind::Msaa] {
+ let fresh = Some(RetainedElement::detached(kind, (x, y), false));
+ let ok = super::resolve_onscreen_point_with_scroll(&fresh, 0, 1, x, y, "clicking");
+ assert_eq!(ok.ok(), Some((x, y)));
+
+ let stale = Some(RetainedElement::detached(kind, (x, y), true));
+ let err = super::resolve_onscreen_point_with_scroll(&stale, 0, 1, x, y, "clicking")
+ .expect_err("flagged element must be refused");
+ let sc = err.structured_content.expect("structured refusal");
+ assert_eq!(sc["status"], "refused");
+ assert_eq!(sc["code"], "element_not_visible");
+ assert_eq!(sc["refusal"]["code"], "element_not_visible");
+ }
+ }
+}