Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): move the pointer for macOS foreground pixel clicks (#4208)
Francesco Bonaccif0da502ac3f4
diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.mdindex b748a23797..79d004d414 100644--- a/libs/cua-driver/docs/test-harnesses-guide.md+++ b/libs/cua-driver/docs/test-harnesses-guide.md@@ -318,6 +318,7 @@ the Lume gate runs after its repo-local matrix.| Installed app launch | `installed_app_launch_macos_test.rs` | Calculator and TextEdit || Installed app AX delivery | `installed_app_textedit_macos_test.rs` | TextEdit || Exact window activation | `bring_to_front_macos_test.rs` | Repo-local AppKit and SwiftUI apps |+| Pointer-reading toolkit | `tk_pointer_click_macos_test.rs` | Tk visual-only canvas (skips sans Tk) || Capture contract | `capture_contract_test.rs` | Installed driver and macOS capture APIs || Desktop scope | `desktop_scope_macos_test.rs` | macOS window and desktop scope || Standalone browsers | `standalone_browser_behavior_test.rs` | Installed Google Chrome and Edge |diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex 091203eb5e..5a2356dbf7 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -300,13 +300,24 @@ surface that only accepts events while frontmost (the canvas/viewport/gamecase below). Unmodified element-indexed (AX) actions remain background-capableand hold the no-foreground contract without the flag.+A foreground window-scoped **pixel** `click`, `double_click`, or+`right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driver+activates the exact window (refusing with `foreground_unavailable` when that+window never becomes focused), moves the hardware pointer to the mapped+screen point, and posts through the HID event tap. The pointer stays at the+target afterward, matching Windows `SendInput` and X11 XTest foreground+clicks, and the prior frontmost app is restored. The result reports+`route: "global_input"` and says the pointer moved.+Modified clicks are the deliberate exception: pass`delivery_mode:"foreground"` and a concrete `window_id`. macOS applicationscan discard PID-routed modifier state after initially publishing a transientselection, so Cua Driver refuses that background combination. The foreground-rung holds physical HID modifier keys around the click, restores the hardware-cursor and prior foreground app, and confirms list-like selection changes with-a stable AX readback.+rung holds physical HID modifier keys around the click, restores the prior+foreground app, and confirms list-like selection changes with a stable AX+readback. Element-addressed modified clicks also restore the hardware cursor;+pixel modified clicks leave it at the target like other foreground pixel+clicks.macOS-specific residuals worth knowing (the rest of the capture/dispatch/addressing params are a shared cross-platform contract — see `SKILL.md` →@@ -350,6 +361,17 @@ There is no backgrounded path that reaches these apps today.### Known pixel-click limits+- **Toolkits that read the hardware pointer (Tk, some Java and game+ toolkits)** derive a click's location from the real pointer position, not+ from the delivered event. Background delivery never moves the pointer, so a+ background pixel click lands wherever the pointer happens to be. When the+ target maps a Tk library or Python's `_tkinter`, a background window pixel+ click returns `background_unavailable` with `reason:+ "pointer_reading_toolkit"`. Other pointer-reading apps cannot be detected+ cheaply; their background result stays "not driver-verified" and says the+ pointer was not moved. Retry with `delivery_mode:"foreground"`, which moves+ the pointer first.+- **Chromium `<video>` play/pause**: pixel click is often rejectedby HTML5's click-to-play handler on some builds. Use keyboardinstead: `press_key({pid, key: "k"})` (YouTube) ordiff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rsindex 3eb011c729..82c6267b71 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs@@ -397,7 +397,7 @@ fn harness_appkit_foreground_single_click_has_one_ordered_native_pair() {"appkit","single_click_native_pair",Targeting::Px,- DriverRoute::MacosCgEventPid,+ DriverRoute::MacosCgEventHid,);execute_case(case, |evidence| {let mut driver =diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rsnew file mode 100644index 0000000000..d496152ff0--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rs@@ -0,0 +1,285 @@+//! macOS window-scoped pixel clicks on a toolkit that reads the hardware+//! pointer (D-MAC-6).+//!+//! Tk's macOS backend derives a mouse event's location from the global+//! pointer, not from the delivered CGEvent. The repository's Tk+//! `visual-only-canvas` fixture therefore logged `ignored click outside cards`+//! for window-scoped clicks even at the correct screenshot point, because the+//! PID-routed event never moved the pointer. These rows prove the contract:+//!+//! * `delivery_mode:"foreground"` activates the exact window, moves the+//! hardware pointer to the mapped point, and posts through the HID tap, so+//! the fixture's own loopback oracle records exactly one `send` selection+//! and the pointer is left at the target (as on Windows and X11).+//! * `delivery_mode:"background"` returns the structured+//! `background_unavailable` refusal instead of an unverifiable success, and+//! the oracle is unchanged.+//!+//! The rows skip with an explicit reason when `python3` lacks Tk support+//! (GitHub-hosted runners may not ship it) unless `CUA_TEST_REQUIRE_TK` is+//! set.+//!+//! Run with:+//! `cargo test -p cua-driver-e2e --test tk_pointer_click_macos_test -- --ignored --nocapture --test-threads=1`++#![cfg(target_os = "macos")]++use std::path::{Path, PathBuf};+use std::process::{Command, Stdio};+use std::thread;+use std::time::{Duration, Instant};++use cua_driver_testkit::{spawn_in_job, Driver, FixtureJournal, McpDriver};+use serde_json::{json, Value};++/// Canvas size and the Send card center from the fixture's `CARDS` table.+const CANVAS_WIDTH: f64 = 760.0;+const CANVAS_HEIGHT: f64 = 460.0;+const SEND_CENTER: (f64, f64) = (394.0, 221.0);+const SEND_BOUNDS: (f64, f64, f64, f64) = (292.0, 132.0, 496.0, 310.0);++fn fixture_path() -> PathBuf {+ Path::new(env!("CARGO_MANIFEST_DIR"))+ .join("../../../tests/fixtures/apps/cross-platform/visual-only-canvas/main.py")+}++/// Return a skip reason when this host cannot run the Tk fixture.+fn tk_unavailable() -> Option<String> {+ let probe = Command::new("python3")+ .args(["-c", "import tkinter; print(tkinter.TkVersion)"])+ .stdin(Stdio::null())+ .output();+ let reason = match probe {+ Err(error) => format!("python3 is unavailable: {error}"),+ Ok(output) if !output.status.success() => format!(+ "python3 has no importable tkinter: {}",+ String::from_utf8_lossy(&output.stderr).trim()+ ),+ Ok(_) => return None,+ };+ if std::env::var_os("CUA_TEST_REQUIRE_TK").is_some() {+ panic!("CUA_TEST_REQUIRE_TK is set but {reason}");+ }+ Some(reason)+}++struct TkFixture {+ driver: McpDriver,+ journal: FixtureJournal,+ pid: i64,+ window_id: u64,+ session: String,+}++fn launch(label: &str) -> Option<TkFixture> {+ if let Some(reason) = tk_unavailable() {+ eprintln!("[tk-pointer] SKIP {label}: {reason}");+ return None;+ }+ let journal = FixtureJournal::start();+ let title = format!("Cua Tk Pointer Fixture [{label}]");+ let child = spawn_in_job(+ Command::new("python3")+ .arg(fixture_path())+ .args(["--journal-url", journal.url(), "--title", &title])+ .stdin(Stdio::null())+ .stdout(Stdio::null())+ .stderr(Stdio::inherit()),+ )+ .expect("start the Tk visual-only-canvas fixture");+ let mut driver =+ McpDriver::spawn_macos_daemon_proxy_named(label).expect("connect to the macOS daemon");+ driver.reaper().push(child);+ wait_for(+ || journal.snapshot()["ready"].as_bool() == Some(true),+ "Tk fixture never published readiness",+ );+ let pid = journal.snapshot()["pid"]+ .as_i64()+ .filter(|pid| *pid > 0)+ .expect("fixture journal reports its pid");+ let (window_id, _) = driver+ .find_window(pid, &title)+ .expect("find the exact Tk fixture window");+ let session = format!("{label}-session");+ let started = driver.call(+ "start_session",+ json!({"session": session, "capture_scope": "window"}),+ );+ assert!(!started.is_error(), "start_session: {}", started.text());+ Some(TkFixture {+ driver,+ journal,+ pid,+ window_id,+ session,+ })+}++fn wait_for(mut predicate: impl FnMut() -> bool, message: &str) {+ let deadline = Instant::now() + Duration::from_secs(10);+ while !predicate() {+ assert!(Instant::now() < deadline, "{message}");+ thread::sleep(Duration::from_millis(50));+ }+}++/// Window-local screenshot pixel for the Send card center. The capture spans+/// the full window: the canvas fills its width and sits under the title bar.+fn send_screenshot_point(fixture: &mut TkFixture) -> (f64, f64) {+ let state = fixture.driver.call(+ "get_window_state",+ json!({+ "session": fixture.session,+ "pid": fixture.pid,+ "window_id": fixture.window_id,+ }),+ );+ assert!(!state.is_error(), "get_window_state: {}", state.text());+ let width = state.structured()["screenshot_width"]+ .as_f64()+ .expect("screenshot width");+ let height = state.structured()["screenshot_height"]+ .as_f64()+ .expect("screenshot height");+ let scale = width / CANVAS_WIDTH;+ let title_bar = height - CANVAS_HEIGHT * scale;+ assert!(+ scale > 0.0 && (0.0..=80.0 * scale).contains(&title_bar),+ "unexpected Tk window capture geometry {width}x{height}"+ );+ (SEND_CENTER.0 * scale, title_bar + SEND_CENTER.1 * scale)+}++fn window_bounds(fixture: &mut TkFixture) -> Value {+ let windows = fixture+ .driver+ .call("list_windows", json!({ "pid": fixture.pid }));+ windows.structured()["windows"]+ .as_array()+ .and_then(|windows| {+ windows+ .iter()+ .find(|window| window["window_id"].as_u64() == Some(fixture.window_id))+ })+ .map(|window| window["bounds"].clone())+ .expect("fixture window bounds")+}++fn pointer(fixture: &mut TkFixture) -> (f64, f64) {+ let position = fixture.driver.call("get_cursor_position", json!({}));+ assert!(+ !position.is_error(),+ "get_cursor_position: {}",+ position.text()+ );+ (+ position.structured()["x"].as_f64().expect("pointer x"),+ position.structured()["y"].as_f64().expect("pointer y"),+ )+}++#[test]+#[ignore]+fn tk_window_pixel_click_foreground_moves_pointer_and_reaches_target() {+ let Some(mut fixture) = launch("tk-pointer-foreground") else {+ return;+ };+ let (x, y) = send_screenshot_point(&mut fixture);+ let click = fixture.driver.call(+ "click",+ json!({+ "pid": fixture.pid,+ "window_id": fixture.window_id,+ "x": x,+ "y": y,+ "delivery_mode": "foreground",+ }),+ );+ assert!(!click.is_error(), "foreground click failed: {}", click.raw);+ assert_eq!(click.action_route(), Some("global_input"), "{}", click.raw);+ assert_eq!(+ click.action_delivery_mode(),+ Some("foreground"),+ "{}",+ click.raw+ );+ assert!(+ click.text().contains("hardware pointer moved"),+ "result must say the pointer moved: {}",+ click.text()+ );+ wait_for(+ || {+ let state = fixture.journal.snapshot();+ state["selected"] == "send" && state["action_count"] == 1+ },+ &format!(+ "Tk oracle did not record exactly one Send selection: {}",+ fixture.journal.snapshot()+ ),+ );++ // The foreground contract leaves the hardware pointer at the target, like+ // Windows SendInput and X11 XTest. Allow a small title-bar tolerance.+ let bounds = window_bounds(&mut fixture);+ let (px, py) = pointer(&mut fixture);+ let left = bounds["x"].as_f64().unwrap();+ let bottom = bounds["y"].as_f64().unwrap() + bounds["height"].as_f64().unwrap();+ let canvas_top = bottom - CANVAS_HEIGHT;+ let inside_send = px >= left + SEND_BOUNDS.0+ && px <= left + SEND_BOUNDS.2+ && py >= canvas_top + SEND_BOUNDS.1 - 4.0+ && py <= canvas_top + SEND_BOUNDS.3 + 4.0;+ assert!(+ inside_send,+ "hardware pointer ({px},{py}) is not over the Send card; window bounds {bounds}"+ );+}++#[test]+#[ignore]+fn tk_window_pixel_click_background_is_refused_without_delivery() {+ let Some(mut fixture) = launch("tk-pointer-background") else {+ return;+ };+ let (x, y) = send_screenshot_point(&mut fixture);+ let click = fixture.driver.call(+ "click",+ json!({+ "pid": fixture.pid,+ "window_id": fixture.window_id,+ "x": x,+ "y": y,+ "delivery_mode": "background",+ }),+ );+ assert!(+ click.is_error(),+ "background Tk click must refuse: {}",+ click.raw+ );+ let structured = click.structured();+ assert_eq!(+ structured["code"], "background_unavailable",+ "{}",+ click.raw+ );+ assert_eq!(+ structured["reason"], "pointer_reading_toolkit",+ "{}",+ click.raw+ );+ assert_eq!(structured["toolkit"], "tk", "{}", click.raw);+ assert_eq!(+ structured["escalation"]["recommended"], "foreground",+ "{}",+ click.raw+ );+ thread::sleep(Duration::from_millis(500));+ let state = fixture.journal.snapshot();+ assert!(+ state["selected"].is_null() && state["action_count"] == 0,+ "a refused background click must not reach the fixture: {state}"+ );+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rsindex 7411233b96..123289a225 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs@@ -12,6 +12,7 @@ pub mod ax_actions;pub mod interactive;pub mod keyboard;pub mod mouse;+pub mod pointer_toolkit;pub mod skylight;pub use ax_actions::perform_ax_action;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rsnew file mode 100644index 0000000000..5b4309014c--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rs@@ -0,0 +1,257 @@+//! Detect target processes whose UI toolkit derives pointer-event locations+//! from the hardware pointer rather than from the delivered event.+//!+//! Tk's macOS backend reads the global pointer position when it translates an+//! incoming mouse event into Tk coordinates. A PID-routed background CGEvent+//! never moves the hardware pointer, so such an app receives the click wherever+//! the real pointer happens to be (for example the visual-only canvas fixture's+//! `ignored click outside cards at canvas (-86, -114)`), while the driver can+//! only report the post as unverifiable. Background delivery therefore cannot+//! satisfy these targets; the explicit foreground rung moves the pointer first.+//!+//! Detection is intentionally cheap and conservative: walk the target's+//! file-backed memory regions with `proc_pidinfo` (same-user, no task port) and+//! look for a mapped Tk library or Python's `_tkinter` extension. A toolkit+//! loaded only from the dyld shared cache is not visible this way; such targets+//! fall back to the honest "not driver-verified" background result.++/// A UI toolkit known to derive click locations from the hardware pointer.+#[derive(Clone, Copy, Debug, Eq, PartialEq)]+pub enum PointerReadingToolkit {+ Tk,+}++impl PointerReadingToolkit {+ pub fn name(self) -> &'static str {+ match self {+ Self::Tk => "tk",+ }+ }+}++/// Classify one mapped image path. Pure so the matcher is unit-testable.+///+/// Matches `Tk.framework` bundles, Tk shared libraries (`libtk8.6.dylib`,+/// Tk 9's `libtcl9tk9.0.dylib`), and CPython's `_tkinter` extension module,+/// which either links Tk dynamically or embeds it statically.+pub fn classify_image_path(path: &str) -> Option<PointerReadingToolkit> {+ if path.contains("/Tk.framework/") {+ return Some(PointerReadingToolkit::Tk);+ }+ let file = path.rsplit('/').next().unwrap_or(path);+ let lower = file.to_ascii_lowercase();+ if lower.starts_with("_tkinter") && lower.ends_with(".so") {+ return Some(PointerReadingToolkit::Tk);+ }+ if !lower.ends_with(".dylib") {+ return None;+ }+ let rest = lower.strip_prefix("lib")?;+ // Tk 9 prefixes the Tcl major version: `libtcl9tk9.0.dylib`.+ let rest = match rest.strip_prefix("tcl") {+ Some(after_tcl) => {+ let digits = after_tcl.bytes().take_while(u8::is_ascii_digit).count();+ if digits == 0 {+ return None;+ }+ &after_tcl[digits..]+ }+ None => rest,+ };+ let after_tk = rest.strip_prefix("tk")?;+ after_tk+ .bytes()+ .next()+ .filter(u8::is_ascii_digit)+ .map(|_| PointerReadingToolkit::Tk)+}++/// Return the pointer-reading toolkit mapped into `pid`, if any.+#[cfg(target_os = "macos")]+pub fn detect(pid: i32) -> Option<PointerReadingToolkit> {+ // Prefer the vnode-only flavor, which skips anonymous regions; fall back+ // to the full walk on kernels that reject it.+ match walk_regions(pid, PROC_PIDREGIONPATHINFO2) {+ Some(found) => found,+ None => walk_regions(pid, PROC_PIDREGIONPATHINFO).flatten(),+ }+}++#[cfg(not(target_os = "macos"))]+pub fn detect(_pid: i32) -> Option<PointerReadingToolkit> {+ None+}++#[cfg(target_os = "macos")]+const PROC_PIDREGIONPATHINFO: libc::c_int = 8;+#[cfg(target_os = "macos")]+const PROC_PIDREGIONPATHINFO2: libc::c_int = 22;+/// Bound the walk so a pathological address space cannot stall a click.+#[cfg(target_os = "macos")]+const MAX_REGIONS: usize = 65_536;++/// `struct proc_regioninfo` from `<sys/proc_info.h>`.+#[cfg(target_os = "macos")]+#[repr(C)]+#[derive(Clone, Copy)]+struct ProcRegionInfo {+ pri_protection: u32,+ pri_max_protection: u32,+ pri_inheritance: u32,+ pri_flags: u32,+ pri_offset: u64,+ pri_behavior: u32,+ pri_user_wired_count: u32,+ pri_user_tag: u32,+ pri_pages_resident: u32,+ pri_pages_shared_now_private: u32,+ pri_pages_swapped_out: u32,+ pri_pages_dirtied: u32,+ pri_ref_count: u32,+ pri_shadow_depth: u32,+ pri_share_mode: u32,+ pri_private_pages_resident: u32,+ pri_shared_pages_resident: u32,+ pri_obj_id: u32,+ pri_depth: u32,+ pri_address: u64,+ pri_size: u64,+}++/// `struct proc_regionwithpathinfo`: region info followed by+/// `struct vnode_info_path` (152-byte `vnode_info` + `MAXPATHLEN` path).+#[cfg(target_os = "macos")]+#[repr(C)]+struct ProcRegionWithPathInfo {+ prp_prinfo: ProcRegionInfo,+ prp_vip_vi: [u8; 152],+ prp_vip_path: [libc::c_char; 1024],+}++/// Walk `pid`'s regions with `flavor`. `None` means the flavor was rejected+/// before any region was read; `Some(result)` is a completed walk.+#[cfg(target_os = "macos")]+fn walk_regions(pid: i32, flavor: libc::c_int) -> Option<Option<PointerReadingToolkit>> {+ let size = std::mem::size_of::<ProcRegionWithPathInfo>();+ let mut address: u64 = 0;+ let mut read_any = false;+ for _ in 0..MAX_REGIONS {+ // SAFETY: an all-zero bit pattern is valid for this plain C struct.+ let mut info: ProcRegionWithPathInfo = unsafe { std::mem::zeroed() };+ // SAFETY: the buffer is exactly the kernel structure size and lives for+ // the duration of the call.+ let written = unsafe {+ libc::proc_pidinfo(+ pid,+ flavor,+ address,+ (&mut info as *mut ProcRegionWithPathInfo).cast(),+ size as libc::c_int,+ )+ };+ if written <= 0 || (written as usize) < size {+ break;+ }+ read_any = true;+ // SAFETY: the kernel NUL-terminates `vip_path` within MAXPATHLEN.+ let path = unsafe { std::ffi::CStr::from_ptr(info.prp_vip_path.as_ptr()) };+ if let Some(toolkit) = classify_image_path(&path.to_string_lossy()) {+ return Some(Some(toolkit));+ }+ let region = info.prp_prinfo;+ let next = region.pri_address.saturating_add(region.pri_size);+ if next <= address {+ break;+ }+ address = next;+ }+ if read_any {+ Some(None)+ } else {+ None+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn classifies_tk_images_and_rejects_neighbours() {+ for path in [+ "/Library/Frameworks/Tk.framework/Versions/8.6/Tk",+ "/opt/homebrew/Cellar/tcl-tk/9.0.2/lib/libtcl9tk9.0.dylib",+ "/opt/homebrew/opt/tcl-tk@8/lib/libtk8.6.dylib",+ "/usr/local/lib/python3.12/lib-dynload/_tkinter.cpython-312-darwin.so",+ "/Users/me/.local/share/uv/python/cpython-3.13/lib/python3.13/lib-dynload/_tkinter.cpython-313-darwin.so",+ ] {+ assert_eq!(+ classify_image_path(path),+ Some(PointerReadingToolkit::Tk),+ "{path}"+ );+ }+ for path in [+ "",+ "/usr/lib/libSystem.B.dylib",+ "/opt/homebrew/lib/libtcl9.0.dylib",+ "/opt/homebrew/lib/libtcl8.6.dylib",+ "/opt/homebrew/lib/libtkrzw.dylib",+ "/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit",+ "/usr/lib/python3/_tkinter_helper.py",+ "/Applications/Foo.app/Contents/MacOS/tk8.6",+ ] {+ assert_eq!(classify_image_path(path), None, "{path}");+ }+ }++ #[cfg(target_os = "macos")]+ #[test]+ fn kernel_region_structure_matches_proc_info_layout() {+ assert_eq!(std::mem::size_of::<ProcRegionInfo>(), 96);+ assert_eq!(std::mem::size_of::<ProcRegionWithPathInfo>(), 1272);+ }++ #[cfg(target_os = "macos")]+ #[test]+ fn detect_is_negative_for_a_non_tk_process() {+ // The test binary maps no Tk image. A completed negative walk proves the+ // region flavor is readable for a same-user process.+ assert_eq!(detect(std::process::id() as i32), None);+ assert!(walk_regions(std::process::id() as i32, PROC_PIDREGIONPATHINFO).is_some());+ }++ /// Loads `_tkinter` without creating any window, so this never touches the+ /// desktop. Skips when the host Python lacks Tk support.+ #[cfg(target_os = "macos")]+ #[test]+ fn detect_finds_tk_in_a_python_process_that_imported_tkinter() {+ use std::io::BufRead;+ let spawned = std::process::Command::new("python3")+ .args([+ "-c",+ "import _tkinter, sys, time; print('ready', flush=True); time.sleep(30)",+ ])+ .stdout(std::process::Stdio::piped())+ .stderr(std::process::Stdio::null())+ .spawn();+ let Ok(mut child) = spawned else {+ eprintln!("skipping: python3 is unavailable");+ return;+ };+ let mut line = String::new();+ let ready = child+ .stdout+ .take()+ .map(|stdout| std::io::BufReader::new(stdout).read_line(&mut line))+ .is_some_and(|read| read.is_ok())+ && line.trim() == "ready";+ let detected = ready.then(|| detect(child.id() as i32));+ let _ = child.kill();+ let _ = child.wait();+ match detected {+ None => eprintln!("skipping: python3 has no importable _tkinter"),+ Some(found) => assert_eq!(found, Some(PointerReadingToolkit::Tk)),+ }+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex e9c300ad83..fcf8bd969e 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -33,6 +33,7 @@ use crate::focus_guard;use crate::window_change_detector::WindowChangeDetector;use core_foundation::base::{CFRelease, CFTypeRef, TCFType};+use super::pixel_route::PixelClickRoute;use super::ToolState;pub struct ClickTool {@@ -1074,6 +1075,12 @@ impl Tool for ClickTool {// requested foreground click without a window id still degrades to// background, matching the existing contract and result label.let fg = delivery_mode.is_foreground() && window_id.is_some();+ // Background delivery cannot satisfy a toolkit that reads the+ // hardware pointer; refuse before any activation or dispatch.+ let route = match super::pixel_route::resolve(pid, fg, window_id, "mouse_click").await {+ Ok(route) => route,+ Err(refusal) => return refusal,+ };let activation_policy = pixel_activation_policy(&button_str, fg, window_id.is_some());// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──@@ -1149,11 +1156,14 @@ impl Tool for ClickTool {"click.pixel",|| async move {tokio::task::spawn_blocking(move || {- let has_modifiers = !mods_owned.is_empty();let do_click = move || -> anyhow::Result<()> {let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();- if fg && !m.is_empty() {- return crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(+ if route == PixelClickRoute::ForegroundHid {+ // Warp the hardware pointer to the mapped global+ // point and post at the HID tap. The pointer stays+ // at the target, as on Windows and X11, so apps that+ // read the pointer when handling the event see it.+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(screen_x,screen_y,count,@@ -1196,26 +1206,19 @@ impl Tool for ClickTool {}}};- // Foreground rung: brief front → click → restore.- // Returns whether the window was ACTUALLY fronted, so the- // reported `path` honestly reflects the rung that ran.- match (fg, window_id, has_modifiers) {- (true, Some(wid), true) => {+ // Foreground rung: front the exact window → HID click →+ // restore the prior front process. The HID tap has no+ // pid addressing, so activation must be proven (the+ // window is AX-focused) or no input is sent.+ match (route, window_id) {+ (PixelClickRoute::ForegroundHid, Some(wid)) => {crate::input::skylight::with_foreground_hid_activation(pid as libc::pid_t,wid,do_click,)- .map(|_| true)}- (true, Some(wid), false) => {- crate::input::skylight::with_foreground_assist(- pid as libc::pid_t,- wid,- do_click,- )- }- _ => do_click().map(|_| false),+ _ => do_click(),}}).await@@ -1268,27 +1271,31 @@ impl Tool for ClickTool {_ => "click",};match result {- Ok(Ok(fronted)) => {- // `with_foreground_assist` returns `false` when the fronting SPIs- // were unavailable and it clicked WITHOUT activation — report the- // background path in that case so `path` reflects the rung that ran.- let (path, mode_label) = if fg && fronted {- ("cgevent_fg", "foreground CGEvent")+ Ok(Ok(())) => {+ let target = if route == PixelClickRoute::ForegroundHid {+ format!("at screen-point ({screen_x:.0},{screen_y:.0}) for pid {pid}")} else {- ("cgevent", "background CGEvent")+ format!("to pid {pid}")};ToolResult::text(format!(- "✅ Posted {button_label} to pid {pid} ({mode_label}; \- not driver-verified — confirm via screenshot).{}",+ "✅ Posted {button_label} {target} ({}).{}",+ super::pixel_route::delivery_note(route),changes.result_suffix())).with_structured(serde_json::json!({- "path": path,+ "path": super::pixel_route::path_label(route),"verified": false,"effect": "unverifiable","focus_without_raise": focus_without_raise}))}+ Ok(Err(e)) if route == PixelClickRoute::ForegroundHid => {+ super::pixel_route::foreground_unavailable(+ button_label,+ window_id.unwrap_or_default(),+ &e.to_string(),+ )+ }Ok(Err(e)) => ToolResult::error(format!("{button_label} failed: {e}")),Err(e) => ToolResult::error(format!("Task error: {e}")),}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rsindex 03190160ed..4e8bf84ec1 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs@@ -247,8 +247,24 @@ impl Tool for DoubleClickTool {);let fg = delivery_mode.is_foreground() && window_id.is_some();+ let route =+ match super::pixel_route::resolve(pid, fg, window_id, "mouse_double_click").await {+ Ok(route) => route,+ Err(refusal) => return refusal,+ };let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {let do_click = move || -> anyhow::Result<()> {+ if route == super::pixel_route::PixelClickRoute::ForegroundHid {+ // Warp the hardware pointer and post at the HID tap; see+ // `pixel_route` for the cross-platform foreground contract.+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(+ screen_x,+ screen_y,+ 2,+ "left",+ &[],+ );+ }if let Some(wid) = window_id {crate::input::mouse::click_at_xy_with_window_local(pid,@@ -265,31 +281,37 @@ impl Tool for DoubleClickTool {crate::input::mouse::click_at_xy(pid, screen_x, screen_y, 2, &[])}};- // Foreground rung: brief front → double-click → restore prior frontmost.- match (fg, window_id) {- (true, Some(wid)) => {- crate::input::skylight::with_foreground_assist(+ // Foreground rung: front the exact window → HID double-click →+ // restore the prior frontmost. No input is sent unless the exact+ // window is proven focused.+ match (route, window_id) {+ (super::pixel_route::PixelClickRoute::ForegroundHid, Some(wid)) => {+ crate::input::skylight::with_foreground_hid_activation(pid as libc::pid_t,wid,do_click,- )?;- Ok(())+ )}_ => do_click(),}}).await;- let mode_label = if fg {- " (delivery_mode:foreground)"- } else {- ""- };match result {- Ok(Ok(())) => ToolResult::text(format!("✅ Double-clicked at ({screen_x:.1}, {screen_y:.1}){mode_label}."))- .with_structured(serde_json::json!({- "path": if fg { "cgevent_fg" } else { "cgevent" }, "verified": false, "effect": "unverifiable"- })),+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Double-clicked at ({screen_x:.1}, {screen_y:.1}) ({}).",+ super::pixel_route::delivery_note(route)+ ))+ .with_structured(serde_json::json!({+ "path": super::pixel_route::path_label(route), "verified": false, "effect": "unverifiable"+ })),+ Ok(Err(e)) if route == super::pixel_route::PixelClickRoute::ForegroundHid => {+ super::pixel_route::foreground_unavailable(+ "Double-click",+ window_id.unwrap_or_default(),+ &e.to_string(),+ )+ }Ok(Err(e)) => ToolResult::error(format!("Double-click failed: {e}")),Err(e) => ToolResult::error(format!("Task error: {e}")),}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex de12ecb254..b6ec2b0fa4 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -34,6 +34,7 @@ pub(crate) mod get_screen_size;mod health_report;mod move_cursor;mod page;+mod pixel_route;pub(crate) mod px_frame;mod set_config;mod type_text_chars;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rsnew file mode 100644index 0000000000..36a4df3ff3--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rs@@ -0,0 +1,209 @@+//! Transport selection for window-addressed pixel clicks (`click`,+//! `double_click`, `right_click`) on macOS.+//!+//! The cross-platform foreground contract is that a foreground pixel click+//! moves the system pointer to the target and delivers real input there:+//! Windows uses `SetCursorPos` + `SendInput`, X11 uses an XTest warp + button,+//! and macOS desktop scope and foreground drags warp the pointer and post at the+//! HID tap. Window-scoped foreground pixel clicks follow the same model, so apps+//! that derive a click location from the hardware pointer (Tk, some Java and+//! game toolkits) receive it at the right point. Background delivery stays+//! PID-routed and never moves the pointer; a detectable pointer-reading toolkit+//! refuses with the shared `background_unavailable` contract instead of+//! reporting an unverifiable success.++use cua_driver_core::protocol::ToolResult;++use crate::input::pointer_toolkit::PointerReadingToolkit;++/// Transport for a pixel click once any background AX hit-test backend has+/// declined (or was not eligible).+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub(crate) enum PixelClickRoute {+ /// Explicit foreground rung with an exact window: activate that window,+ /// move the hardware pointer to the mapped global point, and post through+ /// the global HID event tap. The pointer stays at the target.+ ForegroundHid,+ /// Background delivery routed to the target pid. The hardware pointer is+ /// never moved, so the result stays "not driver-verified".+ BackgroundPid,+ /// Background delivery refused: the target's toolkit reads the hardware+ /// pointer, so a PID-routed event would land wherever the pointer is.+ RefuseBackground(PointerReadingToolkit),+}++/// Pure route decision. `effective_foreground` is the requested foreground+/// rung; it only selects the HID route with an exact window, because the HID+/// tap has no pid addressing.+pub(crate) fn pixel_click_route(+ effective_foreground: bool,+ has_window: bool,+ toolkit: Option<PointerReadingToolkit>,+) -> PixelClickRoute {+ if effective_foreground && has_window {+ return PixelClickRoute::ForegroundHid;+ }+ match toolkit {+ Some(toolkit) if has_window => PixelClickRoute::RefuseBackground(toolkit),+ _ => PixelClickRoute::BackgroundPid,+ }+}++/// Resolve the route for a live target, probing the toolkit only for+/// window-addressed background clicks. Returns the structured refusal when+/// background delivery cannot reach the target.+pub(crate) async fn resolve(+ pid: i32,+ effective_foreground: bool,+ window_id: Option<u32>,+ event_kind: &str,+) -> Result<PixelClickRoute, ToolResult> {+ let has_window = window_id.is_some();+ let toolkit = if !effective_foreground && has_window {+ tokio::task::spawn_blocking(move || crate::input::pointer_toolkit::detect(pid))+ .await+ .ok()+ .flatten()+ } else {+ None+ };+ match pixel_click_route(effective_foreground, has_window, toolkit) {+ PixelClickRoute::RefuseBackground(toolkit) => {+ Err(pointer_reading_background_refusal(pid, toolkit, event_kind))+ }+ route => Ok(route),+ }+}++pub(crate) fn pointer_reading_background_refusal(+ pid: i32,+ toolkit: PointerReadingToolkit,+ event_kind: &str,+) -> ToolResult {+ let name = toolkit.name();+ cua_driver_core::delivery::background_unavailable_result(+ format!(+ "Background pixel click is not available for pid {pid}: its {name} toolkit \+ derives click locations from the hardware pointer, and background delivery \+ never moves the pointer, so the click would land wherever the pointer \+ currently is. Retry this action with delivery_mode:\"foreground\"; Cua \+ Driver will activate the window, move the pointer to the target, and click."+ ),+ "background_unavailable",+ "the target toolkit reads the hardware pointer position, which background \+ PID-routed events cannot move; retry with delivery_mode:\"foreground\".",+ serde_json::json!({+ "event_kind": event_kind,+ "reason": "pointer_reading_toolkit",+ "toolkit": name,+ }),+ )+}++/// Human-readable delivery note appended to a successful pixel click result.+pub(crate) fn delivery_note(route: PixelClickRoute) -> &'static str {+ match route {+ PixelClickRoute::ForegroundHid => {+ "foreground: exact window activated, hardware pointer moved to the target, \+ HID event tap; not driver-verified — confirm via screenshot"+ }+ _ => {+ "background CGEvent routed to the pid; the hardware pointer was not moved, \+ so apps that read the pointer position may not receive it at the target; \+ not driver-verified — confirm via screenshot and retry with \+ delivery_mode:\"foreground\" if nothing changed"+ }+ }+}++/// Legacy `path` label consumed by the action-truth normalizer:+/// `cgevent_fg` maps to the HID transport with foreground delivery and+/// `cgevent` to the PID transport with background delivery.+pub(crate) fn path_label(route: PixelClickRoute) -> &'static str {+ match route {+ PixelClickRoute::ForegroundHid => "cgevent_fg",+ _ => "cgevent",+ }+}++/// Structured error for a foreground HID click whose exact-window activation+/// or dispatch failed. No input reaches another window in that case.+pub(crate) fn foreground_unavailable(action: &str, window_id: u32, cause: &str) -> ToolResult {+ ToolResult::error(format!(+ "{action} failed: foreground HID delivery to window {window_id} was not possible: \+ {cause}"+ ))+ .with_structured(serde_json::json!({ "code": "foreground_unavailable" }))+}++#[cfg(test)]+mod tests {+ use super::*;++ /// D-MAC-6: a foreground window pixel click must use the HID tap with a+ /// real pointer move (Tk reads the hardware pointer), background clicks+ /// keep PID routing, and a known pointer-reading toolkit refuses+ /// background delivery instead of reporting an unverifiable success.+ #[test]+ fn pixel_click_route_moves_pointer_only_for_foreground_and_refuses_tk_background() {+ let tk = Some(PointerReadingToolkit::Tk);+ assert_eq!(+ pixel_click_route(true, true, None),+ PixelClickRoute::ForegroundHid+ );+ assert_eq!(+ pixel_click_route(true, true, tk),+ PixelClickRoute::ForegroundHid,+ "foreground delivery moves the pointer, so Tk is supported"+ );+ assert_eq!(+ pixel_click_route(false, true, None),+ PixelClickRoute::BackgroundPid+ );+ assert_eq!(+ pixel_click_route(false, true, tk),+ PixelClickRoute::RefuseBackground(PointerReadingToolkit::Tk)+ );+ assert_eq!(+ pixel_click_route(false, false, tk),+ PixelClickRoute::BackgroundPid,+ "legacy window-less pid clicks keep their established route"+ );+ assert_eq!(+ pixel_click_route(true, false, None),+ PixelClickRoute::BackgroundPid,+ "foreground without an exact window cannot use the unaddressed HID tap"+ );+ }++ #[test]+ fn route_labels_say_what_actually_happened() {+ assert_eq!(path_label(PixelClickRoute::ForegroundHid), "cgevent_fg");+ assert_eq!(path_label(PixelClickRoute::BackgroundPid), "cgevent");+ assert!(delivery_note(PixelClickRoute::ForegroundHid).contains("hardware pointer moved"));+ assert!(delivery_note(PixelClickRoute::BackgroundPid).contains("pointer was not moved"));+ }++ #[test]+ fn pointer_reading_refusal_uses_the_shared_background_unavailable_contract() {+ let result =+ pointer_reading_background_refusal(42, PointerReadingToolkit::Tk, "mouse_click");+ assert_eq!(result.is_error, Some(true));+ let structured = result.structured_content.expect("structured refusal");+ assert_eq!(structured["code"], "background_unavailable");+ assert_eq!(structured["reason"], "pointer_reading_toolkit");+ assert_eq!(structured["toolkit"], "tk");+ assert_eq!(structured["event_kind"], "mouse_click");+ assert_eq!(structured["escalation"]["recommended"], "foreground");+ }++ #[test]+ fn foreground_failure_is_structured() {+ let result = foreground_unavailable("click", 7, "window never focused");+ assert_eq!(result.is_error, Some(true));+ assert_eq!(+ result.structured_content.expect("structured")["code"],+ "foreground_unavailable"+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rsindex 1a05770851..70ed22ebb0 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs@@ -247,9 +247,21 @@ impl Tool for RightClickTool {};let fg = delivery_mode.is_foreground() && window_id.is_some();+ let route = match super::pixel_route::resolve(pid, fg, window_id, "mouse_right_click").await+ {+ Ok(route) => route,+ Err(refusal) => return refusal,+ };let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {let do_it = move || -> anyhow::Result<()> {let m: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ if route == super::pixel_route::PixelClickRoute::ForegroundHid {+ // Warp the hardware pointer and post at the HID tap; see+ // `pixel_route` for the cross-platform foreground contract.+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(+ screen_x, screen_y, 1, "right", &m,+ );+ }if let Some(wid) = window_id {crate::input::mouse::right_click_at_xy_with_window_local(pid,@@ -264,26 +276,36 @@ impl Tool for RightClickTool {crate::input::mouse::right_click_at_xy(pid, screen_x, screen_y, &m)}};- // Foreground rung: brief front → right-click → restore prior frontmost.- match (fg, window_id) {- (true, Some(wid)) => {- crate::input::skylight::with_foreground_assist(pid as libc::pid_t, wid, do_it)?;- Ok(())+ // Foreground rung: front the exact window → HID right-click →+ // restore the prior frontmost. No input is sent unless the exact+ // window is proven focused.+ match (route, window_id) {+ (super::pixel_route::PixelClickRoute::ForegroundHid, Some(wid)) => {+ crate::input::skylight::with_foreground_hid_activation(+ pid as libc::pid_t,+ wid,+ do_it,+ )}_ => do_it(),}}).await;- let mode_label = if fg {- " (delivery_mode:foreground)"- } else {- ""- };match result {- Ok(Ok(())) => ToolResult::text(format!("Right-clicked{mod_suffix} at ({screen_x:.1}, {screen_y:.1}){mode_label}."))- .with_structured(serde_json::json!({- "path": if fg { "cgevent_fg" } else { "cgevent" }, "verified": false, "effect": "unverifiable"- })),+ Ok(Ok(())) => ToolResult::text(format!(+ "Right-clicked{mod_suffix} at ({screen_x:.1}, {screen_y:.1}) ({}).",+ super::pixel_route::delivery_note(route)+ ))+ .with_structured(serde_json::json!({+ "path": super::pixel_route::path_label(route), "verified": false, "effect": "unverifiable"+ })),+ Ok(Err(e)) if route == super::pixel_route::PixelClickRoute::ForegroundHid => {+ super::pixel_route::foreground_unavailable(+ "Right-click",+ window_id.unwrap_or_default(),+ &e.to_string(),+ )+ }Ok(Err(e)) => ToolResult::error(format!("Right-click failed: {e}")),Err(e) => ToolResult::error(format!("Task error: {e}")),}diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.shindex e4dc7c3ceb..0ab5b0d6c2 100755--- a/scripts/ci/macos/run-rust-e2e.sh+++ b/scripts/ci/macos/run-rust-e2e.sh@@ -401,6 +401,12 @@ EOFrun_test bring-to-front env -u CUA_E2E_RECORDINGS_ROOT \cargo test -p cua-driver-e2e --test bring_to_front_macos_test -- \--ignored --nocapture --test-threads=1+ # Tk derives click locations from the hardware pointer. The rows skip with+ # an explicit reason when python3 lacks tkinter (hosted images may not ship+ # it); set CUA_TEST_REQUIRE_TK=1 where Tk is provisioned.+ run_test tk-pointer-click env -u CUA_E2E_RECORDINGS_ROOT \+ cargo test -p cua-driver-e2e --test tk_pointer_click_macos_test -- \+ --ignored --nocapture --test-threads=1fifiif [[ "${SUITE}" == capture || "${SUITE}" == all ]]; then