Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): move the pointer for macOS foreground pixel clicks (#4208)

Francesco Bonaccif0da502ac3f4
diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.md
index b748a23797..79d004d414 100644
--- a/libs/cua-driver/docs/test-harnesses-guide.md
+++ b/libs/cua-driver/docs/test-harnesses-guide.md
@@ -318,6 +318,7 @@ the Lume gate runs after its repo-local matrix.
| Installed app launch | `installed_app_launch_macos_test.rs` | Calculator and TextEdit |
| Installed app AX delivery | `installed_app_textedit_macos_test.rs` | TextEdit |
| Exact window activation | `bring_to_front_macos_test.rs` | Repo-local AppKit and SwiftUI apps |
+| Pointer-reading toolkit | `tk_pointer_click_macos_test.rs` | Tk visual-only canvas (skips sans Tk) |
| Capture contract | `capture_contract_test.rs` | Installed driver and macOS capture APIs |
| Desktop scope | `desktop_scope_macos_test.rs` | macOS window and desktop scope |
| Standalone browsers | `standalone_browser_behavior_test.rs` | Installed Google Chrome and Edge |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 091203eb5e..5a2356dbf7 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -300,13 +300,24 @@ surface that only accepts events while frontmost (the canvas/viewport/game
case below). Unmodified element-indexed (AX) actions remain background-capable
and hold the no-foreground contract without the flag.
+A foreground window-scoped **pixel** `click`, `double_click`, or
+`right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driver
+activates the exact window (refusing with `foreground_unavailable` when that
+window never becomes focused), moves the hardware pointer to the mapped
+screen point, and posts through the HID event tap. The pointer stays at the
+target afterward, matching Windows `SendInput` and X11 XTest foreground
+clicks, and the prior frontmost app is restored. The result reports
+`route: "global_input"` and says the pointer moved.
+
Modified clicks are the deliberate exception: pass
`delivery_mode:"foreground"` and a concrete `window_id`. macOS applications
can discard PID-routed modifier state after initially publishing a transient
selection, so Cua Driver refuses that background combination. The foreground
-rung holds physical HID modifier keys around the click, restores the hardware
-cursor and prior foreground app, and confirms list-like selection changes with
-a stable AX readback.
+rung holds physical HID modifier keys around the click, restores the prior
+foreground app, and confirms list-like selection changes with a stable AX
+readback. Element-addressed modified clicks also restore the hardware cursor;
+pixel modified clicks leave it at the target like other foreground pixel
+clicks.
macOS-specific residuals worth knowing (the rest of the capture/dispatch/
addressing params are a shared cross-platform contract — see `SKILL.md` →
@@ -350,6 +361,17 @@ There is no backgrounded path that reaches these apps today.
### Known pixel-click limits
+- **Toolkits that read the hardware pointer (Tk, some Java and game
+ toolkits)** derive a click's location from the real pointer position, not
+ from the delivered event. Background delivery never moves the pointer, so a
+ background pixel click lands wherever the pointer happens to be. When the
+ target maps a Tk library or Python's `_tkinter`, a background window pixel
+ click returns `background_unavailable` with `reason:
+ "pointer_reading_toolkit"`. Other pointer-reading apps cannot be detected
+ cheaply; their background result stays "not driver-verified" and says the
+ pointer was not moved. Retry with `delivery_mode:"foreground"`, which moves
+ the pointer first.
+
- **Chromium `<video>` play/pause**: pixel click is often rejected
by HTML5's click-to-play handler on some builds. Use keyboard
instead: `press_key({pid, key: "k"})` (YouTube) or
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
index 3eb011c729..82c6267b71 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
@@ -397,7 +397,7 @@ fn harness_appkit_foreground_single_click_has_one_ordered_native_pair() {
"appkit",
"single_click_native_pair",
Targeting::Px,
- DriverRoute::MacosCgEventPid,
+ DriverRoute::MacosCgEventHid,
);
execute_case(case, |evidence| {
let mut driver =
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rs
new file mode 100644
index 0000000000..d496152ff0
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/tk_pointer_click_macos_test.rs
@@ -0,0 +1,285 @@
+//! macOS window-scoped pixel clicks on a toolkit that reads the hardware
+//! pointer (D-MAC-6).
+//!
+//! Tk's macOS backend derives a mouse event's location from the global
+//! pointer, not from the delivered CGEvent. The repository's Tk
+//! `visual-only-canvas` fixture therefore logged `ignored click outside cards`
+//! for window-scoped clicks even at the correct screenshot point, because the
+//! PID-routed event never moved the pointer. These rows prove the contract:
+//!
+//! * `delivery_mode:"foreground"` activates the exact window, moves the
+//! hardware pointer to the mapped point, and posts through the HID tap, so
+//! the fixture's own loopback oracle records exactly one `send` selection
+//! and the pointer is left at the target (as on Windows and X11).
+//! * `delivery_mode:"background"` returns the structured
+//! `background_unavailable` refusal instead of an unverifiable success, and
+//! the oracle is unchanged.
+//!
+//! The rows skip with an explicit reason when `python3` lacks Tk support
+//! (GitHub-hosted runners may not ship it) unless `CUA_TEST_REQUIRE_TK` is
+//! set.
+//!
+//! Run with:
+//! `cargo test -p cua-driver-e2e --test tk_pointer_click_macos_test -- --ignored --nocapture --test-threads=1`
+
+#![cfg(target_os = "macos")]
+
+use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
+use std::thread;
+use std::time::{Duration, Instant};
+
+use cua_driver_testkit::{spawn_in_job, Driver, FixtureJournal, McpDriver};
+use serde_json::{json, Value};
+
+/// Canvas size and the Send card center from the fixture's `CARDS` table.
+const CANVAS_WIDTH: f64 = 760.0;
+const CANVAS_HEIGHT: f64 = 460.0;
+const SEND_CENTER: (f64, f64) = (394.0, 221.0);
+const SEND_BOUNDS: (f64, f64, f64, f64) = (292.0, 132.0, 496.0, 310.0);
+
+fn fixture_path() -> PathBuf {
+ Path::new(env!("CARGO_MANIFEST_DIR"))
+ .join("../../../tests/fixtures/apps/cross-platform/visual-only-canvas/main.py")
+}
+
+/// Return a skip reason when this host cannot run the Tk fixture.
+fn tk_unavailable() -> Option<String> {
+ let probe = Command::new("python3")
+ .args(["-c", "import tkinter; print(tkinter.TkVersion)"])
+ .stdin(Stdio::null())
+ .output();
+ let reason = match probe {
+ Err(error) => format!("python3 is unavailable: {error}"),
+ Ok(output) if !output.status.success() => format!(
+ "python3 has no importable tkinter: {}",
+ String::from_utf8_lossy(&output.stderr).trim()
+ ),
+ Ok(_) => return None,
+ };
+ if std::env::var_os("CUA_TEST_REQUIRE_TK").is_some() {
+ panic!("CUA_TEST_REQUIRE_TK is set but {reason}");
+ }
+ Some(reason)
+}
+
+struct TkFixture {
+ driver: McpDriver,
+ journal: FixtureJournal,
+ pid: i64,
+ window_id: u64,
+ session: String,
+}
+
+fn launch(label: &str) -> Option<TkFixture> {
+ if let Some(reason) = tk_unavailable() {
+ eprintln!("[tk-pointer] SKIP {label}: {reason}");
+ return None;
+ }
+ let journal = FixtureJournal::start();
+ let title = format!("Cua Tk Pointer Fixture [{label}]");
+ let child = spawn_in_job(
+ Command::new("python3")
+ .arg(fixture_path())
+ .args(["--journal-url", journal.url(), "--title", &title])
+ .stdin(Stdio::null())
+ .stdout(Stdio::null())
+ .stderr(Stdio::inherit()),
+ )
+ .expect("start the Tk visual-only-canvas fixture");
+ let mut driver =
+ McpDriver::spawn_macos_daemon_proxy_named(label).expect("connect to the macOS daemon");
+ driver.reaper().push(child);
+ wait_for(
+ || journal.snapshot()["ready"].as_bool() == Some(true),
+ "Tk fixture never published readiness",
+ );
+ let pid = journal.snapshot()["pid"]
+ .as_i64()
+ .filter(|pid| *pid > 0)
+ .expect("fixture journal reports its pid");
+ let (window_id, _) = driver
+ .find_window(pid, &title)
+ .expect("find the exact Tk fixture window");
+ let session = format!("{label}-session");
+ let started = driver.call(
+ "start_session",
+ json!({"session": session, "capture_scope": "window"}),
+ );
+ assert!(!started.is_error(), "start_session: {}", started.text());
+ Some(TkFixture {
+ driver,
+ journal,
+ pid,
+ window_id,
+ session,
+ })
+}
+
+fn wait_for(mut predicate: impl FnMut() -> bool, message: &str) {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ while !predicate() {
+ assert!(Instant::now() < deadline, "{message}");
+ thread::sleep(Duration::from_millis(50));
+ }
+}
+
+/// Window-local screenshot pixel for the Send card center. The capture spans
+/// the full window: the canvas fills its width and sits under the title bar.
+fn send_screenshot_point(fixture: &mut TkFixture) -> (f64, f64) {
+ let state = fixture.driver.call(
+ "get_window_state",
+ json!({
+ "session": fixture.session,
+ "pid": fixture.pid,
+ "window_id": fixture.window_id,
+ }),
+ );
+ assert!(!state.is_error(), "get_window_state: {}", state.text());
+ let width = state.structured()["screenshot_width"]
+ .as_f64()
+ .expect("screenshot width");
+ let height = state.structured()["screenshot_height"]
+ .as_f64()
+ .expect("screenshot height");
+ let scale = width / CANVAS_WIDTH;
+ let title_bar = height - CANVAS_HEIGHT * scale;
+ assert!(
+ scale > 0.0 && (0.0..=80.0 * scale).contains(&title_bar),
+ "unexpected Tk window capture geometry {width}x{height}"
+ );
+ (SEND_CENTER.0 * scale, title_bar + SEND_CENTER.1 * scale)
+}
+
+fn window_bounds(fixture: &mut TkFixture) -> Value {
+ let windows = fixture
+ .driver
+ .call("list_windows", json!({ "pid": fixture.pid }));
+ windows.structured()["windows"]
+ .as_array()
+ .and_then(|windows| {
+ windows
+ .iter()
+ .find(|window| window["window_id"].as_u64() == Some(fixture.window_id))
+ })
+ .map(|window| window["bounds"].clone())
+ .expect("fixture window bounds")
+}
+
+fn pointer(fixture: &mut TkFixture) -> (f64, f64) {
+ let position = fixture.driver.call("get_cursor_position", json!({}));
+ assert!(
+ !position.is_error(),
+ "get_cursor_position: {}",
+ position.text()
+ );
+ (
+ position.structured()["x"].as_f64().expect("pointer x"),
+ position.structured()["y"].as_f64().expect("pointer y"),
+ )
+}
+
+#[test]
+#[ignore]
+fn tk_window_pixel_click_foreground_moves_pointer_and_reaches_target() {
+ let Some(mut fixture) = launch("tk-pointer-foreground") else {
+ return;
+ };
+ let (x, y) = send_screenshot_point(&mut fixture);
+ let click = fixture.driver.call(
+ "click",
+ json!({
+ "pid": fixture.pid,
+ "window_id": fixture.window_id,
+ "x": x,
+ "y": y,
+ "delivery_mode": "foreground",
+ }),
+ );
+ assert!(!click.is_error(), "foreground click failed: {}", click.raw);
+ assert_eq!(click.action_route(), Some("global_input"), "{}", click.raw);
+ assert_eq!(
+ click.action_delivery_mode(),
+ Some("foreground"),
+ "{}",
+ click.raw
+ );
+ assert!(
+ click.text().contains("hardware pointer moved"),
+ "result must say the pointer moved: {}",
+ click.text()
+ );
+ wait_for(
+ || {
+ let state = fixture.journal.snapshot();
+ state["selected"] == "send" && state["action_count"] == 1
+ },
+ &format!(
+ "Tk oracle did not record exactly one Send selection: {}",
+ fixture.journal.snapshot()
+ ),
+ );
+
+ // The foreground contract leaves the hardware pointer at the target, like
+ // Windows SendInput and X11 XTest. Allow a small title-bar tolerance.
+ let bounds = window_bounds(&mut fixture);
+ let (px, py) = pointer(&mut fixture);
+ let left = bounds["x"].as_f64().unwrap();
+ let bottom = bounds["y"].as_f64().unwrap() + bounds["height"].as_f64().unwrap();
+ let canvas_top = bottom - CANVAS_HEIGHT;
+ let inside_send = px >= left + SEND_BOUNDS.0
+ && px <= left + SEND_BOUNDS.2
+ && py >= canvas_top + SEND_BOUNDS.1 - 4.0
+ && py <= canvas_top + SEND_BOUNDS.3 + 4.0;
+ assert!(
+ inside_send,
+ "hardware pointer ({px},{py}) is not over the Send card; window bounds {bounds}"
+ );
+}
+
+#[test]
+#[ignore]
+fn tk_window_pixel_click_background_is_refused_without_delivery() {
+ let Some(mut fixture) = launch("tk-pointer-background") else {
+ return;
+ };
+ let (x, y) = send_screenshot_point(&mut fixture);
+ let click = fixture.driver.call(
+ "click",
+ json!({
+ "pid": fixture.pid,
+ "window_id": fixture.window_id,
+ "x": x,
+ "y": y,
+ "delivery_mode": "background",
+ }),
+ );
+ assert!(
+ click.is_error(),
+ "background Tk click must refuse: {}",
+ click.raw
+ );
+ let structured = click.structured();
+ assert_eq!(
+ structured["code"], "background_unavailable",
+ "{}",
+ click.raw
+ );
+ assert_eq!(
+ structured["reason"], "pointer_reading_toolkit",
+ "{}",
+ click.raw
+ );
+ assert_eq!(structured["toolkit"], "tk", "{}", click.raw);
+ assert_eq!(
+ structured["escalation"]["recommended"], "foreground",
+ "{}",
+ click.raw
+ );
+ thread::sleep(Duration::from_millis(500));
+ let state = fixture.journal.snapshot();
+ assert!(
+ state["selected"].is_null() && state["action_count"] == 0,
+ "a refused background click must not reach the fixture: {state}"
+ );
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
index 7411233b96..123289a225 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
@@ -12,6 +12,7 @@ pub mod ax_actions;
pub mod interactive;
pub mod keyboard;
pub mod mouse;
+pub mod pointer_toolkit;
pub mod skylight;
pub use ax_actions::perform_ax_action;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rs
new file mode 100644
index 0000000000..5b4309014c
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/pointer_toolkit.rs
@@ -0,0 +1,257 @@
+//! Detect target processes whose UI toolkit derives pointer-event locations
+//! from the hardware pointer rather than from the delivered event.
+//!
+//! Tk's macOS backend reads the global pointer position when it translates an
+//! incoming mouse event into Tk coordinates. A PID-routed background CGEvent
+//! never moves the hardware pointer, so such an app receives the click wherever
+//! the real pointer happens to be (for example the visual-only canvas fixture's
+//! `ignored click outside cards at canvas (-86, -114)`), while the driver can
+//! only report the post as unverifiable. Background delivery therefore cannot
+//! satisfy these targets; the explicit foreground rung moves the pointer first.
+//!
+//! Detection is intentionally cheap and conservative: walk the target's
+//! file-backed memory regions with `proc_pidinfo` (same-user, no task port) and
+//! look for a mapped Tk library or Python's `_tkinter` extension. A toolkit
+//! loaded only from the dyld shared cache is not visible this way; such targets
+//! fall back to the honest "not driver-verified" background result.
+
+/// A UI toolkit known to derive click locations from the hardware pointer.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum PointerReadingToolkit {
+ Tk,
+}
+
+impl PointerReadingToolkit {
+ pub fn name(self) -> &'static str {
+ match self {
+ Self::Tk => "tk",
+ }
+ }
+}
+
+/// Classify one mapped image path. Pure so the matcher is unit-testable.
+///
+/// Matches `Tk.framework` bundles, Tk shared libraries (`libtk8.6.dylib`,
+/// Tk 9's `libtcl9tk9.0.dylib`), and CPython's `_tkinter` extension module,
+/// which either links Tk dynamically or embeds it statically.
+pub fn classify_image_path(path: &str) -> Option<PointerReadingToolkit> {
+ if path.contains("/Tk.framework/") {
+ return Some(PointerReadingToolkit::Tk);
+ }
+ let file = path.rsplit('/').next().unwrap_or(path);
+ let lower = file.to_ascii_lowercase();
+ if lower.starts_with("_tkinter") && lower.ends_with(".so") {
+ return Some(PointerReadingToolkit::Tk);
+ }
+ if !lower.ends_with(".dylib") {
+ return None;
+ }
+ let rest = lower.strip_prefix("lib")?;
+ // Tk 9 prefixes the Tcl major version: `libtcl9tk9.0.dylib`.
+ let rest = match rest.strip_prefix("tcl") {
+ Some(after_tcl) => {
+ let digits = after_tcl.bytes().take_while(u8::is_ascii_digit).count();
+ if digits == 0 {
+ return None;
+ }
+ &after_tcl[digits..]
+ }
+ None => rest,
+ };
+ let after_tk = rest.strip_prefix("tk")?;
+ after_tk
+ .bytes()
+ .next()
+ .filter(u8::is_ascii_digit)
+ .map(|_| PointerReadingToolkit::Tk)
+}
+
+/// Return the pointer-reading toolkit mapped into `pid`, if any.
+#[cfg(target_os = "macos")]
+pub fn detect(pid: i32) -> Option<PointerReadingToolkit> {
+ // Prefer the vnode-only flavor, which skips anonymous regions; fall back
+ // to the full walk on kernels that reject it.
+ match walk_regions(pid, PROC_PIDREGIONPATHINFO2) {
+ Some(found) => found,
+ None => walk_regions(pid, PROC_PIDREGIONPATHINFO).flatten(),
+ }
+}
+
+#[cfg(not(target_os = "macos"))]
+pub fn detect(_pid: i32) -> Option<PointerReadingToolkit> {
+ None
+}
+
+#[cfg(target_os = "macos")]
+const PROC_PIDREGIONPATHINFO: libc::c_int = 8;
+#[cfg(target_os = "macos")]
+const PROC_PIDREGIONPATHINFO2: libc::c_int = 22;
+/// Bound the walk so a pathological address space cannot stall a click.
+#[cfg(target_os = "macos")]
+const MAX_REGIONS: usize = 65_536;
+
+/// `struct proc_regioninfo` from `<sys/proc_info.h>`.
+#[cfg(target_os = "macos")]
+#[repr(C)]
+#[derive(Clone, Copy)]
+struct ProcRegionInfo {
+ pri_protection: u32,
+ pri_max_protection: u32,
+ pri_inheritance: u32,
+ pri_flags: u32,
+ pri_offset: u64,
+ pri_behavior: u32,
+ pri_user_wired_count: u32,
+ pri_user_tag: u32,
+ pri_pages_resident: u32,
+ pri_pages_shared_now_private: u32,
+ pri_pages_swapped_out: u32,
+ pri_pages_dirtied: u32,
+ pri_ref_count: u32,
+ pri_shadow_depth: u32,
+ pri_share_mode: u32,
+ pri_private_pages_resident: u32,
+ pri_shared_pages_resident: u32,
+ pri_obj_id: u32,
+ pri_depth: u32,
+ pri_address: u64,
+ pri_size: u64,
+}
+
+/// `struct proc_regionwithpathinfo`: region info followed by
+/// `struct vnode_info_path` (152-byte `vnode_info` + `MAXPATHLEN` path).
+#[cfg(target_os = "macos")]
+#[repr(C)]
+struct ProcRegionWithPathInfo {
+ prp_prinfo: ProcRegionInfo,
+ prp_vip_vi: [u8; 152],
+ prp_vip_path: [libc::c_char; 1024],
+}
+
+/// Walk `pid`'s regions with `flavor`. `None` means the flavor was rejected
+/// before any region was read; `Some(result)` is a completed walk.
+#[cfg(target_os = "macos")]
+fn walk_regions(pid: i32, flavor: libc::c_int) -> Option<Option<PointerReadingToolkit>> {
+ let size = std::mem::size_of::<ProcRegionWithPathInfo>();
+ let mut address: u64 = 0;
+ let mut read_any = false;
+ for _ in 0..MAX_REGIONS {
+ // SAFETY: an all-zero bit pattern is valid for this plain C struct.
+ let mut info: ProcRegionWithPathInfo = unsafe { std::mem::zeroed() };
+ // SAFETY: the buffer is exactly the kernel structure size and lives for
+ // the duration of the call.
+ let written = unsafe {
+ libc::proc_pidinfo(
+ pid,
+ flavor,
+ address,
+ (&mut info as *mut ProcRegionWithPathInfo).cast(),
+ size as libc::c_int,
+ )
+ };
+ if written <= 0 || (written as usize) < size {
+ break;
+ }
+ read_any = true;
+ // SAFETY: the kernel NUL-terminates `vip_path` within MAXPATHLEN.
+ let path = unsafe { std::ffi::CStr::from_ptr(info.prp_vip_path.as_ptr()) };
+ if let Some(toolkit) = classify_image_path(&path.to_string_lossy()) {
+ return Some(Some(toolkit));
+ }
+ let region = info.prp_prinfo;
+ let next = region.pri_address.saturating_add(region.pri_size);
+ if next <= address {
+ break;
+ }
+ address = next;
+ }
+ if read_any {
+ Some(None)
+ } else {
+ None
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn classifies_tk_images_and_rejects_neighbours() {
+ for path in [
+ "/Library/Frameworks/Tk.framework/Versions/8.6/Tk",
+ "/opt/homebrew/Cellar/tcl-tk/9.0.2/lib/libtcl9tk9.0.dylib",
+ "/opt/homebrew/opt/tcl-tk@8/lib/libtk8.6.dylib",
+ "/usr/local/lib/python3.12/lib-dynload/_tkinter.cpython-312-darwin.so",
+ "/Users/me/.local/share/uv/python/cpython-3.13/lib/python3.13/lib-dynload/_tkinter.cpython-313-darwin.so",
+ ] {
+ assert_eq!(
+ classify_image_path(path),
+ Some(PointerReadingToolkit::Tk),
+ "{path}"
+ );
+ }
+ for path in [
+ "",
+ "/usr/lib/libSystem.B.dylib",
+ "/opt/homebrew/lib/libtcl9.0.dylib",
+ "/opt/homebrew/lib/libtcl8.6.dylib",
+ "/opt/homebrew/lib/libtkrzw.dylib",
+ "/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit",
+ "/usr/lib/python3/_tkinter_helper.py",
+ "/Applications/Foo.app/Contents/MacOS/tk8.6",
+ ] {
+ assert_eq!(classify_image_path(path), None, "{path}");
+ }
+ }
+
+ #[cfg(target_os = "macos")]
+ #[test]
+ fn kernel_region_structure_matches_proc_info_layout() {
+ assert_eq!(std::mem::size_of::<ProcRegionInfo>(), 96);
+ assert_eq!(std::mem::size_of::<ProcRegionWithPathInfo>(), 1272);
+ }
+
+ #[cfg(target_os = "macos")]
+ #[test]
+ fn detect_is_negative_for_a_non_tk_process() {
+ // The test binary maps no Tk image. A completed negative walk proves the
+ // region flavor is readable for a same-user process.
+ assert_eq!(detect(std::process::id() as i32), None);
+ assert!(walk_regions(std::process::id() as i32, PROC_PIDREGIONPATHINFO).is_some());
+ }
+
+ /// Loads `_tkinter` without creating any window, so this never touches the
+ /// desktop. Skips when the host Python lacks Tk support.
+ #[cfg(target_os = "macos")]
+ #[test]
+ fn detect_finds_tk_in_a_python_process_that_imported_tkinter() {
+ use std::io::BufRead;
+ let spawned = std::process::Command::new("python3")
+ .args([
+ "-c",
+ "import _tkinter, sys, time; print('ready', flush=True); time.sleep(30)",
+ ])
+ .stdout(std::process::Stdio::piped())
+ .stderr(std::process::Stdio::null())
+ .spawn();
+ let Ok(mut child) = spawned else {
+ eprintln!("skipping: python3 is unavailable");
+ return;
+ };
+ let mut line = String::new();
+ let ready = child
+ .stdout
+ .take()
+ .map(|stdout| std::io::BufReader::new(stdout).read_line(&mut line))
+ .is_some_and(|read| read.is_ok())
+ && line.trim() == "ready";
+ let detected = ready.then(|| detect(child.id() as i32));
+ let _ = child.kill();
+ let _ = child.wait();
+ match detected {
+ None => eprintln!("skipping: python3 has no importable _tkinter"),
+ Some(found) => assert_eq!(found, Some(PointerReadingToolkit::Tk)),
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index e9c300ad83..fcf8bd969e 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -33,6 +33,7 @@ use crate::focus_guard;
use crate::window_change_detector::WindowChangeDetector;
use core_foundation::base::{CFRelease, CFTypeRef, TCFType};
+use super::pixel_route::PixelClickRoute;
use super::ToolState;
pub struct ClickTool {
@@ -1074,6 +1075,12 @@ impl Tool for ClickTool {
// requested foreground click without a window id still degrades to
// background, matching the existing contract and result label.
let fg = delivery_mode.is_foreground() && window_id.is_some();
+ // Background delivery cannot satisfy a toolkit that reads the
+ // hardware pointer; refuse before any activation or dispatch.
+ let route = match super::pixel_route::resolve(pid, fg, window_id, "mouse_click").await {
+ Ok(route) => route,
+ Err(refusal) => return refusal,
+ };
let activation_policy = pixel_activation_policy(&button_str, fg, window_id.is_some());
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
@@ -1149,11 +1156,14 @@ impl Tool for ClickTool {
"click.pixel",
|| async move {
tokio::task::spawn_blocking(move || {
- let has_modifiers = !mods_owned.is_empty();
let do_click = move || -> anyhow::Result<()> {
let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- if fg && !m.is_empty() {
- return crate::input::mouse::click_at_xy_desktop_with_modifiers_preserving_cursor(
+ if route == PixelClickRoute::ForegroundHid {
+ // Warp the hardware pointer to the mapped global
+ // point and post at the HID tap. The pointer stays
+ // at the target, as on Windows and X11, so apps that
+ // read the pointer when handling the event see it.
+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(
screen_x,
screen_y,
count,
@@ -1196,26 +1206,19 @@ impl Tool for ClickTool {
}
}
};
- // Foreground rung: brief front → click → restore.
- // Returns whether the window was ACTUALLY fronted, so the
- // reported `path` honestly reflects the rung that ran.
- match (fg, window_id, has_modifiers) {
- (true, Some(wid), true) => {
+ // Foreground rung: front the exact window → HID click →
+ // restore the prior front process. The HID tap has no
+ // pid addressing, so activation must be proven (the
+ // window is AX-focused) or no input is sent.
+ match (route, window_id) {
+ (PixelClickRoute::ForegroundHid, Some(wid)) => {
crate::input::skylight::with_foreground_hid_activation(
pid as libc::pid_t,
wid,
do_click,
)
- .map(|_| true)
}
- (true, Some(wid), false) => {
- crate::input::skylight::with_foreground_assist(
- pid as libc::pid_t,
- wid,
- do_click,
- )
- }
- _ => do_click().map(|_| false),
+ _ => do_click(),
}
})
.await
@@ -1268,27 +1271,31 @@ impl Tool for ClickTool {
_ => "click",
};
match result {
- Ok(Ok(fronted)) => {
- // `with_foreground_assist` returns `false` when the fronting SPIs
- // were unavailable and it clicked WITHOUT activation — report the
- // background path in that case so `path` reflects the rung that ran.
- let (path, mode_label) = if fg && fronted {
- ("cgevent_fg", "foreground CGEvent")
+ Ok(Ok(())) => {
+ let target = if route == PixelClickRoute::ForegroundHid {
+ format!("at screen-point ({screen_x:.0},{screen_y:.0}) for pid {pid}")
} else {
- ("cgevent", "background CGEvent")
+ format!("to pid {pid}")
};
ToolResult::text(format!(
- "✅ Posted {button_label} to pid {pid} ({mode_label}; \
- not driver-verified — confirm via screenshot).{}",
+ "✅ Posted {button_label} {target} ({}).{}",
+ super::pixel_route::delivery_note(route),
changes.result_suffix()
))
.with_structured(serde_json::json!({
- "path": path,
+ "path": super::pixel_route::path_label(route),
"verified": false,
"effect": "unverifiable",
"focus_without_raise": focus_without_raise
}))
}
+ Ok(Err(e)) if route == PixelClickRoute::ForegroundHid => {
+ super::pixel_route::foreground_unavailable(
+ button_label,
+ window_id.unwrap_or_default(),
+ &e.to_string(),
+ )
+ }
Ok(Err(e)) => ToolResult::error(format!("{button_label} failed: {e}")),
Err(e) => ToolResult::error(format!("Task error: {e}")),
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
index 03190160ed..4e8bf84ec1 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
@@ -247,8 +247,24 @@ impl Tool for DoubleClickTool {
);
let fg = delivery_mode.is_foreground() && window_id.is_some();
+ let route =
+ match super::pixel_route::resolve(pid, fg, window_id, "mouse_double_click").await {
+ Ok(route) => route,
+ Err(refusal) => return refusal,
+ };
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
let do_click = move || -> anyhow::Result<()> {
+ if route == super::pixel_route::PixelClickRoute::ForegroundHid {
+ // Warp the hardware pointer and post at the HID tap; see
+ // `pixel_route` for the cross-platform foreground contract.
+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(
+ screen_x,
+ screen_y,
+ 2,
+ "left",
+ &[],
+ );
+ }
if let Some(wid) = window_id {
crate::input::mouse::click_at_xy_with_window_local(
pid,
@@ -265,31 +281,37 @@ impl Tool for DoubleClickTool {
crate::input::mouse::click_at_xy(pid, screen_x, screen_y, 2, &[])
}
};
- // Foreground rung: brief front → double-click → restore prior frontmost.
- match (fg, window_id) {
- (true, Some(wid)) => {
- crate::input::skylight::with_foreground_assist(
+ // Foreground rung: front the exact window → HID double-click →
+ // restore the prior frontmost. No input is sent unless the exact
+ // window is proven focused.
+ match (route, window_id) {
+ (super::pixel_route::PixelClickRoute::ForegroundHid, Some(wid)) => {
+ crate::input::skylight::with_foreground_hid_activation(
pid as libc::pid_t,
wid,
do_click,
- )?;
- Ok(())
+ )
}
_ => do_click(),
}
})
.await;
- let mode_label = if fg {
- " (delivery_mode:foreground)"
- } else {
- ""
- };
match result {
- Ok(Ok(())) => ToolResult::text(format!("✅ Double-clicked at ({screen_x:.1}, {screen_y:.1}){mode_label}."))
- .with_structured(serde_json::json!({
- "path": if fg { "cgevent_fg" } else { "cgevent" }, "verified": false, "effect": "unverifiable"
- })),
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Double-clicked at ({screen_x:.1}, {screen_y:.1}) ({}).",
+ super::pixel_route::delivery_note(route)
+ ))
+ .with_structured(serde_json::json!({
+ "path": super::pixel_route::path_label(route), "verified": false, "effect": "unverifiable"
+ })),
+ Ok(Err(e)) if route == super::pixel_route::PixelClickRoute::ForegroundHid => {
+ super::pixel_route::foreground_unavailable(
+ "Double-click",
+ window_id.unwrap_or_default(),
+ &e.to_string(),
+ )
+ }
Ok(Err(e)) => ToolResult::error(format!("Double-click failed: {e}")),
Err(e) => ToolResult::error(format!("Task error: {e}")),
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index de12ecb254..b6ec2b0fa4 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -34,6 +34,7 @@ pub(crate) mod get_screen_size;
mod health_report;
mod move_cursor;
mod page;
+mod pixel_route;
pub(crate) mod px_frame;
mod set_config;
mod type_text_chars;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rs
new file mode 100644
index 0000000000..36a4df3ff3
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/pixel_route.rs
@@ -0,0 +1,209 @@
+//! Transport selection for window-addressed pixel clicks (`click`,
+//! `double_click`, `right_click`) on macOS.
+//!
+//! The cross-platform foreground contract is that a foreground pixel click
+//! moves the system pointer to the target and delivers real input there:
+//! Windows uses `SetCursorPos` + `SendInput`, X11 uses an XTest warp + button,
+//! and macOS desktop scope and foreground drags warp the pointer and post at the
+//! HID tap. Window-scoped foreground pixel clicks follow the same model, so apps
+//! that derive a click location from the hardware pointer (Tk, some Java and
+//! game toolkits) receive it at the right point. Background delivery stays
+//! PID-routed and never moves the pointer; a detectable pointer-reading toolkit
+//! refuses with the shared `background_unavailable` contract instead of
+//! reporting an unverifiable success.
+
+use cua_driver_core::protocol::ToolResult;
+
+use crate::input::pointer_toolkit::PointerReadingToolkit;
+
+/// Transport for a pixel click once any background AX hit-test backend has
+/// declined (or was not eligible).
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(crate) enum PixelClickRoute {
+ /// Explicit foreground rung with an exact window: activate that window,
+ /// move the hardware pointer to the mapped global point, and post through
+ /// the global HID event tap. The pointer stays at the target.
+ ForegroundHid,
+ /// Background delivery routed to the target pid. The hardware pointer is
+ /// never moved, so the result stays "not driver-verified".
+ BackgroundPid,
+ /// Background delivery refused: the target's toolkit reads the hardware
+ /// pointer, so a PID-routed event would land wherever the pointer is.
+ RefuseBackground(PointerReadingToolkit),
+}
+
+/// Pure route decision. `effective_foreground` is the requested foreground
+/// rung; it only selects the HID route with an exact window, because the HID
+/// tap has no pid addressing.
+pub(crate) fn pixel_click_route(
+ effective_foreground: bool,
+ has_window: bool,
+ toolkit: Option<PointerReadingToolkit>,
+) -> PixelClickRoute {
+ if effective_foreground && has_window {
+ return PixelClickRoute::ForegroundHid;
+ }
+ match toolkit {
+ Some(toolkit) if has_window => PixelClickRoute::RefuseBackground(toolkit),
+ _ => PixelClickRoute::BackgroundPid,
+ }
+}
+
+/// Resolve the route for a live target, probing the toolkit only for
+/// window-addressed background clicks. Returns the structured refusal when
+/// background delivery cannot reach the target.
+pub(crate) async fn resolve(
+ pid: i32,
+ effective_foreground: bool,
+ window_id: Option<u32>,
+ event_kind: &str,
+) -> Result<PixelClickRoute, ToolResult> {
+ let has_window = window_id.is_some();
+ let toolkit = if !effective_foreground && has_window {
+ tokio::task::spawn_blocking(move || crate::input::pointer_toolkit::detect(pid))
+ .await
+ .ok()
+ .flatten()
+ } else {
+ None
+ };
+ match pixel_click_route(effective_foreground, has_window, toolkit) {
+ PixelClickRoute::RefuseBackground(toolkit) => {
+ Err(pointer_reading_background_refusal(pid, toolkit, event_kind))
+ }
+ route => Ok(route),
+ }
+}
+
+pub(crate) fn pointer_reading_background_refusal(
+ pid: i32,
+ toolkit: PointerReadingToolkit,
+ event_kind: &str,
+) -> ToolResult {
+ let name = toolkit.name();
+ cua_driver_core::delivery::background_unavailable_result(
+ format!(
+ "Background pixel click is not available for pid {pid}: its {name} toolkit \
+ derives click locations from the hardware pointer, and background delivery \
+ never moves the pointer, so the click would land wherever the pointer \
+ currently is. Retry this action with delivery_mode:\"foreground\"; Cua \
+ Driver will activate the window, move the pointer to the target, and click."
+ ),
+ "background_unavailable",
+ "the target toolkit reads the hardware pointer position, which background \
+ PID-routed events cannot move; retry with delivery_mode:\"foreground\".",
+ serde_json::json!({
+ "event_kind": event_kind,
+ "reason": "pointer_reading_toolkit",
+ "toolkit": name,
+ }),
+ )
+}
+
+/// Human-readable delivery note appended to a successful pixel click result.
+pub(crate) fn delivery_note(route: PixelClickRoute) -> &'static str {
+ match route {
+ PixelClickRoute::ForegroundHid => {
+ "foreground: exact window activated, hardware pointer moved to the target, \
+ HID event tap; not driver-verified — confirm via screenshot"
+ }
+ _ => {
+ "background CGEvent routed to the pid; the hardware pointer was not moved, \
+ so apps that read the pointer position may not receive it at the target; \
+ not driver-verified — confirm via screenshot and retry with \
+ delivery_mode:\"foreground\" if nothing changed"
+ }
+ }
+}
+
+/// Legacy `path` label consumed by the action-truth normalizer:
+/// `cgevent_fg` maps to the HID transport with foreground delivery and
+/// `cgevent` to the PID transport with background delivery.
+pub(crate) fn path_label(route: PixelClickRoute) -> &'static str {
+ match route {
+ PixelClickRoute::ForegroundHid => "cgevent_fg",
+ _ => "cgevent",
+ }
+}
+
+/// Structured error for a foreground HID click whose exact-window activation
+/// or dispatch failed. No input reaches another window in that case.
+pub(crate) fn foreground_unavailable(action: &str, window_id: u32, cause: &str) -> ToolResult {
+ ToolResult::error(format!(
+ "{action} failed: foreground HID delivery to window {window_id} was not possible: \
+ {cause}"
+ ))
+ .with_structured(serde_json::json!({ "code": "foreground_unavailable" }))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// D-MAC-6: a foreground window pixel click must use the HID tap with a
+ /// real pointer move (Tk reads the hardware pointer), background clicks
+ /// keep PID routing, and a known pointer-reading toolkit refuses
+ /// background delivery instead of reporting an unverifiable success.
+ #[test]
+ fn pixel_click_route_moves_pointer_only_for_foreground_and_refuses_tk_background() {
+ let tk = Some(PointerReadingToolkit::Tk);
+ assert_eq!(
+ pixel_click_route(true, true, None),
+ PixelClickRoute::ForegroundHid
+ );
+ assert_eq!(
+ pixel_click_route(true, true, tk),
+ PixelClickRoute::ForegroundHid,
+ "foreground delivery moves the pointer, so Tk is supported"
+ );
+ assert_eq!(
+ pixel_click_route(false, true, None),
+ PixelClickRoute::BackgroundPid
+ );
+ assert_eq!(
+ pixel_click_route(false, true, tk),
+ PixelClickRoute::RefuseBackground(PointerReadingToolkit::Tk)
+ );
+ assert_eq!(
+ pixel_click_route(false, false, tk),
+ PixelClickRoute::BackgroundPid,
+ "legacy window-less pid clicks keep their established route"
+ );
+ assert_eq!(
+ pixel_click_route(true, false, None),
+ PixelClickRoute::BackgroundPid,
+ "foreground without an exact window cannot use the unaddressed HID tap"
+ );
+ }
+
+ #[test]
+ fn route_labels_say_what_actually_happened() {
+ assert_eq!(path_label(PixelClickRoute::ForegroundHid), "cgevent_fg");
+ assert_eq!(path_label(PixelClickRoute::BackgroundPid), "cgevent");
+ assert!(delivery_note(PixelClickRoute::ForegroundHid).contains("hardware pointer moved"));
+ assert!(delivery_note(PixelClickRoute::BackgroundPid).contains("pointer was not moved"));
+ }
+
+ #[test]
+ fn pointer_reading_refusal_uses_the_shared_background_unavailable_contract() {
+ let result =
+ pointer_reading_background_refusal(42, PointerReadingToolkit::Tk, "mouse_click");
+ assert_eq!(result.is_error, Some(true));
+ let structured = result.structured_content.expect("structured refusal");
+ assert_eq!(structured["code"], "background_unavailable");
+ assert_eq!(structured["reason"], "pointer_reading_toolkit");
+ assert_eq!(structured["toolkit"], "tk");
+ assert_eq!(structured["event_kind"], "mouse_click");
+ assert_eq!(structured["escalation"]["recommended"], "foreground");
+ }
+
+ #[test]
+ fn foreground_failure_is_structured() {
+ let result = foreground_unavailable("click", 7, "window never focused");
+ assert_eq!(result.is_error, Some(true));
+ assert_eq!(
+ result.structured_content.expect("structured")["code"],
+ "foreground_unavailable"
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
index 1a05770851..70ed22ebb0 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
@@ -247,9 +247,21 @@ impl Tool for RightClickTool {
};
let fg = delivery_mode.is_foreground() && window_id.is_some();
+ let route = match super::pixel_route::resolve(pid, fg, window_id, "mouse_right_click").await
+ {
+ Ok(route) => route,
+ Err(refusal) => return refusal,
+ };
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
let do_it = move || -> anyhow::Result<()> {
let m: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ if route == super::pixel_route::PixelClickRoute::ForegroundHid {
+ // Warp the hardware pointer and post at the HID tap; see
+ // `pixel_route` for the cross-platform foreground contract.
+ return crate::input::mouse::click_at_xy_desktop_with_modifiers(
+ screen_x, screen_y, 1, "right", &m,
+ );
+ }
if let Some(wid) = window_id {
crate::input::mouse::right_click_at_xy_with_window_local(
pid,
@@ -264,26 +276,36 @@ impl Tool for RightClickTool {
crate::input::mouse::right_click_at_xy(pid, screen_x, screen_y, &m)
}
};
- // Foreground rung: brief front → right-click → restore prior frontmost.
- match (fg, window_id) {
- (true, Some(wid)) => {
- crate::input::skylight::with_foreground_assist(pid as libc::pid_t, wid, do_it)?;
- Ok(())
+ // Foreground rung: front the exact window → HID right-click →
+ // restore the prior frontmost. No input is sent unless the exact
+ // window is proven focused.
+ match (route, window_id) {
+ (super::pixel_route::PixelClickRoute::ForegroundHid, Some(wid)) => {
+ crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ do_it,
+ )
}
_ => do_it(),
}
})
.await;
- let mode_label = if fg {
- " (delivery_mode:foreground)"
- } else {
- ""
- };
match result {
- Ok(Ok(())) => ToolResult::text(format!("Right-clicked{mod_suffix} at ({screen_x:.1}, {screen_y:.1}){mode_label}."))
- .with_structured(serde_json::json!({
- "path": if fg { "cgevent_fg" } else { "cgevent" }, "verified": false, "effect": "unverifiable"
- })),
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Right-clicked{mod_suffix} at ({screen_x:.1}, {screen_y:.1}) ({}).",
+ super::pixel_route::delivery_note(route)
+ ))
+ .with_structured(serde_json::json!({
+ "path": super::pixel_route::path_label(route), "verified": false, "effect": "unverifiable"
+ })),
+ Ok(Err(e)) if route == super::pixel_route::PixelClickRoute::ForegroundHid => {
+ super::pixel_route::foreground_unavailable(
+ "Right-click",
+ window_id.unwrap_or_default(),
+ &e.to_string(),
+ )
+ }
Ok(Err(e)) => ToolResult::error(format!("Right-click failed: {e}")),
Err(e) => ToolResult::error(format!("Task error: {e}")),
}
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index e4dc7c3ceb..0ab5b0d6c2 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -401,6 +401,12 @@ EOF
run_test bring-to-front env -u CUA_E2E_RECORDINGS_ROOT \
cargo test -p cua-driver-e2e --test bring_to_front_macos_test -- \
--ignored --nocapture --test-threads=1
+ # Tk derives click locations from the hardware pointer. The rows skip with
+ # an explicit reason when python3 lacks tkinter (hosted images may not ship
+ # it); set CUA_TEST_REQUIRE_TK=1 where Tk is provisioned.
+ run_test tk-pointer-click env -u CUA_E2E_RECORDINGS_ROOT \
+ cargo test -p cua-driver-e2e --test tk_pointer_click_macos_test -- \
+ --ignored --nocapture --test-threads=1
fi
fi
if [[ "${SUITE}" == capture || "${SUITE}" == all ]]; then