Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): observe agent cursor moves/presses and report the system cursor shape (#3883)
ddupont7fe7c33f741e
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_hook.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_hook.rsnew file mode 100644index 0000000000..dca0598c34--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_hook.rs@@ -0,0 +1,83 @@+//! Cursor hook — a per-process callback fired whenever the agent cursor moves+//! or a press happens, so an embedder (for example a remote-desktop host) can+//! observe where every cursor is without polling or driving the overlay itself.+//!+//! Mirrors the `pip_hook` / `session` hook idiom: a single registered closure,+//! fired from the platform cursor-state write path. Coordinates are SCREEN+//! points (the space the overlay works in); the embedder maps them to whatever+//! window/target space it needs. No-op until an embedder registers a hook, so+//! there is zero cost in the common (daemon / CLI) case.++use std::sync::atomic::{AtomicBool, Ordering};+use std::sync::OnceLock;++/// One cursor update: the cursor identified by `cursor_id` is at screen point+/// (`x`, `y`), optionally in a pressed state (a click/drag press). Emitted on+/// every commanded move and on press edges.+#[derive(Debug, Clone)]+pub struct CursorHookEvent {+ pub cursor_id: String,+ pub x: f64,+ pub y: f64,+ pub pressed: bool,+}++type CursorHookFnBox = Box<dyn Fn(CursorHookEvent) + Send + Sync>;+static CURSOR_HOOK_FN: OnceLock<CursorHookFnBox> = OnceLock::new();++/// Register the process-wide cursor observer. Call once at startup.+///+/// Returns `true` if this call installed the observer, `false` if one was+/// already registered — in which case `f` is dropped and the existing observer+/// keeps receiving events.+///+/// Check the result. Registration is one-shot and there is no deregistration,+/// so a second caller silently gets nothing: a host that assumed it had the+/// cursor stream would render no pointers and have no way to find out why. If+/// this returns `false`, something else in the process owns the stream. Two+/// consumers in one process need a fan-out observer registered once, not two+/// calls here.+pub fn set_cursor_hook_fn(f: impl Fn(CursorHookEvent) + Send + Sync + 'static) -> bool {+ CURSOR_HOOK_FN.set(Box::new(f)).is_ok()+}++/// True when an observer is registered (lets hot paths skip building an event).+///+/// This answers "does anybody want these events", **not** "will any arrive".+/// See [`cursor_hook_supported`] for the latter.+pub fn cursor_hook_enabled() -> bool {+ CURSOR_HOOK_FN.get().is_some()+}++static EMITTER_DECLARED: AtomicBool = AtomicBool::new(false);++/// Declare that this platform adapter emits cursor events. Called once by an+/// adapter that actually drives [`push_cursor_event`] from its cursor write+/// path, during tool registration.+pub fn declare_cursor_hook_emitter() {+ EMITTER_DECLARED.store(true, Ordering::Release);+}++/// Whether this build emits cursor events at all.+///+/// Registering a hook always succeeds, so `cursor_hook_enabled()` cannot tell+/// an embedder whether events will ever arrive. Today only the macOS adapter+/// emits; the Windows, X11 and Wayland adapters have no cursor write path+/// wired to this hook, so on those hosts a registered hook stays silent+/// forever.+///+/// Per the cross-platform contract, a host must publish that limitation+/// explicitly — "cursor tracking unavailable on this platform" — rather than+/// showing a viewer a pointer that never moves and letting it look like a+/// hung stream. Check this at startup, not by waiting for a first event that+/// is not coming.+pub fn cursor_hook_supported() -> bool {+ EMITTER_DECLARED.load(Ordering::Acquire)+}++/// Fire a cursor update. No-op when nothing is registered.+pub fn push_cursor_event(event: CursorHookEvent) {+ if let Some(f) = CURSOR_HOOK_FN.get() {+ f(event);+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_shape.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_shape.rsnew file mode 100644index 0000000000..e65ee1b9e8--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/cursor_shape.rs@@ -0,0 +1,128 @@+//! What the operating system is currently drawing as the pointer.+//!+//! This is the *system* cursor — the I-beam over a text field, the resize+//! cursor on a window edge, the pointing hand over a link. It is distinct from+//! the agent-cursor overlay (`cursor_overlay`), which draws Cua's own cursor,+//! and from [`crate::cursor_hook`], which reports where a cursor *is*. A remote+//! viewer needs both: position says where to draw, shape says what to draw.+//!+//! ## Shape of this module+//!+//! Per the cross-platform contract, the vocabulary and the dispatch live here+//! in the common crate and each platform adapter stays thin: an adapter+//! installs a probe with [`set_cursor_shape_probe`], and everything else —+//! the enum, the default, the "nobody answered" behaviour — is shared.+//!+//! A platform that cannot read the system cursor simply never installs a+//! probe, and [`current_system_cursor_shape`] reports+//! [`SystemCursorShape::Unknown`]. `Unknown` is deliberately distinct from+//! [`SystemCursorShape::Default`]: it means "this host cannot tell you", not+//! "the pointer is an arrow", so a consumer can publish the limitation instead+//! of rendering a confidently wrong arrow.++use std::sync::OnceLock;++/// A platform-neutral system cursor shape.+///+/// Intentionally a small closed vocabulary plus a `Custom` escape. Consumers+/// map each variant onto their own native cursor, which keeps the pointer+/// crisp at any scale and lets the viewer honour its own accessibility+/// settings. Only a cursor with no portable equivalent needs to ship pixels.+#[derive(Debug, Clone, PartialEq, Default)]+pub enum SystemCursorShape {+ /// The ordinary arrow.+ #[default]+ Default,+ /// Text insertion.+ Text,+ /// Vertical-text insertion.+ VerticalText,+ /// A link or other clickable affordance.+ Pointer,+ /// Open hand over a draggable surface.+ Grab,+ /// Closed hand, drag in progress.+ Grabbing,+ /// Crosshair.+ Crosshair,+ /// Busy.+ Wait,+ /// The action is not permitted here.+ NotAllowed,+ /// A resize affordance on the given edge or corner.+ Resize(ResizeAxis),+ /// A cursor with no portable equivalent. `png` is the cursor image,+ /// `hotspot` is in that image's pixel space, `scale` is device pixels per+ /// point.+ Custom {+ png: Vec<u8>,+ hotspot_x: f64,+ hotspot_y: f64,+ scale: f64,+ },+ /// This host cannot report the system cursor shape. Not the same as+ /// `Default`.+ Unknown,+}++/// Which edge or corner a [`SystemCursorShape::Resize`] refers to.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum ResizeAxis {+ NorthSouth,+ EastWest,+ NorthEastSouthWest,+ NorthWestSouthEast,+ All,+ Column,+ Row,+}++type ProbeFn = Box<dyn Fn() -> SystemCursorShape + Send + Sync>;+static PROBE: OnceLock<ProbeFn> = OnceLock::new();++/// Install the platform probe. Call once, from the platform adapter's tool+/// registration. Subsequent calls are ignored, matching `set_cursor_hook_fn`.+pub fn set_cursor_shape_probe(probe: impl Fn() -> SystemCursorShape + Send + Sync + 'static) {+ let _ = PROBE.set(Box::new(probe));+}++/// Whether this build can report the system cursor shape at all.+///+/// Callers should use this to publish the limitation explicitly rather than+/// letting `Unknown` be mistaken for a transient read failure.+pub fn cursor_shape_supported() -> bool {+ PROBE.get().is_some()+}++/// The shape the OS is drawing right now, or [`SystemCursorShape::Unknown`]+/// when this platform installed no probe.+pub fn current_system_cursor_shape() -> SystemCursorShape {+ match PROBE.get() {+ Some(probe) => probe(),+ None => SystemCursorShape::Unknown,+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ /// The distinction the whole module rests on: an unsupported platform must+ /// report `Unknown`, never `Default`. Reporting `Default` would render a+ /// confident arrow on a host that has no idea what the pointer looks like.+ #[test]+ fn unknown_is_not_default() {+ assert_ne!(SystemCursorShape::Unknown, SystemCursorShape::Default);+ assert_eq!(SystemCursorShape::default(), SystemCursorShape::Default);+ }++ /// With no probe installed the accessor must say so rather than guess.+ /// This runs in a fresh test process for platforms whose adapter is not+ /// linked into this crate's tests (which is all of them: core never+ /// depends on a platform crate).+ #[test]+ fn no_probe_reports_unknown_and_unsupported() {+ assert!(!cursor_shape_supported());+ assert_eq!(current_system_cursor_shape(), SystemCursorShape::Unknown);+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex 11c7fb34c0..cb080f5cf8 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -52,7 +52,9 @@ pub mod cdp;pub mod clipboard;pub mod consent;pub mod cursor_events;+pub mod cursor_hook;pub mod cursor_sampler;+pub mod cursor_shape;pub mod daemon;pub mod element_cache;pub mod element_query;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_inert.rs b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_inert.rsnew file mode 100644index 0000000000..ee9407f95b--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_inert.rs@@ -0,0 +1,62 @@+//! The cursor hook must cost nothing for the users who never register one.+//!+//! That is the overwhelming majority of cua-driver consumers: the daemon, the+//! CLI, every SDK caller that does not embed a remote-desktop host. This file+//! is a *separate test binary* on purpose: `set_cursor_hook_fn` writes a+//! process-wide `OnceLock`, so "no hook is registered" is only observable in a+//! process where nothing ever registers one. Merging these assertions into a+//! binary that also exercises registration would silently stop testing the+//! unregistered case the moment test order changed.++use cua_driver_core::cursor_hook::{cursor_hook_enabled, push_cursor_event, CursorHookEvent};++#[test]+fn hook_is_inert_and_silent_when_nothing_is_registered() {+ assert!(+ !cursor_hook_enabled(),+ "no hook was registered in this process, so the hot-path fast check must report disabled"+ );++ // Pushing must be a no-op rather than a panic, an unwrap of an empty+ // OnceLock, or a deadlock. The macOS cursor write path calls this on every+ // commanded move, so a fault here would take down every consumer that never+ // asked for the feature.+ for i in 0..1000 {+ push_cursor_event(CursorHookEvent {+ cursor_id: format!("cursor-{i}"),+ x: i as f64,+ y: -(i as f64),+ pressed: i % 2 == 0,+ });+ }++ assert!(+ !cursor_hook_enabled(),+ "pushing events must not implicitly install a hook"+ );+}++/// Concurrent pushes with no hook installed must stay a no-op and must not+/// contend on anything. `OnceLock::get` on an unset cell is a relaxed load, so+/// this also documents that the unregistered path takes no lock.+#[test]+fn concurrent_pushes_without_a_hook_are_safe() {+ let threads: Vec<_> = (0..8)+ .map(|t| {+ std::thread::spawn(move || {+ for i in 0..500 {+ push_cursor_event(CursorHookEvent {+ cursor_id: format!("t{t}"),+ x: i as f64,+ y: i as f64,+ pressed: false,+ });+ }+ })+ })+ .collect();+ for t in threads {+ t.join().expect("no-op pushes must not panic");+ }+ assert!(!cursor_hook_enabled());+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_platform_support.rs b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_platform_support.rsnew file mode 100644index 0000000000..8d422c2201--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_platform_support.rs@@ -0,0 +1,80 @@+//! The cursor hook must publish its platform limitation explicitly.+//!+//! `AGENTS.md`: "When a platform cannot support the same contract, return or+//! publish that limitation explicitly instead of substituting misleading+//! behavior."+//!+//! Only the macOS adapter emits cursor events; the Windows and Linux adapters+//! have no cursor write path wired to the hook. But `set_cursor_hook_fn`+//! succeeds everywhere, so an embedder that trusted `cursor_hook_enabled()`+//! would conclude on Windows that tracking was live and then wait forever for a+//! first event. The viewer would show a pointer frozen at the origin — a+//! confidently wrong picture, indistinguishable from a hung stream.+//!+//! `cursor_hook_supported()` is the explicit publication of that limitation.+//! Its own binary, because both flags it reads are process-global one-shots and+//! this file must observe the *undeclared* state.++use cua_driver_core::cursor_hook::{+ cursor_hook_enabled, cursor_hook_supported, declare_cursor_hook_emitter, push_cursor_event,+ set_cursor_hook_fn, CursorHookEvent,+};+use std::sync::{+ atomic::{AtomicUsize, Ordering},+ Arc,+};++#[test]+fn support_is_about_the_platform_not_about_registration() {+ // A bare core build — which is what the Windows and Linux adapters link+ // against for this module — declares no emitter.+ assert!(+ !cursor_hook_supported(),+ "no adapter declared an emitter, so this build must report cursor \+ tracking as unsupported"+ );++ // Registering a hook must NOT make the platform look supported. This is the+ // whole point: the two questions are independent, and conflating them is+ // what makes a silent Windows host look like a working one.+ let seen = Arc::new(AtomicUsize::new(0));+ let counter = Arc::clone(&seen);+ set_cursor_hook_fn(move |_| {+ counter.fetch_add(1, Ordering::SeqCst);+ });++ assert!(+ cursor_hook_enabled(),+ "an embedder did register, so the hook is enabled"+ );+ assert!(+ !cursor_hook_supported(),+ "registering an observer must not be mistaken for the platform being \+ able to produce events; an embedder must still be able to tell the \+ user that this host cannot track cursors"+ );++ // ...and once an adapter declares itself, support flips — while the two+ // remain separately observable.+ declare_cursor_hook_emitter();+ assert!(cursor_hook_supported());+ assert!(cursor_hook_enabled());++ // Declaring an emitter must not synthesise events by itself.+ assert_eq!(+ seen.load(Ordering::SeqCst),+ 0,+ "declaring support must not fabricate cursor events"+ );+ push_cursor_event(CursorHookEvent {+ cursor_id: "s".into(),+ x: 0.0,+ y: 0.0,+ pressed: false,+ });+ assert_eq!(seen.load(Ordering::SeqCst), 1);++ // Idempotent: repeated declaration is harmless.+ declare_cursor_hook_emitter();+ assert!(cursor_hook_supported());+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_registered.rs b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_registered.rsnew file mode 100644index 0000000000..dd9906b4d2--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/tests/cursor_hook_registered.rs@@ -0,0 +1,147 @@+//! Behaviour of the cursor hook once an embedder has registered one.+//!+//! Separate binary from `cursor_hook_inert.rs` because `set_cursor_hook_fn`+//! backs onto a process-wide `OnceLock`: registration is irreversible, so the+//! registered and unregistered contracts cannot be asserted in one process.+//!+//! The single `#[test]` below is deliberate for the same reason. Rust runs the+//! tests in a binary on parallel threads; splitting these assertions into+//! several tests would race over the one-shot registration and the shared sink.++use cua_driver_core::cursor_hook::{+ cursor_hook_enabled, push_cursor_event, set_cursor_hook_fn, CursorHookEvent,+};+use std::sync::{Arc, Mutex};++#[test]+fn registered_hook_receives_events_verbatim_and_keeps_sources_distinct() {+ let sink: Arc<Mutex<Vec<CursorHookEvent>>> = Arc::new(Mutex::new(Vec::new()));++ assert!(+ !cursor_hook_enabled(),+ "precondition: nothing registered yet in this process"+ );++ let recorder = Arc::clone(&sink);+ assert!(+ set_cursor_hook_fn(move |ev| recorder.lock().unwrap().push(ev)),+ "the first registration in a process must report that it took effect"+ );++ assert!(+ cursor_hook_enabled(),+ "after registration the hot-path check must report enabled"+ );++ // ── Fields survive the trip unmodified ────────────────────────────────+ // A host renders the pointer from exactly these numbers; a hook that+ // rounded, swapped or dropped a field would misplace every cursor.+ push_cursor_event(CursorHookEvent {+ cursor_id: "session-alpha".into(),+ x: 12.5,+ y: -40.25,+ pressed: false,+ });+ push_cursor_event(CursorHookEvent {+ cursor_id: "session-alpha".into(),+ x: 12.5,+ y: -40.25,+ pressed: true,+ });++ {+ let got = sink.lock().unwrap();+ assert_eq!(got.len(), 2);+ assert_eq!(got[0].cursor_id, "session-alpha");+ assert_eq!(got[0].x, 12.5);+ assert_eq!(got[0].y, -40.25);+ assert!(!got[0].pressed, "a move must arrive as pressed=false");+ assert!(got[1].pressed, "a press edge must arrive as pressed=true");+ }+ sink.lock().unwrap().clear();++ // ── Identity: N participants must not collapse into one cursor ────────+ // The hook carries `cursor_id` precisely so a multi-participant host can+ // draw one pointer per agent. A consumer that stamped a constant id (or an+ // emitter that dropped the caller's id) would render every agent on top of+ // every other. This asserts the driver hands out the caller's own id.+ for id in ["agent-a", "agent-b", "agent-c"] {+ push_cursor_event(CursorHookEvent {+ cursor_id: id.into(),+ x: 1.0,+ y: 2.0,+ pressed: false,+ });+ }+ {+ let got = sink.lock().unwrap();+ let ids: Vec<&str> = got.iter().map(|e| e.cursor_id.as_str()).collect();+ assert_eq!(+ ids,+ vec!["agent-a", "agent-b", "agent-c"],+ "each source must keep its own cursor id; collapsing them merges every \+ participant's pointer into one"+ );+ }+ sink.lock().unwrap().clear();++ // ── Re-registration is ignored, not honoured ──────────────────────────+ // Documented as "call once at startup". If a second registration silently+ // replaced the first, a library initialising late would steal the host's+ // cursor stream and the host would go blind with no error.+ let hijack = Arc::new(Mutex::new(0usize));+ let hijack_probe = Arc::clone(&hijack);+ assert!(+ !set_cursor_hook_fn(move |_| {+ *hijack_probe.lock().unwrap() += 1;+ }),+ "a second registration must REPORT that it did not take effect. \+ Silently returning as though it had is how a host ends up rendering \+ no pointers with no way to discover why it owns no stream"+ );+ push_cursor_event(CursorHookEvent {+ cursor_id: "after-second-register".into(),+ x: 0.0,+ y: 0.0,+ pressed: false,+ });+ assert_eq!(+ *hijack.lock().unwrap(),+ 0,+ "a second set_cursor_hook_fn must be ignored"+ );+ assert_eq!(+ sink.lock().unwrap().len(),+ 1,+ "the originally registered hook must keep receiving events"+ );+ sink.lock().unwrap().clear();++ // ── Concurrency: every event from every thread is delivered exactly once ──+ let threads: Vec<_> = (0..8)+ .map(|t| {+ std::thread::spawn(move || {+ for i in 0..100 {+ push_cursor_event(CursorHookEvent {+ cursor_id: format!("thread-{t}"),+ x: i as f64,+ y: t as f64,+ pressed: false,+ });+ }+ })+ })+ .collect();+ for t in threads {+ t.join().unwrap();+ }+ let got = sink.lock().unwrap();+ assert_eq!(got.len(), 800, "no event may be dropped or duplicated");+ for t in 0..8 {+ let n = got+ .iter()+ .filter(|e| e.cursor_id == format!("thread-{t}"))+ .count();+ assert_eq!(n, 100, "thread {t}: all of its events must survive intact");+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/Cargo.toml b/libs/cua-driver/rust/crates/platform-macos/Cargo.tomlindex 96196ad6c1..3fa046fc8c 100644--- a/libs/cua-driver/rust/crates/platform-macos/Cargo.toml+++ b/libs/cua-driver/rust/crates/platform-macos/Cargo.toml@@ -39,6 +39,8 @@ objc2-foundation = { version = "0.2", features = ["NSThread", "NSObject", "NSValue","NSAppleEventDescriptor", "NSNotification", "NSOperation","NSURL", "NSDate", "NSError",+ # Cursor image bytes for system cursor shape classification.+ "NSData","block2", "libc",] }objc2-app-kit = { version = "0.2", features = [@@ -50,6 +52,8 @@ objc2-app-kit = { version = "0.2", features = [# one-line action label, NSPanel/NSImageCell are reachable via# transitive deps."NSImage", "NSImageView", "NSTextField", "NSControl", "NSCell",+ # System cursor shape reporting (cursor_shape adapter).+ "NSCursor","block2", "libc",] }diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/mod.rsindex 72c2ded15f..cb5012c56c 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/mod.rs@@ -6,6 +6,8 @@//! - `overlay::send_command(cmd)` — called from tool implementationspub mod overlay;+/// System cursor shape reporting (what the OS draws, not what Cua draws).+pub mod shape;pub mod state;// Re-export the legacy per-instance cursor state (used by tools for multi-cursor tracking).diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/shape.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/shape.rsnew file mode 100644index 0000000000..4a14b1489e--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/shape.rs@@ -0,0 +1,326 @@+//! macOS adapter for [`cua_driver_core::cursor_shape`].+//!+//! Thin by design: the vocabulary, the dispatch and the "unsupported" story+//! live in the common crate. All this file does is read AppKit's current+//! system cursor and classify it.+//!+//! ## How the classification works, and why it is not pointer equality+//!+//! `+[NSCursor currentSystemCursor]` builds a *new* `NSCursor` describing+//! whatever the Window Server is drawing. The standard cursors+//! (`+[NSCursor IBeamCursor]` and friends) are cached singletons. So the+//! obvious `currentSystemCursor == IBeamCursor` pointer test is always false+//! and would silently report `Default` for every shape — the exact+//! "check that cannot fail" trap. There is a unit test below pinning that.+//!+//! Instead each standard cursor is fingerprinted once by its hot spot and its+//! image's TIFF bytes, and the current cursor is matched against that table.+//! An unmatched cursor is reported as `Custom` with its own PNG, so a bespoke+//! application cursor still reaches the viewer rather than degrading to an+//! arrow.++use std::collections::HashMap;+use std::sync::OnceLock;++use cua_driver_core::cursor_shape::{ResizeAxis, SystemCursorShape};+use objc2::rc::Retained;+use objc2_app_kit::{NSCursor, NSImage};+use objc2_foundation::NSData;++/// A cursor's identity: hot spot rounded to whole pixels plus a hash of its+/// image bytes. Two cursors with the same fingerprint are the same cursor.+type Fingerprint = (i64, i64, u64);++fn hash_bytes(bytes: &[u8]) -> u64 {+ // FNV-1a. The table is tiny and built once; a cryptographic hash would be+ // pointless here and a collision only mislabels one cursor.+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;+ for byte in bytes {+ hash ^= u64::from(*byte);+ hash = hash.wrapping_mul(0x0000_0100_0000_01b3);+ }+ hash+}++/// Fetch one standard cursor, or `None` if AppKit will not supply it.+///+/// `+[NSCursor currentSystemCursor]` reads Window Server state and works+/// early, but the standard singletons (`arrowCursor` and friends) stay NULL+/// until an `NSApplication` exists -- the two are NOT equivalent, and gating+/// the singletons on `currentSystemCursor` aborts the process. objc2's+/// generated accessors `expect()` non-NULL, so this goes through a raw+/// `msg_send!` and null-checks the result itself.+macro_rules! standard_cursor {+ ($selector:ident) => {{+ let raw: *mut objc2::runtime::AnyObject =+ unsafe { objc2::msg_send![objc2::class!(NSCursor), $selector] };+ if raw.is_null() {+ None+ } else {+ unsafe { Retained::retain(raw.cast::<NSCursor>()) }+ }+ }};+}++/// True when AppKit will hand out the standard cursor singletons, which is+/// what the fingerprint table is built from.+pub fn appkit_cursors_available() -> bool {+ standard_cursor!(arrowCursor).is_some()+}++/// `-[NSCursor image]` as an `Option`.+///+/// objc2's generated `image()` accessor `expect()`s a non-NULL result and so+/// aborts the process if AppKit hands back nil -- which it does for a cursor+/// built from a Window Server description that carries no image. Every read of+/// the image goes through here so a missing image degrades to `Unknown` or+/// `None`, never a crash in the embedder.+fn cursor_image(cursor: &NSCursor) -> Option<Retained<NSImage>> {+ unsafe { objc2::msg_send_id![cursor, image] }+}++/// `-[NSImage TIFFRepresentation]` as an `Option`. Nil for an image with no+/// bitmap representation.+fn tiff_data(image: &NSImage) -> Option<Retained<NSData>> {+ unsafe { objc2::msg_send_id![image, TIFFRepresentation] }+}++fn tiff_bytes(cursor: &NSCursor) -> Option<Vec<u8>> {+ let image = cursor_image(cursor)?;+ let data = tiff_data(&image)?;+ Some(unsafe { data.bytes() }.to_vec())+}++fn fingerprint(cursor: &NSCursor) -> Option<Fingerprint> {+ let bytes = tiff_bytes(cursor)?;+ let spot = unsafe { cursor.hotSpot() };+ Some((+ spot.x.round() as i64,+ spot.y.round() as i64,+ hash_bytes(&bytes),+ ))+}++/// The standard-cursor fingerprint table, built once on first use.+///+/// Built from AppKit's own accessors rather than from hardcoded bytes, so it+/// tracks whatever the running OS version actually draws (cursor artwork has+/// changed across releases) and across accessibility cursor-size settings, as+/// long as the size does not change mid-process.+fn standard_table() -> &'static HashMap<Fingerprint, SystemCursorShape> {+ static TABLE: OnceLock<HashMap<Fingerprint, SystemCursorShape>> = OnceLock::new();+ TABLE.get_or_init(|| {+ let mut table = HashMap::new();+ // Each entry: an AppKit accessor and the neutral shape it means.+ // `resizeUpDown`/`resizeLeftRight` are the portable resize cursors;+ // the diagonal ones are not public API on macOS, so a window-corner+ // resize falls through to `Custom` and still renders correctly.+ // Nothing to fingerprint until AppKit is up. Leaving the table+ // empty here is what makes `current_shape` report Unknown rather than+ // mislabelling every cursor as Custom.+ if !appkit_cursors_available() {+ return table;+ }+ let entries: Vec<(Option<Retained<NSCursor>>, SystemCursorShape)> = vec![+ (standard_cursor!(arrowCursor), SystemCursorShape::Default),+ (standard_cursor!(IBeamCursor), SystemCursorShape::Text),+ (+ standard_cursor!(IBeamCursorForVerticalLayout),+ SystemCursorShape::VerticalText,+ ),+ (+ standard_cursor!(pointingHandCursor),+ SystemCursorShape::Pointer,+ ),+ (standard_cursor!(openHandCursor), SystemCursorShape::Grab),+ (+ standard_cursor!(closedHandCursor),+ SystemCursorShape::Grabbing,+ ),+ (+ standard_cursor!(crosshairCursor),+ SystemCursorShape::Crosshair,+ ),+ (+ standard_cursor!(operationNotAllowedCursor),+ SystemCursorShape::NotAllowed,+ ),+ (+ standard_cursor!(resizeUpDownCursor),+ SystemCursorShape::Resize(ResizeAxis::NorthSouth),+ ),+ (+ standard_cursor!(resizeLeftRightCursor),+ SystemCursorShape::Resize(ResizeAxis::EastWest),+ ),+ (+ standard_cursor!(resizeUpCursor),+ SystemCursorShape::Resize(ResizeAxis::NorthSouth),+ ),+ (+ standard_cursor!(resizeDownCursor),+ SystemCursorShape::Resize(ResizeAxis::NorthSouth),+ ),+ (+ standard_cursor!(resizeLeftCursor),+ SystemCursorShape::Resize(ResizeAxis::EastWest),+ ),+ (+ standard_cursor!(resizeRightCursor),+ SystemCursorShape::Resize(ResizeAxis::EastWest),+ ),+ ];+ for (cursor, shape) in entries {+ let Some(cursor) = cursor else { continue };+ if let Some(print) = fingerprint(&cursor) {+ // First writer wins: `resizeUpCursor` and `resizeDownCursor`+ // can share artwork with `resizeUpDownCursor`, and they all+ // mean the same neutral shape anyway.+ table.entry(print).or_insert(shape);+ }+ }+ table+ })+}++/// Read the shape the Window Server is currently drawing.+pub fn current_shape() -> SystemCursorShape {+ let Some(cursor) = (unsafe { NSCursor::currentSystemCursor() }) else {+ // No Window Server session, or AppKit declined to describe the+ // cursor. "I cannot tell" -- not "it is an arrow".+ return SystemCursorShape::Unknown;+ };+ let Some(print) = fingerprint(&cursor) else {+ return SystemCursorShape::Unknown;+ };+ if standard_table().is_empty() {+ // AppKit came up after the table was first built, or fingerprinting+ // failed wholesale. Reporting Custom for every cursor here would look+ // like it worked while being useless, so say Unknown.+ return SystemCursorShape::Unknown;+ }+ if let Some(shape) = standard_table().get(&print) {+ return shape.clone();+ }+ // An application-specific cursor. Ship its pixels so the viewer can draw+ // the real thing instead of falling back to an arrow.+ match custom_shape(&cursor) {+ Some(shape) => shape,+ None => SystemCursorShape::Unknown,+ }+}++fn custom_shape(cursor: &NSCursor) -> Option<SystemCursorShape> {+ unsafe {+ // Every step here can legitimately return nil -- a cursor with no+ // image, an image with no bitmap representation, bytes AppKit declines+ // to decode. Each is `Option` + `?` so an unclassifiable cursor falls+ // back to `Unknown` rather than aborting the host process. `custom_shape`+ // already returns `Option` for exactly this reason.+ let image = cursor_image(cursor)?;+ let tiff = tiff_data(&image)?;+ // NSBitmapImageRep -> PNG. NSBitmapImageFileTypePNG = 4.+ let rep: Option<Retained<objc2::runtime::AnyObject>> = objc2::msg_send_id![+ objc2::class!(NSBitmapImageRep),+ imageRepWithData: &*tiff+ ];+ let rep = rep?;+ let empty: Option<Retained<objc2_foundation::NSDictionary>> =+ objc2::msg_send_id![objc2::class!(NSDictionary), dictionary];+ let empty = empty?;+ let png: Option<Retained<NSData>> = objc2::msg_send_id![+ &*rep,+ representationUsingType: 4usize,+ properties: &*empty+ ];+ let png = png?.bytes().to_vec();+ let spot = cursor.hotSpot();+ let size = image.size();+ let pixels_wide: i64 = objc2::msg_send![&*rep, pixelsWide];+ // Device pixels per point. `size` is in points; `pixelsWide` is the+ // backing store, so their ratio is the cursor's scale.+ let scale = if size.width > 0.0 {+ pixels_wide as f64 / size.width+ } else {+ 1.0+ };+ Some(SystemCursorShape::Custom {+ png,+ hotspot_x: spot.x * scale,+ hotspot_y: spot.y * scale,+ scale,+ })+ }+}++/// Install this adapter as the process-wide probe.+pub fn install() {+ cua_driver_core::cursor_shape::set_cursor_shape_probe(current_shape);+}++#[cfg(test)]+mod tests {+ use super::*;++ /// AppKit's cursor class methods return NULL until an `NSApplication`+ /// exists. Rust's test harness runs each test on a spawned thread with no+ /// AppKit, so the classification tests below cannot run here -- they are+ /// proven in-guest against the real daemon instead (see the commit+ /// message). What IS provable here, and what actually regressed during+ /// development, is that the probe DEGRADES rather than panics.+ ///+ /// This test is meaningful precisely because AppKit is unavailable: the+ /// first implementation used objc2's generated accessors, which `expect()`+ /// a non-NULL result, and this exact call aborted the process.+ #[test]+ fn probe_degrades_to_unknown_without_appkit() {+ if appkit_cursors_available() {+ // An AppKit-hosted runner: then the probe must return a real+ // answer rather than Unknown, which is the opposite assertion.+ assert_ne!(current_shape(), SystemCursorShape::Unknown);+ return;+ }+ assert_eq!(current_shape(), SystemCursorShape::Unknown);+ assert!(standard_table().is_empty());+ }++ /// The fingerprint must discriminate: if arrow and I-beam hashed alike,+ /// every shape would collapse to one value and the probe would be a+ /// confident liar.+ ///+ /// `#[ignore]` because this needs a live `NSApplication`, which the Rust+ /// test harness does not provide. It is marked rather than early-returned+ /// so CI reports it as *ignored* instead of *passed*: a test that returns+ /// before its first assertion still prints "ok", which reads as coverage+ /// that does not exist. Run it on an AppKit-hosted machine with+ /// `cargo test -p platform-macos -- --ignored`.+ #[test]+ #[ignore = "requires a live NSApplication; run with --ignored on an AppKit host"]+ fn fingerprint_distinguishes_arrow_from_ibeam() {+ assert!(+ appkit_cursors_available(),+ "this test was run without AppKit; it can only assert anything \+ where the standard cursor singletons are available"+ );+ let (Some(arrow), Some(ibeam)) =+ (standard_cursor!(arrowCursor), standard_cursor!(IBeamCursor))+ else {+ return;+ };+ let arrow_print = fingerprint(&arrow).expect("arrow fingerprint");+ let ibeam_print = fingerprint(&ibeam).expect("ibeam fingerprint");+ assert_ne!(+ arrow_print, ibeam_print,+ "arrow and I-beam fingerprint identically; every shape would \+ collapse to one value"+ );+ assert_eq!(+ standard_table().get(&arrow_print),+ Some(&SystemCursorShape::Default)+ );+ assert_eq!(+ standard_table().get(&ibeam_print),+ Some(&SystemCursorShape::Text)+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rsindex 89b3319d65..5a7d655b96 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs@@ -142,6 +142,50 @@ impl CursorRegistry {position: None,});state.position = Some(CursorPosition { x, y });+ drop(inner);+ // Notify any embedder of the cursor move so it can render this cursor as+ // an overlay without driving the overlay itself.+ self.emit_cursor_event(cursor_id, x, y, false);+ }++ /// Report a press edge (click / drag start) for `cursor_id` at screen point+ /// (`x`, `y`) to the embedder's cursor hook.+ ///+ /// Presses go through the registry rather than being pushed directly from+ /// each tool so that they obey the same write-boundary resurrection guard as+ /// [`Self::update_position`]. A click whose session has already ended must+ /// not reach the hook: the registry has cleared that cursor, and an embedder+ /// that rendered the press would resurrect a cursor the driver considers+ /// gone. Unlike `update_position` this records no position — a press is an+ /// event, not a new resting place, and the move that precedes it has already+ /// stored the coordinates.+ pub fn note_press(&self, cursor_id: &str, x: f64, y: f64) {+ self.emit_cursor_event(cursor_id, x, y, true);+ }++ /// Single choke point for cursor-hook emission, so the suppression rule+ /// cannot drift between the move path and the press path.+ fn emit_cursor_event(&self, cursor_id: &str, x: f64, y: f64, pressed: bool) {+ // Cheapest check first. `CursorHookEvent` owns its id, so building one+ // costs a heap allocation on every commanded move — and almost every+ // cua-driver consumer (the daemon, the CLI, every SDK caller that is not+ // a remote-desktop host) never registers a hook, so that allocation+ // would be pure waste for them. This is what `cursor_hook_enabled` is+ // for, and it is the same order the `pip_hook` idiom uses.+ if !cua_driver_core::cursor_hook::cursor_hook_enabled() {+ return;+ }+ if cursor_id.is_empty() || cua_driver_core::session::is_session_ended(cursor_id) {+ return;+ }+ cua_driver_core::cursor_hook::push_cursor_event(+ cua_driver_core::cursor_hook::CursorHookEvent {+ cursor_id: cursor_id.to_owned(),+ x,+ y,+ pressed,+ },+ );}pub fn set_enabled(&self, cursor_id: &str, enabled: bool) {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rsindex f27237357e..84e5a7be5b 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs@@ -116,6 +116,20 @@ pub fn register_tools_with_cursor_and_provider(if cursor_overlay_available {cursor::overlay::init(cfg);}+ // System cursor shape reporting is independent of the agent-cursor+ // overlay: a remote viewer wants the I-beam even when Cua draws no+ // cursor of its own. It needs only a Window Server session, so it is+ // gated on graphic access rather than on `cfg.enabled`.+ if session::has_graphic_access() {+ cursor::shape::install();+ }+ // This adapter drives `push_cursor_event` from the cursor write path in+ // `cursor::state`, so declare cursor tracking as supported. The Windows+ // and Linux adapters make no such declaration, which is how an embedder+ // learns it must publish the limitation instead of waiting for events+ // that will never arrive. Unconditional: emission depends on the+ // registry write path, not on the overlay or on graphic access.+ cua_driver_core::cursor_hook::declare_cursor_hook_emitter();let mut r = ToolRegistry::new_with_protected_consent_provider(provider);tools::register_all(&mut r,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex f0e211e35f..af99862cee 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -299,6 +299,9 @@ impl Tool for ClickTool {self.state.cursor_registry.update_position(&cursor_key, sx, sy);+ // Press edge for the agent-cursor overlay (renders a click pulse on+ // viewers via the cursor hook).+ self.state.cursor_registry.note_press(&cursor_key, sx, sy);let btn = button.clone();let desktop_modifiers: Vec<String> = args.str_array("modifier");@@ -487,6 +490,7 @@ impl Tool for ClickTool {self.state.cursor_registry.update_position(&cursor_key, cx, cy);+ self.state.cursor_registry.note_press(&cursor_key, cx, cy);let mods_owned = modifiers.clone();let foreground = delivery_mode.is_foreground();@@ -531,6 +535,7 @@ impl Tool for ClickTool {self.state.cursor_registry.update_position(&cursor_key, cx, cy);+ self.state.cursor_registry.note_press(&cursor_key, cx, cy);}// Finder icon/list items can expose a readable AXSelected state@@ -963,6 +968,9 @@ impl Tool for ClickTool {self.state.cursor_registry.update_position(&cursor_key, screen_x, screen_y);+ self.state+ .cursor_registry+ .note_press(&cursor_key, screen_x, screen_y);// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// A pixel click can land on a "Sign In" button that opens a sheetdiff --git a/libs/cua-driver/rust/crates/platform-macos/tests/cursor_hook_registry.rs b/libs/cua-driver/rust/crates/platform-macos/tests/cursor_hook_registry.rsnew file mode 100644index 0000000000..47ae2fef64--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/tests/cursor_hook_registry.rs@@ -0,0 +1,166 @@+//! The cursor hook must obey the registry's write-boundary resurrection guard.+//!+//! `CursorRegistry` refuses to create or touch a cursor whose id is empty or+//! whose session has already ended — the "resurrection guard" repeated at every+//! write boundary in `cursor/state.rs`. The cursor hook is a new write+//! boundary: it publishes cursor activity to an embedder that draws pointers on+//! a remote viewer. If the hook escaped the guard, a click arriving after+//! `session_end` would make the embedder re-draw a cursor the driver has+//! already cleared, and the viewer would show a ghost pointer that nothing can+//! remove.+//!+//! Before the fix this was live: `update_position` applied the guard, but the+//! press edge was pushed straight to `push_cursor_event` from four call sites+//! in `tools/click.rs`, bypassing it. The session_end and empty-id assertions+//! below each fail against that version and pass against this one.+//!+//! One test binary, one `#[test]`: `set_cursor_hook_fn` is a process-wide+//! one-shot, so the registration and the assertions must share a process and+//! must not race.++use cua_driver_core::cursor_hook::{set_cursor_hook_fn, CursorHookEvent};+use platform_macos::cursor::state::CursorRegistry;+use std::sync::{Arc, Mutex};++#[test]+fn registry_cursor_hook_honours_the_resurrection_guard() {+ let sink: Arc<Mutex<Vec<CursorHookEvent>>> = Arc::new(Mutex::new(Vec::new()));+ let recorder = Arc::clone(&sink);+ set_cursor_hook_fn(move |ev| recorder.lock().unwrap().push(ev));++ let take = || -> Vec<CursorHookEvent> { std::mem::take(&mut *sink.lock().unwrap()) };++ let registry = CursorRegistry::new();++ // ── A live cursor reports both its move and its press ─────────────────+ registry.update_position("session-live", 10.0, 20.0);+ registry.note_press("session-live", 10.0, 20.0);+ let got = take();+ assert_eq!(got.len(), 2, "a live cursor must report move and press");+ assert_eq!(got[0].cursor_id, "session-live");+ assert_eq!((got[0].x, got[0].y), (10.0, 20.0));+ assert!(!got[0].pressed);+ assert!(got[1].pressed, "note_press must mark the event as a press");+ assert_eq!(+ (got[1].x, got[1].y),+ (10.0, 20.0),+ "a press must carry the point it happened at"+ );++ // ── Distinct sessions stay distinct ───────────────────────────────────+ // A multi-participant host draws one pointer per session id. Were the+ // registry to normalise or share ids, every participant would land on the+ // same pointer.+ registry.update_position("session-a", 1.0, 1.0);+ registry.update_position("session-b", 2.0, 2.0);+ registry.note_press("session-b", 2.0, 2.0);+ let got = take();+ let ids: Vec<&str> = got.iter().map(|e| e.cursor_id.as_str()).collect();+ assert_eq!(ids, vec!["session-a", "session-b", "session-b"]);+ assert_eq!(+ got[0].x, 1.0,+ "session-a must keep its own position, not session-b's"+ );++ // ── The empty-id sentinel means "no cursor", so it must emit nothing ───+ // `cursor_key` is an empty string when there is no cursor to speak of.+ // `update_position` already treated it that way; the press path must agree,+ // otherwise the same value means "anonymous default cursor" on one path and+ // "no cursor at all" on the other.+ registry.update_position("", 5.0, 5.0);+ registry.note_press("", 5.0, 5.0);+ assert!(+ take().is_empty(),+ "the empty cursor id is the no-cursor sentinel: neither a move nor a \+ press may reach the hook"+ );++ // ── A press after session_end must be suppressed (the regression) ──────+ registry.update_position("session-ending", 7.0, 7.0);+ assert_eq!(take().len(), 1, "precondition: the session is live");++ cua_driver_core::session::end_session("session-ending");++ registry.update_position("session-ending", 8.0, 8.0);+ registry.note_press("session-ending", 8.0, 8.0);+ assert!(+ take().is_empty(),+ "after session_end the registry has cleared this cursor; neither an \+ in-flight move nor an in-flight press may reach the embedder, or the \+ viewer resurrects a pointer the driver considers gone"+ );++ // ── Suppression is scoped to the ended session only ────────────────────+ // A guard that over-fired would silence every other participant's cursor.+ registry.update_position("session-live", 9.0, 9.0);+ registry.note_press("session-live", 9.0, 9.0);+ assert_eq!(+ take().len(),+ 2,+ "ending one session must not silence any other cursor"+ );+}++/// Structural lock on the fix above.+///+/// The guard is only worth anything if it cannot be walked around. The original+/// bug was not a wrong guard, it was four tool call sites that never reached+/// one — and the natural way to wire up the next tool is to copy an existing+/// `push_cursor_event(...)` block, which is exactly how the bypass would come+/// back. Keeping emission to a single private choke point inside+/// `CursorRegistry` means any new tool must go through `update_position` or+/// `note_press` and is guarded automatically.+///+/// This asserts the invariant on the source itself because there is no runtime+/// signal for "a tool pushed an event directly": such a call simply works, and+/// silently skips the guard.+#[test]+fn cursor_events_are_emitted_from_exactly_one_place_in_this_adapter() {+ fn rs_files(dir: &std::path::Path, out: &mut Vec<std::path::PathBuf>) {+ for entry in std::fs::read_dir(dir).expect("readable source dir") {+ let path = entry.expect("readable entry").path();+ if path.is_dir() {+ rs_files(&path, out);+ } else if path.extension().is_some_and(|e| e == "rs") {+ out.push(path);+ }+ }+ }++ let src = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");+ let mut files = Vec::new();+ rs_files(&src, &mut files);+ assert!(!files.is_empty(), "expected to find adapter sources");++ let mut offenders = Vec::new();+ for file in &files {+ let text = std::fs::read_to_string(file).expect("readable source file");+ for (i, line) in text.lines().enumerate() {+ // Skip prose: the call is named in doc comments explaining the rule.+ let trimmed = line.trim_start();+ if trimmed.starts_with("//") {+ continue;+ }+ if line.contains("push_cursor_event") {+ let _ = i;+ offenders.push(+ file.strip_prefix(&src)+ .unwrap_or(file)+ .display()+ .to_string(),+ );+ }+ }+ }++ assert_eq!(+ offenders,+ vec!["cursor/state.rs".to_string()],+ "cursor events must be pushed only from CursorRegistry::emit_cursor_event. \+ A direct push_cursor_event call elsewhere bypasses the empty-id and \+ session_end resurrection guards, which is the exact regression \+ registry_cursor_hook_honours_the_resurrection_guard exists to prevent. \+ Call update_position or note_press instead. \+ (If emit_cursor_event legitimately moved, update the expected location.)"+ );+}