Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): keep cursor badges and drags in sync (#2676)
Francesco Bonaccib249df857994
diff --git a/AGENTS.md b/AGENTS.mdindex 16b8e8e940..5c04dfb36b 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -23,6 +23,19 @@ Preserve contributor credit when external code or design ships in Cua.Do not reimplement submitted work solely to remove its authorship history.+## Cross-platform Cua Driver behavior++Treat user-visible Cua Driver behavior as a cross-platform contract. Implement+shared state, geometry, timing, and protocol semantics in the common crates+whenever possible, then keep each macOS, Windows, X11, and Wayland adapter thin.+Do not silently land a platform-specific cursor or interaction change as though+it were universal.++For every affected platform, add focused coverage and either verify the native+behavior or document a concrete operating-system or compositor limitation.+When a platform cannot support the same contract, return or publish that+limitation explicitly instead of substituting misleading behavior.+## Pull request titles and component releasesPull requests are squash-merged, so the pull request title becomes the commitdiff --git a/libs/cua-driver/docs/cursor-themes.md b/libs/cua-driver/docs/cursor-themes.mdindex 1e05aac380..a213e913f4 100644--- a/libs/cua-driver/docs/cursor-themes.md+++ b/libs/cua-driver/docs/cursor-themes.md@@ -20,7 +20,14 @@ The native renderer places the sanitized public session name in a compact badgebelow the built-in pointer. The badge uses a session-accent gradient and acircular session marker. It appears when the cursor is first revealed, remainsfully visible for two seconds, then fades over 400 milliseconds while the-cursor stays available. Hiding and revealing the cursor shows the badge again.+cursor stays available. Moving the human pointer over the synthetic cursor+temporarily reveals the badge again, and moving it away hides the badge.+Hiding and revealing the cursor also shows the badge again. Hover reveal uses+the native global-pointer APIs on macOS, Windows, and X11. Stock Wayland does+not expose another client's global pointer position, so native Wayland keeps+the timed reveal but cannot offer hover reveal without a compositor-owned+adapter. The click-through overlay never captures pointer input to fake this+capability.The badge is not part of the dotLottie theme artifact, so custom theme authorsdo not need to provide badge artwork or a font. Cua Driver strips controlcharacters, collapses whitespace, and shortens labels longer than 28 characters@@ -31,6 +38,10 @@ screen target, including indexed or token-addressed text and value operations,move the agent cursor to that target before dispatch. Actions without a spatialtarget, such as typing into the already focused desktop application, keep thecursor at its current location and only change the semantic animation.+During a drag, the native gesture's endpoints, step count, and wall-clock+duration drive the same shared pointer-anchor transform on macOS, Windows, X11,+and supported Wayland input routes, so the synthetic cursor follows the held+gesture instead of moving only after release.The cursor is a visual aid for people watching an agent. It is not a securityindicator, an authorization prompt, or evidence that a tool call succeeded.diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rsindex ba960822ca..e8d4583dd4 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs@@ -359,3 +359,38 @@ pub enum OverlayCommand {/// `None` clears the highlight.ShowFocusRect(Option<[f64; 4]>),}++/// Build the shared overlay command for one native pointer position.+///+/// Native drag implementations report the actual event coordinate while the+/// cursor artwork is centred 16 points down-right so its tip lands on that+/// coordinate. Keeping this transform here prevents platform-specific drag+/// loops from drifting apart.+pub fn track_pointer_command(x: f64, y: f64) -> OverlayCommand {+ const CLICK_OFFSET: f64 = 16.0;+ let heading = std::f64::consts::FRAC_PI_4;+ OverlayCommand::SnapTo {+ x: x + heading.cos() * CLICK_OFFSET,+ y: y + heading.sin() * CLICK_OFFSET,+ heading_radians: Some(heading),+ }+}++#[cfg(test)]+mod pointer_tracking_tests {+ use super::*;++ #[test]+ fn tracked_artwork_keeps_its_tip_on_the_native_pointer() {+ let OverlayCommand::SnapTo {+ x,+ y,+ heading_radians: Some(heading),+ } = track_pointer_command(120.0, 80.0)+ else {+ panic!("pointer tracking must produce an anchored snap");+ };+ assert!((x - (120.0 + heading.cos() * 16.0)).abs() < f64::EPSILON);+ assert!((y - (80.0 + heading.sin() * 16.0)).abs() < f64::EPSILON);+ }+}diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rsindex 6955828053..dc6dd4f3df 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs@@ -86,6 +86,10 @@ pub struct RenderStateCore {pub session_label: Option<String>,/// Elapsed time since the session label was revealed with the cursor.pub session_badge_secs: f64,+ /// Whether the user's hardware pointer is currently over this synthetic+ /// cursor. Hover temporarily reveals an already-faded session badge+ /// without changing its one-shot reveal timer.+ pub session_badge_hovered: bool,}impl RenderStateCore {@@ -130,6 +134,7 @@ impl RenderStateCore {pinned_wid: None,session_label: None,session_badge_secs: SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS,+ session_badge_hovered: false,}}@@ -147,6 +152,9 @@ impl RenderStateCore {if self.session_label.is_none() {return 0.0;}+ if self.session_badge_hovered {+ return 1.0;+ }if self.session_badge_secs <= SESSION_BADGE_HOLD_SECS {return 1.0;}@@ -156,6 +164,38 @@ impl RenderStateCore {(1.0 - smooth) as f32}+ pub fn session_badge_needs_frame_tick(&self) -> bool {+ self.session_label.is_some()+ && self.session_badge_secs < SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS+ && self.cursor_is_revealed()+ }++ /// Whether the platform overlay should keep a low-frequency hardware+ /// pointer poll alive for hover-to-reveal. This is deliberately separate+ /// from [`Self::session_badge_needs_frame_tick`]: a faded badge needs hover+ /// hit-testing, not continuous 60 fps repainting.+ pub fn session_badge_needs_hover_poll(&self) -> bool {+ self.session_label.is_some() && self.cursor_is_revealed()+ }++ /// Update hover state from a platform-native hardware pointer sample.+ ///+ /// `self.pos` is the centre of the cursor artwork. The hit radius is a+ /// little larger than the 42 point production artwork so the interaction+ /// remains comfortable around the white outline and glow.+ pub fn update_session_badge_hover(&mut self, pointer: Option<(f64, f64)>) -> bool {+ const HOVER_RADIUS: f64 = crate::theme::DISPLAY_SIZE as f64 * 0.82;+ let hovered = self.session_badge_needs_hover_poll()+ && pointer.is_some_and(|(x, y)| {+ let dx = x - self.pos.0;+ let dy = y - self.pos.1;+ dx * dx + dy * dy <= HOVER_RADIUS * HOVER_RADIUS+ });+ let changed = hovered != self.session_badge_hovered;+ self.session_badge_hovered = hovered;+ changed+ }+/// Return the theme that is actually being painted, including any/// non-fatal fallback from an unavailable launch-time selection.pub fn active_theme_metadata(&self) -> (String, String, String, Option<String>) {@@ -606,8 +646,11 @@ impl RenderStateCore {true}OverlayCommand::SetSessionLabel(label) => {- self.session_label = crate::sanitize_session_label(&label);- self.session_badge_secs = 0.0;+ let session_label = crate::sanitize_session_label(&label);+ if session_label != self.session_label {+ self.session_label = session_label;+ self.session_badge_secs = 0.0;+ }true}OverlayCommand::ShowFocusRect(_) => false, // caller-specific@@ -881,6 +924,32 @@ mod session_badge_and_action_tests {assert_eq!(core.session_badge_alpha(), 0.0);}+ #[test]+ fn repeated_session_label_metadata_does_not_restart_badge_timer() {+ let mut core = RenderStateCore::new(CursorConfig::default());+ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ );+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);+ assert_eq!(core.session_badge_alpha(), 0.0);++ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ );+ assert_eq!(core.session_badge_alpha(), 0.0);++ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Writing".into()),+ false,+ false,+ );+ assert_eq!(core.session_badge_alpha(), 1.0);+ }+#[test]fn revealing_hidden_cursor_restarts_badge_without_restarting_on_every_move() {let mut core = RenderStateCore::new(CursorConfig::default());@@ -902,6 +971,7 @@ mod session_badge_and_action_tests {false,);assert_eq!(core.session_badge_alpha(), 1.0);+ assert!(core.session_badge_needs_frame_tick());core.tick_motion(0.5);let elapsed = core.session_badge_secs;core.apply_command_base(@@ -914,6 +984,30 @@ mod session_badge_and_action_tests {false,);assert_eq!(core.session_badge_secs, elapsed);+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);+ assert!(!core.session_badge_needs_frame_tick());+ }++ #[test]+ fn hardware_pointer_hover_reveals_only_while_over_cursor() {+ let mut core = RenderStateCore::new(CursorConfig::default());+ core.pos = (300.0, 240.0);+ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ );+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);+ assert_eq!(core.session_badge_alpha(), 0.0);+ assert!(core.session_badge_needs_hover_poll());++ assert!(core.update_session_badge_hover(Some((302.0, 238.0))));+ assert_eq!(core.session_badge_alpha(), 1.0);+ assert!(!core.update_session_badge_hover(Some((304.0, 241.0))));+ assert_eq!(core.session_badge_alpha(), 1.0);++ assert!(core.update_session_badge_hover(Some((500.0, 500.0))));+ assert_eq!(core.session_badge_alpha(), 0.0);}#[test]diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rsindex 0df93a1570..d3fd32db84 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs@@ -460,6 +460,7 @@ impl RenderState {self.core.path.is_some()|| self.core.spring.is_some()|| self.core.click_t.is_some()+ || self.core.session_badge_needs_frame_tick()|| (self.core.motion.idle_hide_ms > 0.0&& self.core.visible&& self.core.pos.0 >= -100.0@@ -775,12 +776,18 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMslet now = Instant::now();let elapsed_dt = now.duration_since(last_tick).as_secs_f64();last_tick = now;+ let hardware_pointer = conn+ .query_pointer(root)+ .ok()+ .and_then(|cookie| cookie.reply().ok())+ .map(|reply| (f64::from(reply.root_x), f64::from(reply.root_y)));// Drain commands and tick.let (arrived,pinned_wid,had_msg,+ hover_changed,next_frame_tick_needed,next_z_order_tick_needed,next_idle_wait_interval,@@ -795,6 +802,10 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMsmaintenance_timeout,frame_tick_needed,);+ let mut hover_changed = false;+ for rs in map.cursors.values_mut() {+ hover_changed |= rs.core.update_session_badge_hover(hardware_pointer);+ }let pinned_wid = map.last_active.as_ref()@@ -807,6 +818,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMsarrived,pinned_wid,had_msg,+ hover_changed,next_frame_tick_needed,next_z_order_tick_needed,next_idle_wait_interval,@@ -829,7 +841,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMs// Render after a command, during an active animation/fade, and once// more as the final active state settles. This leaves the X11 window in// its completed/cleared state before the next blocking receive.- if had_msg || frame_tick_needed || next_frame_tick_needed {+ if had_msg || hover_changed || frame_tick_needed || next_frame_tick_needed {let tiles = {let guard = RENDER.lock().unwrap();guard.as_ref().map(render_x11_tiles)diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex 7d9121bebd..df57376b3c 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -1576,6 +1576,37 @@ async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {crate::overlay::animate_cursor_to_for(cursor_id.to_owned(), sx, sy).await;}+async fn track_overlay_drag_for(+ cursor_id: String,+ from: (f64, f64),+ to: (f64, f64),+ duration_ms: u64,+ steps: usize,+) {+ if !crate::overlay::is_enabled_for(&cursor_id) {+ return;+ }+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );+ let steps = steps.max(1);+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);+ for index in 0..=steps {+ let t = index as f64 / steps as f64;+ let x = from.0 + (to.0 - from.0) * t;+ let y = from.1 + (to.1 - from.1) * t;+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::track_pointer_command(x, y),+ );+ if index < steps && !step_delay.is_zero() {+ tokio::time::sleep(step_delay).await;+ }+ }+ crate::overlay::send_command_for(cursor_id, cursor_overlay::OverlayCommand::SetPressed(false));+}+fn process_name(pid: u32) -> Option<String> {let cmdline = fs::read(format!("/proc/{pid}/cmdline")).ok()?;let first = String::from_utf8_lossy(&cmdline)@@ -4441,8 +4472,20 @@ impl Tool for DragTool {steps,)}- })- .await;+ });+ let visual_drag = track_overlay_drag_for(+ cursor_id.clone(),+ (from_x, from_y),+ (to_x, to_y),+ duration_ms,+ steps,+ );+ let (result, ()) = tokio::join!(result, visual_drag);+ if matches!(&result, Ok(Ok(()))) {+ self.state+ .cursor_registry+ .update_position(&cursor_id, to_x, to_y);+ }return match result {Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(json!({"scope":"desktop","path":path,"effect":"unverifiable"}),@@ -4563,11 +4606,6 @@ impl Tool for DragTool {},);}- crate::overlay::send_command_for(- cursor_id.clone(),- cursor_overlay::OverlayCommand::SetPressed(true),- );-// Native Wayland: emit press + interpolated motion + release as one// virtual-pointer (wlroots) or libei (GNOME/KDE) sequence, output-relative// coords. Returns early so we don't fall into the X11 XSendEvent loop below.@@ -4584,7 +4622,6 @@ impl Tool for DragTool {button as u32,)})- .await} else {let ((fxi, fyi), (txi, tyi)) = wayland_points.unwrap_or(((from_x.round() as i32, from_y.round() as i32),@@ -4593,12 +4630,27 @@ impl Tool for DragTool {tokio::task::spawn_blocking(move || {crate::wayland::drag(xid, fxi, fyi, txi, tyi, steps_u32, duration_ms, button)})- .await};- crate::overlay::send_command_for(+ let ((from_output_x, from_output_y), (to_output_x, to_output_y)) = wayland_points+ .unwrap_or((+ (from_x.round() as i32, from_y.round() as i32),+ (to_x.round() as i32, to_y.round() as i32),+ ));+ let visual_drag = track_overlay_drag_for(cursor_id.clone(),- cursor_overlay::OverlayCommand::SetPressed(false),+ (f64::from(from_output_x), f64::from(from_output_y)),+ (f64::from(to_output_x), f64::from(to_output_y)),+ duration_ms,+ steps,);+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);+ if matches!(&drag_result, Ok(Ok(()))) {+ self.state.cursor_registry.update_position(+ &cursor_id,+ f64::from(to_output_x),+ f64::from(to_output_y),+ );+ }return match drag_result {Ok(Ok(())) => ToolResult::text(format!("✅ Posted drag ({button_str}) to pid {pid} \@@ -4635,12 +4687,20 @@ impl Tool for DragTool {steps,)})- })- .await;- crate::overlay::send_command_for(+ });+ let visual_drag = track_overlay_drag_for(cursor_id.clone(),- cursor_overlay::OverlayCommand::SetPressed(false),+ (screen_from_x, screen_from_y),+ (screen_to_x, screen_to_y),+ duration_ms,+ steps,);+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);+ if matches!(&drag_result, Ok(Ok(()))) {+ self.state+ .cursor_registry+ .update_position(&cursor_id, screen_to_x, screen_to_y);+ }return match drag_result {Ok(Ok(())) => ToolResult::text(format!("Dragged ({button_str}) to pid {pid} from ({from_x:.0}, {from_y:.0}) \@@ -4657,6 +4717,10 @@ impl Tool for DragTool {};}+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );let press_result = tokio::task::spawn_blocking(move || {crate::input::send_button_down(xid,@@ -4678,8 +4742,6 @@ impl Tool for DragTool {} else {duration_ms};- let mut prev_x = from_x;- let mut prev_y = from_y;for i in 1..=steps {let t = i as f64 / steps.max(1) as f64;let ix = from_x + (to_x - from_x) * t;@@ -4699,20 +4761,14 @@ impl Tool for DragTool {tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy)).await{- let heading = if (ix - prev_x).abs() > f64::EPSILON- || (iy - prev_y).abs() > f64::EPSILON- {- Some((iy - prev_y).atan2(ix - prev_x))- } else {- None- };self.state.cursor_registry.update_position(&cursor_id, sx, sy);- overlay_move_to_for(&cursor_id, sx, sy, heading);+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::track_pointer_command(sx, sy),+ );}- prev_x = ix;- prev_y = iy;if step_delay_ms > 0 {tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms)).await;@@ -4750,19 +4806,13 @@ impl Tool for DragTool {if let Ok(Ok((sx_to, sy_to))) =tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y)).await{- self.state- .cursor_registry- .update_position(&cursor_id, sx_to, sy_to);- overlay_snap_to_for(- &cursor_id,- sx_to,- sy_to,- Some((to_y - from_y).atan2(to_x - from_x)),- );crate::overlay::send_command_for(cursor_id.clone(),- cursor_overlay::OverlayCommand::ClickPulse { x: sx_to, y: sy_to },+ cursor_overlay::track_pointer_command(sx_to, sy_to),);+ self.state+ .cursor_registry+ .update_position(&cursor_id, sx_to, sy_to);}}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rsindex 411f5eb40f..e3a1c70e23 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs@@ -216,7 +216,11 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {layer_surface.set_exclusive_zone(-1);layer_surface.set_keyboard_interactivity(KeyboardInteractivity::None);- // Click-through: empty input region.+ // Click-through: empty input region. Standard Wayland intentionally does+ // not expose another client's global pointer position, so this surface+ // cannot implement the macOS/Windows/X11 badge-hover reveal without a+ // compositor-owned adapter. Giving it an input region would steal the+ // user's pointer events instead of observing them.let region: WlRegion = compositor.create_region(&qh, ());surface.set_input_region(Some(®ion));region.destroy();diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rsindex f0332cf63c..9089525e80 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs@@ -522,6 +522,7 @@ impl RenderState {|| self.core.spring.is_some()|| self.core.click_t.is_some()|| self.focus_rect.is_some()+ || self.core.session_badge_needs_frame_tick()|| (self.core.motion.idle_hide_ms > 0.0&& self.core.visible&& self.core.pos.0 >= -100.0@@ -666,8 +667,10 @@ fn render_loop(_win_h: f64,) {let target_frame_ms = Duration::from_millis(16); // ~60 fps while pixels can change+ let hover_poll_ms = Duration::from_millis(80);let mut last_tick = Instant::now();let mut frame_tick_needed = false;+ let mut hover_poll_needed = false;// Repin bookkeeping: track last pinned wid and a frame counter for// the periodic defensive-repin (every ~60 active frames ≈ 1 s).let mut last_pinned: Option<u64> = None;@@ -677,11 +680,17 @@ fn render_loop(// When no cursor animation/fade is active, block until the MCP side// sends a command. This is the idle-server fast path: no fullscreen// pixmap allocation, no CGImage conversion, no 60fps wakeup.- let first_msg = if frame_tick_needed {- None+ let (first_msg, hover_poll_tick) = if frame_tick_needed {+ (None, hover_poll_needed)+ } else if hover_poll_needed {+ match rx.recv_timeout(hover_poll_ms) {+ Ok(msg) => (Some(msg), true),+ Err(std::sync::mpsc::RecvTimeoutError::Timeout) => (None, true),+ Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,+ }} else {match rx.recv() {- Ok(msg) => Some(msg),+ Ok(msg) => (Some(msg), false),Err(_) => break,}};@@ -702,7 +711,17 @@ fn render_loop(// `pinned_wid` follows the most-recently-updated cursor: a single// NSWindow can occupy only one z-band, so the last-active cursor's// target wins. `arrived` collects the keys whose path just ended.- let (pinned_wid, raise_unpinned, arrived, win_w, win_h, had_msg, next_frame_tick_needed) = {+ let (+ pinned_wid,+ raise_unpinned,+ arrived,+ win_w,+ win_h,+ had_msg,+ hover_changed,+ next_frame_tick_needed,+ next_hover_poll_needed,+ ) = {let mut guard = RENDER.lock().unwrap();match guard.as_mut() {Some(map) => {@@ -734,6 +753,22 @@ fn render_loop(}}}+ let pointer = if hover_poll_tick+ || map+ .cursors+ .values()+ .any(|rs| rs.core.session_badge_needs_hover_poll())+ {+ hardware_cursor_position()+ } else {+ None+ };+ let mut hover_changed = false;+ if pointer.is_some() || hover_poll_tick {+ for rs in map.cursors.values_mut() {+ hover_changed |= rs.core.update_session_badge_hover(pointer);+ }+ }let pinned = last_key.as_ref().and_then(|k| map.cursors.get(k))@@ -745,6 +780,10 @@ fn render_loop(.is_some_and(cursor_is_externally_visible)&& pinned.is_none();let next_frame_tick_needed = render_map_needs_frame_tick(map);+ let next_hover_poll_needed = map+ .cursors+ .values()+ .any(|rs| rs.core.session_badge_needs_hover_poll());(pinned,raise_unpinned,@@ -752,7 +791,9 @@ fn render_loop(map.win_w,map.win_h,had_msg,+ hover_changed,next_frame_tick_needed,+ next_hover_poll_needed,)}None => break,@@ -790,7 +831,7 @@ fn render_loop(// Render only when a command arrived or the previous/next tick can// change pixels. A final frame is emitted as animations/fades finish so// the layer is left in the completed/cleared state before blocking.- if had_msg || frame_tick_needed || next_frame_tick_needed {+ if had_msg || hover_changed || frame_tick_needed || next_frame_tick_needed {let pixmap = {let guard = RENDER.lock().unwrap();if let Some(map) = guard.as_ref() {@@ -830,6 +871,7 @@ fn render_loop(}frame_tick_needed = next_frame_tick_needed;+ hover_poll_needed = next_hover_poll_needed;if frame_tick_needed {// Sleep remainder of frame budget.let elapsed = Instant::now().duration_since(last_tick);@@ -840,6 +882,18 @@ fn render_loop(}}+fn hardware_cursor_position() -> Option<(f64, f64)> {+ use core_graphics::{+ event::CGEvent,+ event_source::{CGEventSource, CGEventSourceStateID},+ };++ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState).ok()?;+ let event = CGEvent::new(source).ok()?;+ let location = event.location();+ Some((location.x, location.y))+}+fn cursor_is_externally_visible(state: &RenderState) -> bool {state.core.cfg.enabled&& state.core.visiblediff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rsindex 5ab4e11b4d..6a210126e0 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs@@ -487,6 +487,49 @@ pub fn drag_at_xy(button: DragButton,foreground_release: bool,) -> anyhow::Result<()> {+ drag_at_xy_observed(+ pid,+ from_x,+ from_y,+ to_x,+ to_y,+ from_local,+ to_local,+ wid,+ duration_ms,+ steps,+ modifiers,+ button,+ foreground_release,+ |_, _| {},+ )+}++/// PID-routed drag with an observer called for every native pointer position.+///+/// The cursor overlay uses this for the same reason as+/// [`drag_at_xy_foreground_observed`]: the synthetic cursor should follow the+/// actual dispatched path rather than jumping to the endpoint after release.+#[allow(clippy::too_many_arguments)]+pub fn drag_at_xy_observed<F>(+ pid: i32,+ from_x: f64,+ from_y: f64,+ to_x: f64,+ to_y: f64,+ from_local: Option<(f64, f64)>,+ to_local: Option<(f64, f64)>,+ wid: Option<u32>,+ duration_ms: u64,+ steps: usize,+ modifiers: &[&str],+ button: DragButton,+ foreground_release: bool,+ mut observe: F,+) -> anyhow::Result<()>+where+ F: FnMut(f64, f64),+{use core_graphics::event::CGEventTapLocation;use std::time::{SystemTime, UNIX_EPOCH};@@ -552,6 +595,7 @@ pub fn drag_at_xy(button_number,0,);+ observe(from_x, from_y);std::thread::sleep(std::time::Duration::from_millis(16));// Interpolated drag steps.@@ -569,6 +613,7 @@ pub fn drag_at_xy(drag.set_flags(flags);}post_mouse_event(pid, &drag, il, wid, click_group_id, 1, button_number, 0);+ observe(ix, iy);if step_delay_ms > 0 {std::thread::sleep(std::time::Duration::from_millis(step_delay_ms));}@@ -618,6 +663,39 @@ pub fn drag_at_xy_foreground(modifiers: &[&str],button: DragButton,) -> anyhow::Result<()> {+ drag_at_xy_foreground_observed(+ from_x,+ from_y,+ to_x,+ to_y,+ duration_ms,+ steps,+ modifiers,+ button,+ |_, _| {},+ )+}++/// Foreground drag with an observer called for every native pointer position.+///+/// The cursor overlay uses this to follow the same interpolated path and+/// cadence as the HID gesture instead of gliding only after the real pointer+/// has already completed the drag.+#[allow(clippy::too_many_arguments)]+pub fn drag_at_xy_foreground_observed<F>(+ from_x: f64,+ from_y: f64,+ to_x: f64,+ to_y: f64,+ duration_ms: u64,+ steps: usize,+ modifiers: &[&str],+ button: DragButton,+ mut observe: F,+) -> anyhow::Result<()>+where+ F: FnMut(f64, f64),+{use core_graphics::display::CGDisplay;use core_graphics::event::CGEventTapLocation;@@ -658,6 +736,7 @@ pub fn drag_at_xy_foreground(// when the HID event carries an explicit location.let _ = CGDisplay::warp_mouse_cursor_position(CGPoint::new(from_x, from_y));unsafe { CGAssociateMouseAndMouseCursorPosition(true) };+ observe(from_x, from_y);std::thread::sleep(std::time::Duration::from_millis(40));// Prime the renderer's tracking state with a genuine HID mouse move.@@ -687,18 +766,17 @@ pub fn drag_at_xy_foreground(for i in 1..=steps {let t = i as f64 / steps as f64;- let event = CGEvent::new_mouse_event(- source.clone(),- dragged_type,- CGPoint::new(from_x + (to_x - from_x) * t, from_y + (to_y - from_y) * t),- cg_button,- )- .map_err(|_| anyhow::anyhow!("foreground drag mouseDragged failed"))?;+ let x = from_x + (to_x - from_x) * t;+ let y = from_y + (to_y - from_y) * t;+ let event =+ CGEvent::new_mouse_event(source.clone(), dragged_type, CGPoint::new(x, y), cg_button)+ .map_err(|_| anyhow::anyhow!("foreground drag mouseDragged failed"))?;if flags != CGEventFlags::CGEventFlagNull {event.set_flags(flags);}event.set_integer_value_field(core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE, 1);post(&event);+ observe(x, y);if step_delay_ms > 0 {std::thread::sleep(std::time::Duration::from_millis(step_delay_ms));}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex 950a4f876a..4d7b690e09 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -112,6 +112,7 @@ impl Tool for DragTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);if args.opt_str("scope").as_deref() == Some("desktop")&& args.get("pid").is_none()&& args.get("window_id").is_none()@@ -131,9 +132,14 @@ impl Tool for DragTool {ClickButton::Right => DragButton::Right,ClickButton::Middle => DragButton::Middle,};+ let cursor_for_drag = cursor_key.clone();+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );let result = tokio::task::spawn_blocking(move || {let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();- crate::input::mouse::drag_at_xy_foreground(+ crate::input::mouse::drag_at_xy_foreground_observed(from_x,from_y,to_x,@@ -142,9 +148,24 @@ impl Tool for DragTool {steps,&modifier_refs,button,+ move |x, y| {+ crate::cursor::overlay::send_command(+ cursor_for_drag.clone(),+ cursor_overlay::track_pointer_command(x, y),+ );+ },)}).await;+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(false),+ );+ if matches!(&result, Ok(Ok(()))) {+ self.state+ .cursor_registry+ .update_position(&cursor_key, to_x, to_y);+ }return match result {Ok(Ok(())) => ToolResult::text(format!("Dragged desktop from ({from_x:.1}, {from_y:.1}) to ({to_x:.1}, {to_y:.1})."@@ -174,8 +195,6 @@ impl Tool for DragTool {).with_structured(serde_json::json!({ "code": "background_unavailable" }));}- let cursor_key = super::cursor_tools::resolve_cursor_key(&args);-// Coerce integer or float from JSON for coordinate fields.let coerce = |key: &str| -> Option<f64> {args.opt_f64(key)@@ -279,7 +298,12 @@ impl Tool for DragTool {// Dispatch blocking drag synthesis.let mods_owned = modifiers.clone();let fg = delivery_mode.is_foreground() && window_id.is_some();- let result = focus_guard::with_focus_suppressed(+ let cursor_for_drag = cursor_key.clone();+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );+ let drag_input = focus_guard::with_focus_suppressed(// Foreground drag deliberately activates the target so the global// HID stream carries the pressed-button state. A suppression lease// here would race that activation and restore the prior app before@@ -299,7 +323,8 @@ impl Tool for DragTool {// documented Cocoa activation is the fallback.apps::activate_pid(pid);std::thread::sleep(std::time::Duration::from_millis(40));- return crate::input::mouse::drag_at_xy_foreground(+ let observed_cursor = cursor_for_drag.clone();+ return crate::input::mouse::drag_at_xy_foreground_observed(from_sx,from_sy,to_sx,@@ -308,9 +333,15 @@ impl Tool for DragTool {steps,&m,button,+ move |x, y| {+ crate::cursor::overlay::send_command(+ observed_cursor.clone(),+ cursor_overlay::track_pointer_command(x, y),+ );+ },);}- crate::input::mouse::drag_at_xy(+ crate::input::mouse::drag_at_xy_observed(pid,from_sx,from_sy,@@ -324,6 +355,12 @@ impl Tool for DragTool {&m,button,fg,+ move |x, y| {+ crate::cursor::overlay::send_command(+ cursor_for_drag.clone(),+ cursor_overlay::track_pointer_command(x, y),+ );+ },)};// Foreground rung: activate for the complete HID gesture,@@ -347,11 +384,19 @@ impl Tool for DragTool {},).await;+ let result = drag_input;+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(false),+ );+ if matches!(&result, Ok(Ok(()))) {+ self.state+ .cursor_registry+ .update_position(&cursor_key, to_sx, to_sy);+ }let changes = super::finish_window_observation(snapshot, &args).await;- // Animate cursor to end position.- crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), to_sx, to_sy).await;if let Some(wid) = window_id {crate::cursor::overlay::send_command(cursor_key.clone(),diff --git a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rsindex 5495b3a660..cb0ed0f7b0 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs@@ -543,6 +543,7 @@ impl RenderState {self.core.path.is_some()|| self.core.spring.is_some()|| self.core.click_t.is_some()+ || self.core.session_badge_needs_frame_tick()|| (self.core.motion.idle_hide_ms > 0.0&& self.core.visible&& self.core.pos.0 >= -100.0@@ -1063,9 +1064,11 @@ unsafe fn present_surface(hwnd: windows::Win32::Foundation::HWND, dirty: DirtyRe// ── Idle render gate (issue #1808) ────────────────────────────────────────//-// The overlay window timer is re-armed between two cadences:+// The overlay window timer is re-armed between three cadences:// * ACTIVE (`TIMER_MS_ACTIVE`, ~125 Hz) while any cursor is animating /// fading — this is what produces a smooth glide + click pulse.+// * HOVER (`TIMER_MS_HOVER`, 12.5 Hz) while a revealed cursor can show its+// session badge again under the user's hardware pointer.// * IDLE (`TIMER_MS_IDLE`, a slow heartbeat) when every cursor is// quiescent — the handler then only drains the command channel cheaply// and re-arms ACTIVE the instant a command arrives. No full-screen pixmap@@ -1078,9 +1081,10 @@ unsafe fn present_surface(hwnd: windows::Win32::Foundation::HWND, dirty: DirtyRe// armed at; the WM_TIMER handler flips it based on `render_map_needs_frame_tick`.const TIMER_ID: usize = 1;const TIMER_MS_ACTIVE: u32 = 8; // ~125 Hz, matches the C# reference render rate+const TIMER_MS_HOVER: u32 = 80; // low-cost hardware-pointer hover samplingconst TIMER_MS_IDLE: u32 = 250; // slow heartbeat: drain channel, stay responsive/// Current armed timer cadence in ms. Compared against the desired cadence each-/// WM_TIMER so we only call `SetTimer` (re-arm) on an actual active↔idle flip.+/// WM_TIMER so we only call `SetTimer` when the desired cadence changes.static TIMER_PERIOD_MS: std::sync::atomic::AtomicU32 =std::sync::atomic::AtomicU32::new(TIMER_MS_ACTIVE);@@ -1159,7 +1163,7 @@ unsafe extern "system" fn wnd_proc(// the slow IDLE cadence below.let was_active =TIMER_PERIOD_MS.load(std::sync::atomic::Ordering::Relaxed) == TIMER_MS_ACTIVE;- let (upload, arrived, pinned_wid, needs_tick) = {+ let (upload, arrived, pinned_wid, needs_tick, needs_hover_poll) = {let mut guard = RENDER.lock().unwrap();if let Some(map) = guard.as_mut() {// Drain the channel via get-or-create; track the last-touched@@ -1204,19 +1208,31 @@ unsafe extern "system" fn wnd_proc((rs.core.idle_secs + (real_dt - dt)).min(fade_start);}}+ let mut pointer = POINT::default();+ let pointer = GetCursorPos(&mut pointer)+ .ok()+ .map(|_| (f64::from(pointer.x), f64::from(pointer.y)));+ let mut hover_changed = false;+ for rs in map.cursors.values_mut() {+ hover_changed |= rs.core.update_session_badge_hover(pointer);+ }// After ticking: does any cursor still need frame ticks?let needs_tick = render_map_needs_frame_tick(map);+ let needs_hover_poll = map+ .cursors+ .values()+ .any(|rs| rs.core.session_badge_needs_hover_poll());// Render only when something can have changed pixels this// frame: a fresh command, a still-running animation, or the// final settle frame as the previous animation winds down// (`was_active && !needs_tick`). A fully-quiescent idle tick// returns `None` here and does no compositing at all.- let should_render = had_msg || needs_tick || was_active;+ let should_render = had_msg || hover_changed || needs_tick || was_active;if !should_render {- (None, arrived, None, needs_tick)+ (None, arrived, None, needs_tick, needs_hover_poll)} else {// Decide where to pin the single overlay window in z.//@@ -1261,10 +1277,10 @@ unsafe extern "system" fn wnd_proc(// preserves pre-retina-fix behaviour on Windows).let upload = composite_dirty(map);- (upload, arrived, pinned, needs_tick)+ (upload, arrived, pinned, needs_tick, needs_hover_poll)}} else {- (None, Vec::new(), None, false)+ (None, Vec::new(), None, false, false)}};@@ -1299,6 +1315,8 @@ unsafe extern "system" fn wnd_proc(// same period every tick would itself be needless work.let desired_ms = if needs_tick {TIMER_MS_ACTIVE+ } else if needs_hover_poll {+ TIMER_MS_HOVER} else {TIMER_MS_IDLE};diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 379b79b7be..351b197018 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -208,6 +208,34 @@ async fn overlay_glide_to(key: &str, sx: f64, sy: f64) {}crate::overlay::animate_cursor_to(key.to_owned(), sx, sy).await;}++async fn track_overlay_drag(+ key: String,+ from: (f64, f64),+ to: (f64, f64),+ duration_ms: u64,+ steps: usize,+) {+ if key.is_empty() || !crate::overlay::is_enabled(&key) {+ return;+ }+ crate::overlay::send_command(+ key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );+ let steps = steps.max(1);+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);+ for index in 0..=steps {+ let t = index as f64 / steps as f64;+ let x = from.0 + (to.0 - from.0) * t;+ let y = from.1 + (to.1 - from.1) * t;+ crate::overlay::send_command(key.clone(), cursor_overlay::track_pointer_command(x, y));+ if index < steps && !step_delay.is_zero() {+ tokio::time::sleep(step_delay).await;+ }+ }+ crate::overlay::send_command(key, cursor_overlay::OverlayCommand::SetPressed(false));+}use cua_driver_contract::{ClickButton, ClickInput, DragInput, GetCursorPositionInput, GetDesktopStateInput,GetScreenSizeInput, HotkeyInput, MoveCursorInput, PressKeyInput, ScrollDirection, ScrollInput,@@ -6263,6 +6291,7 @@ impl Tool for DragTool {async fn invoke(&self, args: Value) -> ToolResult {use crate::input::delivery::{DeliveryMode, EventKind};use cua_driver_core::tool_args::ArgsExt;+ let cursor_key = resolve_cursor_key(&args);if args.get("scope").and_then(Value::as_str) == Some("desktop")&& args.get("pid").is_none()&& args.get("window_id").is_none()@@ -6282,7 +6311,7 @@ impl Tool for DragTool {Ok(hwnd) => hwnd,Err(error) => return ToolResult::error(error.to_string()),};- return match tokio::task::spawn_blocking(move || {+ let native_drag = tokio::task::spawn_blocking(move || {crate::input::mouse::send_drag_synthesized(hwnd,from_x,@@ -6293,9 +6322,16 @@ impl Tool for DragTool {steps,&button,)- })- .await- {+ });+ let visual_drag = track_overlay_drag(+ cursor_key.clone(),+ (f64::from(from_x), f64::from(from_y)),+ (f64::from(to_x), f64::from(to_y)),+ duration_ms,+ steps,+ );+ let (native_drag, ()) = tokio::join!(native_drag, visual_drag);+ return match native_drag {Ok(Ok(())) => ToolResult::text(format!("Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."))@@ -6316,7 +6352,6 @@ impl Tool for DragTool {let pid = raw_pid as u32;let delivery = DeliveryMode::from_args(&args);- let cursor_key = resolve_cursor_key(&args);// Accepts numeric JSON as either float or integer — coerce both to f64.let coerce = |key: &str| -> Option<f64> {args.opt_f64(key)@@ -6435,24 +6470,21 @@ impl Tool for DragTool {steps.max(8),&btn,)- })- .await;+ });+ let visual_drag = track_overlay_drag(+ cursor_key.clone(),+ (f64::from(sx_from), f64::from(sy_from)),+ (f64::from(sx_to), f64::from(sy_to)),+ duration_ms,+ steps,+ );+ let (inj, ()) = tokio::join!(inj, visual_drag);return match inj {- Ok(Ok(())) => {- overlay_glide_to(&cursor_key, sx_to as f64, sy_to as f64).await;- crate::overlay::send_command(- cursor_key.clone(),- cursor_overlay::OverlayCommand::ClickPulse {- x: sx_to as f64,- y: sy_to as f64,- },- );- ToolResult::text(format!(- "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \+ Ok(Ok(())) => ToolResult::text(format!(+ "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \(delivery_mode:background, PostMessage would have been dropped)."- ))- }+ )),Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(hwnd,EventKind::MouseMove,@@ -6469,6 +6501,7 @@ impl Tool for DragTool {// foreground-lock constraint as send_click_synthesized.if delivery == DeliveryMode::Foreground {let btn_fg = button.clone();+ pin_overlay_above(&cursor_key, hwnd);let prev_fg_addr = unsafe {windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize};@@ -6483,8 +6516,15 @@ impl Tool for DragTool {steps,&btn_fg,)- })- .await;+ });+ let visual_drag = track_overlay_drag(+ cursor_key.clone(),+ (f64::from(sx_from), f64::from(sy_from)),+ (f64::from(sx_to), f64::from(sy_to)),+ duration_ms,+ steps,+ );+ let (send_result, ()) = tokio::join!(send_result, visual_drag);tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));let button_suffix = if button == "left" {String::new()@@ -6535,13 +6575,9 @@ impl Tool for DragTool {// sufficient — the 80 ms z-order tick keeps it asserted thereafter.pin_overlay_above(&cursor_key, hwnd);- // Animate the agent cursor to the drag-start, fire a press pulse,- // run the actual drag synthesis, then glide to the drag-end and- // fire a release pulse. Mirrors macOS `tools/drag.rs:191-242`.- // Cursor doesn't follow the interpolated PostMessage path during- // the drag itself (the timing coordination would be invasive);- // pre- and post-glides plus the press/release pulses are enough- // signal for a user watching the agent operate.+ // Move the agent cursor to the drag-start, show the press state, and+ // track the same interpolated step count and duration while the+ // posted-message drag runs.overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;crate::overlay::send_command(cursor_key.clone(),@@ -6566,22 +6602,15 @@ impl Tool for DragTool {steps,&button_c,)- })- .await;-- // Drag finished — glide the visual cursor to the drag-end and- // pulse the release. Skipped on error so the cursor doesn't lie- // about a successful endpoint.- if matches!(&result, Ok(Ok(()))) {- overlay_glide_to(&cursor_key, sx_to as f64, sy_to as f64).await;- crate::overlay::send_command(- cursor_key.clone(),- cursor_overlay::OverlayCommand::ClickPulse {- x: sx_to as f64,- y: sy_to as f64,- },- );- }+ });+ let visual_drag = track_overlay_drag(+ cursor_key.clone(),+ (f64::from(sx_from), f64::from(sy_from)),+ (f64::from(sx_to), f64::from(sy_to)),+ duration_ms,+ steps,+ );+ let (result, ()) = tokio::join!(result, visual_drag);match result {Ok(Ok(())) => {