Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): keep cursor badges and drags in sync (#2676)

Francesco Bonaccib249df857994
diff --git a/AGENTS.md b/AGENTS.md
index 16b8e8e940..5c04dfb36b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -23,6 +23,19 @@ Preserve contributor credit when external code or design ships in Cua.
Do not reimplement submitted work solely to remove its authorship history.
+## Cross-platform Cua Driver behavior
+
+Treat user-visible Cua Driver behavior as a cross-platform contract. Implement
+shared state, geometry, timing, and protocol semantics in the common crates
+whenever possible, then keep each macOS, Windows, X11, and Wayland adapter thin.
+Do not silently land a platform-specific cursor or interaction change as though
+it were universal.
+
+For every affected platform, add focused coverage and either verify the native
+behavior or document a concrete operating-system or compositor limitation.
+When a platform cannot support the same contract, return or publish that
+limitation explicitly instead of substituting misleading behavior.
+
## Pull request titles and component releases
Pull requests are squash-merged, so the pull request title becomes the commit
diff --git a/libs/cua-driver/docs/cursor-themes.md b/libs/cua-driver/docs/cursor-themes.md
index 1e05aac380..a213e913f4 100644
--- a/libs/cua-driver/docs/cursor-themes.md
+++ b/libs/cua-driver/docs/cursor-themes.md
@@ -20,7 +20,14 @@ The native renderer places the sanitized public session name in a compact badge
below the built-in pointer. The badge uses a session-accent gradient and a
circular session marker. It appears when the cursor is first revealed, remains
fully visible for two seconds, then fades over 400 milliseconds while the
-cursor stays available. Hiding and revealing the cursor shows the badge again.
+cursor stays available. Moving the human pointer over the synthetic cursor
+temporarily reveals the badge again, and moving it away hides the badge.
+Hiding and revealing the cursor also shows the badge again. Hover reveal uses
+the native global-pointer APIs on macOS, Windows, and X11. Stock Wayland does
+not expose another client's global pointer position, so native Wayland keeps
+the timed reveal but cannot offer hover reveal without a compositor-owned
+adapter. The click-through overlay never captures pointer input to fake this
+capability.
The badge is not part of the dotLottie theme artifact, so custom theme authors
do not need to provide badge artwork or a font. Cua Driver strips control
characters, collapses whitespace, and shortens labels longer than 28 characters
@@ -31,6 +38,10 @@ screen target, including indexed or token-addressed text and value operations,
move the agent cursor to that target before dispatch. Actions without a spatial
target, such as typing into the already focused desktop application, keep the
cursor at its current location and only change the semantic animation.
+During a drag, the native gesture's endpoints, step count, and wall-clock
+duration drive the same shared pointer-anchor transform on macOS, Windows, X11,
+and supported Wayland input routes, so the synthetic cursor follows the held
+gesture instead of moving only after release.
The cursor is a visual aid for people watching an agent. It is not a security
indicator, an authorization prompt, or evidence that a tool call succeeded.
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
index ba960822ca..e8d4583dd4 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
@@ -359,3 +359,38 @@ pub enum OverlayCommand {
/// `None` clears the highlight.
ShowFocusRect(Option<[f64; 4]>),
}
+
+/// Build the shared overlay command for one native pointer position.
+///
+/// Native drag implementations report the actual event coordinate while the
+/// cursor artwork is centred 16 points down-right so its tip lands on that
+/// coordinate. Keeping this transform here prevents platform-specific drag
+/// loops from drifting apart.
+pub fn track_pointer_command(x: f64, y: f64) -> OverlayCommand {
+ const CLICK_OFFSET: f64 = 16.0;
+ let heading = std::f64::consts::FRAC_PI_4;
+ OverlayCommand::SnapTo {
+ x: x + heading.cos() * CLICK_OFFSET,
+ y: y + heading.sin() * CLICK_OFFSET,
+ heading_radians: Some(heading),
+ }
+}
+
+#[cfg(test)]
+mod pointer_tracking_tests {
+ use super::*;
+
+ #[test]
+ fn tracked_artwork_keeps_its_tip_on_the_native_pointer() {
+ let OverlayCommand::SnapTo {
+ x,
+ y,
+ heading_radians: Some(heading),
+ } = track_pointer_command(120.0, 80.0)
+ else {
+ panic!("pointer tracking must produce an anchored snap");
+ };
+ assert!((x - (120.0 + heading.cos() * 16.0)).abs() < f64::EPSILON);
+ assert!((y - (80.0 + heading.sin() * 16.0)).abs() < f64::EPSILON);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
index 6955828053..dc6dd4f3df 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
@@ -86,6 +86,10 @@ pub struct RenderStateCore {
pub session_label: Option<String>,
/// Elapsed time since the session label was revealed with the cursor.
pub session_badge_secs: f64,
+ /// Whether the user's hardware pointer is currently over this synthetic
+ /// cursor. Hover temporarily reveals an already-faded session badge
+ /// without changing its one-shot reveal timer.
+ pub session_badge_hovered: bool,
}
impl RenderStateCore {
@@ -130,6 +134,7 @@ impl RenderStateCore {
pinned_wid: None,
session_label: None,
session_badge_secs: SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS,
+ session_badge_hovered: false,
}
}
@@ -147,6 +152,9 @@ impl RenderStateCore {
if self.session_label.is_none() {
return 0.0;
}
+ if self.session_badge_hovered {
+ return 1.0;
+ }
if self.session_badge_secs <= SESSION_BADGE_HOLD_SECS {
return 1.0;
}
@@ -156,6 +164,38 @@ impl RenderStateCore {
(1.0 - smooth) as f32
}
+ pub fn session_badge_needs_frame_tick(&self) -> bool {
+ self.session_label.is_some()
+ && self.session_badge_secs < SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS
+ && self.cursor_is_revealed()
+ }
+
+ /// Whether the platform overlay should keep a low-frequency hardware
+ /// pointer poll alive for hover-to-reveal. This is deliberately separate
+ /// from [`Self::session_badge_needs_frame_tick`]: a faded badge needs hover
+ /// hit-testing, not continuous 60 fps repainting.
+ pub fn session_badge_needs_hover_poll(&self) -> bool {
+ self.session_label.is_some() && self.cursor_is_revealed()
+ }
+
+ /// Update hover state from a platform-native hardware pointer sample.
+ ///
+ /// `self.pos` is the centre of the cursor artwork. The hit radius is a
+ /// little larger than the 42 point production artwork so the interaction
+ /// remains comfortable around the white outline and glow.
+ pub fn update_session_badge_hover(&mut self, pointer: Option<(f64, f64)>) -> bool {
+ const HOVER_RADIUS: f64 = crate::theme::DISPLAY_SIZE as f64 * 0.82;
+ let hovered = self.session_badge_needs_hover_poll()
+ && pointer.is_some_and(|(x, y)| {
+ let dx = x - self.pos.0;
+ let dy = y - self.pos.1;
+ dx * dx + dy * dy <= HOVER_RADIUS * HOVER_RADIUS
+ });
+ let changed = hovered != self.session_badge_hovered;
+ self.session_badge_hovered = hovered;
+ changed
+ }
+
/// Return the theme that is actually being painted, including any
/// non-fatal fallback from an unavailable launch-time selection.
pub fn active_theme_metadata(&self) -> (String, String, String, Option<String>) {
@@ -606,8 +646,11 @@ impl RenderStateCore {
true
}
OverlayCommand::SetSessionLabel(label) => {
- self.session_label = crate::sanitize_session_label(&label);
- self.session_badge_secs = 0.0;
+ let session_label = crate::sanitize_session_label(&label);
+ if session_label != self.session_label {
+ self.session_label = session_label;
+ self.session_badge_secs = 0.0;
+ }
true
}
OverlayCommand::ShowFocusRect(_) => false, // caller-specific
@@ -881,6 +924,32 @@ mod session_badge_and_action_tests {
assert_eq!(core.session_badge_alpha(), 0.0);
}
+ #[test]
+ fn repeated_session_label_metadata_does_not_restart_badge_timer() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ );
+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);
+ assert_eq!(core.session_badge_alpha(), 0.0);
+
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ );
+ assert_eq!(core.session_badge_alpha(), 0.0);
+
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Writing".into()),
+ false,
+ false,
+ );
+ assert_eq!(core.session_badge_alpha(), 1.0);
+ }
+
#[test]
fn revealing_hidden_cursor_restarts_badge_without_restarting_on_every_move() {
let mut core = RenderStateCore::new(CursorConfig::default());
@@ -902,6 +971,7 @@ mod session_badge_and_action_tests {
false,
);
assert_eq!(core.session_badge_alpha(), 1.0);
+ assert!(core.session_badge_needs_frame_tick());
core.tick_motion(0.5);
let elapsed = core.session_badge_secs;
core.apply_command_base(
@@ -914,6 +984,30 @@ mod session_badge_and_action_tests {
false,
);
assert_eq!(core.session_badge_secs, elapsed);
+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);
+ assert!(!core.session_badge_needs_frame_tick());
+ }
+
+ #[test]
+ fn hardware_pointer_hover_reveals_only_while_over_cursor() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ core.pos = (300.0, 240.0);
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ );
+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);
+ assert_eq!(core.session_badge_alpha(), 0.0);
+ assert!(core.session_badge_needs_hover_poll());
+
+ assert!(core.update_session_badge_hover(Some((302.0, 238.0))));
+ assert_eq!(core.session_badge_alpha(), 1.0);
+ assert!(!core.update_session_badge_hover(Some((304.0, 241.0))));
+ assert_eq!(core.session_badge_alpha(), 1.0);
+
+ assert!(core.update_session_badge_hover(Some((500.0, 500.0))));
+ assert_eq!(core.session_badge_alpha(), 0.0);
}
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
index 0df93a1570..d3fd32db84 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
@@ -460,6 +460,7 @@ impl RenderState {
self.core.path.is_some()
|| self.core.spring.is_some()
|| self.core.click_t.is_some()
+ || self.core.session_badge_needs_frame_tick()
|| (self.core.motion.idle_hide_ms > 0.0
&& self.core.visible
&& self.core.pos.0 >= -100.0
@@ -775,12 +776,18 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMs
let now = Instant::now();
let elapsed_dt = now.duration_since(last_tick).as_secs_f64();
last_tick = now;
+ let hardware_pointer = conn
+ .query_pointer(root)
+ .ok()
+ .and_then(|cookie| cookie.reply().ok())
+ .map(|reply| (f64::from(reply.root_x), f64::from(reply.root_y)));
// Drain commands and tick.
let (
arrived,
pinned_wid,
had_msg,
+ hover_changed,
next_frame_tick_needed,
next_z_order_tick_needed,
next_idle_wait_interval,
@@ -795,6 +802,10 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMs
maintenance_timeout,
frame_tick_needed,
);
+ let mut hover_changed = false;
+ for rs in map.cursors.values_mut() {
+ hover_changed |= rs.core.update_session_badge_hover(hardware_pointer);
+ }
let pinned_wid = map
.last_active
.as_ref()
@@ -807,6 +818,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMs
arrived,
pinned_wid,
had_msg,
+ hover_changed,
next_frame_tick_needed,
next_z_order_tick_needed,
next_idle_wait_interval,
@@ -829,7 +841,7 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver<OverlayMs
// Render after a command, during an active animation/fade, and once
// more as the final active state settles. This leaves the X11 window in
// its completed/cleared state before the next blocking receive.
- if had_msg || frame_tick_needed || next_frame_tick_needed {
+ if had_msg || hover_changed || frame_tick_needed || next_frame_tick_needed {
let tiles = {
let guard = RENDER.lock().unwrap();
guard.as_ref().map(render_x11_tiles)
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 7d9121bebd..df57376b3c 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1576,6 +1576,37 @@ async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {
crate::overlay::animate_cursor_to_for(cursor_id.to_owned(), sx, sy).await;
}
+async fn track_overlay_drag_for(
+ cursor_id: String,
+ from: (f64, f64),
+ to: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+) {
+ if !crate::overlay::is_enabled_for(&cursor_id) {
+ return;
+ }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let steps = steps.max(1);
+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
+ for index in 0..=steps {
+ let t = index as f64 / steps as f64;
+ let x = from.0 + (to.0 - from.0) * t;
+ let y = from.1 + (to.1 - from.1) * t;
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::track_pointer_command(x, y),
+ );
+ if index < steps && !step_delay.is_zero() {
+ tokio::time::sleep(step_delay).await;
+ }
+ }
+ crate::overlay::send_command_for(cursor_id, cursor_overlay::OverlayCommand::SetPressed(false));
+}
+
fn process_name(pid: u32) -> Option<String> {
let cmdline = fs::read(format!("/proc/{pid}/cmdline")).ok()?;
let first = String::from_utf8_lossy(&cmdline)
@@ -4441,8 +4472,20 @@ impl Tool for DragTool {
steps,
)
}
- })
- .await;
+ });
+ let visual_drag = track_overlay_drag_for(
+ cursor_id.clone(),
+ (from_x, from_y),
+ (to_x, to_y),
+ duration_ms,
+ steps,
+ );
+ let (result, ()) = tokio::join!(result, visual_drag);
+ if matches!(&result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, to_x, to_y);
+ }
return match result {
Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(
json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
@@ -4563,11 +4606,6 @@ impl Tool for DragTool {
},
);
}
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(true),
- );
-
// Native Wayland: emit press + interpolated motion + release as one
// virtual-pointer (wlroots) or libei (GNOME/KDE) sequence, output-relative
// coords. Returns early so we don't fall into the X11 XSendEvent loop below.
@@ -4584,7 +4622,6 @@ impl Tool for DragTool {
button as u32,
)
})
- .await
} else {
let ((fxi, fyi), (txi, tyi)) = wayland_points.unwrap_or((
(from_x.round() as i32, from_y.round() as i32),
@@ -4593,12 +4630,27 @@ impl Tool for DragTool {
tokio::task::spawn_blocking(move || {
crate::wayland::drag(xid, fxi, fyi, txi, tyi, steps_u32, duration_ms, button)
})
- .await
};
- crate::overlay::send_command_for(
+ let ((from_output_x, from_output_y), (to_output_x, to_output_y)) = wayland_points
+ .unwrap_or((
+ (from_x.round() as i32, from_y.round() as i32),
+ (to_x.round() as i32, to_y.round() as i32),
+ ));
+ let visual_drag = track_overlay_drag_for(
cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
+ (f64::from(from_output_x), f64::from(from_output_y)),
+ (f64::from(to_output_x), f64::from(to_output_y)),
+ duration_ms,
+ steps,
);
+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
+ if matches!(&drag_result, Ok(Ok(()))) {
+ self.state.cursor_registry.update_position(
+ &cursor_id,
+ f64::from(to_output_x),
+ f64::from(to_output_y),
+ );
+ }
return match drag_result {
Ok(Ok(())) => ToolResult::text(format!(
"✅ Posted drag ({button_str}) to pid {pid} \
@@ -4635,12 +4687,20 @@ impl Tool for DragTool {
steps,
)
})
- })
- .await;
- crate::overlay::send_command_for(
+ });
+ let visual_drag = track_overlay_drag_for(
cursor_id.clone(),
- cursor_overlay::OverlayCommand::SetPressed(false),
+ (screen_from_x, screen_from_y),
+ (screen_to_x, screen_to_y),
+ duration_ms,
+ steps,
);
+ let (drag_result, ()) = tokio::join!(drag_result, visual_drag);
+ if matches!(&drag_result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, screen_to_x, screen_to_y);
+ }
return match drag_result {
Ok(Ok(())) => ToolResult::text(format!(
"Dragged ({button_str}) to pid {pid} from ({from_x:.0}, {from_y:.0}) \
@@ -4657,6 +4717,10 @@ impl Tool for DragTool {
};
}
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
let press_result = tokio::task::spawn_blocking(move || {
crate::input::send_button_down(
xid,
@@ -4678,8 +4742,6 @@ impl Tool for DragTool {
} else {
duration_ms
};
- let mut prev_x = from_x;
- let mut prev_y = from_y;
for i in 1..=steps {
let t = i as f64 / steps.max(1) as f64;
let ix = from_x + (to_x - from_x) * t;
@@ -4699,20 +4761,14 @@ impl Tool for DragTool {
tokio::task::spawn_blocking(move || window_local_to_screen(xid, ix, iy))
.await
{
- let heading = if (ix - prev_x).abs() > f64::EPSILON
- || (iy - prev_y).abs() > f64::EPSILON
- {
- Some((iy - prev_y).atan2(ix - prev_x))
- } else {
- None
- };
self.state
.cursor_registry
.update_position(&cursor_id, sx, sy);
- overlay_move_to_for(&cursor_id, sx, sy, heading);
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::track_pointer_command(sx, sy),
+ );
}
- prev_x = ix;
- prev_y = iy;
if step_delay_ms > 0 {
tokio::time::sleep(std::time::Duration::from_millis(step_delay_ms))
.await;
@@ -4750,19 +4806,13 @@ impl Tool for DragTool {
if let Ok(Ok((sx_to, sy_to))) =
tokio::task::spawn_blocking(move || window_local_to_screen(xid, to_x, to_y)).await
{
- self.state
- .cursor_registry
- .update_position(&cursor_id, sx_to, sy_to);
- overlay_snap_to_for(
- &cursor_id,
- sx_to,
- sy_to,
- Some((to_y - from_y).atan2(to_x - from_x)),
- );
crate::overlay::send_command_for(
cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx_to, y: sy_to },
+ cursor_overlay::track_pointer_command(sx_to, sy_to),
);
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, sx_to, sy_to);
}
}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
index 411f5eb40f..e3a1c70e23 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
@@ -216,7 +216,11 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
layer_surface.set_exclusive_zone(-1);
layer_surface.set_keyboard_interactivity(KeyboardInteractivity::None);
- // Click-through: empty input region.
+ // Click-through: empty input region. Standard Wayland intentionally does
+ // not expose another client's global pointer position, so this surface
+ // cannot implement the macOS/Windows/X11 badge-hover reveal without a
+ // compositor-owned adapter. Giving it an input region would steal the
+ // user's pointer events instead of observing them.
let region: WlRegion = compositor.create_region(&qh, ());
surface.set_input_region(Some(&region));
region.destroy();
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
index f0332cf63c..9089525e80 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs
@@ -522,6 +522,7 @@ impl RenderState {
|| self.core.spring.is_some()
|| self.core.click_t.is_some()
|| self.focus_rect.is_some()
+ || self.core.session_badge_needs_frame_tick()
|| (self.core.motion.idle_hide_ms > 0.0
&& self.core.visible
&& self.core.pos.0 >= -100.0
@@ -666,8 +667,10 @@ fn render_loop(
_win_h: f64,
) {
let target_frame_ms = Duration::from_millis(16); // ~60 fps while pixels can change
+ let hover_poll_ms = Duration::from_millis(80);
let mut last_tick = Instant::now();
let mut frame_tick_needed = false;
+ let mut hover_poll_needed = false;
// Repin bookkeeping: track last pinned wid and a frame counter for
// the periodic defensive-repin (every ~60 active frames ≈ 1 s).
let mut last_pinned: Option<u64> = None;
@@ -677,11 +680,17 @@ fn render_loop(
// When no cursor animation/fade is active, block until the MCP side
// sends a command. This is the idle-server fast path: no fullscreen
// pixmap allocation, no CGImage conversion, no 60fps wakeup.
- let first_msg = if frame_tick_needed {
- None
+ let (first_msg, hover_poll_tick) = if frame_tick_needed {
+ (None, hover_poll_needed)
+ } else if hover_poll_needed {
+ match rx.recv_timeout(hover_poll_ms) {
+ Ok(msg) => (Some(msg), true),
+ Err(std::sync::mpsc::RecvTimeoutError::Timeout) => (None, true),
+ Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,
+ }
} else {
match rx.recv() {
- Ok(msg) => Some(msg),
+ Ok(msg) => (Some(msg), false),
Err(_) => break,
}
};
@@ -702,7 +711,17 @@ fn render_loop(
// `pinned_wid` follows the most-recently-updated cursor: a single
// NSWindow can occupy only one z-band, so the last-active cursor's
// target wins. `arrived` collects the keys whose path just ended.
- let (pinned_wid, raise_unpinned, arrived, win_w, win_h, had_msg, next_frame_tick_needed) = {
+ let (
+ pinned_wid,
+ raise_unpinned,
+ arrived,
+ win_w,
+ win_h,
+ had_msg,
+ hover_changed,
+ next_frame_tick_needed,
+ next_hover_poll_needed,
+ ) = {
let mut guard = RENDER.lock().unwrap();
match guard.as_mut() {
Some(map) => {
@@ -734,6 +753,22 @@ fn render_loop(
}
}
}
+ let pointer = if hover_poll_tick
+ || map
+ .cursors
+ .values()
+ .any(|rs| rs.core.session_badge_needs_hover_poll())
+ {
+ hardware_cursor_position()
+ } else {
+ None
+ };
+ let mut hover_changed = false;
+ if pointer.is_some() || hover_poll_tick {
+ for rs in map.cursors.values_mut() {
+ hover_changed |= rs.core.update_session_badge_hover(pointer);
+ }
+ }
let pinned = last_key
.as_ref()
.and_then(|k| map.cursors.get(k))
@@ -745,6 +780,10 @@ fn render_loop(
.is_some_and(cursor_is_externally_visible)
&& pinned.is_none();
let next_frame_tick_needed = render_map_needs_frame_tick(map);
+ let next_hover_poll_needed = map
+ .cursors
+ .values()
+ .any(|rs| rs.core.session_badge_needs_hover_poll());
(
pinned,
raise_unpinned,
@@ -752,7 +791,9 @@ fn render_loop(
map.win_w,
map.win_h,
had_msg,
+ hover_changed,
next_frame_tick_needed,
+ next_hover_poll_needed,
)
}
None => break,
@@ -790,7 +831,7 @@ fn render_loop(
// Render only when a command arrived or the previous/next tick can
// change pixels. A final frame is emitted as animations/fades finish so
// the layer is left in the completed/cleared state before blocking.
- if had_msg || frame_tick_needed || next_frame_tick_needed {
+ if had_msg || hover_changed || frame_tick_needed || next_frame_tick_needed {
let pixmap = {
let guard = RENDER.lock().unwrap();
if let Some(map) = guard.as_ref() {
@@ -830,6 +871,7 @@ fn render_loop(
}
frame_tick_needed = next_frame_tick_needed;
+ hover_poll_needed = next_hover_poll_needed;
if frame_tick_needed {
// Sleep remainder of frame budget.
let elapsed = Instant::now().duration_since(last_tick);
@@ -840,6 +882,18 @@ fn render_loop(
}
}
+fn hardware_cursor_position() -> Option<(f64, f64)> {
+ use core_graphics::{
+ event::CGEvent,
+ event_source::{CGEventSource, CGEventSourceStateID},
+ };
+
+ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState).ok()?;
+ let event = CGEvent::new(source).ok()?;
+ let location = event.location();
+ Some((location.x, location.y))
+}
+
fn cursor_is_externally_visible(state: &RenderState) -> bool {
state.core.cfg.enabled
&& state.core.visible
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
index 5ab4e11b4d..6a210126e0 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
@@ -487,6 +487,49 @@ pub fn drag_at_xy(
button: DragButton,
foreground_release: bool,
) -> anyhow::Result<()> {
+ drag_at_xy_observed(
+ pid,
+ from_x,
+ from_y,
+ to_x,
+ to_y,
+ from_local,
+ to_local,
+ wid,
+ duration_ms,
+ steps,
+ modifiers,
+ button,
+ foreground_release,
+ |_, _| {},
+ )
+}
+
+/// PID-routed drag with an observer called for every native pointer position.
+///
+/// The cursor overlay uses this for the same reason as
+/// [`drag_at_xy_foreground_observed`]: the synthetic cursor should follow the
+/// actual dispatched path rather than jumping to the endpoint after release.
+#[allow(clippy::too_many_arguments)]
+pub fn drag_at_xy_observed<F>(
+ pid: i32,
+ from_x: f64,
+ from_y: f64,
+ to_x: f64,
+ to_y: f64,
+ from_local: Option<(f64, f64)>,
+ to_local: Option<(f64, f64)>,
+ wid: Option<u32>,
+ duration_ms: u64,
+ steps: usize,
+ modifiers: &[&str],
+ button: DragButton,
+ foreground_release: bool,
+ mut observe: F,
+) -> anyhow::Result<()>
+where
+ F: FnMut(f64, f64),
+{
use core_graphics::event::CGEventTapLocation;
use std::time::{SystemTime, UNIX_EPOCH};
@@ -552,6 +595,7 @@ pub fn drag_at_xy(
button_number,
0,
);
+ observe(from_x, from_y);
std::thread::sleep(std::time::Duration::from_millis(16));
// Interpolated drag steps.
@@ -569,6 +613,7 @@ pub fn drag_at_xy(
drag.set_flags(flags);
}
post_mouse_event(pid, &drag, il, wid, click_group_id, 1, button_number, 0);
+ observe(ix, iy);
if step_delay_ms > 0 {
std::thread::sleep(std::time::Duration::from_millis(step_delay_ms));
}
@@ -618,6 +663,39 @@ pub fn drag_at_xy_foreground(
modifiers: &[&str],
button: DragButton,
) -> anyhow::Result<()> {
+ drag_at_xy_foreground_observed(
+ from_x,
+ from_y,
+ to_x,
+ to_y,
+ duration_ms,
+ steps,
+ modifiers,
+ button,
+ |_, _| {},
+ )
+}
+
+/// Foreground drag with an observer called for every native pointer position.
+///
+/// The cursor overlay uses this to follow the same interpolated path and
+/// cadence as the HID gesture instead of gliding only after the real pointer
+/// has already completed the drag.
+#[allow(clippy::too_many_arguments)]
+pub fn drag_at_xy_foreground_observed<F>(
+ from_x: f64,
+ from_y: f64,
+ to_x: f64,
+ to_y: f64,
+ duration_ms: u64,
+ steps: usize,
+ modifiers: &[&str],
+ button: DragButton,
+ mut observe: F,
+) -> anyhow::Result<()>
+where
+ F: FnMut(f64, f64),
+{
use core_graphics::display::CGDisplay;
use core_graphics::event::CGEventTapLocation;
@@ -658,6 +736,7 @@ pub fn drag_at_xy_foreground(
// when the HID event carries an explicit location.
let _ = CGDisplay::warp_mouse_cursor_position(CGPoint::new(from_x, from_y));
unsafe { CGAssociateMouseAndMouseCursorPosition(true) };
+ observe(from_x, from_y);
std::thread::sleep(std::time::Duration::from_millis(40));
// Prime the renderer's tracking state with a genuine HID mouse move.
@@ -687,18 +766,17 @@ pub fn drag_at_xy_foreground(
for i in 1..=steps {
let t = i as f64 / steps as f64;
- let event = CGEvent::new_mouse_event(
- source.clone(),
- dragged_type,
- CGPoint::new(from_x + (to_x - from_x) * t, from_y + (to_y - from_y) * t),
- cg_button,
- )
- .map_err(|_| anyhow::anyhow!("foreground drag mouseDragged failed"))?;
+ let x = from_x + (to_x - from_x) * t;
+ let y = from_y + (to_y - from_y) * t;
+ let event =
+ CGEvent::new_mouse_event(source.clone(), dragged_type, CGPoint::new(x, y), cg_button)
+ .map_err(|_| anyhow::anyhow!("foreground drag mouseDragged failed"))?;
if flags != CGEventFlags::CGEventFlagNull {
event.set_flags(flags);
}
event.set_integer_value_field(core_graphics::event::EventField::MOUSE_EVENT_CLICK_STATE, 1);
post(&event);
+ observe(x, y);
if step_delay_ms > 0 {
std::thread::sleep(std::time::Duration::from_millis(step_delay_ms));
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index 950a4f876a..4d7b690e09 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -112,6 +112,7 @@ impl Tool for DragTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
if args.opt_str("scope").as_deref() == Some("desktop")
&& args.get("pid").is_none()
&& args.get("window_id").is_none()
@@ -131,9 +132,14 @@ impl Tool for DragTool {
ClickButton::Right => DragButton::Right,
ClickButton::Middle => DragButton::Middle,
};
+ let cursor_for_drag = cursor_key.clone();
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
let result = tokio::task::spawn_blocking(move || {
let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
- crate::input::mouse::drag_at_xy_foreground(
+ crate::input::mouse::drag_at_xy_foreground_observed(
from_x,
from_y,
to_x,
@@ -142,9 +148,24 @@ impl Tool for DragTool {
steps,
&modifier_refs,
button,
+ move |x, y| {
+ crate::cursor::overlay::send_command(
+ cursor_for_drag.clone(),
+ cursor_overlay::track_pointer_command(x, y),
+ );
+ },
)
})
.await;
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ if matches!(&result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, to_x, to_y);
+ }
return match result {
Ok(Ok(())) => ToolResult::text(format!(
"Dragged desktop from ({from_x:.1}, {from_y:.1}) to ({to_x:.1}, {to_y:.1})."
@@ -174,8 +195,6 @@ impl Tool for DragTool {
)
.with_structured(serde_json::json!({ "code": "background_unavailable" }));
}
- let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
-
// Coerce integer or float from JSON for coordinate fields.
let coerce = |key: &str| -> Option<f64> {
args.opt_f64(key)
@@ -279,7 +298,12 @@ impl Tool for DragTool {
// Dispatch blocking drag synthesis.
let mods_owned = modifiers.clone();
let fg = delivery_mode.is_foreground() && window_id.is_some();
- let result = focus_guard::with_focus_suppressed(
+ let cursor_for_drag = cursor_key.clone();
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let drag_input = focus_guard::with_focus_suppressed(
// Foreground drag deliberately activates the target so the global
// HID stream carries the pressed-button state. A suppression lease
// here would race that activation and restore the prior app before
@@ -299,7 +323,8 @@ impl Tool for DragTool {
// documented Cocoa activation is the fallback.
apps::activate_pid(pid);
std::thread::sleep(std::time::Duration::from_millis(40));
- return crate::input::mouse::drag_at_xy_foreground(
+ let observed_cursor = cursor_for_drag.clone();
+ return crate::input::mouse::drag_at_xy_foreground_observed(
from_sx,
from_sy,
to_sx,
@@ -308,9 +333,15 @@ impl Tool for DragTool {
steps,
&m,
button,
+ move |x, y| {
+ crate::cursor::overlay::send_command(
+ observed_cursor.clone(),
+ cursor_overlay::track_pointer_command(x, y),
+ );
+ },
);
}
- crate::input::mouse::drag_at_xy(
+ crate::input::mouse::drag_at_xy_observed(
pid,
from_sx,
from_sy,
@@ -324,6 +355,12 @@ impl Tool for DragTool {
&m,
button,
fg,
+ move |x, y| {
+ crate::cursor::overlay::send_command(
+ cursor_for_drag.clone(),
+ cursor_overlay::track_pointer_command(x, y),
+ );
+ },
)
};
// Foreground rung: activate for the complete HID gesture,
@@ -347,11 +384,19 @@ impl Tool for DragTool {
},
)
.await;
+ let result = drag_input;
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ if matches!(&result, Ok(Ok(()))) {
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, to_sx, to_sy);
+ }
let changes = super::finish_window_observation(snapshot, &args).await;
- // Animate cursor to end position.
- crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), to_sx, to_sy).await;
if let Some(wid) = window_id {
crate::cursor::overlay::send_command(
cursor_key.clone(),
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
index 5495b3a660..cb0ed0f7b0 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs
@@ -543,6 +543,7 @@ impl RenderState {
self.core.path.is_some()
|| self.core.spring.is_some()
|| self.core.click_t.is_some()
+ || self.core.session_badge_needs_frame_tick()
|| (self.core.motion.idle_hide_ms > 0.0
&& self.core.visible
&& self.core.pos.0 >= -100.0
@@ -1063,9 +1064,11 @@ unsafe fn present_surface(hwnd: windows::Win32::Foundation::HWND, dirty: DirtyRe
// ── Idle render gate (issue #1808) ────────────────────────────────────────
//
-// The overlay window timer is re-armed between two cadences:
+// The overlay window timer is re-armed between three cadences:
// * ACTIVE (`TIMER_MS_ACTIVE`, ~125 Hz) while any cursor is animating /
// fading — this is what produces a smooth glide + click pulse.
+// * HOVER (`TIMER_MS_HOVER`, 12.5 Hz) while a revealed cursor can show its
+// session badge again under the user's hardware pointer.
// * IDLE (`TIMER_MS_IDLE`, a slow heartbeat) when every cursor is
// quiescent — the handler then only drains the command channel cheaply
// and re-arms ACTIVE the instant a command arrives. No full-screen pixmap
@@ -1078,9 +1081,10 @@ unsafe fn present_surface(hwnd: windows::Win32::Foundation::HWND, dirty: DirtyRe
// armed at; the WM_TIMER handler flips it based on `render_map_needs_frame_tick`.
const TIMER_ID: usize = 1;
const TIMER_MS_ACTIVE: u32 = 8; // ~125 Hz, matches the C# reference render rate
+const TIMER_MS_HOVER: u32 = 80; // low-cost hardware-pointer hover sampling
const TIMER_MS_IDLE: u32 = 250; // slow heartbeat: drain channel, stay responsive
/// Current armed timer cadence in ms. Compared against the desired cadence each
-/// WM_TIMER so we only call `SetTimer` (re-arm) on an actual active↔idle flip.
+/// WM_TIMER so we only call `SetTimer` when the desired cadence changes.
static TIMER_PERIOD_MS: std::sync::atomic::AtomicU32 =
std::sync::atomic::AtomicU32::new(TIMER_MS_ACTIVE);
@@ -1159,7 +1163,7 @@ unsafe extern "system" fn wnd_proc(
// the slow IDLE cadence below.
let was_active =
TIMER_PERIOD_MS.load(std::sync::atomic::Ordering::Relaxed) == TIMER_MS_ACTIVE;
- let (upload, arrived, pinned_wid, needs_tick) = {
+ let (upload, arrived, pinned_wid, needs_tick, needs_hover_poll) = {
let mut guard = RENDER.lock().unwrap();
if let Some(map) = guard.as_mut() {
// Drain the channel via get-or-create; track the last-touched
@@ -1204,19 +1208,31 @@ unsafe extern "system" fn wnd_proc(
(rs.core.idle_secs + (real_dt - dt)).min(fade_start);
}
}
+ let mut pointer = POINT::default();
+ let pointer = GetCursorPos(&mut pointer)
+ .ok()
+ .map(|_| (f64::from(pointer.x), f64::from(pointer.y)));
+ let mut hover_changed = false;
+ for rs in map.cursors.values_mut() {
+ hover_changed |= rs.core.update_session_badge_hover(pointer);
+ }
// After ticking: does any cursor still need frame ticks?
let needs_tick = render_map_needs_frame_tick(map);
+ let needs_hover_poll = map
+ .cursors
+ .values()
+ .any(|rs| rs.core.session_badge_needs_hover_poll());
// Render only when something can have changed pixels this
// frame: a fresh command, a still-running animation, or the
// final settle frame as the previous animation winds down
// (`was_active && !needs_tick`). A fully-quiescent idle tick
// returns `None` here and does no compositing at all.
- let should_render = had_msg || needs_tick || was_active;
+ let should_render = had_msg || hover_changed || needs_tick || was_active;
if !should_render {
- (None, arrived, None, needs_tick)
+ (None, arrived, None, needs_tick, needs_hover_poll)
} else {
// Decide where to pin the single overlay window in z.
//
@@ -1261,10 +1277,10 @@ unsafe extern "system" fn wnd_proc(
// preserves pre-retina-fix behaviour on Windows).
let upload = composite_dirty(map);
- (upload, arrived, pinned, needs_tick)
+ (upload, arrived, pinned, needs_tick, needs_hover_poll)
}
} else {
- (None, Vec::new(), None, false)
+ (None, Vec::new(), None, false, false)
}
};
@@ -1299,6 +1315,8 @@ unsafe extern "system" fn wnd_proc(
// same period every tick would itself be needless work.
let desired_ms = if needs_tick {
TIMER_MS_ACTIVE
+ } else if needs_hover_poll {
+ TIMER_MS_HOVER
} else {
TIMER_MS_IDLE
};
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 379b79b7be..351b197018 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -208,6 +208,34 @@ async fn overlay_glide_to(key: &str, sx: f64, sy: f64) {
}
crate::overlay::animate_cursor_to(key.to_owned(), sx, sy).await;
}
+
+async fn track_overlay_drag(
+ key: String,
+ from: (f64, f64),
+ to: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+) {
+ if key.is_empty() || !crate::overlay::is_enabled(&key) {
+ return;
+ }
+ crate::overlay::send_command(
+ key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let steps = steps.max(1);
+ let step_delay = std::time::Duration::from_millis(duration_ms / steps as u64);
+ for index in 0..=steps {
+ let t = index as f64 / steps as f64;
+ let x = from.0 + (to.0 - from.0) * t;
+ let y = from.1 + (to.1 - from.1) * t;
+ crate::overlay::send_command(key.clone(), cursor_overlay::track_pointer_command(x, y));
+ if index < steps && !step_delay.is_zero() {
+ tokio::time::sleep(step_delay).await;
+ }
+ }
+ crate::overlay::send_command(key, cursor_overlay::OverlayCommand::SetPressed(false));
+}
use cua_driver_contract::{
ClickButton, ClickInput, DragInput, GetCursorPositionInput, GetDesktopStateInput,
GetScreenSizeInput, HotkeyInput, MoveCursorInput, PressKeyInput, ScrollDirection, ScrollInput,
@@ -6263,6 +6291,7 @@ impl Tool for DragTool {
async fn invoke(&self, args: Value) -> ToolResult {
use crate::input::delivery::{DeliveryMode, EventKind};
use cua_driver_core::tool_args::ArgsExt;
+ let cursor_key = resolve_cursor_key(&args);
if args.get("scope").and_then(Value::as_str) == Some("desktop")
&& args.get("pid").is_none()
&& args.get("window_id").is_none()
@@ -6282,7 +6311,7 @@ impl Tool for DragTool {
Ok(hwnd) => hwnd,
Err(error) => return ToolResult::error(error.to_string()),
};
- return match tokio::task::spawn_blocking(move || {
+ let native_drag = tokio::task::spawn_blocking(move || {
crate::input::mouse::send_drag_synthesized(
hwnd,
from_x,
@@ -6293,9 +6322,16 @@ impl Tool for DragTool {
steps,
&button,
)
- })
- .await
- {
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(from_x), f64::from(from_y)),
+ (f64::from(to_x), f64::from(to_y)),
+ duration_ms,
+ steps,
+ );
+ let (native_drag, ()) = tokio::join!(native_drag, visual_drag);
+ return match native_drag {
Ok(Ok(())) => ToolResult::text(format!(
"Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."
))
@@ -6316,7 +6352,6 @@ impl Tool for DragTool {
let pid = raw_pid as u32;
let delivery = DeliveryMode::from_args(&args);
- let cursor_key = resolve_cursor_key(&args);
// Accepts numeric JSON as either float or integer — coerce both to f64.
let coerce = |key: &str| -> Option<f64> {
args.opt_f64(key)
@@ -6435,24 +6470,21 @@ impl Tool for DragTool {
steps.max(8),
&btn,
)
- })
- .await;
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (inj, ()) = tokio::join!(inj, visual_drag);
return match inj {
- Ok(Ok(())) => {
- overlay_glide_to(&cursor_key, sx_to as f64, sy_to as f64).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx_to as f64,
- y: sy_to as f64,
- },
- );
- ToolResult::text(format!(
- "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \
+ Ok(Ok(())) => ToolResult::text(format!(
+ "✅ Sent drag via synthetic-pen injection on pid {raw_pid} \
from screen ({sx_from},{sy_from}) → ({sx_to},{sy_to}) \
(delivery_mode:background, PostMessage would have been dropped)."
- ))
- }
+ )),
Ok(Err(e)) => crate::input::delivery::background_unavailable_error_with_cause(
hwnd,
EventKind::MouseMove,
@@ -6469,6 +6501,7 @@ impl Tool for DragTool {
// foreground-lock constraint as send_click_synthesized.
if delivery == DeliveryMode::Foreground {
let btn_fg = button.clone();
+ pin_overlay_above(&cursor_key, hwnd);
let prev_fg_addr = unsafe {
windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
};
@@ -6483,8 +6516,15 @@ impl Tool for DragTool {
steps,
&btn_fg,
)
- })
- .await;
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (send_result, ()) = tokio::join!(send_result, visual_drag);
tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
let button_suffix = if button == "left" {
String::new()
@@ -6535,13 +6575,9 @@ impl Tool for DragTool {
// sufficient — the 80 ms z-order tick keeps it asserted thereafter.
pin_overlay_above(&cursor_key, hwnd);
- // Animate the agent cursor to the drag-start, fire a press pulse,
- // run the actual drag synthesis, then glide to the drag-end and
- // fire a release pulse. Mirrors macOS `tools/drag.rs:191-242`.
- // Cursor doesn't follow the interpolated PostMessage path during
- // the drag itself (the timing coordination would be invasive);
- // pre- and post-glides plus the press/release pulses are enough
- // signal for a user watching the agent operate.
+ // Move the agent cursor to the drag-start, show the press state, and
+ // track the same interpolated step count and duration while the
+ // posted-message drag runs.
overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await;
crate::overlay::send_command(
cursor_key.clone(),
@@ -6566,22 +6602,15 @@ impl Tool for DragTool {
steps,
&button_c,
)
- })
- .await;
-
- // Drag finished — glide the visual cursor to the drag-end and
- // pulse the release. Skipped on error so the cursor doesn't lie
- // about a successful endpoint.
- if matches!(&result, Ok(Ok(()))) {
- overlay_glide_to(&cursor_key, sx_to as f64, sy_to as f64).await;
- crate::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::ClickPulse {
- x: sx_to as f64,
- y: sy_to as f64,
- },
- );
- }
+ });
+ let visual_drag = track_overlay_drag(
+ cursor_key.clone(),
+ (f64::from(sx_from), f64::from(sy_from)),
+ (f64::from(sx_to), f64::from(sy_to)),
+ duration_ms,
+ steps,
+ );
+ let (result, ()) = tokio::join!(result, visual_drag);
match result {
Ok(Ok(())) => {