Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): background drag on macOS (CUA-1207) (#5309)

Francesco Bonacci1dc9b52bd090
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index c4313ff592..01fc949d9f 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -41,14 +41,14 @@ Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbin
When `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.
-macOS has no background drag: a window-scoped drag needs delivery_mode:"foreground" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:"foreground" the call is refused with background_unavailable and nothing is sent.
+A window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:"drag" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:"foreground", which briefly fronts the window and moves the physical pointer.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="drag--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button used for the drag. Default: left. |
-| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Window-scoped drag on macOS supports only "foreground": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default "background" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field. |
+| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Window-scoped drag on macOS: "background" (default) posts the press, drag and release to the exact window (window_id required) without moving the pointer, raising the window or taking the user's focus. "foreground" briefly fronts the window, drives the physical pointer through the gesture, then restores the prior frontmost app; use it only when a background drag had no effect. Desktop scope ignores this field. |
| <span id="drag--macos--duration_ms"></span>`duration_ms` | `integer` | | Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500. Range: 0 to 10000. |
| <span id="drag--macos--from_x"></span>`from_x` | `number` | required | Drag-start X in window-local screenshot pixels. Top-left origin. |
| <span id="drag--macos--from_y"></span>`from_y` | `number` | required | Drag-start Y in window-local screenshot pixels. Top-left origin. |
diff --git a/libs/cua-driver/docs/action-support.md b/libs/cua-driver/docs/action-support.md
index f66f3b00e5..e239670667 100644
--- a/libs/cua-driver/docs/action-support.md
+++ b/libs/cua-driver/docs/action-support.md
@@ -63,9 +63,9 @@ integrity-level harness can exercise it.
| Harness | Proven contracts | Refusals and gaps |
| --- | --- | --- |
-| AppKit | AX tree/capture; AX background left click, set value, and type text; PX background left/right/double click; PX foreground right/double click and slider drag; AX foreground/background scroll; desktop PX foreground left click | PX background slider drag returns exact `background_unavailable`. Native press key, hotkey, AX-addressed right/double click, and broader control combinations remain unproven. |
+| AppKit | AX tree/capture; AX background left click, set value, and type text; PX background left/right/double click and slider drag; PX foreground right/double click and slider drag; AX foreground/background scroll; desktop PX foreground left click | A background drag that starts an AppKit drag-and-drop session (table reordering) returns `background_unavailable` with `reason: drag_and_drop_session`. Native press key, hotkey, AX-addressed right/double click, and broader control combinations remain unproven. |
| SwiftUI | AX tree/capture; AX background left click and set value; foreground popover-trigger activation | The fixture proves `popover_open=true`, but the transient panel remains absent from targeted AX enumeration. Other native pointer and keyboard combinations remain unproven. |
-| WKWebView | Full 40-cell shared-web catalog through the dedicated repo-local native host | PX background drag returns exact `background_unavailable`; the other 39 shared cells deliver. Native host-specific controls are outside this fixture. |
+| WKWebView | Full 40-cell shared-web catalog through the dedicated repo-local native host | PX background drag is delivered to the window without moving the pointer (CUA-1207); not yet re-run on this host. The other 39 shared cells deliver. Native host-specific controls are outside this fixture. |
## Native Linux
diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.md
index e16f850a72..a6cf57db16 100644
--- a/libs/cua-driver/docs/test-matrix.md
+++ b/libs/cua-driver/docs/test-matrix.md
@@ -190,7 +190,7 @@ background delivery and attach desktop-side-effect oracles.
| Harness | Source test | Coverage |
| --- | --- | --- |
-| AppKit | `harness_appkit_test.rs` | AX tree/capture, AX value/text, AX scroll, PX clicks, and foreground slider drag across the proven delivery modes; background drag is an exact refusal |
+| AppKit | `harness_appkit_test.rs` | AX tree/capture, AX value/text, AX scroll, PX clicks, and slider drag in both background and foreground |
| SwiftUI | `harness_swiftui_test.rs` | AX tree/capture, background click/value, and foreground popover-trigger state |
| WKWebView | `cross_platform_behavior_test.rs` | Dedicated native host running the full 40-cell shared web catalog |
| Desktop target | `desktop_scope_macos_test.rs` | Per-call window/desktop modality, screen-absolute action delivery, and strict rejection |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index bb3f39dea1..b4241b9975 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -122,7 +122,7 @@ is safe even for apps that normally foreground on media-load
| Move or resize one exact window | `set_window_frame({pid, window_id, x, y, width, height})` | `osascript` position/size writes or title-bar dragging |
| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |
| Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls |
-| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |
+| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y})` in the background (pixel-only: macOS AX has no semantic drag). Verify with a screenshot; `delivery_mode:"foreground"` only if nothing changed. A bound browser tab: `browser_pointer` `action:"drag"` | `cliclick dd:`, `osascript drag`, refusing a drag-only step as "needs the window in front" |
| Hover (reveal a hover-only control) | Window-targeted `move_cursor`; its result says whether it hovered (AX alternate UI or an Electron page). A web page in Chrome, Edge or Brave: `browser_pointer` hover after `get_browser_state`. If not, look for the same command elsewhere (menu item, context menu, shortcut, help text in `get_window_state`); only `move_cursor({scope:"desktop", delivery_mode:"foreground"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovered when its result says the app sees no hover; a desktop `move_cursor` over a background browser page |
| Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` |
| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` |
@@ -383,12 +383,31 @@ surface that only accepts events while frontmost (the canvas/viewport/game
case below). Unmodified `element_token` (AX) actions remain background-capable
and hold the no-foreground contract without the flag.
-`drag` is the exception: macOS has no background drag. A window-scoped
-`drag` needs `delivery_mode:"foreground"` and `window_id`; without them it
-refuses with `background_unavailable` and sends nothing. The foreground drag
-fronts the exact window, moves the physical pointer along the path, and then
-restores the prior frontmost app. Use it only when a drag is the task, and
-expect the hardware pointer to move.
+`drag` runs in the background too. A window-scoped `drag` (`pid` and
+`window_id`) makes the window key without raising it, posts the press, the
+interpolated drags and the release to that window's process, and hands focus
+back to the user's window. The real pointer does not move. It reaches
+controls that follow the drag events: native sliders and canvases, and
+pointer-event sliders, canvases and boards on web pages in Chrome. The
+result is not driver-verified: check the outcome with a screenshot or the
+tree. Only if nothing changed, retry with `delivery_mode:"foreground"`,
+which fronts the window, moves the physical pointer along the path, and
+restores the prior frontmost app.
+
+A native `<input type="range">` in Chrome does not follow a background drag
+(pointer-event widgets do). Use `set_value` on its slider element, or
+`browser_pointer` `action:"drag"` on a bound tab, which sends the drag
+through CDP `Input.dispatchMouseEvent` with the button held.
+
+A drag that starts a macOS drag-and-drop session (reordering a list by drag
+and drop, moving a file, HTML5 `draggable` content) cannot drop in the
+background: macOS tracks those sessions with the real pointer. The driver
+sees the session start, releases at once, and returns
+`background_unavailable` with `reason: "drag_and_drop_session"`. Look for a
+non-drag route first (Move Up/Down, a menu or context-menu command, a
+shortcut, cut and paste), then foreground if the user allows it. Toolkits that read the hardware pointer (Tk) or drop routed
+mouse events (LibreOffice) refuse the background drag with
+`background_unavailable` before anything is sent.
A foreground window-scoped **pixel** `click`, `double_click`, or
`right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driver
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
index fcf5071599..9877e57314 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
@@ -1656,7 +1656,8 @@ fn shared_case_with_native_hyprland(
}
// Electron scroll is delivered in the background through
// accessibility (reveal scroll), including inner scrollers.
- (_, "drag", Targeting::Px) => vec![RefusalCode::BackgroundUnavailable],
+ // The shared drag cell is a pointer-event drag, which the
+ // background window-routed drag delivers (CUA-1207).
_ => Vec::new(),
}
} else if cfg!(target_os = "linux")
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
index 418276cb96..d62d0d816c 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/harness_appkit_test.rs
@@ -2173,12 +2173,14 @@ fn harness_appkit_slider_drag_px_background() {
"slider_drag",
Targeting::Px,
DriverRoute::MacosCgEventPid,
- )
- .expecting_refusal(vec![RefusalCode::BackgroundUnavailable]);
+ );
run_case(case, |pid, wid, driver| {
let pre = snapshot_elements(driver, pid, wid);
assert!(pre.tree_text().contains("slider_value=0"));
let (x, y, width, height) = element_pixel_frame(&pre, "sld-value");
+ // The background oracles fail the case if the drag moves the user's
+ // pointer, raises or activates the fixture, or takes key focus from
+ // the foreground sentinel.
let (response, mut passed) = run_with_background_oracles(
driver,
TargetWindow {
@@ -2204,30 +2206,20 @@ fn harness_appkit_slider_drag_px_background() {
)
.unwrap_or_else(|error| panic!("background desktop contract failed: {error}"));
assert!(
- response.is_error(),
- "AppKit background drag unexpectedly reported delivery: {}",
- response.text()
- );
- assert_eq!(
- response.structured()["code"].as_str(),
- Some("background_unavailable"),
- "AppKit background drag returned the wrong refusal: {}",
+ !response.is_error(),
+ "AppKit background drag failed: {}",
response.text()
);
- std::thread::sleep(Duration::from_millis(200));
+ assert_eq!(response.structured()["path"].as_str(), Some("cgevent"));
+ std::thread::sleep(Duration::from_millis(300));
assert!(
- snapshot_elements(driver, pid, wid)
+ !snapshot_elements(driver, pid, wid)
.tree_text()
.contains("slider_value=0"),
- "refused AppKit background drag changed the slider"
+ "AppKit background drag did not move the slider"
);
passed.push(OracleKind::FixtureState);
- Observation::refused(
- RefusalCode::BackgroundUnavailable,
- passed,
- response.text(),
- Evidence::default(),
- )
+ Observation::delivered_with_fixture_state(passed)
});
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
index f588c65749..1b34151cb8 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
@@ -807,13 +807,11 @@ pub fn drag_at_xy_observed<F>(
modifiers: &[&str],
button: DragButton,
foreground_hid: bool,
- mut observe: F,
+ observe: F,
) -> anyhow::Result<()>
where
F: FnMut(f64, f64),
{
- use std::time::{SystemTime, UNIX_EPOCH};
-
if foreground_hid {
return drag_at_xy_window_foreground_observed(
pid,
@@ -832,117 +830,231 @@ where
);
}
+ let wid = wid.ok_or_else(|| {
+ anyhow::anyhow!("a background drag needs the exact target window (window_id)")
+ })?;
+ drag_at_xy_background(
+ pid,
+ (from_x, from_y),
+ (to_x, to_y),
+ wid,
+ duration_ms,
+ steps,
+ modifiers,
+ button,
+ observe,
+ || true,
+ )
+ .map(|_| ())
+}
+
+/// How a background drag gesture ended.
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub enum BackgroundDragOutcome {
+ /// Every planned event was posted and the release landed on the drop point.
+ Completed,
+ /// The caller's `keep_going` check failed during the path; the release
+ /// was posted at once at this screen point instead of the drop point.
+ StoppedEarly { at: (f64, f64) },
+}
+
+/// Event kinds in the background drag plan.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum BackgroundDragEventKind {
+ /// Stamped `mouseMoved` at the start point (cursor-tracking primer).
+ Move,
+ /// Off-screen left down/up at (-1, -1): the same Chromium user-activation
+ /// primer the background click recipe sends before its target pair.
+ PrimerDown,
+ PrimerUp,
+ Down,
+ Dragged,
+ Up,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq)]
+struct BackgroundDragStep {
+ kind: BackgroundDragEventKind,
+ /// Screen point. It is both the event location and the stamped window
+ /// location, as in the background click recipe.
+ point: (f64, f64),
+ /// Field 0 gesture phase: move 2, primer down 1, primer up 2, gesture 3.
+ phase: i64,
+ click_state: i64,
+ delay_after_ms: u64,
+}
+
+/// The complete background drag gesture, as data.
+///
+/// One stamped move primes the target's cursor-tracking state, an off-screen
+/// primer click satisfies Chromium's user-activation gate, and then one
+/// down, `steps` interpolated drags and one up make the gesture. Every event
+/// is posted exactly once.
+fn background_drag_plan(
+ from: (f64, f64),
+ to: (f64, f64),
+ steps: usize,
+ duration_ms: u64,
+) -> Vec<BackgroundDragStep> {
+ use BackgroundDragEventKind::*;
+ let steps = steps.max(1);
+ let step_delay_ms = duration_ms / steps as u64;
+ let off_screen = (-1.0, -1.0);
+ let step = |kind, point, phase, click_state, delay_after_ms| BackgroundDragStep {
+ kind,
+ point,
+ phase,
+ click_state,
+ delay_after_ms,
+ };
+ let mut plan = Vec::with_capacity(steps + 5);
+ plan.push(step(Move, from, 2, 0, 15));
+ plan.push(step(PrimerDown, off_screen, 1, 1, 1));
+ plan.push(step(PrimerUp, off_screen, 2, 1, 100));
+ // A short hold before the first drag lets drag sources (NSTableView,
+ // Chromium pointer capture) arm their tracking.
+ plan.push(step(Down, from, 3, 1, 30));
+ for i in 1..=steps {
+ let t = i as f64 / steps as f64;
+ let point = (from.0 + (to.0 - from.0) * t, from.1 + (to.1 - from.1) * t);
+ // The last drag lands on the drop point; give the target one
+ // run-loop turn to process it before the release.
+ let delay = if i == steps {
+ step_delay_ms.max(50)
+ } else {
+ step_delay_ms
+ };
+ plan.push(step(Dragged, point, 3, 1, delay));
+ }
+ plan.push(step(Up, to, 3, 1, 100));
+ plan
+}
+
+/// Background press-drag-release addressed to one window of `pid`.
+///
+/// This is the background click route stretched over a gesture: every event
+/// is posted to the process (SkyLight `SLEventPostToPid`, or the public
+/// per-pid API only when the SPI is absent), carries the exact window in the
+/// routing fields (f51/f91/f92), the target pid (f40) and one gesture ID
+/// (f58), and is stamped with its own location. The hardware pointer is never
+/// warped and nothing is posted to the global HID tap, so the user's pointer,
+/// frontmost app and window order stay as they were. The caller makes the
+/// window AppKit-key without raising it first
+/// ([`prepare_background_pixel_click`]).
+///
+/// `keep_going` runs after each drag step; when it returns false the rest of
+/// the path is skipped and the release is posted at once where the gesture
+/// stands. The drag tool uses it to cut short a gesture that started a system
+/// drag-and-drop session, which macOS tracks with the hardware pointer.
+#[allow(clippy::too_many_arguments)]
+pub fn drag_at_xy_background<F, G>(
+ pid: i32,
+ from: (f64, f64),
+ to: (f64, f64),
+ wid: u32,
+ duration_ms: u64,
+ steps: usize,
+ modifiers: &[&str],
+ button: DragButton,
+ mut observe: F,
+ mut keep_going: G,
+) -> anyhow::Result<BackgroundDragOutcome>
+where
+ F: FnMut(f64, f64),
+ G: FnMut() -> bool,
+{
+ use std::time::{SystemTime, UNIX_EPOCH};
+ use BackgroundDragEventKind::*;
+
let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
let flags = parse_modifier_flags(modifiers);
-
- let (cg_button, down_type, dragged_type, up_type) = match button {
+ let (cg_button, down_type, dragged_type, up_type, button_number) = match button {
DragButton::Left => (
CGMouseButton::Left,
CGEventType::LeftMouseDown,
CGEventType::LeftMouseDragged,
CGEventType::LeftMouseUp,
+ 0,
),
DragButton::Right => (
CGMouseButton::Right,
CGEventType::RightMouseDown,
CGEventType::RightMouseDragged,
CGEventType::RightMouseUp,
+ 1,
),
DragButton::Middle => (
CGMouseButton::Center,
CGEventType::OtherMouseDown,
CGEventType::OtherMouseDragged,
CGEventType::OtherMouseUp,
+ 2,
),
};
- // f3 button number must match the dragged button (0=left, 1=right, 2=middle).
- let button_number: i64 = match button {
- DragButton::Left => 0,
- DragButton::Right => 1,
- DragButton::Middle => 2,
- };
-
- let click_group_id: Option<i64> = wid.map(|_| {
- SystemTime::now()
- .duration_since(UNIX_EPOCH)
- .unwrap_or_default()
- .subsec_nanos() as i64
- });
-
- let steps = steps.max(1);
- let step_delay_ms = if steps > 1 {
- duration_ms / steps as u64
- } else {
- duration_ms
- };
-
- // Prime the routed target's cursor-tracking state immediately before the
- // press, matching the click path's stamped mouseMoved + settle interval.
- let from_pt = CGPoint::new(from_x, from_y);
- post_mouse_moved_primer(
- pid,
- &source,
- from_pt,
- from_local,
- wid,
- click_group_id,
- MousePostMode::Both,
- );
- std::thread::sleep(std::time::Duration::from_millis(12));
-
- // MouseDown at start.
- let down = CGEvent::new_mouse_event(source.clone(), down_type, from_pt, cg_button)
- .map_err(|_| anyhow::anyhow!("drag mouseDown failed"))?;
- set_explicit_flags(&down, flags);
- post_mouse_event(
- pid,
- &down,
- from_local,
- wid,
- click_group_id,
- 1,
- button_number,
- 0,
- );
- observe(from_x, from_y);
- std::thread::sleep(std::time::Duration::from_millis(16));
+ let gesture_id = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .unwrap_or_default()
+ .subsec_nanos() as i64;
+ let route = post_route(MousePostMode::SkyLightPreferred, skylight_available());
+ let window_id = wid as i64;
- // Interpolated drag steps.
- for i in 1..=steps {
- let t = i as f64 / steps as f64;
- let ix = from_x + (to_x - from_x) * t;
- let iy = from_y + (to_y - from_y) * t;
- let il = from_local
- .zip(to_local)
- .map(|((fx, fy), (tx, ty))| (fx + (tx - fx) * t, fy + (ty - fy) * t));
- let drag_pt = CGPoint::new(ix, iy);
- let drag = CGEvent::new_mouse_event(source.clone(), dragged_type, drag_pt, cg_button)
- .map_err(|_| anyhow::anyhow!("drag mouseDragged failed"))?;
- set_explicit_flags(&drag, flags);
- post_mouse_event(pid, &drag, il, wid, click_group_id, 1, button_number, 0);
- observe(ix, iy);
- if step_delay_ms > 0 {
- std::thread::sleep(std::time::Duration::from_millis(step_delay_ms));
+ let mut stopped_at: Option<(f64, f64)> = None;
+ let mut last_point = from;
+ for mut step in background_drag_plan(from, to, steps, duration_ms) {
+ if let Some(at) = stopped_at {
+ match step.kind {
+ Dragged => continue,
+ Up => step.point = at,
+ _ => {}
+ }
+ }
+ // (type, button, button number, subtype). Primers keep the click
+ // recipe's left button and touch subtype; the gesture itself is a
+ // plain mouse drag (subtype 0), like the foreground drag.
+ let (event_type, event_button, event_button_number, subtype) = match step.kind {
+ Move => (CGEventType::MouseMoved, CGMouseButton::Left, 0, 3),
+ PrimerDown => (CGEventType::LeftMouseDown, CGMouseButton::Left, 0, 3),
+ PrimerUp => (CGEventType::LeftMouseUp, CGMouseButton::Left, 0, 3),
+ Down => (down_type, cg_button, button_number, 0),
+ Dragged => (dragged_type, cg_button, button_number, 0),
+ Up => (up_type, cg_button, button_number, 0),
+ };
+ let point = CGPoint::new(step.point.0, step.point.1);
+ let event = CGEvent::new_mouse_event(source.clone(), event_type, point, event_button)
+ .map_err(|_| {
+ anyhow::anyhow!("background drag {:?} event creation failed", step.kind)
+ })?;
+ let ptr = event.as_ptr() as *mut std::ffi::c_void;
+ let set =
+ |field: u32, value: i64| crate::input::skylight::set_integer_field(ptr, field, value);
+ set(0, step.phase);
+ set(1, step.click_state);
+ set(3, event_button_number);
+ set(7, subtype);
+ set(40, pid as i64);
+ set(51, window_id);
+ set(91, window_id);
+ set(92, window_id);
+ set(58, gesture_id);
+ crate::input::skylight::set_window_location(ptr, step.point.0, step.point.1);
+ set_explicit_flags(&event, flags);
+ dispatch_mouse_event(route, pid, &event);
+ if matches!(step.kind, Down | Dragged) {
+ observe(step.point.0, step.point.1);
+ last_point = step.point;
+ }
+ if step.delay_after_ms > 0 {
+ std::thread::sleep(std::time::Duration::from_millis(step.delay_after_ms));
+ }
+ if step.kind == Dragged && stopped_at.is_none() && !keep_going() {
+ stopped_at = Some(last_point);
}
}
-
- // MouseUp at end.
- // Give Chromium one run-loop turn to process the final dragged event at
- // the drop point before releasing pointer capture.
- std::thread::sleep(std::time::Duration::from_millis(50));
- let to_pt = CGPoint::new(to_x, to_y);
- let up = CGEvent::new_mouse_event(source.clone(), up_type, to_pt, cg_button)
- .map_err(|_| anyhow::anyhow!("drag mouseUp failed"))?;
- set_explicit_flags(&up, flags);
- post_mouse_event(pid, &up, to_local, wid, click_group_id, 1, button_number, 0);
- // Chromium may process the final pointerup on the next run-loop turn. In
- // the foreground rung the caller restores the previous app immediately
- // after this function returns, so let the target consume the release and
- // complete pointer capture before that restore.
- std::thread::sleep(std::time::Duration::from_millis(100));
-
- Ok(())
+ Ok(match stopped_at {
+ Some(at) => BackgroundDragOutcome::StoppedEarly { at },
+ None => BackgroundDragOutcome::Completed,
+ })
}
#[allow(clippy::too_many_arguments)]
@@ -2221,6 +2333,203 @@ mod tests {
CGEventFlags::CGEventFlagCommand.bits() | CGEventFlags::CGEventFlagShift.bits();
assert_eq!(&modifiers[1..], [expected, expected]);
}
+ #[test]
+ fn background_drag_plan_is_one_gesture_from_start_to_drop() {
+ use BackgroundDragEventKind::*;
+ let plan = background_drag_plan((100.0, 200.0), (300.0, 100.0), 4, 400);
+ let kinds: Vec<_> = plan.iter().map(|step| step.kind).collect();
+ assert_eq!(
+ kinds,
+ [Move, PrimerDown, PrimerUp, Down, Dragged, Dragged, Dragged, Dragged, Up]
+ );
+ assert_eq!(plan[0].point, (100.0, 200.0));
+ assert_eq!(
+ plan[1].point,
+ (-1.0, -1.0),
+ "the primer click stays off-screen"
+ );
+ assert_eq!(plan[3].point, (100.0, 200.0));
+ assert_eq!(plan[4].point, (150.0, 175.0));
+ assert_eq!(
+ plan[7].point,
+ (300.0, 100.0),
+ "the last drag lands on the drop point"
+ );
+ assert_eq!(plan[8].point, (300.0, 100.0));
+ for step in &plan[3..] {
+ assert_eq!(step.phase, 3, "down, drags and up share one gesture phase");
+ assert_eq!(step.click_state, 1);
+ }
+ assert_eq!(plan[4].delay_after_ms, 100);
+ assert!(
+ plan[7].delay_after_ms >= 50,
+ "the drop point settles before release"
+ );
+ }
+
+ #[test]
+ fn background_drag_plan_keeps_one_drag_for_zero_steps() {
+ let plan = background_drag_plan((0.0, 0.0), (10.0, 0.0), 0, 0);
+ let drags: Vec<_> = plan
+ .iter()
+ .filter(|step| step.kind == BackgroundDragEventKind::Dragged)
+ .collect();
+ assert_eq!(drags.len(), 1);
+ assert_eq!(drags[0].point, (10.0, 0.0));
+ }
+
+ #[test]
+ fn background_drag_posts_each_event_once_to_the_pid_with_window_routing() {
+ let capture = Capture::start(true);
+ let mut observed = Vec::new();
+ drag_at_xy_background(
+ 42,
+ (100.0, 100.0),
+ (200.0, 100.0),
+ 7,
+ 0,
+ 2,
+ &["shift"],
+ DragButton::Left,
+ |x, y| observed.push((x, y)),
+ || true,
+ )
+ .unwrap();
+ let posted = capture.posted();
+ let events: Vec<_> = posted.iter().map(|posted| posted.event.as_str()).collect();
+ assert_eq!(
+ events,
+ [
+ "MouseMoved",
+ "LeftMouseDown",
+ "LeftMouseUp",
+ "LeftMouseDown",
+ "LeftMouseDragged",
+ "LeftMouseDragged",
+ "LeftMouseUp"
+ ]
+ );
+ assert!(
+ posted
+ .iter()
+ .all(|posted| posted.route == PostRoute::SkyLight),
+ "a background drag never reaches the HID tap or a second transport"
+ );
+ let routing = capture.window_routing();
+ let gesture = routing[0][3];
+ for fields in routing {
+ assert_eq!(
+ &fields[..3],
+ &[7, 7, 7],
+ "every event names the exact window"
+ );
+ assert_eq!(fields[3], gesture, "every event carries one gesture ID");
+ }
+ assert!(capture
+ .modifiers()
+ .iter()
+ .all(|flags| *flags == 0x0002_0000));
+ assert_eq!(observed, [(100.0, 100.0), (150.0, 100.0), (200.0, 100.0)]);
+ }
+
+ #[test]
+ fn background_right_drag_uses_right_button_events_and_public_fallback() {
+ let capture = Capture::start(false);
+ drag_at_xy_background(
+ 42,
+ (0.0, 0.0),
+ (10.0, 10.0),
+ 7,
+ 0,
+ 1,
+ &[],
+ DragButton::Right,
+ |_, _| {},
+ || true,
+ )
+ .unwrap();
+ let posted = capture.posted();
+ assert!(posted
+ .iter()
+ .all(|posted| posted.route == PostRoute::PublicPid));
+ let gesture: Vec<_> = posted[3..]
+ .iter()
+ .map(|posted| (posted.event.as_str(), posted.button))
+ .collect();
+ assert_eq!(
+ gesture,
+ [
+ ("RightMouseDown", 1),
+ ("RightMouseDragged", 1),
+ ("RightMouseUp", 1)
+ ]
+ );
+ }
+
+ #[test]
+ fn background_drag_stopped_early_releases_where_it_stands() {
+ let capture = Capture::start(true);
+ let mut checks = 0;
+ let outcome = drag_at_xy_background(
+ 42,
+ (0.0, 0.0),
+ (100.0, 0.0),
+ 7,
+ 0,
+ 4,
+ &[],
+ DragButton::Left,
+ |_, _| {},
+ || {
+ checks += 1;
+ checks < 2
+ },
+ )
+ .unwrap();
+ assert_eq!(
+ outcome,
+ BackgroundDragOutcome::StoppedEarly { at: (50.0, 0.0) }
+ );
+ let events: Vec<_> = capture
+ .posted()
+ .iter()
+ .map(|posted| posted.event.clone())
+ .collect();
+ assert_eq!(
+ events[3..],
+ [
+ "LeftMouseDown",
+ "LeftMouseDragged",
+ "LeftMouseDragged",
+ "LeftMouseUp"
+ ],
+ "no drag after the stop, and the release still goes out"
+ );
+ }
+
+ #[test]
+ fn window_less_background_drag_is_refused_before_posting() {
+ let capture = Capture::start(true);
+ let error = drag_at_xy_observed(
+ 42,
+ 0.0,
+ 0.0,
+ 10.0,
+ 10.0,
+ None,
+ None,
+ None,
+ 0,
+ 1,
+ &[],
+ DragButton::Left,
+ false,
+ |_, _| {},
+ )
+ .unwrap_err();
+ assert!(error.to_string().contains("window_id"));
+ assert!(capture.posted().is_empty());
+ }
}
#[cfg(test)]
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index dc8c4745cb..8dceb7c5f7 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -222,6 +222,59 @@ fn background_pixel_restore_pid(
}
}
+/// Hand the user's frontmost app and key window back after a raw background
+/// pixel action that ran the focus-without-raise prologue (click, drag).
+///
+/// The no-raise record can make NSWorkspace report the target as active even
+/// though its window never moved in z-order. Once the input has been queued,
+/// restore the prior app if the target is still reported frontmost. Base this
+/// on observed state, not `focus_without_raise`: the private recipe can report
+/// failure after partially activating the target, and the raw input can
+/// self-activate even when that recipe is unavailable. Do not overwrite a
+/// different app here; the wildcard suppression lease handles genuine side
+/// effects.
+pub(super) async fn restore_user_focus_after_background_pixel(
+ pid: i32,
+ prior_front: Option<i32>,
+ prior_key_wid: Option<u32>,
+ window_id: Option<u32>,
+ focus_without_raise: bool,
+) {
+ if prior_front == Some(pid) {
+ return;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+ let reactivated = if let Some(previous_pid) = background_pixel_restore_pid(
+ PixelActivationPolicy::AllowTargetWithoutRaise,
+ prior_front,
+ pid,
+ apps::frontmost_pid(),
+ ) {
+ let _ = apps::activate_pid(previous_pid);
+ wait_until_frontmost(previous_pid, std::time::Duration::from_millis(300)).await;
+ true
+ } else {
+ false
+ };
+ // The prologue leaves the user's window key, but the input can still make
+ // the target self-activate, and re-activating the prior app does not by
+ // itself make its window key again. Post the focus record to the user's
+ // window in both cases so their typing keeps landing there.
+ if let (Some(previous_pid), Some(wid)) = (prior_front, window_id) {
+ if (focus_without_raise || reactivated) && apps::frontmost_pid() == Some(previous_pid) {
+ let _ = tokio::task::spawn_blocking(move || {
+ crate::input::skylight::restore_focus_after_without_raise(
+ previous_pid,
+ prior_key_wid,
+ pid,
+ wid,
+ )
+ })
+ .await;
+ }
+ }
+}
+
/// Poll until `pid` is the frontmost application or `budget` runs out.
async fn wait_until_frontmost(pid: i32, budget: std::time::Duration) -> bool {
let deadline = std::time::Instant::now() + budget;
@@ -1322,51 +1375,15 @@ impl Tool for ClickTool {
)
.await;
- // The no-raise record can make NSWorkspace report the target as
- // active even though its window never moved in z-order. Once the
- // click has been queued, restore the prior app if the target is
- // still reported frontmost. Base this on observed state, not
- // `focus_without_raise`: the private recipe can report failure
- // after partially activating the target, and the raw click can
- // self-activate even when that recipe is unavailable. Do not
- // overwrite a different app here; the wildcard suppression lease
- // handles genuine side effects.
- if activation_policy == PixelActivationPolicy::AllowTargetWithoutRaise
- && prior_front != Some(pid)
- {
- tokio::time::sleep(std::time::Duration::from_millis(50)).await;
- let reactivated = if let Some(previous_pid) = background_pixel_restore_pid(
- activation_policy,
- prior_front,
+ if activation_policy == PixelActivationPolicy::AllowTargetWithoutRaise {
+ restore_user_focus_after_background_pixel(
pid,
- apps::frontmost_pid(),
- ) {
- let _ = apps::activate_pid(previous_pid);
- wait_until_frontmost(previous_pid, std::time::Duration::from_millis(300)).await;
- true
- } else {
- false
- };
- // The prologue leaves the user's window key, but the click can
- // still make the target self-activate, and re-activating the
- // prior app does not by itself make its window key again.
- // Post the focus record to the user's window in both cases so
- // their typing keeps landing there.
- if let (Some(previous_pid), Some(wid)) = (prior_front, window_id) {
- if (focus_without_raise || reactivated)
- && apps::frontmost_pid() == Some(previous_pid)
- {
- let _ = tokio::task::spawn_blocking(move || {
- crate::input::skylight::restore_focus_after_without_raise(
- previous_pid,
- prior_key_wid,
- pid,
- wid,
- )
- })
- .await;
- }
- }
+ prior_front,
+ prior_key_wid,
+ window_id,
+ focus_without_raise,
+ )
+ .await;
}
let changes = super::finish_window_observation(snapshot).await;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index d08518eb06..6c8157b9ba 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -35,33 +35,12 @@ impl DragTool {
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
-const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS supports only \"foreground\": \
- briefly front the exact window (window_id required), drive the physical pointer through the \
- gesture, then restore the prior frontmost app. The default \"background\" is refused with \
- background_unavailable because macOS has no background drag route. Desktop scope ignores this field.";
-
-const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: \
- WindowServer tracks a drag (including drag-and-drop) with the real pointer over the \
- frontmost window, so it cannot run without fronting the target. Nothing was sent. First \
- look for a route that is not a drag: a Move Up/Down button or menu item, a context-menu \
- command, a keyboard shortcut on the selected item, or set_value on a slider. If the drag is \
- the only route and the user allows the window to come forward, retry with \
- delivery_mode:\"foreground\" and window_id: the driver briefly fronts that window, moves \
- the pointer along the path and restores the prior frontmost app. Otherwise stop and report \
- that the step needs the window in front.";
-
-fn background_drag_refusal() -> ToolResult {
- cua_driver_core::delivery::background_unavailable_result(
- BACKGROUND_DRAG_UNAVAILABLE,
- "background_unavailable",
- "macOS has no background drag route; drag moves the physical pointer",
- serde_json::json!({
- "effect": "refused",
- "event_kind": "mouse_drag",
- "requires": ["delivery_mode:foreground", "window_id"],
- }),
- )
-}
+const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS: \"background\" (default) \
+ posts the press, drag and release to the exact window (window_id required) without moving \
+ the pointer, raising the window or taking the user's focus. \"foreground\" briefly fronts \
+ the window, drives the physical pointer through the gesture, then restores the prior \
+ frontmost app; use it only when a background drag had no effect. Desktop scope ignores \
+ this field.";
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
@@ -79,11 +58,20 @@ fn def() -> &'static ToolDef {
`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\n\
When `from_zoom` is true, coordinates are in the last zoom image for this \
pid; the driver maps them back to window coordinates before dispatching.\n\n\
- macOS has no background drag: a window-scoped drag needs \
- delivery_mode:\"foreground\" and window_id. It briefly fronts the exact \
- window, moves the physical pointer along the path, then restores the prior \
- frontmost app. Without delivery_mode:\"foreground\" the call is refused with \
- background_unavailable and nothing is sent."
+ A window-scoped drag (pid + window_id) runs in the background by default: \
+ the gesture is posted to that window, the real pointer does not move, and the \
+ user's frontmost window keeps focus. It works for controls that follow the \
+ drag events: sliders, canvases, and pointer-event widgets on web pages in \
+ Chrome. A drag that starts a macOS drag-and-drop session (reordering a list \
+ by drag and drop, moving a file, HTML5 draggable content) cannot drop in the \
+ background, because macOS tracks those with the real pointer; the driver \
+ detects it, releases at once and returns background_unavailable with reason \
+ drag_and_drop_session. For a page in a browser \
+ tab already bound with get_browser_state, browser_pointer action:\"drag\" \
+ drives the same gesture through CDP with page refs. The result is not \
+ driver-verified: confirm with a screenshot or the accessibility tree, and only \
+ if nothing changed retry with delivery_mode:\"foreground\", which briefly \
+ fronts the window and moves the physical pointer."
.into(),
input_schema: serde_json::json!({
"type": "object",
@@ -225,14 +213,13 @@ impl Tool for DragTool {
Ok(v) => v,
Err(e) => return e,
};
- // delivery_mode: foreground briefly fronts the window before the
- // press-drag-release gesture (the explicit last resort for surfaces
- // that drop background CGEvents), via the same skylight assist click
- // uses. Requires a window_id to have a window to front.
+ // Background (default): post the gesture to the exact window without
+ // moving the pointer or raising it, after the focus-without-raise
+ // prologue background clicks use. Foreground: briefly front the
+ // window and drive the physical pointer (the explicit last resort for
+ // surfaces that drop background events). Both need the exact window.
let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());
- if !delivery_mode.is_foreground() {
- return background_drag_refusal();
- }
+ let fg = delivery_mode.is_foreground();
// Coerce integer or float from JSON for coordinate fields.
let coerce = |key: &str| -> Option<f64> {
args.opt_f64(key)
@@ -256,18 +243,12 @@ impl Tool for DragTool {
None => return ToolResult::error("Missing required parameter: to_y"),
};
- let window_id = args.opt_u64("window_id").map(|v| v as u32);
- if delivery_mode.is_foreground() && window_id.is_none() {
- return ToolResult::error(
- "delivery_mode=foreground requires window_id for drag on macOS".to_owned(),
- )
- .with_structured(serde_json::json!({
- "code": "window_id_required",
- "effect": "refused"
- }));
- }
- let duration_ms = args.u64_or("duration_ms", 500);
- let steps = args.u64_or("steps", 20) as usize;
+ let Some(wid) = args.opt_u64("window_id").map(|v| v as u32) else {
+ return window_id_required(fg);
+ };
+ let window_id = Some(wid);
+ let duration_ms = args.u64_or("duration_ms", 500).min(10_000);
+ let steps = (args.u64_or("steps", 20) as usize).clamp(1, 200);
let from_zoom = args.bool_or("from_zoom", false);
let button_str = args.str_or("button", "left");
let modifiers: Vec<String> = args.str_array("modifier");
@@ -308,141 +289,208 @@ impl Tool for DragTool {
}
// Translate window-local screenshot pixels → screen coordinates, and
- // window-local logical coords for CGEventSetWindowLocation. Both ends of
- // the drag resolve against ONE frame, so a window that closed mid-call
- // refuses rather than dragging across the desktop behind it.
+ // window-local logical coords for the foreground HID stamp. Both ends
+ // of the drag resolve against ONE frame, so a window that closed
+ // mid-call refuses rather than dragging across the desktop behind it.
let (from_sx, from_sy, from_lx, from_ly, to_sx, to_sy, to_lx, to_ly) =
- if let Some(wid) = window_id {
- match super::px_frame::resolve_or_refuse(wid).await {
- Ok(frame) => {
- let (fsx, fsy, flx, fly) = frame.to_screen(from_x, from_y);
- let (tsx, tsy, tlx, tly) = frame.to_screen(to_x, to_y);
- (fsx, fsy, flx, fly, tsx, tsy, tlx, tly)
- }
- Err(refusal) => return refusal,
+ match super::px_frame::resolve_or_refuse(wid).await {
+ Ok(frame) => {
+ let (fsx, fsy, flx, fly) = frame.to_screen(from_x, from_y);
+ let (tsx, tsy, tlx, tly) = frame.to_screen(to_x, to_y);
+ (fsx, fsy, flx, fly, tsx, tsy, tlx, tly)
}
- } else {
- (from_x, from_y, from_x, from_y, to_x, to_y, to_x, to_y)
+ Err(refusal) => return refusal,
};
- // Animate agent cursor along drag path (start → end).
- if let Some(wid) = window_id {
- crate::cursor::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::PinAbove(wid as u64),
- );
+ // A toolkit that reads the hardware pointer (Tk) or drops routed
+ // mouse events (LibreOffice VCL) cannot take a background gesture;
+ // refuse before any activation or dispatch.
+ if !fg {
+ if let Err(refusal) =
+ super::pixel_route::resolve(pid, false, window_id, "mouse_drag").await
+ {
+ return refusal;
+ }
}
+
+ // Animate agent cursor along drag path (start → end).
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), from_sx, from_sy).await;
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Drags can trigger drag-and-drop side-effects that spawn helper
- // windows (drop on Dock, drop on background app icon) and the
- // mouseDown half-event alone can activate the target app on some
- // Chromium builds. Wrap to catch + report both.
+ // windows (drop on Dock, drop on background app icon). In the
+ // background the target itself may become AppKit-active without
+ // raising; anything else that takes focus is reported and undone.
let prior_front = apps::frontmost_pid();
- let snapshot = WindowChangeDetector::snapshot(prior_front);
+ let snapshot = if fg {
+ WindowChangeDetector::snapshot(prior_front)
+ } else {
+ WindowChangeDetector::snapshot_allowing_activation(prior_front, pid)
+ };
+
+ // Background prologue, as for a background pixel click: keep the
+ // user's key focus if the gesture opens a menu, remember their key
+ // window, and make the target window AppKit-key without raising it.
+ let _key_focus = if fg {
+ None
+ } else {
+ tokio::task::spawn_blocking(move || crate::key_focus_guard::protect(pid))
+ .await
+ .ok()
+ .flatten()
+ };
+ let prior_key_wid = match prior_front {
+ Some(prior) if !fg && prior != pid => tokio::task::spawn_blocking(move || {
+ crate::input::skylight::key_window_of_pid(prior)
+ })
+ .await
+ .ok()
+ .flatten(),
+ _ => None,
+ };
+ let focus_without_raise = if fg {
+ false
+ } else {
+ match tokio::task::spawn_blocking(move || {
+ crate::input::mouse::prepare_background_pixel_click(pid, wid)
+ })
+ .await
+ {
+ Ok(activated) => {
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
+ activated
+ }
+ Err(error) => {
+ return ToolResult::error(format!(
+ "Background drag activation task failed: {error}"
+ ))
+ }
+ }
+ };
// Dispatch blocking drag synthesis.
let mods_owned = modifiers.clone();
- let fg = delivery_mode.is_foreground();
- let foreground_window_id = window_id;
let cursor_for_drag = cursor_key.clone();
crate::cursor::overlay::send_command(
cursor_key.clone(),
cursor_overlay::OverlayCommand::SetPressed(true),
);
- let drag_input = focus_guard::with_focus_suppressed(
- // Foreground drag deliberately activates the target while the
- // HID gesture is delivered. A suppression lease here
- // would race that activation and restore the prior app before
- // Chromium receives the gesture.
- if fg { None } else { Some(pid) },
+ let result = focus_guard::with_focus_suppressed(
+ // Neither mode suppresses the target: foreground deliberately
+ // activates it for the HID gesture, and background lets the
+ // no-raise prologue make it active; the restore below hands focus
+ // back. Other apps that take focus are still suppressed.
+ None,
prior_front,
"drag.CGEvent",
|| async move {
- tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
- let do_it = move || -> anyhow::Result<()> {
- let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
- if let Some(wid) = foreground_window_id {
- // Keep the exact target window active for the
- // complete HID gesture, then restore the prior
- // front process.
- return crate::input::skylight::with_foreground_hid_activation(
- pid as libc::pid_t,
- wid,
- || {
- let observed_cursor = cursor_for_drag.clone();
- crate::input::mouse::drag_at_xy_observed(
- pid,
- from_sx,
- from_sy,
- to_sx,
- to_sy,
- Some((from_lx, from_ly)),
- Some((to_lx, to_ly)),
- Some(wid),
- duration_ms,
- steps,
- &m,
- button,
- true,
- move |x, y| {
- crate::cursor::overlay::send_command(
- observed_cursor.clone(),
- cursor_overlay::track_pointer_command(x, y),
- );
- },
- )
- },
- );
- }
- crate::input::mouse::drag_at_xy_observed(
- pid,
- from_sx,
- from_sy,
- to_sx,
- to_sy,
- Some((from_lx, from_ly)),
- Some((to_lx, to_ly)),
- window_id,
- duration_ms,
- steps,
- &m,
- button,
- fg,
- move |x, y| {
- crate::cursor::overlay::send_command(
- cursor_for_drag.clone(),
- cursor_overlay::track_pointer_command(x, y),
- );
+ tokio::task::spawn_blocking(move || -> anyhow::Result<bool> {
+ let m: Vec<&str> = mods_owned.iter().map(String::as_str).collect();
+ let observe = move |x: f64, y: f64| {
+ crate::cursor::overlay::send_command(
+ cursor_for_drag.clone(),
+ cursor_overlay::track_pointer_command(x, y),
+ );
+ };
+ if fg {
+ // Keep the exact target window active for the
+ // complete HID gesture, then restore the prior
+ // front process.
+ return crate::input::skylight::with_foreground_hid_activation(
+ pid as libc::pid_t,
+ wid,
+ || {
+ crate::input::mouse::drag_at_xy_observed(
+ pid,
+ from_sx,
+ from_sy,
+ to_sx,
+ to_sy,
+ Some((from_lx, from_ly)),
+ Some((to_lx, to_ly)),
+ Some(wid),
+ duration_ms,
+ steps,
+ &m,
+ button,
+ true,
+ observe,
+ )
},
)
- };
- do_it()
+ .map(|()| false);
+ }
+ // A drag that starts a system drag-and-drop session
+ // (list reordering, file drags, HTML5 draggable) is
+ // tracked by macOS with the hardware pointer, so the
+ // drop cannot land at the requested point. AppKit shows
+ // the drag image in a window of the target at the
+ // dragging level; when one appears, release at once so
+ // the session ends where the user's pointer is not
+ // moving, and report it.
+ let before = drag_image_windows(pid);
+ let mut session_seen = false;
+ let outcome = crate::input::mouse::drag_at_xy_background(
+ pid,
+ (from_sx, from_sy),
+ (to_sx, to_sy),
+ wid,
+ duration_ms,
+ steps,
+ &m,
+ button,
+ observe,
+ || {
+ session_seen = !drag_image_windows(pid).is_subset(&before);
+ !session_seen
+ },
+ )?;
+ let session = session_seen
+ || !drag_image_windows(pid).is_subset(&before)
+ || matches!(
+ outcome,
+ crate::input::mouse::BackgroundDragOutcome::StoppedEarly { .. }
+ );
+ Ok(session)
})
.await
},
)
.await;
- let result = drag_input;
crate::cursor::overlay::send_command(
cursor_key.clone(),
cursor_overlay::OverlayCommand::SetPressed(false),
);
- if matches!(&result, Ok(Ok(()))) {
+ if matches!(&result, Ok(Ok(false))) {
self.state
.cursor_registry
.update_position(&cursor_key, to_sx, to_sy);
}
+ if !fg {
+ super::click::restore_user_focus_after_background_pixel(
+ pid,
+ prior_front,
+ prior_key_wid,
+ window_id,
+ focus_without_raise,
+ )
+ .await;
+ }
+
let changes = super::finish_window_observation(snapshot).await;
- if let Some(wid) = window_id {
- crate::cursor::overlay::send_command(
- cursor_key.clone(),
- cursor_overlay::OverlayCommand::PinAbove(wid as u64),
- );
- }
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
let mod_suffix = if modifiers.is_empty() {
String::new()
@@ -454,19 +502,25 @@ impl Tool for DragTool {
} else {
format!(" ({button_str} button)")
};
-
- let mode_label = if fg {
- " (delivery_mode:foreground)"
+ let (route_note, path) = if fg {
+ (
+ "delivery_mode:foreground; window fronted, hardware pointer moved, HID event tap",
+ "cgevent_hid",
+ )
} else {
- ""
+ (
+ "background: routed to the window's pid, pointer not moved, window not raised",
+ "cgevent",
+ )
};
match result {
- Ok(Ok(())) => ToolResult::text(format!(
- "✅ Posted drag{btn_suffix}{mod_suffix} to pid {pid} \
+ Ok(Ok(true)) => drag_and_drop_session_result(pid, wid, changes.result_suffix()),
+ Ok(Ok(false)) => ToolResult::text(format!(
+ "✅ Posted drag{btn_suffix}{mod_suffix} to pid {pid} window {wid} \
from window-pixel ({}, {}) → ({}, {}), \
screen ({}, {}) → ({}, {}) \
- in {duration_ms}ms / {steps} steps{mode_label} \
- (foreground HID CGEvent; not driver-verified — confirm via screenshot).{}",
+ in {duration_ms}ms / {steps} steps ({route_note}; not driver-verified — \
+ confirm via screenshot).{}",
from_x as i64,
from_y as i64,
to_x as i64,
@@ -478,7 +532,11 @@ impl Tool for DragTool {
changes.result_suffix(),
))
.with_structured(serde_json::json!({
- "path": "cgevent_hid", "verified": false, "effect": "unverifiable"
+ "path": path,
+ "delivery_mode": if fg { "foreground" } else { "background" },
+ "verified": false,
+ "effect": "unverifiable",
+ "focus_without_raise": focus_without_raise
})),
Ok(Err(e)) => ToolResult::error(format!("drag failed: {e}")),
Err(e) => ToolResult::error(format!("Task error: {e}")),
@@ -486,17 +544,66 @@ impl Tool for DragTool {
}
}
+/// `kCGDraggingWindowLevel`: AppKit shows the image of a live (or sliding
+/// back) drag-and-drop session in a window of the source app at this level.
+const DRAGGING_WINDOW_LEVEL: i32 = 500;
+
+fn drag_image_windows(pid: i32) -> std::collections::HashSet<u32> {
+ crate::windows::pid_on_screen_windows_any_layer(pid)
+ .into_iter()
+ .filter(|window| window.layer == DRAGGING_WINDOW_LEVEL)
+ .map(|window| window.window_id)
+ .collect()
+}
+
+fn drag_and_drop_session_result(pid: i32, wid: u32, suffix: String) -> ToolResult {
+ cua_driver_core::delivery::background_unavailable_result(
+ format!(
+ "The background drag in pid {pid} window {wid} started a macOS drag-and-drop \
+ session (for example reordering a list, moving a file or dragging page content). \
+ macOS tracks drag-and-drop with the real pointer, so the drop did not land at the \
+ requested point; the driver released at once. Look for a route that is not a drag \
+ (a Move Up/Down button or menu item, a context-menu command, a keyboard shortcut, \
+ cut and paste). If the drag is the only route and the user allows the window to \
+ come forward, retry with delivery_mode:\"foreground\".{suffix}"
+ ),
+ "background_unavailable",
+ "drag-and-drop sessions follow the hardware pointer on macOS",
+ serde_json::json!({
+ "effect": "no_drop",
+ "event_kind": "mouse_drag",
+ "reason": "drag_and_drop_session",
+ "requires": ["delivery_mode:foreground", "window_id"],
+ }),
+ )
+}
+
+fn window_id_required(foreground: bool) -> ToolResult {
+ let mode = if foreground {
+ "foreground"
+ } else {
+ "background"
+ };
+ ToolResult::error(format!(
+ "A {mode} drag on macOS needs window_id: the gesture is addressed to one exact \
+ window. Read it from list_windows or get_window_state and retry."
+ ))
+ .with_structured(serde_json::json!({
+ "code": "window_id_required",
+ "effect": "refused"
+ }))
+}
+
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
- async fn background_drag_refusal_is_structured_and_names_the_retry() {
+ async fn window_drag_without_window_id_is_refused_in_both_modes() {
let tool = DragTool::new(Arc::new(ToolState::new(false, false, None)));
- for delivery_mode in [None, Some("background")] {
+ for delivery_mode in [None, Some("background"), Some("foreground")] {
let mut args = serde_json::json!({
"pid": 4242,
- "window_id": 7,
"from_x": 1, "from_y": 2, "to_x": 3, "to_y": 4,
});
if let Some(mode) = delivery_mode {
@@ -505,24 +612,23 @@ mod tests {
let result = tool.invoke(args).await;
assert_eq!(result.is_error, Some(true));
let structured = result.structured_content.expect("structured refusal");
- assert_eq!(structured["code"], "background_unavailable");
+ assert_eq!(structured["code"], "window_id_required");
assert_eq!(structured["effect"], "refused");
- assert_eq!(structured["escalation"]["recommended"], "foreground");
- assert_eq!(
- structured["suggestion"],
- cua_driver_core::delivery::FOREGROUND_RETRY_SUGGESTION
- );
}
}
#[test]
- fn schema_says_macos_drag_is_foreground_only() {
+ fn schema_says_macos_drag_runs_in_the_background() {
let def = def();
- assert!(def.description.contains("macOS has no background drag"));
- let mode = &def.input_schema["properties"]["delivery_mode"]["description"];
- assert!(mode
+ assert!(def
+ .description
+ .contains("runs in the background by default"));
+ assert!(def.description.contains("browser_pointer"));
+ assert!(!def.description.contains("no background drag"));
+ let mode = def.input_schema["properties"]["delivery_mode"]["description"]
.as_str()
- .unwrap()
- .contains("supports only \"foreground\""));
+ .unwrap();
+ assert!(mode.contains("\"background\" (default)"));
+ assert!(mode.contains("without moving the pointer"));
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
index d598429241..0b5e24b195 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
@@ -106,6 +106,19 @@ pub(crate) fn pid_windows_with_space_snapshot(pid: i32, on_screen_only: bool) ->
)
}
+/// On-screen windows of `pid` on every layer, without the Space snapshot.
+/// Cheap enough to poll during a gesture (the drag tool watches for a
+/// drag-image window at the dragging level).
+pub(crate) fn pid_on_screen_windows_any_layer(pid: i32) -> Vec<WindowInfo> {
+ enumerate_windows(
+ kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements,
+ LayerFilter::AnyLayer,
+ Some(pid),
+ false,
+ )
+ .windows
+}
+
pub(crate) fn prefer_layer0(windows: &mut Vec<WindowInfo>) {
if windows.iter().any(|w| w.layer == 0) {
windows.retain(|w| w.layer == 0);
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index a3d853284e..974143df62 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -710,7 +710,7 @@
"read_only": false
},
{
- "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nmacOS has no background drag: a window-scoped drag needs delivery_mode:\"foreground\" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:\"foreground\" the call is refused with background_unavailable and nothing is sent.",
+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nA window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:\"drag\" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:\"foreground\", which briefly fronts the window and moves the physical pointer.",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -726,7 +726,7 @@
"type": "string"
},
"delivery_mode": {
- "description": "Window-scoped drag on macOS supports only \"foreground\": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default \"background\" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field.",
+ "description": "Window-scoped drag on macOS: \"background\" (default) posts the press, drag and release to the exact window (window_id required) without moving the pointer, raising the window or taking the user's focus. \"foreground\" briefly fronts the window, drives the physical pointer through the gesture, then restores the prior frontmost app; use it only when a background drag had no effect. Desktop scope ignores this field.",
"enum": [
"background",
"foreground"