Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): deliver background drags in Electron windows through the main process (CUA-1207) (#5351)
Francesco Bonacci24f6fc04d892
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdxindex 3d4e7f5926..deed8f894a 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx@@ -41,7 +41,7 @@ Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbinWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.-A window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:"drag" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:"foreground", which briefly fronts the window and moves the physical pointer.+A window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. In an Electron window the driver sends the gesture from the app's main process, so pointer-capture widgets and HTML5 drag and drop also drop there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:"drag" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:"foreground", which briefly fronts the window and moves the physical pointer.**macOS parameters**diff --git a/libs/cua-driver/docs/action-support.md b/libs/cua-driver/docs/action-support.mdindex e239670667..a94369a600 100644--- a/libs/cua-driver/docs/action-support.md+++ b/libs/cua-driver/docs/action-support.md@@ -31,7 +31,7 @@ Foreground rows listed here are delivered for both AX and PX.| --- | --- | --- | --- || Windows | Electron | left click and child window AX/PX | right click and double click AX/PX, plus drag PX: `background_occluded`; type text, press key, hotkey, and scroll AX/PX plus editor save AX: `background_unavailable`; `start_minimized` launch: `background_unavailable` when Windows denies the foreground lock, with no process spawned || Windows | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, child window AX/PX, scroll AX, editor save AX | hotkey AX/PX and scroll PX: `background_unavailable`; drag PX: `background_occluded` |-| macOS | Electron | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, child window AX/PX, editor save AX | scroll AX/PX and drag PX: `background_unavailable` |+| macOS | Electron | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, child window AX/PX, editor save AX, drag PX (sent from the app's main process; refuses with `background_unavailable` when the app's inspector is off) | scroll AX/PX: `background_unavailable` || macOS | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` || macOS | WKWebView | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` || Linux X11 | Electron | Background AX left click and child-window actions deliver with strict focus, z-order, cursor, and input-leak oracles | Background PX left click, AX/PX right and double click, PX child window and drag, AX/PX keyboard and scroll, and AX editor save return exact `background_unavailable` |diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex 5f4a2655c8..482c7acbe0 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -413,7 +413,15 @@ background: macOS tracks those sessions with the real pointer. The driversees the session start, releases at once, and returns`background_unavailable` with `reason: "drag_and_drop_session"`. Look for anon-drag route first (Move Up/Down, a menu or context-menu command, a-shortcut, cut and paste), then foreground if the user allows it. Toolkits that read the hardware pointer (Tk) or drop routed+shortcut, cut and paste), then foreground if the user allows it.++In an Electron window, `drag` is sent from the app's main process+(CDP `Input.dispatchMouseEvent` with the button held, or+`webContents.sendInputEvent`), because Chromium reads the held button from+the hardware and a window-routed press cannot hold it. Pointer-capture+widgets and HTML5 drag and drop both drop there, still without the real+pointer, focus or a raise. If the app's inspector is off, the drag refuses+with `background_unavailable`. Toolkits that read the hardware pointer (Tk) or drop routedmouse events (LibreOffice) refuse the background drag with`background_unavailable` before anything is sent.diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rsindex aa94f1a687..d45ba7fe84 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs@@ -1658,8 +1658,8 @@ fn shared_case_with_native_hyprland(// accessibility (reveal scroll), including inner scrollers.// The shared drag cell is a pointer-event drag, which the// background window-routed drag delivers (CUA-1207). Electron- // loses the release, so the driver refuses there.- ("electron", "drag", Targeting::Px) => vec![RefusalCode::BackgroundUnavailable],+ // takes it from the app's main process, which holds the button+ // that a window-routed press cannot._ => Vec::new(),}} else if cfg!(target_os = "linux")diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rsnew file mode 100644index 0000000000..71120c7ea2--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rs@@ -0,0 +1,336 @@+//! Background drag for Electron windows through the app's main process.+//!+//! A window-routed CGEvent drag reaches an Electron page, but Chromium builds+//! each web mouse event's held-button modifiers from+//! `+[NSEvent pressedMouseButtons]`, which reads the HID system button state+//! (`CGSInputButtonState`). A press posted to the process does not change that+//! state. `pointerdown` still reports the pressed button, so the page captures+//! the pointer, but every `mouseDragged` arrives as a `pointermove` with+//! `buttons == 0`. The pointer leaves the "active buttons" state, Blink fires+//! `lostpointercapture` on the first move, and the release is dispatched to+//! the element under the drop point instead of the capturing element: the drop+//! never lands. Widgets that test `event.buttons` during the move fail the+//! same way.+//!+//! This route sends the gesture from the Electron main process instead+//! (through its Node inspector, as the background hover route does), where the+//! button state is part of the event:+//!+//! 1. `webContents.debugger` and CDP `Input.dispatchMouseEvent` with `buttons`+//! set. `Input.setInterceptDrags` keeps an HTML5 drag-and-drop gesture out+//! of the native drag session (which follows the real pointer); the drop is+//! finished with `Input.dispatchDragEvent` at the drop point.+//! 2. When the debugger cannot attach, `webContents.sendInputEvent` with the+//! `leftButtonDown` (or right/middle) modifier on the press and moves.+//!+//! Neither path moves the real pointer, raises the window or takes focus.++/// Mouse button for the Electron drag route.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum ElectronDragButton {+ Left,+ Right,+ Middle,+}++impl ElectronDragButton {+ fn cdp_name(self) -> &'static str {+ match self {+ Self::Left => "left",+ Self::Right => "right",+ Self::Middle => "middle",+ }+ }++ /// CDP `buttons` bitmask while this button is held.+ fn cdp_mask(self) -> u8 {+ match self {+ Self::Left => 1,+ Self::Right => 2,+ Self::Middle => 4,+ }+ }++ /// Electron `sendInputEvent` modifier that marks this button as held.+ fn held_modifier(self) -> &'static str {+ match self {+ Self::Left => "leftButtonDown",+ Self::Right => "rightButtonDown",+ Self::Middle => "middleButtonDown",+ }+ }+}++/// One drag gesture in window-local points (top-left of the window frame,+/// title bar included), as the pixel tools resolve them.+#[derive(Clone, Debug, PartialEq)]+pub struct ElectronDragSpec {+ pub window_id: u32,+ pub from: (f64, f64),+ pub to: (f64, f64),+ pub steps: usize,+ pub duration_ms: u64,+ pub button: ElectronDragButton,+ pub modifiers: Vec<String>,+}++/// The inspector evaluation times out after 10 s; keep the gesture well+/// inside it.+pub const MAX_DURATION_MS: u64 = 5_000;++/// CDP `modifiers` bitmask and Electron modifier names for the held keys.+fn modifier_sets(modifiers: &[String]) -> (u8, Vec<&'static str>) {+ let mut mask = 0;+ let mut names = Vec::new();+ for m in modifiers {+ let (bit, name) = match m.to_ascii_lowercase().as_str() {+ "option" | "alt" | "opt" => (1, "alt"),+ "ctrl" | "control" => (2, "control"),+ "cmd" | "command" | "meta" | "super" => (4, "meta"),+ "shift" => (8, "shift"),+ _ => continue,+ };+ if mask & bit == 0 {+ mask |= bit;+ names.push(name);+ }+ }+ (mask, names)+}++/// Main-process JavaScript that performs the drag in the BrowserWindow whose+/// native window number is `spec.window_id`. Resolves to a JSON string:+/// `{"ok":true,"route":"cdp"|"send_input_event","drag_intercepted":bool,...}`+/// or `{"ok":false,"reason":..}`.+pub fn electron_drag_js(spec: &ElectronDragSpec) -> String {+ let (cdp_modifiers, names) = modifier_sets(&spec.modifiers);+ let held_keys = serde_json::to_string(&names).unwrap_or_else(|_| "[]".into());+ format!(+ r#"(async () => {{+ const req = (typeof require === 'function') ? require : (process.mainModule && process.mainModule.require.bind(process.mainModule));+ if (!req) return JSON.stringify({{ ok: false, reason: 'no_require' }});+ const {{ BrowserWindow }} = req('electron');+ const w = BrowserWindow.getAllWindows().find((w) => {{+ try {{ return String(w.getMediaSourceId()).split(':')[1] === '{wid}'; }} catch (e) {{ return false; }}+ }});+ if (!w) return JSON.stringify({{ ok: false, reason: 'no_window' }});+ const wc = w.webContents;+ const b = w.getBounds(), c = w.getContentBounds();+ const ox = c.x - b.x, oy = c.y - b.y;+ const local = (p) => [p[0] - ox, p[1] - oy];+ const from = local([{fx}, {fy}]), to = local([{tx}, {ty}]);+ for (const p of [from, to]) {{+ if (p[0] < 0 || p[1] < 0 || p[0] >= c.width || p[1] >= c.height) return JSON.stringify({{ ok: false, reason: 'outside_content' }});+ }}+ const steps = {steps}, stepDelay = Math.floor({duration} / steps);+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));+ const path = [];+ for (let i = 1; i <= steps; i++) path.push([from[0] + (to[0] - from[0]) * i / steps, from[1] + (to[1] - from[1]) * i / steps]);+ const dbg = wc.debugger;+ let attached = false;+ try {{ if (!dbg.isAttached()) {{ dbg.attach('1.3'); attached = true; }} }} catch (e) {{}}+ if (dbg.isAttached()) {{+ let drag = null;+ const onMessage = (_e, method, params) => {{ if (method === 'Input.dragIntercepted') drag = params.data; }};+ dbg.on('message', onMessage);+ const zoom = wc.getZoomFactor() || 1;+ const css = (p) => ({{ x: p[0] / zoom, y: p[1] / zoom }});+ const mouse = (type, p, held, extra) => dbg.sendCommand('Input.dispatchMouseEvent', Object.assign({{+ type, ...css(p), modifiers: {cdp_mods}, button: '{button}', buttons: held ? {mask} : 0, clickCount: 1, pointerType: 'mouse' }}, extra || {{}}));+ const dragEvent = (type, p) => dbg.sendCommand('Input.dispatchDragEvent', {{ type, ...css(p), data: drag, modifiers: {cdp_mods} }});+ let dragged = false, dropped = false;+ try {{+ await dbg.sendCommand('Input.setInterceptDrags', {{ enabled: true }});+ await mouse('mouseMoved', from, false, {{ button: 'none', clickCount: 0 }});+ await mouse('mousePressed', from, true);+ await sleep(30);+ for (const p of path) {{+ if (drag) {{+ if (!dragged) {{ await dragEvent('dragEnter', p); dragged = true; }}+ await dragEvent('dragOver', p);+ }} else {{+ await mouse('mouseMoved', p, true, {{ clickCount: 0 }});+ }}+ await sleep(stepDelay);+ }}+ await sleep(30);+ if (drag) {{+ if (!dragged) await dragEvent('dragEnter', to);+ await dragEvent('dragOver', to);+ await dragEvent('drop', to);+ dropped = true;+ }}+ await mouse('mouseReleased', to, false);+ }} finally {{+ dbg.removeListener('message', onMessage);+ try {{ await dbg.sendCommand('Input.setInterceptDrags', {{ enabled: false }}); }} catch (e) {{}}+ if (attached) {{ try {{ dbg.detach(); }} catch (e) {{}} }}+ }}+ return JSON.stringify({{ ok: true, route: 'cdp', drag_intercepted: !!drag, html5_drop: dropped }});+ }}+ const keys = {held_keys};+ const send = (type, p, held, extra) => wc.sendInputEvent(Object.assign({{+ type, x: Math.round(p[0]), y: Math.round(p[1]), button: '{button}',+ modifiers: held ? keys.concat(['{held}']) : keys }}, extra || {{}}));+ send('mouseMove', from, false);+ send('mouseDown', from, true, {{ clickCount: 1 }});+ await sleep(30);+ for (const p of path) {{ send('mouseMove', p, true); await sleep(stepDelay); }}+ await sleep(30);+ send('mouseUp', to, false, {{ clickCount: 1 }});+ return JSON.stringify({{ ok: true, route: 'send_input_event', drag_intercepted: false, html5_drop: false }});+}})()"#,+ wid = spec.window_id,+ fx = spec.from.0,+ fy = spec.from.1,+ tx = spec.to.0,+ ty = spec.to.1,+ steps = spec.steps.clamp(1, 200),+ duration = spec.duration_ms.min(MAX_DURATION_MS),+ cdp_mods = cdp_modifiers,+ button = spec.button.cdp_name(),+ mask = spec.button.cdp_mask(),+ held = spec.button.held_modifier(),+ held_keys = held_keys,+ )+}++/// What the main process reported for a drag.+#[derive(Clone, Debug, PartialEq, Eq)]+pub enum ElectronDragReply {+ Delivered {+ route: String,+ drag_intercepted: bool,+ html5_drop: bool,+ },+ Refused {+ reason: String,+ },+}++/// Parse the main-process reply, raw or as a quoted JSON string.+pub fn parse_electron_drag_reply(reply: &str) -> Option<ElectronDragReply> {+ let trimmed = reply.trim();+ let value: serde_json::Value = serde_json::from_str(trimmed).ok()?;+ let value = match value {+ serde_json::Value::String(inner) => serde_json::from_str(&inner).ok()?,+ other => other,+ };+ if value.get("ok")?.as_bool()? {+ Some(ElectronDragReply::Delivered {+ route: value+ .get("route")+ .and_then(|r| r.as_str())+ .unwrap_or("unknown")+ .to_owned(),+ drag_intercepted: value+ .get("drag_intercepted")+ .and_then(|v| v.as_bool())+ .unwrap_or(false),+ html5_drop: value+ .get("html5_drop")+ .and_then(|v| v.as_bool())+ .unwrap_or(false),+ })+ } else {+ Some(ElectronDragReply::Refused {+ reason: value+ .get("reason")+ .and_then(|r| r.as_str())+ .unwrap_or("unknown")+ .to_owned(),+ })+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn spec() -> ElectronDragSpec {+ ElectronDragSpec {+ window_id: 4242,+ from: (120.0, 140.5),+ to: (360.0, 150.0),+ steps: 20,+ duration_ms: 400,+ button: ElectronDragButton::Left,+ modifiers: vec!["shift".into(), "cmd".into(), "Shift".into()],+ }+ }++ #[test]+ fn js_targets_the_window_and_holds_the_button_through_the_gesture() {+ let js = electron_drag_js(&spec());+ assert!(js.contains("split(':')[1] === '4242'"));+ assert!(js.contains("local([120, 140.5])"));+ assert!(js.contains("local([360, 150])"));+ // CDP: the press and moves carry the held button, the release does not.+ assert!(js.contains("buttons: held ? 1 : 0"));+ assert!(js.contains("mouse('mousePressed', from, true)"));+ assert!(js.contains("mouse('mouseReleased', to, false)"));+ // HTML5 drag and drop is intercepted and finished with drag events.+ assert!(js.contains("Input.setInterceptDrags"));+ assert!(js.contains("dragEvent('drop', to)"));+ // Fallback: Electron's own held-button modifier.+ assert!(js.contains("keys.concat(['leftButtonDown'])"));+ // Shift + Cmd once each: CDP mask 8 | 4, Electron names.+ assert!(js.contains("modifiers: 12"));+ assert!(js.contains(r#"const keys = ["shift","meta"]"#));+ }++ #[test]+ fn js_bounds_the_gesture_inside_the_inspector_timeout() {+ let mut long = spec();+ long.duration_ms = 10_000;+ long.steps = 500;+ let js = electron_drag_js(&long);+ assert!(js.contains("const steps = 200"));+ assert!(js.contains("Math.floor(5000 / steps)"));+ }++ #[test]+ fn right_and_middle_buttons_use_their_own_masks() {+ let mut s = spec();+ s.button = ElectronDragButton::Right;+ let js = electron_drag_js(&s);+ assert!(js.contains("button: 'right'"));+ assert!(js.contains("buttons: held ? 2 : 0"));+ assert!(js.contains("'rightButtonDown'"));+ s.button = ElectronDragButton::Middle;+ let js = electron_drag_js(&s);+ assert!(js.contains("buttons: held ? 4 : 0"));+ assert!(js.contains("'middleButtonDown'"));+ }++ #[test]+ fn replies_parse_raw_and_quoted() {+ assert_eq!(+ parse_electron_drag_reply(+ r#"{"ok":true,"route":"cdp","drag_intercepted":true,"html5_drop":true}"#+ ),+ Some(ElectronDragReply::Delivered {+ route: "cdp".into(),+ drag_intercepted: true,+ html5_drop: true+ })+ );+ assert_eq!(+ parse_electron_drag_reply(+ r#""{\"ok\":true,\"route\":\"send_input_event\",\"drag_intercepted\":false}""#+ ),+ Some(ElectronDragReply::Delivered {+ route: "send_input_event".into(),+ drag_intercepted: false,+ html5_drop: false+ })+ );+ assert_eq!(+ parse_electron_drag_reply(r#"{"ok":false,"reason":"no_window"}"#),+ Some(ElectronDragReply::Refused {+ reason: "no_window".into()+ })+ );+ assert_eq!(parse_electron_drag_reply("not json"), None);+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rsindex e3faeaeee7..85fe32281b 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs@@ -9,6 +9,7 @@//! the activity-monitor tickle required for live-input detection.pub mod ax_actions;+pub mod electron_drag;pub mod hover;pub mod interactive;pub mod keyboard;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex 40a18047a2..54461d8cda 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -62,7 +62,9 @@ fn def() -> &'static ToolDef {the gesture is posted to that window, the real pointer does not move, and the \user's frontmost window keeps focus. It works for controls that follow the \drag events: sliders, canvases, and pointer-event widgets on web pages in \- Chrome. A drag that starts a macOS drag-and-drop session (reordering a list \+ Chrome. In an Electron window the driver sends the gesture from the app's \+ main process, so pointer-capture widgets and HTML5 drag and drop also drop \+ there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list \by drag and drop, moving a file, HTML5 draggable content) cannot drop in the \background, because macOS tracks those with the real pointer; the driver \detects it, releases at once and returns background_unavailable with reason \@@ -311,21 +313,28 @@ impl Tool for DragTool {{return refusal;}- // Electron/Chromium windows take the press and the moves of a- // background drag but lose the release (the page's pointer- // capture is dropped), so the drop never lands. Refuse instead- // of reporting a delivery that did not happen.+ // Chromium reads a mouse event's held buttons from the hardware+ // state, which a window-routed press does not change, so an+ // Electron page sees the drag moves with buttons == 0, releases+ // its pointer capture and loses the drop. Send the gesture from+ // the app's main process instead, where the held button is part+ // of the event; refuse when that route is unavailable.if crate::browser::ElectronJs::is_electron(pid) {- return cua_driver_core::delivery::background_unavailable_result(- ELECTRON_BACKGROUND_DRAG_UNAVAILABLE,- "background_unavailable",- "Electron/Chromium windows lose the release of a background drag on macOS",- serde_json::json!({- "effect": "refused",- "event_kind": "mouse_drag",- "requires": ["delivery_mode:foreground", "window_id"],- }),- );+ return self+ .electron_background_drag(+ &cursor_key,+ pid,+ wid,+ (from_lx, from_ly),+ (to_lx, to_ly),+ (from_sx, from_sy),+ (to_sx, to_sy),+ duration_ms,+ steps,+ &modifiers,+ button,+ )+ .await;}}@@ -563,15 +572,131 @@ impl Tool for DragTool {}}+impl DragTool {+ /// Background drag in an Electron window through its main process+ /// ([`crate::input::electron_drag`]). Refuses when the app's inspector+ /// is unavailable (for example the inspector fuse is off).+ #[allow(clippy::too_many_arguments)]+ async fn electron_background_drag(+ &self,+ cursor_key: &str,+ pid: i32,+ wid: u32,+ from_local: (f64, f64),+ to_local: (f64, f64),+ from_screen: (f64, f64),+ to_screen: (f64, f64),+ duration_ms: u64,+ steps: usize,+ modifiers: &[String],+ button: DragButton,+ ) -> ToolResult {+ use crate::input::electron_drag::{+ electron_drag_js, parse_electron_drag_reply, ElectronDragButton, ElectronDragReply,+ ElectronDragSpec,+ };+ let spec = ElectronDragSpec {+ window_id: wid,+ from: from_local,+ to: to_local,+ steps,+ duration_ms,+ button: match button {+ DragButton::Left => ElectronDragButton::Left,+ DragButton::Right => ElectronDragButton::Right,+ DragButton::Middle => ElectronDragButton::Middle,+ },+ modifiers: modifiers.to_vec(),+ };+ let cursor_key = cursor_key.to_owned();+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),+ );+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), from_screen.0, from_screen.1)+ .await;+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );+ let reply = crate::browser::ElectronJs::execute_main(&electron_drag_js(&spec), pid).await;+ let outcome = match &reply {+ Ok(text) => parse_electron_drag_reply(text),+ Err(_) => None,+ };+ if matches!(outcome, Some(ElectronDragReply::Delivered { .. })) {+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), to_screen.0, to_screen.1)+ .await;+ self.state+ .cursor_registry+ .update_position(&cursor_key, to_screen.0, to_screen.1);+ }+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::SetPressed(false),+ );+ match outcome {+ Some(ElectronDragReply::Delivered {+ route,+ drag_intercepted,+ html5_drop,+ }) => {+ let dnd = if html5_drop {+ " The gesture started an HTML5 drag and drop; the driver kept it out of the \+ native drag session and dropped it at the end point."+ } else {+ ""+ };+ ToolResult::text(format!(+ "✅ Posted drag to Electron pid {pid} window {wid} from window-point \+ ({:.0}, {:.0}) → ({:.0}, {:.0}) through the app's main process \+ ({route}); pointer not moved, window not raised, focus kept.{dnd} \+ Not driver-verified — confirm via screenshot.",+ from_local.0, from_local.1, to_local.0, to_local.1,+ ))+ .with_structured(serde_json::json!({+ // Chromium trusted input (CDP Input or sendInputEvent), the+ // same public route as browser_pointer.+ "path": "trusted",+ "electron_route": route,+ "drag_intercepted": drag_intercepted,+ "html5_drop": html5_drop,+ "delivery_mode": "background",+ "verified": false,+ "effect": "unverifiable",+ }))+ }+ other => {+ let reason = match (other, &reply) {+ (Some(ElectronDragReply::Refused { reason }), _) => reason,+ (_, Err(error)) => error.to_string(),+ (_, Ok(text)) => format!("unexpected inspector reply: {text}"),+ };+ cua_driver_core::delivery::background_unavailable_result(+ format!("{ELECTRON_BACKGROUND_DRAG_UNAVAILABLE} ({reason})"),+ "background_unavailable",+ "the Electron app's main process could not take the drag",+ serde_json::json!({+ "effect": "refused",+ "event_kind": "mouse_drag",+ "reason": "electron_main_process_unavailable",+ "requires": ["delivery_mode:foreground", "window_id"],+ }),+ )+ }+ }+ }+}+/// `kCGDraggingWindowLevel`: AppKit shows the image of a live (or sliding/// back) drag-and-drop session in a window of the source app at this level.const DRAGGING_WINDOW_LEVEL: i32 = 500;const ELECTRON_BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable for this \- Electron/Chromium window on macOS: it takes the press and the moves but loses the release, \- so the drop does not land. Use the app's own controls (a button, menu item or keyboard \- route) or, if the user allows the window to come forward, retry with \- delivery_mode:\"foreground\".";+ Electron window on macOS: a window-routed drag cannot hold the page's mouse button \+ (Chromium reads it from the hardware), and the app's main process did not take the drag. \+ Use the app's own controls (a button, menu item or keyboard route) or, if the user allows \+ the window to come forward, retry with delivery_mode:\"foreground\".";fn drag_image_windows(pid: i32) -> std::collections::HashSet<u32> {crate::windows::pid_on_screen_windows_any_layer(pid)diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 9a27c050e0..749eca7554 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -710,7 +710,7 @@"read_only": false},{- "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nA window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:\"drag\" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:\"foreground\", which briefly fronts the window and moves the physical pointer.",+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nA window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. In an Electron window the driver sends the gesture from the app's main process, so pointer-capture widgets and HTML5 drag and drop also drop there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:\"drag\" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:\"foreground\", which briefly fronts the window and moves the physical pointer.","destructive": true,"idempotent": false,"input_schema": {