Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): deliver background drags in Electron windows through the main process (CUA-1207) (#5351)

Francesco Bonacci24f6fc04d892
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index 3d4e7f5926..deed8f894a 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -41,7 +41,7 @@ Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbin
When `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.
-A window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:"drag" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:"foreground", which briefly fronts the window and moves the physical pointer.
+A window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. In an Electron window the driver sends the gesture from the app's main process, so pointer-capture widgets and HTML5 drag and drop also drop there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:"drag" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:"foreground", which briefly fronts the window and moves the physical pointer.
**macOS parameters**
diff --git a/libs/cua-driver/docs/action-support.md b/libs/cua-driver/docs/action-support.md
index e239670667..a94369a600 100644
--- a/libs/cua-driver/docs/action-support.md
+++ b/libs/cua-driver/docs/action-support.md
@@ -31,7 +31,7 @@ Foreground rows listed here are delivered for both AX and PX.
| --- | --- | --- | --- |
| Windows | Electron | left click and child window AX/PX | right click and double click AX/PX, plus drag PX: `background_occluded`; type text, press key, hotkey, and scroll AX/PX plus editor save AX: `background_unavailable`; `start_minimized` launch: `background_unavailable` when Windows denies the foreground lock, with no process spawned |
| Windows | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, child window AX/PX, scroll AX, editor save AX | hotkey AX/PX and scroll PX: `background_unavailable`; drag PX: `background_occluded` |
-| macOS | Electron | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, child window AX/PX, editor save AX | scroll AX/PX and drag PX: `background_unavailable` |
+| macOS | Electron | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, child window AX/PX, editor save AX, drag PX (sent from the app's main process; refuses with `background_unavailable` when the app's inspector is off) | scroll AX/PX: `background_unavailable` |
| macOS | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` |
| macOS | WKWebView | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` |
| Linux X11 | Electron | Background AX left click and child-window actions deliver with strict focus, z-order, cursor, and input-leak oracles | Background PX left click, AX/PX right and double click, PX child window and drag, AX/PX keyboard and scroll, and AX editor save return exact `background_unavailable` |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 5f4a2655c8..482c7acbe0 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -413,7 +413,15 @@ background: macOS tracks those sessions with the real pointer. The driver
sees the session start, releases at once, and returns
`background_unavailable` with `reason: "drag_and_drop_session"`. Look for a
non-drag route first (Move Up/Down, a menu or context-menu command, a
-shortcut, cut and paste), then foreground if the user allows it. Toolkits that read the hardware pointer (Tk) or drop routed
+shortcut, cut and paste), then foreground if the user allows it.
+
+In an Electron window, `drag` is sent from the app's main process
+(CDP `Input.dispatchMouseEvent` with the button held, or
+`webContents.sendInputEvent`), because Chromium reads the held button from
+the hardware and a window-routed press cannot hold it. Pointer-capture
+widgets and HTML5 drag and drop both drop there, still without the real
+pointer, focus or a raise. If the app's inspector is off, the drag refuses
+with `background_unavailable`. Toolkits that read the hardware pointer (Tk) or drop routed
mouse events (LibreOffice) refuse the background drag with
`background_unavailable` before anything is sent.
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
index aa94f1a687..d45ba7fe84 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
@@ -1658,8 +1658,8 @@ fn shared_case_with_native_hyprland(
// accessibility (reveal scroll), including inner scrollers.
// The shared drag cell is a pointer-event drag, which the
// background window-routed drag delivers (CUA-1207). Electron
- // loses the release, so the driver refuses there.
- ("electron", "drag", Targeting::Px) => vec![RefusalCode::BackgroundUnavailable],
+ // takes it from the app's main process, which holds the button
+ // that a window-routed press cannot.
_ => Vec::new(),
}
} else if cfg!(target_os = "linux")
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rs
new file mode 100644
index 0000000000..71120c7ea2
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/electron_drag.rs
@@ -0,0 +1,336 @@
+//! Background drag for Electron windows through the app's main process.
+//!
+//! A window-routed CGEvent drag reaches an Electron page, but Chromium builds
+//! each web mouse event's held-button modifiers from
+//! `+[NSEvent pressedMouseButtons]`, which reads the HID system button state
+//! (`CGSInputButtonState`). A press posted to the process does not change that
+//! state. `pointerdown` still reports the pressed button, so the page captures
+//! the pointer, but every `mouseDragged` arrives as a `pointermove` with
+//! `buttons == 0`. The pointer leaves the "active buttons" state, Blink fires
+//! `lostpointercapture` on the first move, and the release is dispatched to
+//! the element under the drop point instead of the capturing element: the drop
+//! never lands. Widgets that test `event.buttons` during the move fail the
+//! same way.
+//!
+//! This route sends the gesture from the Electron main process instead
+//! (through its Node inspector, as the background hover route does), where the
+//! button state is part of the event:
+//!
+//! 1. `webContents.debugger` and CDP `Input.dispatchMouseEvent` with `buttons`
+//! set. `Input.setInterceptDrags` keeps an HTML5 drag-and-drop gesture out
+//! of the native drag session (which follows the real pointer); the drop is
+//! finished with `Input.dispatchDragEvent` at the drop point.
+//! 2. When the debugger cannot attach, `webContents.sendInputEvent` with the
+//! `leftButtonDown` (or right/middle) modifier on the press and moves.
+//!
+//! Neither path moves the real pointer, raises the window or takes focus.
+
+/// Mouse button for the Electron drag route.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ElectronDragButton {
+ Left,
+ Right,
+ Middle,
+}
+
+impl ElectronDragButton {
+ fn cdp_name(self) -> &'static str {
+ match self {
+ Self::Left => "left",
+ Self::Right => "right",
+ Self::Middle => "middle",
+ }
+ }
+
+ /// CDP `buttons` bitmask while this button is held.
+ fn cdp_mask(self) -> u8 {
+ match self {
+ Self::Left => 1,
+ Self::Right => 2,
+ Self::Middle => 4,
+ }
+ }
+
+ /// Electron `sendInputEvent` modifier that marks this button as held.
+ fn held_modifier(self) -> &'static str {
+ match self {
+ Self::Left => "leftButtonDown",
+ Self::Right => "rightButtonDown",
+ Self::Middle => "middleButtonDown",
+ }
+ }
+}
+
+/// One drag gesture in window-local points (top-left of the window frame,
+/// title bar included), as the pixel tools resolve them.
+#[derive(Clone, Debug, PartialEq)]
+pub struct ElectronDragSpec {
+ pub window_id: u32,
+ pub from: (f64, f64),
+ pub to: (f64, f64),
+ pub steps: usize,
+ pub duration_ms: u64,
+ pub button: ElectronDragButton,
+ pub modifiers: Vec<String>,
+}
+
+/// The inspector evaluation times out after 10 s; keep the gesture well
+/// inside it.
+pub const MAX_DURATION_MS: u64 = 5_000;
+
+/// CDP `modifiers` bitmask and Electron modifier names for the held keys.
+fn modifier_sets(modifiers: &[String]) -> (u8, Vec<&'static str>) {
+ let mut mask = 0;
+ let mut names = Vec::new();
+ for m in modifiers {
+ let (bit, name) = match m.to_ascii_lowercase().as_str() {
+ "option" | "alt" | "opt" => (1, "alt"),
+ "ctrl" | "control" => (2, "control"),
+ "cmd" | "command" | "meta" | "super" => (4, "meta"),
+ "shift" => (8, "shift"),
+ _ => continue,
+ };
+ if mask & bit == 0 {
+ mask |= bit;
+ names.push(name);
+ }
+ }
+ (mask, names)
+}
+
+/// Main-process JavaScript that performs the drag in the BrowserWindow whose
+/// native window number is `spec.window_id`. Resolves to a JSON string:
+/// `{"ok":true,"route":"cdp"|"send_input_event","drag_intercepted":bool,...}`
+/// or `{"ok":false,"reason":..}`.
+pub fn electron_drag_js(spec: &ElectronDragSpec) -> String {
+ let (cdp_modifiers, names) = modifier_sets(&spec.modifiers);
+ let held_keys = serde_json::to_string(&names).unwrap_or_else(|_| "[]".into());
+ format!(
+ r#"(async () => {{
+ const req = (typeof require === 'function') ? require : (process.mainModule && process.mainModule.require.bind(process.mainModule));
+ if (!req) return JSON.stringify({{ ok: false, reason: 'no_require' }});
+ const {{ BrowserWindow }} = req('electron');
+ const w = BrowserWindow.getAllWindows().find((w) => {{
+ try {{ return String(w.getMediaSourceId()).split(':')[1] === '{wid}'; }} catch (e) {{ return false; }}
+ }});
+ if (!w) return JSON.stringify({{ ok: false, reason: 'no_window' }});
+ const wc = w.webContents;
+ const b = w.getBounds(), c = w.getContentBounds();
+ const ox = c.x - b.x, oy = c.y - b.y;
+ const local = (p) => [p[0] - ox, p[1] - oy];
+ const from = local([{fx}, {fy}]), to = local([{tx}, {ty}]);
+ for (const p of [from, to]) {{
+ if (p[0] < 0 || p[1] < 0 || p[0] >= c.width || p[1] >= c.height) return JSON.stringify({{ ok: false, reason: 'outside_content' }});
+ }}
+ const steps = {steps}, stepDelay = Math.floor({duration} / steps);
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const path = [];
+ for (let i = 1; i <= steps; i++) path.push([from[0] + (to[0] - from[0]) * i / steps, from[1] + (to[1] - from[1]) * i / steps]);
+ const dbg = wc.debugger;
+ let attached = false;
+ try {{ if (!dbg.isAttached()) {{ dbg.attach('1.3'); attached = true; }} }} catch (e) {{}}
+ if (dbg.isAttached()) {{
+ let drag = null;
+ const onMessage = (_e, method, params) => {{ if (method === 'Input.dragIntercepted') drag = params.data; }};
+ dbg.on('message', onMessage);
+ const zoom = wc.getZoomFactor() || 1;
+ const css = (p) => ({{ x: p[0] / zoom, y: p[1] / zoom }});
+ const mouse = (type, p, held, extra) => dbg.sendCommand('Input.dispatchMouseEvent', Object.assign({{
+ type, ...css(p), modifiers: {cdp_mods}, button: '{button}', buttons: held ? {mask} : 0, clickCount: 1, pointerType: 'mouse' }}, extra || {{}}));
+ const dragEvent = (type, p) => dbg.sendCommand('Input.dispatchDragEvent', {{ type, ...css(p), data: drag, modifiers: {cdp_mods} }});
+ let dragged = false, dropped = false;
+ try {{
+ await dbg.sendCommand('Input.setInterceptDrags', {{ enabled: true }});
+ await mouse('mouseMoved', from, false, {{ button: 'none', clickCount: 0 }});
+ await mouse('mousePressed', from, true);
+ await sleep(30);
+ for (const p of path) {{
+ if (drag) {{
+ if (!dragged) {{ await dragEvent('dragEnter', p); dragged = true; }}
+ await dragEvent('dragOver', p);
+ }} else {{
+ await mouse('mouseMoved', p, true, {{ clickCount: 0 }});
+ }}
+ await sleep(stepDelay);
+ }}
+ await sleep(30);
+ if (drag) {{
+ if (!dragged) await dragEvent('dragEnter', to);
+ await dragEvent('dragOver', to);
+ await dragEvent('drop', to);
+ dropped = true;
+ }}
+ await mouse('mouseReleased', to, false);
+ }} finally {{
+ dbg.removeListener('message', onMessage);
+ try {{ await dbg.sendCommand('Input.setInterceptDrags', {{ enabled: false }}); }} catch (e) {{}}
+ if (attached) {{ try {{ dbg.detach(); }} catch (e) {{}} }}
+ }}
+ return JSON.stringify({{ ok: true, route: 'cdp', drag_intercepted: !!drag, html5_drop: dropped }});
+ }}
+ const keys = {held_keys};
+ const send = (type, p, held, extra) => wc.sendInputEvent(Object.assign({{
+ type, x: Math.round(p[0]), y: Math.round(p[1]), button: '{button}',
+ modifiers: held ? keys.concat(['{held}']) : keys }}, extra || {{}}));
+ send('mouseMove', from, false);
+ send('mouseDown', from, true, {{ clickCount: 1 }});
+ await sleep(30);
+ for (const p of path) {{ send('mouseMove', p, true); await sleep(stepDelay); }}
+ await sleep(30);
+ send('mouseUp', to, false, {{ clickCount: 1 }});
+ return JSON.stringify({{ ok: true, route: 'send_input_event', drag_intercepted: false, html5_drop: false }});
+}})()"#,
+ wid = spec.window_id,
+ fx = spec.from.0,
+ fy = spec.from.1,
+ tx = spec.to.0,
+ ty = spec.to.1,
+ steps = spec.steps.clamp(1, 200),
+ duration = spec.duration_ms.min(MAX_DURATION_MS),
+ cdp_mods = cdp_modifiers,
+ button = spec.button.cdp_name(),
+ mask = spec.button.cdp_mask(),
+ held = spec.button.held_modifier(),
+ held_keys = held_keys,
+ )
+}
+
+/// What the main process reported for a drag.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum ElectronDragReply {
+ Delivered {
+ route: String,
+ drag_intercepted: bool,
+ html5_drop: bool,
+ },
+ Refused {
+ reason: String,
+ },
+}
+
+/// Parse the main-process reply, raw or as a quoted JSON string.
+pub fn parse_electron_drag_reply(reply: &str) -> Option<ElectronDragReply> {
+ let trimmed = reply.trim();
+ let value: serde_json::Value = serde_json::from_str(trimmed).ok()?;
+ let value = match value {
+ serde_json::Value::String(inner) => serde_json::from_str(&inner).ok()?,
+ other => other,
+ };
+ if value.get("ok")?.as_bool()? {
+ Some(ElectronDragReply::Delivered {
+ route: value
+ .get("route")
+ .and_then(|r| r.as_str())
+ .unwrap_or("unknown")
+ .to_owned(),
+ drag_intercepted: value
+ .get("drag_intercepted")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false),
+ html5_drop: value
+ .get("html5_drop")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false),
+ })
+ } else {
+ Some(ElectronDragReply::Refused {
+ reason: value
+ .get("reason")
+ .and_then(|r| r.as_str())
+ .unwrap_or("unknown")
+ .to_owned(),
+ })
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn spec() -> ElectronDragSpec {
+ ElectronDragSpec {
+ window_id: 4242,
+ from: (120.0, 140.5),
+ to: (360.0, 150.0),
+ steps: 20,
+ duration_ms: 400,
+ button: ElectronDragButton::Left,
+ modifiers: vec!["shift".into(), "cmd".into(), "Shift".into()],
+ }
+ }
+
+ #[test]
+ fn js_targets_the_window_and_holds_the_button_through_the_gesture() {
+ let js = electron_drag_js(&spec());
+ assert!(js.contains("split(':')[1] === '4242'"));
+ assert!(js.contains("local([120, 140.5])"));
+ assert!(js.contains("local([360, 150])"));
+ // CDP: the press and moves carry the held button, the release does not.
+ assert!(js.contains("buttons: held ? 1 : 0"));
+ assert!(js.contains("mouse('mousePressed', from, true)"));
+ assert!(js.contains("mouse('mouseReleased', to, false)"));
+ // HTML5 drag and drop is intercepted and finished with drag events.
+ assert!(js.contains("Input.setInterceptDrags"));
+ assert!(js.contains("dragEvent('drop', to)"));
+ // Fallback: Electron's own held-button modifier.
+ assert!(js.contains("keys.concat(['leftButtonDown'])"));
+ // Shift + Cmd once each: CDP mask 8 | 4, Electron names.
+ assert!(js.contains("modifiers: 12"));
+ assert!(js.contains(r#"const keys = ["shift","meta"]"#));
+ }
+
+ #[test]
+ fn js_bounds_the_gesture_inside_the_inspector_timeout() {
+ let mut long = spec();
+ long.duration_ms = 10_000;
+ long.steps = 500;
+ let js = electron_drag_js(&long);
+ assert!(js.contains("const steps = 200"));
+ assert!(js.contains("Math.floor(5000 / steps)"));
+ }
+
+ #[test]
+ fn right_and_middle_buttons_use_their_own_masks() {
+ let mut s = spec();
+ s.button = ElectronDragButton::Right;
+ let js = electron_drag_js(&s);
+ assert!(js.contains("button: 'right'"));
+ assert!(js.contains("buttons: held ? 2 : 0"));
+ assert!(js.contains("'rightButtonDown'"));
+ s.button = ElectronDragButton::Middle;
+ let js = electron_drag_js(&s);
+ assert!(js.contains("buttons: held ? 4 : 0"));
+ assert!(js.contains("'middleButtonDown'"));
+ }
+
+ #[test]
+ fn replies_parse_raw_and_quoted() {
+ assert_eq!(
+ parse_electron_drag_reply(
+ r#"{"ok":true,"route":"cdp","drag_intercepted":true,"html5_drop":true}"#
+ ),
+ Some(ElectronDragReply::Delivered {
+ route: "cdp".into(),
+ drag_intercepted: true,
+ html5_drop: true
+ })
+ );
+ assert_eq!(
+ parse_electron_drag_reply(
+ r#""{\"ok\":true,\"route\":\"send_input_event\",\"drag_intercepted\":false}""#
+ ),
+ Some(ElectronDragReply::Delivered {
+ route: "send_input_event".into(),
+ drag_intercepted: false,
+ html5_drop: false
+ })
+ );
+ assert_eq!(
+ parse_electron_drag_reply(r#"{"ok":false,"reason":"no_window"}"#),
+ Some(ElectronDragReply::Refused {
+ reason: "no_window".into()
+ })
+ );
+ assert_eq!(parse_electron_drag_reply("not json"), None);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
index e3faeaeee7..85fe32281b 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mod.rs
@@ -9,6 +9,7 @@
//! the activity-monitor tickle required for live-input detection.
pub mod ax_actions;
+pub mod electron_drag;
pub mod hover;
pub mod interactive;
pub mod keyboard;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index 40a18047a2..54461d8cda 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -62,7 +62,9 @@ fn def() -> &'static ToolDef {
the gesture is posted to that window, the real pointer does not move, and the \
user's frontmost window keeps focus. It works for controls that follow the \
drag events: sliders, canvases, and pointer-event widgets on web pages in \
- Chrome. A drag that starts a macOS drag-and-drop session (reordering a list \
+ Chrome. In an Electron window the driver sends the gesture from the app's \
+ main process, so pointer-capture widgets and HTML5 drag and drop also drop \
+ there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list \
by drag and drop, moving a file, HTML5 draggable content) cannot drop in the \
background, because macOS tracks those with the real pointer; the driver \
detects it, releases at once and returns background_unavailable with reason \
@@ -311,21 +313,28 @@ impl Tool for DragTool {
{
return refusal;
}
- // Electron/Chromium windows take the press and the moves of a
- // background drag but lose the release (the page's pointer
- // capture is dropped), so the drop never lands. Refuse instead
- // of reporting a delivery that did not happen.
+ // Chromium reads a mouse event's held buttons from the hardware
+ // state, which a window-routed press does not change, so an
+ // Electron page sees the drag moves with buttons == 0, releases
+ // its pointer capture and loses the drop. Send the gesture from
+ // the app's main process instead, where the held button is part
+ // of the event; refuse when that route is unavailable.
if crate::browser::ElectronJs::is_electron(pid) {
- return cua_driver_core::delivery::background_unavailable_result(
- ELECTRON_BACKGROUND_DRAG_UNAVAILABLE,
- "background_unavailable",
- "Electron/Chromium windows lose the release of a background drag on macOS",
- serde_json::json!({
- "effect": "refused",
- "event_kind": "mouse_drag",
- "requires": ["delivery_mode:foreground", "window_id"],
- }),
- );
+ return self
+ .electron_background_drag(
+ &cursor_key,
+ pid,
+ wid,
+ (from_lx, from_ly),
+ (to_lx, to_ly),
+ (from_sx, from_sy),
+ (to_sx, to_sy),
+ duration_ms,
+ steps,
+ &modifiers,
+ button,
+ )
+ .await;
}
}
@@ -563,15 +572,131 @@ impl Tool for DragTool {
}
}
+impl DragTool {
+ /// Background drag in an Electron window through its main process
+ /// ([`crate::input::electron_drag`]). Refuses when the app's inspector
+ /// is unavailable (for example the inspector fuse is off).
+ #[allow(clippy::too_many_arguments)]
+ async fn electron_background_drag(
+ &self,
+ cursor_key: &str,
+ pid: i32,
+ wid: u32,
+ from_local: (f64, f64),
+ to_local: (f64, f64),
+ from_screen: (f64, f64),
+ to_screen: (f64, f64),
+ duration_ms: u64,
+ steps: usize,
+ modifiers: &[String],
+ button: DragButton,
+ ) -> ToolResult {
+ use crate::input::electron_drag::{
+ electron_drag_js, parse_electron_drag_reply, ElectronDragButton, ElectronDragReply,
+ ElectronDragSpec,
+ };
+ let spec = ElectronDragSpec {
+ window_id: wid,
+ from: from_local,
+ to: to_local,
+ steps,
+ duration_ms,
+ button: match button {
+ DragButton::Left => ElectronDragButton::Left,
+ DragButton::Right => ElectronDragButton::Right,
+ DragButton::Middle => ElectronDragButton::Middle,
+ },
+ modifiers: modifiers.to_vec(),
+ };
+ let cursor_key = cursor_key.to_owned();
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), from_screen.0, from_screen.1)
+ .await;
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(true),
+ );
+ let reply = crate::browser::ElectronJs::execute_main(&electron_drag_js(&spec), pid).await;
+ let outcome = match &reply {
+ Ok(text) => parse_electron_drag_reply(text),
+ Err(_) => None,
+ };
+ if matches!(outcome, Some(ElectronDragReply::Delivered { .. })) {
+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), to_screen.0, to_screen.1)
+ .await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, to_screen.0, to_screen.1);
+ }
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::SetPressed(false),
+ );
+ match outcome {
+ Some(ElectronDragReply::Delivered {
+ route,
+ drag_intercepted,
+ html5_drop,
+ }) => {
+ let dnd = if html5_drop {
+ " The gesture started an HTML5 drag and drop; the driver kept it out of the \
+ native drag session and dropped it at the end point."
+ } else {
+ ""
+ };
+ ToolResult::text(format!(
+ "✅ Posted drag to Electron pid {pid} window {wid} from window-point \
+ ({:.0}, {:.0}) → ({:.0}, {:.0}) through the app's main process \
+ ({route}); pointer not moved, window not raised, focus kept.{dnd} \
+ Not driver-verified — confirm via screenshot.",
+ from_local.0, from_local.1, to_local.0, to_local.1,
+ ))
+ .with_structured(serde_json::json!({
+ // Chromium trusted input (CDP Input or sendInputEvent), the
+ // same public route as browser_pointer.
+ "path": "trusted",
+ "electron_route": route,
+ "drag_intercepted": drag_intercepted,
+ "html5_drop": html5_drop,
+ "delivery_mode": "background",
+ "verified": false,
+ "effect": "unverifiable",
+ }))
+ }
+ other => {
+ let reason = match (other, &reply) {
+ (Some(ElectronDragReply::Refused { reason }), _) => reason,
+ (_, Err(error)) => error.to_string(),
+ (_, Ok(text)) => format!("unexpected inspector reply: {text}"),
+ };
+ cua_driver_core::delivery::background_unavailable_result(
+ format!("{ELECTRON_BACKGROUND_DRAG_UNAVAILABLE} ({reason})"),
+ "background_unavailable",
+ "the Electron app's main process could not take the drag",
+ serde_json::json!({
+ "effect": "refused",
+ "event_kind": "mouse_drag",
+ "reason": "electron_main_process_unavailable",
+ "requires": ["delivery_mode:foreground", "window_id"],
+ }),
+ )
+ }
+ }
+ }
+}
+
/// `kCGDraggingWindowLevel`: AppKit shows the image of a live (or sliding
/// back) drag-and-drop session in a window of the source app at this level.
const DRAGGING_WINDOW_LEVEL: i32 = 500;
const ELECTRON_BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable for this \
- Electron/Chromium window on macOS: it takes the press and the moves but loses the release, \
- so the drop does not land. Use the app's own controls (a button, menu item or keyboard \
- route) or, if the user allows the window to come forward, retry with \
- delivery_mode:\"foreground\".";
+ Electron window on macOS: a window-routed drag cannot hold the page's mouse button \
+ (Chromium reads it from the hardware), and the app's main process did not take the drag. \
+ Use the app's own controls (a button, menu item or keyboard route) or, if the user allows \
+ the window to come forward, retry with delivery_mode:\"foreground\".";
fn drag_image_windows(pid: i32) -> std::collections::HashSet<u32> {
crate::windows::pid_on_screen_windows_any_layer(pid)
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 9a27c050e0..749eca7554 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -710,7 +710,7 @@
"read_only": false
},
{
- "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nA window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:\"drag\" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:\"foreground\", which briefly fronts the window and moves the physical pointer.",
+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nA window-scoped drag (pid + window_id) runs in the background by default: the gesture is posted to that window, the real pointer does not move, and the user's frontmost window keeps focus. It works for controls that follow the drag events: sliders, canvases, and pointer-event widgets on web pages in Chrome. In an Electron window the driver sends the gesture from the app's main process, so pointer-capture widgets and HTML5 drag and drop also drop there. Elsewhere, a drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 draggable content) cannot drop in the background, because macOS tracks those with the real pointer; the driver detects it, releases at once and returns background_unavailable with reason drag_and_drop_session. For a page in a browser tab already bound with get_browser_state, browser_pointer action:\"drag\" drives the same gesture through CDP with page refs. The result is not driver-verified: confirm with a screenshot or the accessibility tree, and only if nothing changed retry with delivery_mode:\"foreground\", which briefly fronts the window and moves the physical pointer.",
"destructive": true,
"idempotent": false,
"input_schema": {