Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): make the macOS background drag refusal explicit (#4533)
ddupont61ec8ac1d80d
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdxindex 4da57608b6..8cd5423ed0 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx@@ -41,12 +41,14 @@ Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbinWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.+macOS has no background drag: a window-scoped drag needs delivery_mode:"foreground" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:"foreground" the call is refused with background_unavailable and nothing is sent.+**macOS parameters**| Parameter | Type | Default | Description || --- | --- | --- | --- || <span id="drag--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button used for the drag. Default: left. |-| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target; no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost; the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. |+| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Window-scoped drag on macOS supports only "foreground": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default "background" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field. || <span id="drag--macos--duration_ms"></span>`duration_ms` | `integer` | | Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500. Range: 0 to 10000. || <span id="drag--macos--from_x"></span>`from_x` | `number` | required | Drag-start X in window-local screenshot pixels. Top-left origin. || <span id="drag--macos--from_y"></span>`from_y` | `number` | required | Drag-start Y in window-local screenshot pixels. Top-left origin. |diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex f813c95784..8feb3f15f6 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -109,7 +109,7 @@ is safe even for apps that normally foreground on media-load| Enumerate an app's windows | `list_windows({pid})`: or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` || Move or resize one exact window | `set_window_frame({pid, window_id, x, y, width, height})` | `osascript` position/size writes or title-bar dragging || Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |-| Drag / drag-and-drop / marquee select | `drag({pid, from_x, from_y, to_x, to_y})` (pixel-only: macOS AX has no semantic drag) | `cliclick dd:`, `osascript drag` |+| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` || Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` || Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` || Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |@@ -300,6 +300,13 @@ surface that only accepts events while frontmost (the canvas/viewport/gamecase below). Unmodified `element_token` (AX) actions remain background-capableand hold the no-foreground contract without the flag.+`drag` is the exception: macOS has no background drag. A window-scoped+`drag` needs `delivery_mode:"foreground"` and `window_id`; without them it+refuses with `background_unavailable` and sends nothing. The foreground drag+fronts the exact window, moves the physical pointer along the path, and then+restores the prior frontmost app. Use it only when a drag is the task, and+expect the hardware pointer to move.+A foreground window-scoped **pixel** `click`, `double_click`, or`right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driveractivates the exact window (refusing with `foreground_unavailable` when thatdiff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex 37dd9da6bb..e1e37b0f22 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -35,6 +35,30 @@ impl DragTool {static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS supports only \"foreground\": \+ briefly front the exact window (window_id required), drive the physical pointer through the \+ gesture, then restore the prior frontmost app. The default \"background\" is refused with \+ background_unavailable because macOS has no background drag route. Desktop scope ignores this field.";++const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: a drag is \+ delivered as real pointer events to the frontmost window, so it cannot run without fronting \+ the target. Nothing was sent. Retry with delivery_mode:\"foreground\" and window_id; the \+ driver briefly fronts that window, moves the pointer along the path, and restores the prior \+ frontmost app.";++fn background_drag_refusal() -> ToolResult {+ cua_driver_core::delivery::background_unavailable_result(+ BACKGROUND_DRAG_UNAVAILABLE,+ "background_unavailable",+ "macOS has no background drag route; drag moves the physical pointer",+ serde_json::json!({+ "effect": "refused",+ "event_kind": "mouse_drag",+ "requires": ["delivery_mode:foreground", "window_id"],+ }),+ )+}+fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "drag".into(),@@ -50,7 +74,12 @@ fn def() -> &'static ToolDef {slower, more human drags; decrease for snap gestures.\n\n\`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\n\When `from_zoom` is true, coordinates are in the last zoom image for this \- pid; the driver maps them back to window coordinates before dispatching."+ pid; the driver maps them back to window coordinates before dispatching.\n\n\+ macOS has no background drag: a window-scoped drag needs \+ delivery_mode:\"foreground\" and window_id. It briefly fronts the exact \+ window, moves the physical pointer along the path, then restores the prior \+ frontmost app. Without delivery_mode:\"foreground\" the call is refused with \+ background_unavailable and nothing is sent.".into(),input_schema: serde_json::json!({"type": "object",@@ -93,7 +122,7 @@ fn def() -> &'static ToolDef {"description": "When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates."},"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates." },- "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()+ "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema_with(DELIVERY_MODE_DESCRIPTION)},"additionalProperties": false}),@@ -189,11 +218,7 @@ impl Tool for DragTool {// uses. Requires a window_id to have a window to front.let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());if !delivery_mode.is_foreground() {- return ToolResult::error(- "Background drag is unavailable on macOS; use delivery_mode:\"foreground\"."- .to_owned(),- )- .with_structured(serde_json::json!({ "code": "background_unavailable" }));+ return background_drag_refusal();}// Coerce integer or float from JSON for coordinate fields.let coerce = |key: &str| -> Option<f64> {@@ -447,3 +472,44 @@ impl Tool for DragTool {}}}++#[cfg(test)]+mod tests {+ use super::*;++ #[tokio::test]+ async fn background_drag_refusal_is_structured_and_names_the_retry() {+ let tool = DragTool::new(Arc::new(ToolState::new(false, false, None)));+ for delivery_mode in [None, Some("background")] {+ let mut args = serde_json::json!({+ "pid": 4242,+ "window_id": 7,+ "from_x": 1, "from_y": 2, "to_x": 3, "to_y": 4,+ });+ if let Some(mode) = delivery_mode {+ args["delivery_mode"] = serde_json::json!(mode);+ }+ let result = tool.invoke(args).await;+ assert_eq!(result.is_error, Some(true));+ let structured = result.structured_content.expect("structured refusal");+ assert_eq!(structured["code"], "background_unavailable");+ assert_eq!(structured["effect"], "refused");+ assert_eq!(structured["escalation"]["recommended"], "foreground");+ assert_eq!(+ structured["suggestion"],+ cua_driver_core::delivery::FOREGROUND_RETRY_SUGGESTION+ );+ }+ }++ #[test]+ fn schema_says_macos_drag_is_foreground_only() {+ let def = def();+ assert!(def.description.contains("macOS has no background drag"));+ let mode = &def.input_schema["properties"]["delivery_mode"]["description"];+ assert!(mode+ .as_str()+ .unwrap()+ .contains("supports only \"foreground\""));+ }+}diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 595ea950d6..05ac7446d1 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -721,7 +721,7 @@"read_only": false},{- "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.",+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nmacOS has no background drag: a window-scoped drag needs delivery_mode:\"foreground\" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:\"foreground\" the call is refused with background_unavailable and nothing is sent.","destructive": true,"idempotent": false,"input_schema": {@@ -737,7 +737,7 @@"type": "string"},"delivery_mode": {- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",+ "description": "Window-scoped drag on macOS supports only \"foreground\": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default \"background\" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field.","enum": ["background","foreground"