Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): make the macOS background drag refusal explicit (#4533)

ddupont61ec8ac1d80d
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
index 4da57608b6..8cd5423ed0 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/pointer.mdx
@@ -41,12 +41,14 @@ Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbin
When `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.
+macOS has no background drag: a window-scoped drag needs delivery_mode:"foreground" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:"foreground" the call is refused with background_unavailable and nothing is sent.
+
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="drag--macos--button"></span>`button` | `"left" \| "right" \| "middle"` | | Mouse button used for the drag. Default: left. |
-| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target; no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost; the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. |
+| <span id="drag--macos--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Window-scoped drag on macOS supports only "foreground": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default "background" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field. |
| <span id="drag--macos--duration_ms"></span>`duration_ms` | `integer` | | Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500. Range: 0 to 10000. |
| <span id="drag--macos--from_x"></span>`from_x` | `number` | required | Drag-start X in window-local screenshot pixels. Top-left origin. |
| <span id="drag--macos--from_y"></span>`from_y` | `number` | required | Drag-start Y in window-local screenshot pixels. Top-left origin. |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index f813c95784..8feb3f15f6 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -109,7 +109,7 @@ is safe even for apps that normally foreground on media-load
| Enumerate an app's windows | `list_windows({pid})`: or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` |
| Move or resize one exact window | `set_window_frame({pid, window_id, x, y, width, height})` | `osascript` position/size writes or title-bar dragging |
| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |
-| Drag / drag-and-drop / marquee select | `drag({pid, from_x, from_y, to_x, to_y})` (pixel-only: macOS AX has no semantic drag) | `cliclick dd:`, `osascript drag` |
+| Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y, delivery_mode:"foreground"})` (pixel-only and foreground-only: macOS AX has no semantic drag, and there is no background drag) | `cliclick dd:`, `osascript drag` |
| Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` |
| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` |
| Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |
@@ -300,6 +300,13 @@ surface that only accepts events while frontmost (the canvas/viewport/game
case below). Unmodified `element_token` (AX) actions remain background-capable
and hold the no-foreground contract without the flag.
+`drag` is the exception: macOS has no background drag. A window-scoped
+`drag` needs `delivery_mode:"foreground"` and `window_id`; without them it
+refuses with `background_unavailable` and sends nothing. The foreground drag
+fronts the exact window, moves the physical pointer along the path, and then
+restores the prior frontmost app. Use it only when a drag is the task, and
+expect the hardware pointer to move.
+
A foreground window-scoped **pixel** `click`, `double_click`, or
`right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driver
activates the exact window (refusing with `foreground_unavailable` when that
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index 37dd9da6bb..e1e37b0f22 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -35,6 +35,30 @@ impl DragTool {
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+const DELIVERY_MODE_DESCRIPTION: &str = "Window-scoped drag on macOS supports only \"foreground\": \
+ briefly front the exact window (window_id required), drive the physical pointer through the \
+ gesture, then restore the prior frontmost app. The default \"background\" is refused with \
+ background_unavailable because macOS has no background drag route. Desktop scope ignores this field.";
+
+const BACKGROUND_DRAG_UNAVAILABLE: &str = "Background drag is unavailable on macOS: a drag is \
+ delivered as real pointer events to the frontmost window, so it cannot run without fronting \
+ the target. Nothing was sent. Retry with delivery_mode:\"foreground\" and window_id; the \
+ driver briefly fronts that window, moves the pointer along the path, and restores the prior \
+ frontmost app.";
+
+fn background_drag_refusal() -> ToolResult {
+ cua_driver_core::delivery::background_unavailable_result(
+ BACKGROUND_DRAG_UNAVAILABLE,
+ "background_unavailable",
+ "macOS has no background drag route; drag moves the physical pointer",
+ serde_json::json!({
+ "effect": "refused",
+ "event_kind": "mouse_drag",
+ "requires": ["delivery_mode:foreground", "window_id"],
+ }),
+ )
+}
+
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "drag".into(),
@@ -50,7 +74,12 @@ fn def() -> &'static ToolDef {
slower, more human drags; decrease for snap gestures.\n\n\
`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\n\
When `from_zoom` is true, coordinates are in the last zoom image for this \
- pid; the driver maps them back to window coordinates before dispatching."
+ pid; the driver maps them back to window coordinates before dispatching.\n\n\
+ macOS has no background drag: a window-scoped drag needs \
+ delivery_mode:\"foreground\" and window_id. It briefly fronts the exact \
+ window, moves the physical pointer along the path, then restores the prior \
+ frontmost app. Without delivery_mode:\"foreground\" the call is refused with \
+ background_unavailable and nothing is sent."
.into(),
input_schema: serde_json::json!({
"type": "object",
@@ -93,7 +122,7 @@ fn def() -> &'static ToolDef {
"description": "When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates."
},
"scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates." },
- "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
+ "delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema_with(DELIVERY_MODE_DESCRIPTION)
},
"additionalProperties": false
}),
@@ -189,11 +218,7 @@ impl Tool for DragTool {
// uses. Requires a window_id to have a window to front.
let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());
if !delivery_mode.is_foreground() {
- return ToolResult::error(
- "Background drag is unavailable on macOS; use delivery_mode:\"foreground\"."
- .to_owned(),
- )
- .with_structured(serde_json::json!({ "code": "background_unavailable" }));
+ return background_drag_refusal();
}
// Coerce integer or float from JSON for coordinate fields.
let coerce = |key: &str| -> Option<f64> {
@@ -447,3 +472,44 @@ impl Tool for DragTool {
}
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[tokio::test]
+ async fn background_drag_refusal_is_structured_and_names_the_retry() {
+ let tool = DragTool::new(Arc::new(ToolState::new(false, false, None)));
+ for delivery_mode in [None, Some("background")] {
+ let mut args = serde_json::json!({
+ "pid": 4242,
+ "window_id": 7,
+ "from_x": 1, "from_y": 2, "to_x": 3, "to_y": 4,
+ });
+ if let Some(mode) = delivery_mode {
+ args["delivery_mode"] = serde_json::json!(mode);
+ }
+ let result = tool.invoke(args).await;
+ assert_eq!(result.is_error, Some(true));
+ let structured = result.structured_content.expect("structured refusal");
+ assert_eq!(structured["code"], "background_unavailable");
+ assert_eq!(structured["effect"], "refused");
+ assert_eq!(structured["escalation"]["recommended"], "foreground");
+ assert_eq!(
+ structured["suggestion"],
+ cua_driver_core::delivery::FOREGROUND_RETRY_SUGGESTION
+ );
+ }
+ }
+
+ #[test]
+ fn schema_says_macos_drag_is_foreground_only() {
+ let def = def();
+ assert!(def.description.contains("macOS has no background drag"));
+ let mode = &def.input_schema["properties"]["delivery_mode"]["description"];
+ assert!(mode
+ .as_str()
+ .unwrap()
+ .contains("supports only \"foreground\""));
+ }
+}
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 595ea950d6..05ac7446d1 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -721,7 +721,7 @@
"read_only": false
},
{
- "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.",
+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nmacOS has no background drag: a window-scoped drag needs delivery_mode:\"foreground\" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:\"foreground\" the call is refused with background_unavailable and nothing is sent.",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -737,7 +737,7 @@
"type": "string"
},
"delivery_mode": {
- "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "description": "Window-scoped drag on macOS supports only \"foreground\": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default \"background\" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field.",
"enum": [
"background",
"foreground"