Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): prefer foreground delivery in refusal guidance (#2623)
Francesco Bonacci6d97684fc4be
diff --git a/.github/workflows/ci-rust-windows.yml b/.github/workflows/ci-rust-windows.ymlindex 44beb7457a..9b720dc3ad 100644--- a/.github/workflows/ci-rust-windows.yml+++ b/.github/workflows/ci-rust-windows.yml@@ -64,6 +64,9 @@ jobs:# Compile every Rust target without executing desktop-dependent integration# tests; interactive behavior belongs in e2e-rust-windows.yml.run: cargo test -p cua-driver -p cua-driver-core -p cua-driver-sdk -p cua-driver-testkit -p platform-windows -p cua-driver-uia --all-targets --no-run --locked+ - name: Run Windows delivery guidance unit tests+ working-directory: libs/cua-driver/rust+ run: 'cargo test -p platform-windows input::delivery::tests:: --lib --locked'- name: Run transport-independent authorization and SDK testsworking-directory: libs/cua-driver/rustrun: |diff --git a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx b/docs/content/docs/concepts/capture-and-delivery-modalities.mdxindex 8d55bf75ce..d51a89f307 100644--- a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx+++ b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx@@ -37,10 +37,12 @@ Set `delivery_mode` per call on the input family (`click`, `double_click`, `righ| `delivery_mode` | Behavior ||---|---|| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched when the target surface supports background delivery. See [Best-effort background](/concepts/the-no-foreground-contract). |-| `foreground` | The target is briefly fronted (pair with `bring_to_front` to avoid a per-call flash), input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). |+| `foreground` | The target is briefly fronted for that action, input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). |Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) address an element instead of the focused window, so they hold the background path without any `delivery_mode` flag. The delivery axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first.+`bring_to_front` is not a normal escalation step. Reserve it for focus-proxy surfaces that must stay foreground across multiple calls, such as a remote desktop session. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:"foreground"`.+### 4. Capture scope: what coordinate space the action targetsSet with `set_config capture_scope=…`.diff --git a/docs/content/docs/reference/cua-driver/contracts.mdx b/docs/content/docs/reference/cua-driver/contracts.mdxindex 5f8ac04559..6b19100028 100644--- a/docs/content/docs/reference/cua-driver/contracts.mdx+++ b/docs/content/docs/reference/cua-driver/contracts.mdx@@ -127,7 +127,8 @@ The rejected combinations are enforced. Desktop scope is foreground and pixel-on|---|---|---|---|| `get_window_state` returns both tree + screenshot (element ax / px actions) | ✅ | ✅ | ✅ || `delivery_mode: "background"` (best-effort background) | ✅ | ✅ | ✅ for semantic AT-SPI actions on X11 and Wayland; native-Wayland raw keyboard input remains limited |-| `delivery_mode: "foreground"` + `bring_to_front` | ✅ explicit activation | ✅ explicit activation (input is already background-capable for most surfaces; activation is for focus-proxy surfaces like RDP) | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained |+| `delivery_mode: "foreground"` (action-scoped activation and restore) | ✅ | ✅ | ✅ X11; Wayland activation is compositor-constrained |+| `bring_to_front` (persistent activation for focus-proxy surfaces) | ✅ | ✅, chiefly for RDP | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained || `get_desktop_state` (desktop capture) | ✅ | ✅ | ✅ || Window-less desktop click (`click{x,y}`, no `pid`) | ✅ | rolling out | rolling out |diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.mdindex 0d04715e02..2d6d8c7498 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md@@ -31,7 +31,7 @@ strict no-foreground:| `delivery_mode` | Behavior on Windows || ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || `"background"` (DEFAULT) | Never fronts and **never raises/restacks** the target — macOS-aligned (mirrors CGEvent-to-pid). **Pixel clicks**: a UIA hit-test at the point first (accessibility-channel Invoke — works on UWP / WinUI3 / Win11 packaged apps, no flash); if that misses, coordinate-injected pen/touch, **but only when the target is the _visible_ window at that point**; PostMessage for plain Win32. It returns a structured `background_unavailable` error — rather than raising or fronting — when the target is **occluded** at the point, or the event kind is known-dropped (Chromium DOM mouse + key-combos, GTK buttons, VCL/LibreOffice accelerators, terminal / WPF text with no `element_index`). **No foreground swap and no z-order raise, ever.** |-| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. The explicit, agent-chosen rung where fronting IS allowed — required to reach occluded targets, Chromium DOM content, GTK buttons, VCL accelerators, WPF drag, terminals, and canvas / custom-drawn surfaces with no UIA peer. Implemented for **every** input tool — `type_text` (SendInput Unicode via `send_text_synthesized`) and `scroll` (SendInput wheel via `send_wheel_synthesized`) included. Flashes the target visible unless `bring_to_front` was called first. |+| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. The explicit, agent-chosen rung where fronting IS allowed — required to reach occluded targets, Chromium DOM content, GTK buttons, VCL accelerators, WPF drag, terminals, and canvas / custom-drawn surfaces with no UIA peer. Implemented for **every** input tool — `type_text` (SendInput Unicode via `send_text_synthesized`) and `scroll` (SendInput wheel via `send_wheel_synthesized`) included. The activation and restoration are scoped to that action. |> **macOS is the source of truth — `background` never alters the screen.**> Earlier Windows builds "cheated" in background with three tricks that this@@ -62,9 +62,9 @@ PostMessage** (UWP Calculator, Win11 Notepad, WinUI3 apps, etc.). cua-driverturns the pixel coord into a UIA Invoke at that point and deliversthrough the accessibility channel — no flash, no focus steal. Onlyescalate to `delivery_mode:"foreground"` when you actually see a-`background_unavailable` structured error, and only with the-`bring_to_front` flow described below so the agent pays the flash cost-once instead of per-call.+`background_unavailable` structured error. Retry only that action with+`delivery_mode:"foreground"`; cua-driver briefly activates the target, delivers+the input, and restores the previous foreground.Empirical: pixel-clicks via `delivery_mode:"background"` against the UWPCalculator on Win11 (Number-pad buttons + operators) consistently@@ -84,27 +84,31 @@ costlier path; only use it for surfaces with no UIA peer."target_class": "Chrome_WidgetWin_1","event_kind": "mouse_click","escalation": { "recommended": "foreground", "reason": "occluded / known-dropped event kind" },- "suggestion": "Either call bring_to_front then retry with delivery_mode:\"foreground\", or accept the foreground swap by setting delivery_mode:\"foreground\" directly."+ "suggestion": "Retry this action with delivery_mode:\"foreground\"."}}```The `escalation` field is the same machine-readable hint the actionresponses carry (see `SKILL.md` → behavior matrix). On Windows the-recommendation is `"foreground"` — the dropped event needs the fronting+recommendation is `"foreground"` because the dropped event needs the frontingrung. (Contrast macOS / X11, where a background px click can still landin the background, so there the hint is `px`.)-The recommended flow when an agent gets that error:+The normal flow when an agent gets that error:-1. `bring_to_front(pid)` — activates the target ONCE (visible flicker).-2. Subsequent input calls with `delivery_mode:"foreground"` deliver via- SendInput WITHOUT a per-call flash (the SetForegroundWindow swap- inside SendInput is a no-op because the target is already frontmost).-3. When done, leave the target as the user's foreground or call- `hotkey({pid: original_fg_pid, keys: ["alt","tab"]})` to put their- prior window back. **There is no "restore" tool** — you brought- the target forward deliberately; restoring is your responsibility.+1. Reissue only the refused action with `delivery_mode:"foreground"`.+2. cua-driver activates the target, delivers through SendInput, and restores+ the previous foreground.+3. Continue with `delivery_mode:"background"` for later actions unless they+ are also refused.++### Persistent focus-proxy exception++`bring_to_front` is not part of the normal input ladder. Use it only when a+focus-proxy surface must remain foreground across multiple calls, such as an+RDP or Windows App session, or when repeated action-scoped activation prevents+the remote surface from accepting input.The `bring_to_front` tool uses an `AttachThreadInput` trick to _attempt_the foreground swap even when the daemon isn't at UIAccess integrity (the@@ -273,11 +277,12 @@ modifier state updated, and PostMessage can't do that.backgrounded click delivers through UIA `Invoke` or `PostMessage`, andneither carries live keyboard state — so a `modifier` (Ctrl/Shift/etc.)passed alongside a `delivery_mode:"background"` click **is not honored**-on Windows. The `modifier` _param_ is part of the shared schema and is+on Windows. The `modifier` parameter is part of the shared schema and isaccepted everywhere; it only takes effect on the SendInput rung, i.e. a-`delivery_mode:"foreground"` (or `bring_to_front`-then-foreground) click,-where SendInput sets real modifier state. If you need a modifier-click on-Windows, escalate that one action to `foreground`.+`delivery_mode:"foreground"` click, where SendInput sets real modifier state.+If you need a modifier-click on Windows, escalate that one action to+`foreground`. Reserve `bring_to_front` for the persistent focus-proxy exception+described above.### Cross-platform schema residuals (Windows)diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rsindex 33f6e26e67..c01d0621aa 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs@@ -82,8 +82,8 @@ pub fn delivery_mode_schema() -> Value {error. 'foreground' is the explicit escalation: activate the target \(X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then \restore the prior active window — a brief focus swap unless the \- target was already active. Call bring_to_front first to avoid the \- flash. Matches the macOS / Windows delivery_mode surface.",+ target was already active. Matches the macOS / Windows delivery_mode \+ surface.",);v["default"] = serde_json::json!("background");v@@ -145,16 +145,19 @@ pub fn background_unavailable_error() -> cua_driver_core::protocol::ToolResult {let detail = reason.detail();cua_driver_core::protocol::ToolResult::error(format!(- "Background delivery is not available: {detail}. Either call bring_to_front \- then retry with delivery_mode:\"foreground\", or accept activation by \- setting delivery_mode:\"foreground\" directly."+ "Background delivery is not available: {detail}. Retry this action with \+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \+ the action and restore the previous foreground afterward.")).with_structured(serde_json::json!({"code": reason.code(),"detail": detail,- "suggestion":- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \- or set delivery_mode:\"foreground\" directly.",+ "suggestion": "Retry this action with delivery_mode:\"foreground\".",+ "escalation": {+ "recommended": "foreground",+ "reason": "background input is unavailable on this surface; retry this \+ action with delivery_mode:\"foreground\".",+ },}))}@@ -213,6 +216,10 @@ mod tests {.collect();assert_eq!(en, vec!["background", "foreground"]);assert_eq!(s["default"], "background");+ let description = s["description"]+ .as_str()+ .expect("delivery_mode description");+ assert!(!description.contains("bring_to_front"));}#[test]@@ -223,5 +230,24 @@ mod tests {r.structured_content.as_ref().unwrap()["code"],serde_json::json!("background_unavailable"));+ let text = match &r.content[0] {+ cua_driver_core::protocol::Content::Text { text, .. } => text,+ _ => panic!("expected text content"),+ };+ let structured = r.structured_content.as_ref().unwrap();+ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));+ assert!(!text.contains("bring_to_front"));+ assert_eq!(+ structured["suggestion"].as_str(),+ Some("Retry this action with delivery_mode:\"foreground\".")+ );+ assert_eq!(+ structured["escalation"]["recommended"].as_str(),+ Some("foreground")+ );+ assert!(!structured["escalation"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("bring_to_front"));}}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rsindex e5c641d0a0..7be6deb348 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs@@ -9,7 +9,7 @@//! - `background` (DEFAULT) — PostMessage / UIA path only, never fronts. If//! `would_be_silently_dropped(target, event_kind)` returns true, the tool//! returns a structured `background_unavailable` error so the caller can-//! `bring_to_front` then retry with `delivery_mode:"foreground"`. This is+//! retry that action with `delivery_mode:"foreground"`. This is//! the default because cua-driver's value proposition is that input never//! steals foreground — surfacing an honest error beats silently fronting.//!@@ -119,7 +119,7 @@ pub fn delivery_mode_schema() -> Value {the tool returns a structured background_unavailable error rather than \fronting. 'foreground' is the explicit escalation: a brief \SetForegroundWindow swap + SendInput, restoring the prior foreground \- afterward (call bring_to_front first to avoid the flash). \+ afterward. \IMPORTANT: 'background' is not a hint to weigh — it is the mandatory \first attempt. Do NOT pass 'foreground' preemptively because a target \'looks like' GTK/Chromium/Electron; the DRIVER decides when background \@@ -331,24 +331,22 @@ pub fn background_unavailable_error(let class = read_class_name(hwnd);let text = format!("Background delivery is not available for target window class \- '{class}' on this event kind ({}). Either call bring_to_front \- then retry with delivery_mode:\"foreground\", or accept the foreground \- swap directly by setting delivery_mode:\"foreground\".",+ '{class}' on this event kind ({}). Retry this action with \+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \+ the action and restore the previous foreground afterward.",kind.name());cua_driver_core::protocol::ToolResult::error(text).with_structured(serde_json::json!({"code": "background_unavailable","target_class": class,"event_kind": kind.name(),- "suggestion":- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \- or accept the foreground swap by setting delivery_mode:\"foreground\" directly.",+ "suggestion": "Retry this action with delivery_mode:\"foreground\".",// Windows analog of the macOS escalation signal: this surface drops// background input, so the deliberate next rung is foreground delivery."escalation": {"recommended": "foreground",- "reason": "background input is dropped by this surface — re-call \- delivery_mode:\"foreground\" (or bring_to_front first).",+ "reason": "background input is dropped by this surface; retry this \+ action with delivery_mode:\"foreground\".",},}))}@@ -374,7 +372,9 @@ pub fn background_unavailable_error_with_cause(};let text = format!("Background delivery is not available for target window class \- '{class}' on this event kind ({}): {cause}",+ '{class}' on this event kind ({}): {cause}. Retry this action with \+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \+ the action and restore the previous foreground afterward.",kind.name());cua_driver_core::protocol::ToolResult::error(text).with_structured(serde_json::json!({@@ -382,13 +382,11 @@ pub fn background_unavailable_error_with_cause("target_class": class,"event_kind": kind.name(),"cause": cause,- "suggestion":- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \- or accept the foreground swap by setting delivery_mode:\"foreground\" directly.",+ "suggestion": "Retry this action with delivery_mode:\"foreground\".","escalation": {"recommended": "foreground",- "reason": "background input could not be delivered by the coordinate actuator — \- re-call delivery_mode:\"foreground\" (or bring_to_front first).",+ "reason": "background input could not be delivered by the coordinate \+ actuator; retry this action with delivery_mode:\"foreground\".",},}))}@@ -457,6 +455,41 @@ mod tests {);}+ #[test]+ fn delivery_mode_schema_keeps_persistent_activation_out_of_the_input_ladder() {+ let schema = delivery_mode_schema();+ let description = schema["description"]+ .as_str()+ .expect("delivery_mode description");+ assert!(description.contains("Only THEN re-issue the same action with 'foreground'"));+ assert!(!description.contains("bring_to_front"));+ }++ #[test]+ fn background_unavailable_prefers_action_scoped_foreground_delivery() {+ let result = background_unavailable_error(0, EventKind::MouseClick);+ let structured = result.structured_content.as_ref().expect("structured");+ let text = match &result.content[0] {+ cua_driver_core::protocol::Content::Text { text, .. } => text,+ _ => panic!("expected text content"),+ };++ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));+ assert!(!text.contains("bring_to_front"));+ assert_eq!(+ structured["suggestion"].as_str(),+ Some("Retry this action with delivery_mode:\"foreground\".")+ );+ assert_eq!(+ structured["escalation"]["recommended"].as_str(),+ Some("foreground")+ );+ assert!(!structured["escalation"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("bring_to_front"));+ }+#[test]fn background_unavailable_with_cause_preserves_actuator_failure() {let result = background_unavailable_error_with_cause(@@ -478,6 +511,12 @@ mod tests {_ => panic!("expected text content"),};assert!(text.contains("occluded"), "result text lost cause: {text}");+ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));+ assert!(!text.contains("bring_to_front"));+ assert!(!structured["suggestion"]+ .as_str()+ .unwrap_or_default()+ .contains("bring_to_front"));let uipi = background_unavailable_error_with_cause(0,diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex ab81f25f48..701a22867e 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -5499,7 +5499,7 @@ fn winui3_uia_multi_invoke(/// island ignores posted `WM_*BUTTON`, and the pointer injector/// click-activates the frame. So we return the structured/// `background_unavailable` error (the honest result — the caller can retry-/// with `delivery_mode:"foreground"` or `bring_to_front`) instead of a silent+/// that action with `delivery_mode:"foreground"`) instead of a silent/// no-op that reports false success.////// Returns `None` for non-WinUI3 targets (caller falls through to its normal@@ -7780,17 +7780,13 @@ impl Tool for BringToFrontTool {name: "bring_to_front".into(),description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \the OS foreground. \n\n\- **This deliberately breaks the no-foreground contract.** Use only when an \- agent is about to drive a sequence of operations against a target whose input \- stack silently drops PostMessage (Chromium DOM content, GTK button widgets) \- and the agent wants to pay the foreground cost once instead of per call. \n\n\- Pairs with the `delivery_mode` field on input tools:\n\- 1. Try `click(..., delivery_mode:\"background\")`. If it returns \- `background_unavailable`, the target needs foreground delivery.\n\- 2. Call `bring_to_front(pid)` so the target is foreground.\n\- 3. Subsequent input calls with `delivery_mode:\"foreground\"` deliver via \- SendInput WITHOUT visible flashing -- the SetForegroundWindow swap \- inside SendInput is a no-op since target is already foreground.\n\n\+ **This deliberately breaks the no-foreground contract.** It is not part of \+ the normal input ladder. For an ordinary `background_unavailable` response, \+ retry only the refused action with `delivery_mode:\"foreground\"`; the input \+ tool performs its own activate, act, and restore sequence. Use \+ `bring_to_front` only for a focus-proxy surface that must remain foreground \+ across multiple calls, such as an RDP or Windows App session, or when repeated \+ action-scoped activation prevents the remote surface from accepting input. \n\n\Implementation uses the `AttachThreadInput` trick to bypass Windows' \foreground-lock when the daemon is not at UIAccess integrity. Returns \structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \