Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): prefer foreground delivery in refusal guidance (#2623)

Francesco Bonacci6d97684fc4be
diff --git a/.github/workflows/ci-rust-windows.yml b/.github/workflows/ci-rust-windows.yml
index 44beb7457a..9b720dc3ad 100644
--- a/.github/workflows/ci-rust-windows.yml
+++ b/.github/workflows/ci-rust-windows.yml
@@ -64,6 +64,9 @@ jobs:
# Compile every Rust target without executing desktop-dependent integration
# tests; interactive behavior belongs in e2e-rust-windows.yml.
run: cargo test -p cua-driver -p cua-driver-core -p cua-driver-sdk -p cua-driver-testkit -p platform-windows -p cua-driver-uia --all-targets --no-run --locked
+ - name: Run Windows delivery guidance unit tests
+ working-directory: libs/cua-driver/rust
+ run: 'cargo test -p platform-windows input::delivery::tests:: --lib --locked'
- name: Run transport-independent authorization and SDK tests
working-directory: libs/cua-driver/rust
run: |
diff --git a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
index 8d55bf75ce..d51a89f307 100644
--- a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
+++ b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
@@ -37,10 +37,12 @@ Set `delivery_mode` per call on the input family (`click`, `double_click`, `righ
| `delivery_mode` | Behavior |
|---|---|
| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched when the target surface supports background delivery. See [Best-effort background](/concepts/the-no-foreground-contract). |
-| `foreground` | The target is briefly fronted (pair with `bring_to_front` to avoid a per-call flash), input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). |
+| `foreground` | The target is briefly fronted for that action, input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). |
Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) address an element instead of the focused window, so they hold the background path without any `delivery_mode` flag. The delivery axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first.
+`bring_to_front` is not a normal escalation step. Reserve it for focus-proxy surfaces that must stay foreground across multiple calls, such as a remote desktop session. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:"foreground"`.
+
### 4. Capture scope: what coordinate space the action targets
Set with `set_config capture_scope=…`.
diff --git a/docs/content/docs/reference/cua-driver/contracts.mdx b/docs/content/docs/reference/cua-driver/contracts.mdx
index 5f8ac04559..6b19100028 100644
--- a/docs/content/docs/reference/cua-driver/contracts.mdx
+++ b/docs/content/docs/reference/cua-driver/contracts.mdx
@@ -127,7 +127,8 @@ The rejected combinations are enforced. Desktop scope is foreground and pixel-on
|---|---|---|---|
| `get_window_state` returns both tree + screenshot (element ax / px actions) | ✅ | ✅ | ✅ |
| `delivery_mode: "background"` (best-effort background) | ✅ | ✅ | ✅ for semantic AT-SPI actions on X11 and Wayland; native-Wayland raw keyboard input remains limited |
-| `delivery_mode: "foreground"` + `bring_to_front` | ✅ explicit activation | ✅ explicit activation (input is already background-capable for most surfaces; activation is for focus-proxy surfaces like RDP) | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained |
+| `delivery_mode: "foreground"` (action-scoped activation and restore) | ✅ | ✅ | ✅ X11; Wayland activation is compositor-constrained |
+| `bring_to_front` (persistent activation for focus-proxy surfaces) | ✅ | ✅, chiefly for RDP | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained |
| `get_desktop_state` (desktop capture) | ✅ | ✅ | ✅ |
| Window-less desktop click (`click{x,y}`, no `pid`) | ✅ | rolling out | rolling out |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index 0d04715e02..2d6d8c7498 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -31,7 +31,7 @@ strict no-foreground:
| `delivery_mode` | Behavior on Windows |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"background"` (DEFAULT) | Never fronts and **never raises/restacks** the target — macOS-aligned (mirrors CGEvent-to-pid). **Pixel clicks**: a UIA hit-test at the point first (accessibility-channel Invoke — works on UWP / WinUI3 / Win11 packaged apps, no flash); if that misses, coordinate-injected pen/touch, **but only when the target is the _visible_ window at that point**; PostMessage for plain Win32. It returns a structured `background_unavailable` error — rather than raising or fronting — when the target is **occluded** at the point, or the event kind is known-dropped (Chromium DOM mouse + key-combos, GTK buttons, VCL/LibreOffice accelerators, terminal / WPF text with no `element_index`). **No foreground swap and no z-order raise, ever.** |
-| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. The explicit, agent-chosen rung where fronting IS allowed — required to reach occluded targets, Chromium DOM content, GTK buttons, VCL accelerators, WPF drag, terminals, and canvas / custom-drawn surfaces with no UIA peer. Implemented for **every** input tool — `type_text` (SendInput Unicode via `send_text_synthesized`) and `scroll` (SendInput wheel via `send_wheel_synthesized`) included. Flashes the target visible unless `bring_to_front` was called first. |
+| `"foreground"` | SendInput with brief `SetForegroundWindow(target)` → restore. The explicit, agent-chosen rung where fronting IS allowed — required to reach occluded targets, Chromium DOM content, GTK buttons, VCL accelerators, WPF drag, terminals, and canvas / custom-drawn surfaces with no UIA peer. Implemented for **every** input tool — `type_text` (SendInput Unicode via `send_text_synthesized`) and `scroll` (SendInput wheel via `send_wheel_synthesized`) included. The activation and restoration are scoped to that action. |
> **macOS is the source of truth — `background` never alters the screen.**
> Earlier Windows builds "cheated" in background with three tricks that this
@@ -62,9 +62,9 @@ PostMessage** (UWP Calculator, Win11 Notepad, WinUI3 apps, etc.). cua-driver
turns the pixel coord into a UIA Invoke at that point and delivers
through the accessibility channel — no flash, no focus steal. Only
escalate to `delivery_mode:"foreground"` when you actually see a
-`background_unavailable` structured error, and only with the
-`bring_to_front` flow described below so the agent pays the flash cost
-once instead of per-call.
+`background_unavailable` structured error. Retry only that action with
+`delivery_mode:"foreground"`; cua-driver briefly activates the target, delivers
+the input, and restores the previous foreground.
Empirical: pixel-clicks via `delivery_mode:"background"` against the UWP
Calculator on Win11 (Number-pad buttons + operators) consistently
@@ -84,27 +84,31 @@ costlier path; only use it for surfaces with no UIA peer.
"target_class": "Chrome_WidgetWin_1",
"event_kind": "mouse_click",
"escalation": { "recommended": "foreground", "reason": "occluded / known-dropped event kind" },
- "suggestion": "Either call bring_to_front then retry with delivery_mode:\"foreground\", or accept the foreground swap by setting delivery_mode:\"foreground\" directly."
+ "suggestion": "Retry this action with delivery_mode:\"foreground\"."
}
}
```
The `escalation` field is the same machine-readable hint the action
responses carry (see `SKILL.md` → behavior matrix). On Windows the
-recommendation is `"foreground"` — the dropped event needs the fronting
+recommendation is `"foreground"` because the dropped event needs the fronting
rung. (Contrast macOS / X11, where a background px click can still land
in the background, so there the hint is `px`.)
-The recommended flow when an agent gets that error:
+The normal flow when an agent gets that error:
-1. `bring_to_front(pid)` — activates the target ONCE (visible flicker).
-2. Subsequent input calls with `delivery_mode:"foreground"` deliver via
- SendInput WITHOUT a per-call flash (the SetForegroundWindow swap
- inside SendInput is a no-op because the target is already frontmost).
-3. When done, leave the target as the user's foreground or call
- `hotkey({pid: original_fg_pid, keys: ["alt","tab"]})` to put their
- prior window back. **There is no "restore" tool** — you brought
- the target forward deliberately; restoring is your responsibility.
+1. Reissue only the refused action with `delivery_mode:"foreground"`.
+2. cua-driver activates the target, delivers through SendInput, and restores
+ the previous foreground.
+3. Continue with `delivery_mode:"background"` for later actions unless they
+ are also refused.
+
+### Persistent focus-proxy exception
+
+`bring_to_front` is not part of the normal input ladder. Use it only when a
+focus-proxy surface must remain foreground across multiple calls, such as an
+RDP or Windows App session, or when repeated action-scoped activation prevents
+the remote surface from accepting input.
The `bring_to_front` tool uses an `AttachThreadInput` trick to _attempt_
the foreground swap even when the daemon isn't at UIAccess integrity (the
@@ -273,11 +277,12 @@ modifier state updated, and PostMessage can't do that.
backgrounded click delivers through UIA `Invoke` or `PostMessage`, and
neither carries live keyboard state — so a `modifier` (Ctrl/Shift/etc.)
passed alongside a `delivery_mode:"background"` click **is not honored**
-on Windows. The `modifier` _param_ is part of the shared schema and is
+on Windows. The `modifier` parameter is part of the shared schema and is
accepted everywhere; it only takes effect on the SendInput rung, i.e. a
-`delivery_mode:"foreground"` (or `bring_to_front`-then-foreground) click,
-where SendInput sets real modifier state. If you need a modifier-click on
-Windows, escalate that one action to `foreground`.
+`delivery_mode:"foreground"` click, where SendInput sets real modifier state.
+If you need a modifier-click on Windows, escalate that one action to
+`foreground`. Reserve `bring_to_front` for the persistent focus-proxy exception
+described above.
### Cross-platform schema residuals (Windows)
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs
index 33f6e26e67..c01d0621aa 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs
@@ -82,8 +82,8 @@ pub fn delivery_mode_schema() -> Value {
error. 'foreground' is the explicit escalation: activate the target \
(X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then \
restore the prior active window — a brief focus swap unless the \
- target was already active. Call bring_to_front first to avoid the \
- flash. Matches the macOS / Windows delivery_mode surface.",
+ target was already active. Matches the macOS / Windows delivery_mode \
+ surface.",
);
v["default"] = serde_json::json!("background");
v
@@ -145,16 +145,19 @@ pub fn background_unavailable_error(
) -> cua_driver_core::protocol::ToolResult {
let detail = reason.detail();
cua_driver_core::protocol::ToolResult::error(format!(
- "Background delivery is not available: {detail}. Either call bring_to_front \
- then retry with delivery_mode:\"foreground\", or accept activation by \
- setting delivery_mode:\"foreground\" directly."
+ "Background delivery is not available: {detail}. Retry this action with \
+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \
+ the action and restore the previous foreground afterward."
))
.with_structured(serde_json::json!({
"code": reason.code(),
"detail": detail,
- "suggestion":
- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \
- or set delivery_mode:\"foreground\" directly.",
+ "suggestion": "Retry this action with delivery_mode:\"foreground\".",
+ "escalation": {
+ "recommended": "foreground",
+ "reason": "background input is unavailable on this surface; retry this \
+ action with delivery_mode:\"foreground\".",
+ },
}))
}
@@ -213,6 +216,10 @@ mod tests {
.collect();
assert_eq!(en, vec!["background", "foreground"]);
assert_eq!(s["default"], "background");
+ let description = s["description"]
+ .as_str()
+ .expect("delivery_mode description");
+ assert!(!description.contains("bring_to_front"));
}
#[test]
@@ -223,5 +230,24 @@ mod tests {
r.structured_content.as_ref().unwrap()["code"],
serde_json::json!("background_unavailable")
);
+ let text = match &r.content[0] {
+ cua_driver_core::protocol::Content::Text { text, .. } => text,
+ _ => panic!("expected text content"),
+ };
+ let structured = r.structured_content.as_ref().unwrap();
+ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));
+ assert!(!text.contains("bring_to_front"));
+ assert_eq!(
+ structured["suggestion"].as_str(),
+ Some("Retry this action with delivery_mode:\"foreground\".")
+ );
+ assert_eq!(
+ structured["escalation"]["recommended"].as_str(),
+ Some("foreground")
+ );
+ assert!(!structured["escalation"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("bring_to_front"));
}
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs
index e5c641d0a0..7be6deb348 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/delivery.rs
@@ -9,7 +9,7 @@
//! - `background` (DEFAULT) — PostMessage / UIA path only, never fronts. If
//! `would_be_silently_dropped(target, event_kind)` returns true, the tool
//! returns a structured `background_unavailable` error so the caller can
-//! `bring_to_front` then retry with `delivery_mode:"foreground"`. This is
+//! retry that action with `delivery_mode:"foreground"`. This is
//! the default because cua-driver's value proposition is that input never
//! steals foreground — surfacing an honest error beats silently fronting.
//!
@@ -119,7 +119,7 @@ pub fn delivery_mode_schema() -> Value {
the tool returns a structured background_unavailable error rather than \
fronting. 'foreground' is the explicit escalation: a brief \
SetForegroundWindow swap + SendInput, restoring the prior foreground \
- afterward (call bring_to_front first to avoid the flash). \
+ afterward. \
IMPORTANT: 'background' is not a hint to weigh — it is the mandatory \
first attempt. Do NOT pass 'foreground' preemptively because a target \
'looks like' GTK/Chromium/Electron; the DRIVER decides when background \
@@ -331,24 +331,22 @@ pub fn background_unavailable_error(
let class = read_class_name(hwnd);
let text = format!(
"Background delivery is not available for target window class \
- '{class}' on this event kind ({}). Either call bring_to_front \
- then retry with delivery_mode:\"foreground\", or accept the foreground \
- swap directly by setting delivery_mode:\"foreground\".",
+ '{class}' on this event kind ({}). Retry this action with \
+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \
+ the action and restore the previous foreground afterward.",
kind.name()
);
cua_driver_core::protocol::ToolResult::error(text).with_structured(serde_json::json!({
"code": "background_unavailable",
"target_class": class,
"event_kind": kind.name(),
- "suggestion":
- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \
- or accept the foreground swap by setting delivery_mode:\"foreground\" directly.",
+ "suggestion": "Retry this action with delivery_mode:\"foreground\".",
// Windows analog of the macOS escalation signal: this surface drops
// background input, so the deliberate next rung is foreground delivery.
"escalation": {
"recommended": "foreground",
- "reason": "background input is dropped by this surface — re-call \
- delivery_mode:\"foreground\" (or bring_to_front first).",
+ "reason": "background input is dropped by this surface; retry this \
+ action with delivery_mode:\"foreground\".",
},
}))
}
@@ -374,7 +372,9 @@ pub fn background_unavailable_error_with_cause(
};
let text = format!(
"Background delivery is not available for target window class \
- '{class}' on this event kind ({}): {cause}",
+ '{class}' on this event kind ({}): {cause}. Retry this action with \
+ delivery_mode:\"foreground\"; Cua Driver will activate the target for \
+ the action and restore the previous foreground afterward.",
kind.name()
);
cua_driver_core::protocol::ToolResult::error(text).with_structured(serde_json::json!({
@@ -382,13 +382,11 @@ pub fn background_unavailable_error_with_cause(
"target_class": class,
"event_kind": kind.name(),
"cause": cause,
- "suggestion":
- "Either call bring_to_front then retry with delivery_mode:\"foreground\", \
- or accept the foreground swap by setting delivery_mode:\"foreground\" directly.",
+ "suggestion": "Retry this action with delivery_mode:\"foreground\".",
"escalation": {
"recommended": "foreground",
- "reason": "background input could not be delivered by the coordinate actuator — \
- re-call delivery_mode:\"foreground\" (or bring_to_front first).",
+ "reason": "background input could not be delivered by the coordinate \
+ actuator; retry this action with delivery_mode:\"foreground\".",
},
}))
}
@@ -457,6 +455,41 @@ mod tests {
);
}
+ #[test]
+ fn delivery_mode_schema_keeps_persistent_activation_out_of_the_input_ladder() {
+ let schema = delivery_mode_schema();
+ let description = schema["description"]
+ .as_str()
+ .expect("delivery_mode description");
+ assert!(description.contains("Only THEN re-issue the same action with 'foreground'"));
+ assert!(!description.contains("bring_to_front"));
+ }
+
+ #[test]
+ fn background_unavailable_prefers_action_scoped_foreground_delivery() {
+ let result = background_unavailable_error(0, EventKind::MouseClick);
+ let structured = result.structured_content.as_ref().expect("structured");
+ let text = match &result.content[0] {
+ cua_driver_core::protocol::Content::Text { text, .. } => text,
+ _ => panic!("expected text content"),
+ };
+
+ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));
+ assert!(!text.contains("bring_to_front"));
+ assert_eq!(
+ structured["suggestion"].as_str(),
+ Some("Retry this action with delivery_mode:\"foreground\".")
+ );
+ assert_eq!(
+ structured["escalation"]["recommended"].as_str(),
+ Some("foreground")
+ );
+ assert!(!structured["escalation"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("bring_to_front"));
+ }
+
#[test]
fn background_unavailable_with_cause_preserves_actuator_failure() {
let result = background_unavailable_error_with_cause(
@@ -478,6 +511,12 @@ mod tests {
_ => panic!("expected text content"),
};
assert!(text.contains("occluded"), "result text lost cause: {text}");
+ assert!(text.contains("Retry this action with delivery_mode:\"foreground\""));
+ assert!(!text.contains("bring_to_front"));
+ assert!(!structured["suggestion"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("bring_to_front"));
let uipi = background_unavailable_error_with_cause(
0,
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index ab81f25f48..701a22867e 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -5499,7 +5499,7 @@ fn winui3_uia_multi_invoke(
/// island ignores posted `WM_*BUTTON`, and the pointer injector
/// click-activates the frame. So we return the structured
/// `background_unavailable` error (the honest result — the caller can retry
-/// with `delivery_mode:"foreground"` or `bring_to_front`) instead of a silent
+/// that action with `delivery_mode:"foreground"`) instead of a silent
/// no-op that reports false success.
///
/// Returns `None` for non-WinUI3 targets (caller falls through to its normal
@@ -7780,17 +7780,13 @@ impl Tool for BringToFrontTool {
name: "bring_to_front".into(),
description: "Activate `pid`'s window (or `window_id` if specified) -- bring it to \
the OS foreground. \n\n\
- **This deliberately breaks the no-foreground contract.** Use only when an \
- agent is about to drive a sequence of operations against a target whose input \
- stack silently drops PostMessage (Chromium DOM content, GTK button widgets) \
- and the agent wants to pay the foreground cost once instead of per call. \n\n\
- Pairs with the `delivery_mode` field on input tools:\n\
- 1. Try `click(..., delivery_mode:\"background\")`. If it returns \
- `background_unavailable`, the target needs foreground delivery.\n\
- 2. Call `bring_to_front(pid)` so the target is foreground.\n\
- 3. Subsequent input calls with `delivery_mode:\"foreground\"` deliver via \
- SendInput WITHOUT visible flashing -- the SetForegroundWindow swap \
- inside SendInput is a no-op since target is already foreground.\n\n\
+ **This deliberately breaks the no-foreground contract.** It is not part of \
+ the normal input ladder. For an ordinary `background_unavailable` response, \
+ retry only the refused action with `delivery_mode:\"foreground\"`; the input \
+ tool performs its own activate, act, and restore sequence. Use \
+ `bring_to_front` only for a focus-proxy surface that must remain foreground \
+ across multiple calls, such as an RDP or Windows App session, or when repeated \
+ action-scoped activation prevents the remote surface from accepting input. \n\n\
Implementation uses the `AttachThreadInput` trick to bypass Windows' \
foreground-lock when the daemon is not at UIAccess integrity. Returns \
structured `{previous_fg_hwnd, now_fg_hwnd}` so callers can later restore. \