Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): model Linux dialog delivery

trycua-release[bot]31b4f85755da
diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
index 039c7622db..8d307f718d 100644
--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
@@ -284,7 +284,11 @@ Only an accepted prompt may include `prompt_text`. This tool does not handle
browser permission prompts, extension UI, native sheets, or file pickers.
Creating a Chromium native modal can activate its browser window; once you
restore the intended occlusion, inspecting and resolving that exact dialog do
-not activate it again.
+not activate it again on Windows and macOS. Resolution defaults to
+`delivery_mode:"background"`. Linux Chromium cannot resolve the native modal
+while preserving background posture, so the driver returns
+`browser_input_trust_unavailable` before dispatch. Retry with
+`delivery_mode:"foreground"` only when foreground activation is acceptable.
## Assign files without a picker
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 2e3d9be383..726273d5ef 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -830,11 +830,12 @@ Type text into an exactly-bound tab via the Input domain. mode="insert_text" (de
### `browser_dialog`
-Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id.
+Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.
**Arguments:**
- `action` (string, required)
+- `delivery_mode` (string, optional): Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only.
- `dialog_id` (string, optional): Opaque current dialog generation returned by action=inspect.
- `prompt_text` (string, optional): Sensitive response text, valid only when accepting a prompt dialog.
- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.
diff --git a/docs/content/docs/reference/cua-driver/platform-support.mdx b/docs/content/docs/reference/cua-driver/platform-support.mdx
index a57aefd7a3..24398dcbd7 100644
--- a/docs/content/docs/reference/cua-driver/platform-support.mdx
+++ b/docs/content/docs/reference/cua-driver/platform-support.mdx
@@ -40,7 +40,7 @@ harnesses; an unlisted trust class or host shape is not implied.
| --- | --- | --- |
| Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse |
| Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |
-| Standalone Chrome on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |
+| Standalone Chrome and Edge on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, foreground JavaScript-dialog resolution, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input and background JavaScript-dialog resolution return `browser_input_trust_unavailable` before dispatch |
| Electron on Windows, macOS, X11, and validated Sway | Typed mutation while one proven native window maps to one CDP page | A second page or native window invalidates the bounded exact route |
| Tauri, WKWebView, WebKitGTK, and common split-process WebView2 | Browser identity plus a side-effect-free structured refusal | No exact native-host-to-engine relationship is currently proven |
| Safari and Firefox | Native discovery and native AX/PX fallbacks | No typed browser mutation engine is currently advertised |
diff --git a/libs/cua-driver/docs/browser-tool-implementation-journal.md b/libs/cua-driver/docs/browser-tool-implementation-journal.md
index f46b7f345f..836a118187 100644
--- a/libs/cua-driver/docs/browser-tool-implementation-journal.md
+++ b/libs/cua-driver/docs/browser-tool-implementation-journal.md
@@ -738,13 +738,17 @@ browser-wide setting, subscribe before triggering, correlate exact frame and
opaque id events, restore defaults on every outcome, and remove only a proven
direct partial file after failure.
-The real-browser harness adds separate background rows for dialogs, upload,
+The real-browser harness adds separate rows for dialogs, upload,
hover/right-click/double-click/scroll/drag, and downloads. A fixed loopback
-attachment keeps download evidence deterministic. Every row uses the
+attachment keeps download evidence deterministic. Background rows use the
foreground sentinel and fixture-owned state rather than treating a CDP reply
-as delivery. Trajectory recording redacts prompt text, upload paths, download
-directories, and internal approval markers; product telemetry retains only
-fixed operation and refusal categories.
+as delivery. Linux Chromium cannot resolve a native JavaScript modal without
+changing foreground posture, so its matrix records a no-dispatch background
+refusal and a separate explicit foreground delivery row. Windows and macOS
+retain strict background dialog resolution. Trajectory recording redacts
+prompt text, upload paths, download directories, and internal approval
+markers; product telemetry retains only fixed operation and refusal
+categories.
The dialog design adapts the exact-target, no-replay security model introduced
in PR #2166. Its contributor remains credited through a parsed
diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
index 20a6197d78..21592797a1 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
@@ -267,7 +267,11 @@ returned opaque `dialog_id`. A prompt response is allowed only with
`action:"accept"` on a current prompt. Browser permission UI and native dialogs
remain outside this tool. Creating Chromium's native modal can activate the
browser; after the caller restores occlusion, inspecting and resolving the
-exact page-owned dialog do not require another activation.
+exact page-owned dialog do not require another activation on Windows and
+macOS. Resolution defaults to `delivery_mode:"background"`. Linux Chromium
+cannot resolve its native modal while preserving background posture, so the
+driver refuses that mode before dispatch; retry explicitly with
+`delivery_mode:"foreground"` when foreground activation is acceptable.
### File inputs
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
index 4b8a819597..fcbe2bcad3 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
@@ -1422,7 +1422,7 @@ impl BrowserDialogTool {
Self {
def: ToolDef {
name: "browser_dialog".into(),
- description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id.".into(),
+ description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.".into(),
input_schema: json!({
"type": "object",
"properties": {
@@ -1431,7 +1431,13 @@ impl BrowserDialogTool {
"session": schema_session(),
"action": { "type": "string", "enum": ["inspect", "accept", "dismiss"] },
"dialog_id": { "type": "string", "description": "Opaque current dialog generation returned by action=inspect." },
- "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." }
+ "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." },
+ "delivery_mode": {
+ "type": "string",
+ "enum": ["background", "foreground"],
+ "default": "background",
+ "description": "Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only."
+ }
},
"required": ["target_id", "tab_id", "action"],
"additionalProperties": true
@@ -1464,6 +1470,12 @@ impl Tool for BrowserDialogTool {
if !matches!(action.as_str(), "inspect" | "accept" | "dismiss") {
return ToolResult::error("action must be inspect, accept, or dismiss");
}
+ let delivery_mode = args
+ .opt_str("delivery_mode")
+ .unwrap_or_else(|| "background".into());
+ if !matches!(delivery_mode.as_str(), "background" | "foreground") {
+ return ToolResult::error("delivery_mode must be background or foreground");
+ }
let session = match require_explicit_session(&args) {
Ok(session) => session,
Err(error) => return error,
@@ -1484,12 +1496,18 @@ impl Tool for BrowserDialogTool {
Ok(validated) => validated,
Err(refusal) => return refusal.to_tool_result(),
};
+ if cfg!(target_os = "linux") && action != "inspect" && delivery_mode == "background" {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ "Chromium's native JavaScript modal cannot be resolved on Linux without changing foreground posture; retry with delivery_mode=\"foreground\"",
+ )
+ .with_detail(json!({ "supported_delivery_mode": "foreground" }))
+ .to_tool_result();
+ }
let conn = &validated.conn;
let cdp_session = validated.cdp_session.as_str();
let cdp_target_id = validated.tab.cdp_target_id.as_str();
- if conn.dialog_state(cdp_target_id).is_none()
- && !conn.has_dialog_session(cdp_target_id)
- {
+ if conn.dialog_state(cdp_target_id).is_none() && !conn.has_dialog_session(cdp_target_id) {
// Register before Page.enable so an opening event delivered before
// the command reply is still attributed to the exact target.
conn.register_dialog_session(cdp_session, cdp_target_id);
@@ -1908,6 +1926,12 @@ mod tests {
assert!(!prepare_properties.contains_key("allow_restart"));
assert!(!prepare_properties.contains_key(MCP_HOST_APPROVAL_ARG));
+ let dialog = BrowserDialogTool::new(e.clone());
+ assert_eq!(
+ dialog.def().input_schema["properties"]["delivery_mode"]["default"],
+ "background"
+ );
+
for (def, name) in [
(
BrowserPrepareTool::new(e.clone()).def().clone(),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
index 7f700e9b64..43701e107b 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
@@ -734,13 +734,16 @@ fn wait_for_exact_browser_binding(
"session": session,
}),
);
- if state.structured()["binding_quality"] == "exact" {
+ let has_unique_active_tab = state.structured()["tabs"]
+ .as_array()
+ .is_some_and(|tabs| tabs.iter().filter(|tab| tab["active"] == true).count() == 1);
+ if state.structured()["binding_quality"] == "exact" && has_unique_active_tab {
return Some((window_id, state));
}
}
if Instant::now() >= deadline {
eprintln!(
- "no exact browser window binding for prepared pid {pid}; last windows={}",
+ "no exact browser window binding with one active tab for prepared pid {pid}; last windows={}",
windows.raw
);
return None;
@@ -1031,6 +1034,20 @@ fn refusal_case(browser: &str, action: &str, code: RefusalCode) -> CaseSpec {
case(browser, action).expecting_refusal(vec![code])
}
+fn foreground_page_case(browser: &str, action: &str) -> CaseSpec {
+ CaseSpec::delivered(
+ format!("{}-{browser}-standalone-{action}", std::env::consts::OS),
+ browser,
+ "standalone-chromium",
+ action,
+ Targeting::Page,
+ Delivery::Foreground,
+ Scope::Window,
+ DriverRoute::Cdp,
+ vec![OracleKind::FixtureState],
+ )
+}
+
fn run_with_background_oracles(
fixture: &mut BrowserFixture,
action: impl FnOnce(&mut BrowserFixture) -> Observation,
@@ -2257,7 +2274,12 @@ fn run_same_title_tabs(spec: &BrowserSpec) {
fn run_dialogs(spec: &BrowserSpec) {
let scenario = format!("{}-{}-standalone-dialogs", std::env::consts::OS, spec.name);
- execute_case(case(&spec.name, "browser_dialogs"), |evidence| {
+ let spec_case = if cfg!(target_os = "linux") {
+ foreground_page_case(&spec.name, "browser_dialogs")
+ } else {
+ case(&spec.name, "browser_dialogs")
+ };
+ execute_case(spec_case, |evidence| {
let mut fixture =
launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
*evidence = recording_evidence(fixture.driver.recording_dir());
@@ -2274,7 +2296,11 @@ fn run_dialogs(spec: &BrowserSpec) {
);
assert_eq!(primed.structured()["present"], false, "{}", primed.raw);
- let sentinel = ForegroundSentinel::launch(&mut fixture.driver);
+ let sentinel = if cfg!(target_os = "linux") {
+ None
+ } else {
+ Some(ForegroundSentinel::launch(&mut fixture.driver))
+ };
let target = TargetWindow {
pid: fixture.pid,
native_id: fixture.window_id,
@@ -2320,11 +2346,60 @@ fn run_dialogs(spec: &BrowserSpec) {
);
assert_eq!(clicked.structured()["status"], "ok", "{}", clicked.raw);
thread::sleep(Duration::from_millis(100));
+
+ if cfg!(target_os = "linux") {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let current = loop {
+ let current = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "inspect",
+ "session": session,
+ }),
+ );
+ if current.structured()["present"] == true {
+ break current;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "{kind} dialog was not observed: {}",
+ current.raw
+ );
+ thread::sleep(Duration::from_millis(25));
+ };
+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);
+ let dialog_id = current.structured()["dialog_id"]
+ .as_str()
+ .expect("opaque dialog id")
+ .to_owned();
+ let mut args = serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": action,
+ "dialog_id": dialog_id,
+ "delivery_mode": "foreground",
+ "session": session,
+ });
+ if let Some(prompt_text) = prompt_text {
+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());
+ }
+ let handled = fixture.driver.call("browser_dialog", args);
+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);
+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);
+ continue;
+ }
+
sentinel
+ .as_ref()
+ .expect("background dialog sentinel")
.prepare_background_observation(&mut fixture.driver, target)
.expect("occlude the open JavaScript dialog");
let (_, passed) = sentinel
+ .as_ref()
+ .expect("background dialog sentinel")
.observe_background(target, || {
let deadline = Instant::now() + Duration::from_secs(5);
let current = loop {
@@ -2375,6 +2450,55 @@ fn run_dialogs(spec: &BrowserSpec) {
});
}
+#[cfg(target_os = "linux")]
+fn run_dialog_background_refusal(spec: &BrowserSpec) {
+ let scenario = format!(
+ "{}-{}-standalone-dialog-background-refusal",
+ std::env::consts::OS,
+ spec.name
+ );
+ execute_case(
+ refusal_case(
+ &spec.name,
+ "browser_dialogs_background",
+ RefusalCode::BrowserInputTrustUnavailable,
+ ),
+ |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-dialog-refusal-{}", fixture.pid);
+ let (target_id, tab_id, _) = bind(fixture, &session);
+ let refused = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "accept",
+ "dialog_id": "dialog-unavailable",
+ "session": session,
+ }),
+ );
+ assert_eq!(refused.structured()["status"], "refused", "{}", refused.raw);
+ assert_eq!(
+ refused.structured()["refusal"]["code"],
+ "browser_input_trust_unavailable",
+ "{}",
+ refused.raw
+ );
+ wait_for_text(&fixture.server, "standalone-dialog-state", "dialog=none");
+ Observation::refused(
+ RefusalCode::BrowserInputTrustUnavailable,
+ vec![OracleKind::FixtureState],
+ refused.text(),
+ Evidence::default(),
+ )
+ })
+ },
+ );
+}
+
fn run_upload(spec: &BrowserSpec) {
let scenario = format!("{}-{}-standalone-upload", std::env::consts::OS, spec.name);
execute_case(case(&spec.name, "browser_file_upload"), |evidence| {
@@ -2684,6 +2808,11 @@ standalone_browser_test!(standalone_browser_frames, run_frame_roundtrip);
standalone_browser_test!(standalone_browser_multi_tab, run_multi_tab);
standalone_browser_test!(standalone_browser_same_title_tabs, run_same_title_tabs);
standalone_browser_test!(standalone_browser_dialogs, run_dialogs);
+#[cfg(target_os = "linux")]
+standalone_browser_test!(
+ standalone_browser_dialog_background_refusal,
+ run_dialog_background_refusal
+);
standalone_browser_test!(standalone_browser_upload, run_upload);
standalone_browser_test!(standalone_browser_pointer_actions, run_pointer_actions);
standalone_browser_test!(standalone_browser_download, run_download);
diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.sh
index abbd056794..3289edcd6e 100755
--- a/scripts/ci/run-rust-standalone-browser-e2e.sh
+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh
@@ -91,6 +91,11 @@ fi
tests=(
standalone_browser_background_type
standalone_browser_dialogs
+)
+if [[ "${HOST_OS}" == Linux ]]; then
+ tests+=(standalone_browser_dialog_background_refusal)
+fi
+tests+=(
standalone_browser_download
standalone_browser_existing_profile
standalone_browser_existing_profile_setup