Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): model Linux dialog delivery
trycua-release[bot]31b4f85755da
diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdxindex 039c7622db..8d307f718d 100644--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx@@ -284,7 +284,11 @@ Only an accepted prompt may include `prompt_text`. This tool does not handlebrowser permission prompts, extension UI, native sheets, or file pickers.Creating a Chromium native modal can activate its browser window; once yourestore the intended occlusion, inspecting and resolving that exact dialog do-not activate it again.+not activate it again on Windows and macOS. Resolution defaults to+`delivery_mode:"background"`. Linux Chromium cannot resolve the native modal+while preserving background posture, so the driver returns+`browser_input_trust_unavailable` before dispatch. Retry with+`delivery_mode:"foreground"` only when foreground activation is acceptable.## Assign files without a pickerdiff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 2e3d9be383..726273d5ef 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -830,11 +830,12 @@ Type text into an exactly-bound tab via the Input domain. mode="insert_text" (de### `browser_dialog`-Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id.+Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.**Arguments:**- `action` (string, required)+- `delivery_mode` (string, optional): Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only.- `dialog_id` (string, optional): Opaque current dialog generation returned by action=inspect.- `prompt_text` (string, optional): Sensitive response text, valid only when accepting a prompt dialog.- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.diff --git a/docs/content/docs/reference/cua-driver/platform-support.mdx b/docs/content/docs/reference/cua-driver/platform-support.mdxindex a57aefd7a3..24398dcbd7 100644--- a/docs/content/docs/reference/cua-driver/platform-support.mdx+++ b/docs/content/docs/reference/cua-driver/platform-support.mdx@@ -40,7 +40,7 @@ harnesses; an unlisted trust class or host shape is not implied.| --- | --- | --- || Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse || Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |-| Standalone Chrome on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |+| Standalone Chrome and Edge on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, foreground JavaScript-dialog resolution, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input and background JavaScript-dialog resolution return `browser_input_trust_unavailable` before dispatch || Electron on Windows, macOS, X11, and validated Sway | Typed mutation while one proven native window maps to one CDP page | A second page or native window invalidates the bounded exact route || Tauri, WKWebView, WebKitGTK, and common split-process WebView2 | Browser identity plus a side-effect-free structured refusal | No exact native-host-to-engine relationship is currently proven || Safari and Firefox | Native discovery and native AX/PX fallbacks | No typed browser mutation engine is currently advertised |diff --git a/libs/cua-driver/docs/browser-tool-implementation-journal.md b/libs/cua-driver/docs/browser-tool-implementation-journal.mdindex f46b7f345f..836a118187 100644--- a/libs/cua-driver/docs/browser-tool-implementation-journal.md+++ b/libs/cua-driver/docs/browser-tool-implementation-journal.md@@ -738,13 +738,17 @@ browser-wide setting, subscribe before triggering, correlate exact frame andopaque id events, restore defaults on every outcome, and remove only a provendirect partial file after failure.-The real-browser harness adds separate background rows for dialogs, upload,+The real-browser harness adds separate rows for dialogs, upload,hover/right-click/double-click/scroll/drag, and downloads. A fixed loopback-attachment keeps download evidence deterministic. Every row uses the+attachment keeps download evidence deterministic. Background rows use theforeground sentinel and fixture-owned state rather than treating a CDP reply-as delivery. Trajectory recording redacts prompt text, upload paths, download-directories, and internal approval markers; product telemetry retains only-fixed operation and refusal categories.+as delivery. Linux Chromium cannot resolve a native JavaScript modal without+changing foreground posture, so its matrix records a no-dispatch background+refusal and a separate explicit foreground delivery row. Windows and macOS+retain strict background dialog resolution. Trajectory recording redacts+prompt text, upload paths, download directories, and internal approval+markers; product telemetry retains only fixed operation and refusal+categories.The dialog design adapts the exact-target, no-replay security model introducedin PR #2166. Its contributor remains credited through a parseddiff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.mdindex 20a6197d78..21592797a1 100644--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md@@ -267,7 +267,11 @@ returned opaque `dialog_id`. A prompt response is allowed only with`action:"accept"` on a current prompt. Browser permission UI and native dialogsremain outside this tool. Creating Chromium's native modal can activate thebrowser; after the caller restores occlusion, inspecting and resolving the-exact page-owned dialog do not require another activation.+exact page-owned dialog do not require another activation on Windows and+macOS. Resolution defaults to `delivery_mode:"background"`. Linux Chromium+cannot resolve its native modal while preserving background posture, so the+driver refuses that mode before dispatch; retry explicitly with+`delivery_mode:"foreground"` when foreground activation is acceptable.### File inputsdiff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rsindex 4b8a819597..fcbe2bcad3 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs@@ -1422,7 +1422,7 @@ impl BrowserDialogTool {Self {def: ToolDef {name: "browser_dialog".into(),- description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id.".into(),+ description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.".into(),input_schema: json!({"type": "object","properties": {@@ -1431,7 +1431,13 @@ impl BrowserDialogTool {"session": schema_session(),"action": { "type": "string", "enum": ["inspect", "accept", "dismiss"] },"dialog_id": { "type": "string", "description": "Opaque current dialog generation returned by action=inspect." },- "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." }+ "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." },+ "delivery_mode": {+ "type": "string",+ "enum": ["background", "foreground"],+ "default": "background",+ "description": "Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only."+ }},"required": ["target_id", "tab_id", "action"],"additionalProperties": true@@ -1464,6 +1470,12 @@ impl Tool for BrowserDialogTool {if !matches!(action.as_str(), "inspect" | "accept" | "dismiss") {return ToolResult::error("action must be inspect, accept, or dismiss");}+ let delivery_mode = args+ .opt_str("delivery_mode")+ .unwrap_or_else(|| "background".into());+ if !matches!(delivery_mode.as_str(), "background" | "foreground") {+ return ToolResult::error("delivery_mode must be background or foreground");+ }let session = match require_explicit_session(&args) {Ok(session) => session,Err(error) => return error,@@ -1484,12 +1496,18 @@ impl Tool for BrowserDialogTool {Ok(validated) => validated,Err(refusal) => return refusal.to_tool_result(),};+ if cfg!(target_os = "linux") && action != "inspect" && delivery_mode == "background" {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ "Chromium's native JavaScript modal cannot be resolved on Linux without changing foreground posture; retry with delivery_mode=\"foreground\"",+ )+ .with_detail(json!({ "supported_delivery_mode": "foreground" }))+ .to_tool_result();+ }let conn = &validated.conn;let cdp_session = validated.cdp_session.as_str();let cdp_target_id = validated.tab.cdp_target_id.as_str();- if conn.dialog_state(cdp_target_id).is_none()- && !conn.has_dialog_session(cdp_target_id)- {+ if conn.dialog_state(cdp_target_id).is_none() && !conn.has_dialog_session(cdp_target_id) {// Register before Page.enable so an opening event delivered before// the command reply is still attributed to the exact target.conn.register_dialog_session(cdp_session, cdp_target_id);@@ -1908,6 +1926,12 @@ mod tests {assert!(!prepare_properties.contains_key("allow_restart"));assert!(!prepare_properties.contains_key(MCP_HOST_APPROVAL_ARG));+ let dialog = BrowserDialogTool::new(e.clone());+ assert_eq!(+ dialog.def().input_schema["properties"]["delivery_mode"]["default"],+ "background"+ );+for (def, name) in [(BrowserPrepareTool::new(e.clone()).def().clone(),diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rsindex 7f700e9b64..43701e107b 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs@@ -734,13 +734,16 @@ fn wait_for_exact_browser_binding("session": session,}),);- if state.structured()["binding_quality"] == "exact" {+ let has_unique_active_tab = state.structured()["tabs"]+ .as_array()+ .is_some_and(|tabs| tabs.iter().filter(|tab| tab["active"] == true).count() == 1);+ if state.structured()["binding_quality"] == "exact" && has_unique_active_tab {return Some((window_id, state));}}if Instant::now() >= deadline {eprintln!(- "no exact browser window binding for prepared pid {pid}; last windows={}",+ "no exact browser window binding with one active tab for prepared pid {pid}; last windows={}",windows.raw);return None;@@ -1031,6 +1034,20 @@ fn refusal_case(browser: &str, action: &str, code: RefusalCode) -> CaseSpec {case(browser, action).expecting_refusal(vec![code])}+fn foreground_page_case(browser: &str, action: &str) -> CaseSpec {+ CaseSpec::delivered(+ format!("{}-{browser}-standalone-{action}", std::env::consts::OS),+ browser,+ "standalone-chromium",+ action,+ Targeting::Page,+ Delivery::Foreground,+ Scope::Window,+ DriverRoute::Cdp,+ vec![OracleKind::FixtureState],+ )+}+fn run_with_background_oracles(fixture: &mut BrowserFixture,action: impl FnOnce(&mut BrowserFixture) -> Observation,@@ -2257,7 +2274,12 @@ fn run_same_title_tabs(spec: &BrowserSpec) {fn run_dialogs(spec: &BrowserSpec) {let scenario = format!("{}-{}-standalone-dialogs", std::env::consts::OS, spec.name);- execute_case(case(&spec.name, "browser_dialogs"), |evidence| {+ let spec_case = if cfg!(target_os = "linux") {+ foreground_page_case(&spec.name, "browser_dialogs")+ } else {+ case(&spec.name, "browser_dialogs")+ };+ execute_case(spec_case, |evidence| {let mut fixture =launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());*evidence = recording_evidence(fixture.driver.recording_dir());@@ -2274,7 +2296,11 @@ fn run_dialogs(spec: &BrowserSpec) {);assert_eq!(primed.structured()["present"], false, "{}", primed.raw);- let sentinel = ForegroundSentinel::launch(&mut fixture.driver);+ let sentinel = if cfg!(target_os = "linux") {+ None+ } else {+ Some(ForegroundSentinel::launch(&mut fixture.driver))+ };let target = TargetWindow {pid: fixture.pid,native_id: fixture.window_id,@@ -2320,11 +2346,60 @@ fn run_dialogs(spec: &BrowserSpec) {);assert_eq!(clicked.structured()["status"], "ok", "{}", clicked.raw);thread::sleep(Duration::from_millis(100));++ if cfg!(target_os = "linux") {+ let deadline = Instant::now() + Duration::from_secs(5);+ let current = loop {+ let current = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "inspect",+ "session": session,+ }),+ );+ if current.structured()["present"] == true {+ break current;+ }+ assert!(+ Instant::now() < deadline,+ "{kind} dialog was not observed: {}",+ current.raw+ );+ thread::sleep(Duration::from_millis(25));+ };+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);+ let dialog_id = current.structured()["dialog_id"]+ .as_str()+ .expect("opaque dialog id")+ .to_owned();+ let mut args = serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": action,+ "dialog_id": dialog_id,+ "delivery_mode": "foreground",+ "session": session,+ });+ if let Some(prompt_text) = prompt_text {+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());+ }+ let handled = fixture.driver.call("browser_dialog", args);+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);+ continue;+ }+sentinel+ .as_ref()+ .expect("background dialog sentinel").prepare_background_observation(&mut fixture.driver, target).expect("occlude the open JavaScript dialog");let (_, passed) = sentinel+ .as_ref()+ .expect("background dialog sentinel").observe_background(target, || {let deadline = Instant::now() + Duration::from_secs(5);let current = loop {@@ -2375,6 +2450,55 @@ fn run_dialogs(spec: &BrowserSpec) {});}+#[cfg(target_os = "linux")]+fn run_dialog_background_refusal(spec: &BrowserSpec) {+ let scenario = format!(+ "{}-{}-standalone-dialog-background-refusal",+ std::env::consts::OS,+ spec.name+ );+ execute_case(+ refusal_case(+ &spec.name,+ "browser_dialogs_background",+ RefusalCode::BrowserInputTrustUnavailable,+ ),+ |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-dialog-refusal-{}", fixture.pid);+ let (target_id, tab_id, _) = bind(fixture, &session);+ let refused = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "accept",+ "dialog_id": "dialog-unavailable",+ "session": session,+ }),+ );+ assert_eq!(refused.structured()["status"], "refused", "{}", refused.raw);+ assert_eq!(+ refused.structured()["refusal"]["code"],+ "browser_input_trust_unavailable",+ "{}",+ refused.raw+ );+ wait_for_text(&fixture.server, "standalone-dialog-state", "dialog=none");+ Observation::refused(+ RefusalCode::BrowserInputTrustUnavailable,+ vec![OracleKind::FixtureState],+ refused.text(),+ Evidence::default(),+ )+ })+ },+ );+}+fn run_upload(spec: &BrowserSpec) {let scenario = format!("{}-{}-standalone-upload", std::env::consts::OS, spec.name);execute_case(case(&spec.name, "browser_file_upload"), |evidence| {@@ -2684,6 +2808,11 @@ standalone_browser_test!(standalone_browser_frames, run_frame_roundtrip);standalone_browser_test!(standalone_browser_multi_tab, run_multi_tab);standalone_browser_test!(standalone_browser_same_title_tabs, run_same_title_tabs);standalone_browser_test!(standalone_browser_dialogs, run_dialogs);+#[cfg(target_os = "linux")]+standalone_browser_test!(+ standalone_browser_dialog_background_refusal,+ run_dialog_background_refusal+);standalone_browser_test!(standalone_browser_upload, run_upload);standalone_browser_test!(standalone_browser_pointer_actions, run_pointer_actions);standalone_browser_test!(standalone_browser_download, run_download);diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.shindex abbd056794..3289edcd6e 100755--- a/scripts/ci/run-rust-standalone-browser-e2e.sh+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh@@ -91,6 +91,11 @@ fitests=(standalone_browser_background_typestandalone_browser_dialogs+)+if [[ "${HOST_OS}" == Linux ]]; then+ tests+=(standalone_browser_dialog_background_refusal)+fi+tests+=(standalone_browser_downloadstandalone_browser_existing_profilestandalone_browser_existing_profile_setup