Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): clarify synthetic browser click outcomes (#2866)
Francesco Bonacci197c518c9c36
diff --git a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdxindex c8914a422a..4fc1c9e25b 100644--- a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx+++ b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx@@ -63,8 +63,11 @@ right-click, double-click, scroll, and drag through `browser_pointer`.A caller must explicitly request `input_route: "dom_event"` to invoke anelement's DOM click behavior. That route is synthetic even when it preserves-full-background posture. The driver does not silently change trust models to-make a call appear successful.+full-background posture. A completed JavaScript dispatch does not prove the+control activated: applications may ignore events whose `isTrusted` value is+false. The result therefore remains `effect: "unverifiable"` and recommends a+fresh page-state check. The driver does not silently change trust models or+foreground the browser to make a call appear successful.Page-owned JavaScript dialogs are modeled as short-lived capabilities ratherthan native-window guesses. Inspection returns the kind and an opaque dialogdiff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdxindex 965af2c655..42a09d59dd 100644--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx@@ -315,7 +315,11 @@ browser_click({```Use this only when the page's DOM click semantics are acceptable. The driver-never silently falls back from trusted input to a DOM event.+never silently falls back from trusted input to a DOM event. A successful+`dom_event` dispatch returns `effect: "unverifiable"`: it does not mean the+control activated, and trust-gated controls may ignore it. Take a fresh page+snapshot and verify the expected state before continuing. Cua Driver does not+automatically foreground the browser when that verification fails.Snapshot again, then type into the fresh input ref:diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex adcf273977..29dce0759e 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -876,11 +876,11 @@ Navigate one tab of an exactly-bound browser target to a new URL (http/https/abo### `browser_click`-Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route="dom_event" (synthetic el.click(), ref required) is used only when explicitly requested. Refused for heuristic bindings.+Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route="dom_event" (synthetic el.click(), ref required) is used only when explicitly requested; it proves dispatch, not control activation, because trust-gated controls may ignore synthetic events. Refused for heuristic bindings.**Arguments:**-- `input_route` (string, optional): "trusted" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. "dom_event": synthetic full-background DOM click, only when explicitly requested.+- `input_route` (string, optional): "trusted" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. "dom_event": synthetic full-background DOM click, only when explicitly requested. Dispatch does not prove the control activated; refresh page state and verify the expected postcondition.- `ref` (string, optional): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.mdindex 59b4860660..465ad52b1a 100644--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md@@ -296,9 +296,11 @@ cua-driver browser_click \`dom_event` calls the page element's click behavior without pretending that atrusted pointer event occurred. It requires a ref and is the full-background-alternative where supported. Never silently change trust class after a-refusal. Coordinate clicks accept viewport CSS `x` and `y`, but only on the-trusted route; prefer refs.+alternative where supported. Dispatch is not proof that the control activated:+trust-gated controls can ignore synthetic events, so refresh page state and+verify the expected postcondition. Never silently change trust class or+foreground the browser after a refusal. Coordinate clicks accept viewport CSS+`x` and `y`, but only on the trusted route; prefer refs.### Typediff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rsindex fce9fc8561..6933c5b637 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs@@ -887,7 +887,9 @@ impl BrowserClickTool {(Input.dispatchMouseEvent), and refuses where that route cannot \preserve standalone-browser background posture. \input_route=\"dom_event\" (synthetic \- el.click(), ref required) is used only when explicitly requested. \+ el.click(), ref required) is used only when explicitly requested; \+ it proves dispatch, not control activation, because trust-gated \+ controls may ignore synthetic events. \Refused for heuristic bindings.".into(),input_schema: json!({@@ -905,7 +907,9 @@ impl BrowserClickTool {"description": "\"trusted\" (default): Input.dispatchMouseEvent. \It refuses rather than foregrounding a standalone browser. \\"dom_event\": synthetic full-background DOM click, only \- when explicitly requested."+ when explicitly requested. Dispatch does not prove the \+ control activated; refresh page state and verify the \+ expected postcondition."},},"required": ["target_id", "tab_id"],@@ -1143,16 +1147,23 @@ impl Tool for BrowserClickTool {.await{Ok(_) => ToolResult::text(format!(- "dispatched DOM click on {} in {tab_id}",+ "dispatched synthetic DOM click on {} in {tab_id}; application effect not \+ verified (trust-gated controls may ignore untrusted events). Refresh page \+ state and verify the expected postcondition",ext_ref.as_deref().unwrap_or("?"))).with_structured(json!({"status": "ok",+ "effect": "unverifiable","route": "dom_event","target_id": target_id,"tab_id": tab_id,"ref": ext_ref,"frame": frame_kind,+ "escalation": {+ "recommended": "page",+ "reason": "synthetic DOM dispatch cannot prove control activation; refresh page state and verify the expected postcondition",+ },})),Err(e) => ToolResult::error(format!("DOM click failed: {e}")),};diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rsindex 7d2fb74340..1c7b507f0c 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs@@ -15,6 +15,7 @@ use base64::{engine::general_purpose::STANDARD as BASE64, Engine as _};use image::{DynamicImage, ImageFormat, Rgba, RgbaImage};use serde_json::{json, Value};+use crate::action_record::ActionExecutionRecord;use crate::protocol::{Content, ToolResult};use crate::tool::Tool;@@ -2004,13 +2005,40 @@ async fn trusted_click_refuses_when_standalone_background_posture_is_unavailable);assert!(recorded_calls(&f, "Input.dispatchMouseEvent").is_empty());+ let synthetic_args = json!({+ "target_id": target, "tab_id": tab, "ref": main_ref,+ "input_route": "dom_event", "session": SESSION+ });let synthetic = BrowserClickTool::new(f.engine.clone())- .invoke(json!({- "target_id": target, "tab_id": tab, "ref": main_ref,- "input_route": "dom_event", "session": SESSION- }))+ .invoke(synthetic_args.clone()).await;assert_eq!(structured(&synthetic)["status"], "ok");+ assert_eq!(structured(&synthetic)["effect"], "unverifiable");+ assert_eq!(structured(&synthetic)["escalation"]["recommended"], "page");+ assert!(synthetic.content.iter().any(|content| matches!(+ content,+ crate::protocol::Content::Text { text, .. }+ if text.contains("application effect not verified")+ && text.contains("trust-gated controls")+ )));+ let public = ActionExecutionRecord::from_legacy(+ "browser_click",+ &synthetic_args,+ structured(&synthetic),+ )+ .expect("browser click action record")+ .public_result()+ .expect("public browser click result");+ let public = serde_json::to_value(public).expect("serialize public result");+ assert_eq!(public["effect"], "unverifiable", "{public}");+ assert_eq!(public["route"], "dom", "{public}");+ assert_eq!(public["delivery"]["mode"], "background", "{public}");+ assert_eq!(public["escalation"]["target"], "page", "{public}");+ assert_eq!(+ public["escalation"]["reason"], "effect_unconfirmed",+ "{public}"+ );+ assert!(public.get("status").is_none(), "{public}");assert!(!recorded_calls(&f, "Runtime.callFunctionOn").is_empty());assert!(recorded_calls(&f, "Page.bringToFront").is_empty());assert!(recorded_calls(&f, "Target.activateTarget").is_empty());diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rsindex 4a6c4d0605..a9703be2fc 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs@@ -45,6 +45,31 @@ fn standalone_fixture_html() -> String {)}+/// Synthetic control that records event delivery but deliberately rejects the+/// application action unless the browser marks the click as trusted.+fn standalone_trust_gated_click_html() -> String {+ standalone_fixture_html().replace(+ "</body>",+ r#"<fieldset>+ <legend>trust-gated click</legend>+ <button id="standalone-trust-gated" data-cua-id="standalone-trust-gated">+ Activate trust-gated control+ </button>+ <span id="standalone-trust-gated-state" data-cua-id="standalone-trust-gated-state">+ activation=idle+ </span>+</fieldset>+<script>+ document.getElementById('standalone-trust-gated').addEventListener('click', event => {+ document.getElementById('standalone-trust-gated-state').textContent = event.isTrusted+ ? 'activation=accepted-trusted'+ : 'activation=ignored-untrusted';+ });+</script>+</body>"#,+ )+}+#[cfg(target_os = "macos")]fn standalone_generic_type_text_html() -> String {standalone_fixture_html().replace(@@ -1579,6 +1604,68 @@ fn run_roundtrip(spec: &BrowserSpec) {});}+fn run_trust_gated_dom_click(spec: &BrowserSpec) {+ let scenario = format!(+ "{}-{}-standalone-trust-gated-dom-click",+ std::env::consts::OS,+ spec.name+ );+ execute_case(case(&spec.name, "trust_gated_dom_click"), |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_trust_gated_click_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-trust-gated-dom-click-{}", fixture.pid);+ let (target, tab, snapshot) = bind(fixture, &session);+ let click_ref = ref_by_label(&snapshot, "id=standalone-trust-gated");+ let click = fixture.driver.call(+ "browser_click",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": click_ref,+ "input_route": "dom_event",+ "session": session,+ }),+ );++ assert_eq!(click.action_effect(), Some("unverifiable"), "{}", click.raw);+ assert_eq!(click.action_route(), Some("dom"), "{}", click.raw);+ assert_eq!(+ click.action_delivery_mode(),+ Some("background"),+ "{}",+ click.raw+ );+ assert_eq!(+ click.structured()["escalation"]["target"],+ "page",+ "{}",+ click.raw+ );+ assert_eq!(+ click.structured()["escalation"]["reason"],+ "effect_unconfirmed",+ "{}",+ click.raw+ );+ assert!(+ click.text().contains("application effect not verified")+ && click.text().contains("trust-gated controls"),+ "{}",+ click.raw+ );+ wait_for_text(+ &fixture.server,+ "standalone-trust-gated-state",+ "activation=ignored-untrusted",+ );++ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ });+}+fn run_semantic_state(spec: &BrowserSpec) {let scenario = format!("{}-{}-standalone-semantic-state",@@ -4544,6 +4631,10 @@ macro_rules! standalone_browser_test {}standalone_browser_test!(standalone_browser_roundtrip, run_roundtrip);+standalone_browser_test!(+ standalone_browser_trust_gated_dom_click,+ run_trust_gated_dom_click+);standalone_browser_test!(standalone_browser_semantic_state, run_semantic_state);standalone_browser_test!(standalone_browser_background_type, run_background_type);standalone_browser_test!(standalone_browser_type_replace, run_type_replace);diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.shindex ddaa128434..92a24f4245 100755--- a/scripts/ci/run-rust-standalone-browser-e2e.sh+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh@@ -130,6 +130,7 @@ elsestandalone_browser_roundtripstandalone_browser_semantic_statestandalone_browser_stale_ref+ standalone_browser_trust_gated_dom_clickstandalone_browser_trusted_clickstandalone_browser_uploadstandalone_browser_window_collisiondiff --git a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1 b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1index cf26ab1713..336ccd2d29 100644--- a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1+++ b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1@@ -132,6 +132,7 @@ $tests = @("standalone_browser_roundtrip","standalone_browser_semantic_state","standalone_browser_stale_ref",+ "standalone_browser_trust_gated_dom_click","standalone_browser_trusted_click","standalone_browser_upload","standalone_browser_window_collision"