Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): clarify synthetic browser click outcomes (#2866)

Francesco Bonacci197c518c9c36
diff --git a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
index c8914a422a..4fc1c9e25b 100644
--- a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
+++ b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
@@ -63,8 +63,11 @@ right-click, double-click, scroll, and drag through `browser_pointer`.
A caller must explicitly request `input_route: "dom_event"` to invoke an
element's DOM click behavior. That route is synthetic even when it preserves
-full-background posture. The driver does not silently change trust models to
-make a call appear successful.
+full-background posture. A completed JavaScript dispatch does not prove the
+control activated: applications may ignore events whose `isTrusted` value is
+false. The result therefore remains `effect: "unverifiable"` and recommends a
+fresh page-state check. The driver does not silently change trust models or
+foreground the browser to make a call appear successful.
Page-owned JavaScript dialogs are modeled as short-lived capabilities rather
than native-window guesses. Inspection returns the kind and an opaque dialog
diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
index 965af2c655..42a09d59dd 100644
--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
@@ -315,7 +315,11 @@ browser_click({
```
Use this only when the page's DOM click semantics are acceptable. The driver
-never silently falls back from trusted input to a DOM event.
+never silently falls back from trusted input to a DOM event. A successful
+`dom_event` dispatch returns `effect: "unverifiable"`: it does not mean the
+control activated, and trust-gated controls may ignore it. Take a fresh page
+snapshot and verify the expected state before continuing. Cua Driver does not
+automatically foreground the browser when that verification fails.
Snapshot again, then type into the fresh input ref:
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index adcf273977..29dce0759e 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -876,11 +876,11 @@ Navigate one tab of an exactly-bound browser target to a new URL (http/https/abo
### `browser_click`
-Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route="dom_event" (synthetic el.click(), ref required) is used only when explicitly requested. Refused for heuristic bindings.
+Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route="dom_event" (synthetic el.click(), ref required) is used only when explicitly requested; it proves dispatch, not control activation, because trust-gated controls may ignore synthetic events. Refused for heuristic bindings.
**Arguments:**
-- `input_route` (string, optional): "trusted" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. "dom_event": synthetic full-background DOM click, only when explicitly requested.
+- `input_route` (string, optional): "trusted" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. "dom_event": synthetic full-background DOM click, only when explicitly requested. Dispatch does not prove the control activated; refresh page state and verify the expected postcondition.
- `ref` (string, optional): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.
- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.
- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).
diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
index 59b4860660..465ad52b1a 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
@@ -296,9 +296,11 @@ cua-driver browser_click \
`dom_event` calls the page element's click behavior without pretending that a
trusted pointer event occurred. It requires a ref and is the full-background
-alternative where supported. Never silently change trust class after a
-refusal. Coordinate clicks accept viewport CSS `x` and `y`, but only on the
-trusted route; prefer refs.
+alternative where supported. Dispatch is not proof that the control activated:
+trust-gated controls can ignore synthetic events, so refresh page state and
+verify the expected postcondition. Never silently change trust class or
+foreground the browser after a refusal. Coordinate clicks accept viewport CSS
+`x` and `y`, but only on the trusted route; prefer refs.
### Type
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
index fce9fc8561..6933c5b637 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
@@ -887,7 +887,9 @@ impl BrowserClickTool {
(Input.dispatchMouseEvent), and refuses where that route cannot \
preserve standalone-browser background posture. \
input_route=\"dom_event\" (synthetic \
- el.click(), ref required) is used only when explicitly requested. \
+ el.click(), ref required) is used only when explicitly requested; \
+ it proves dispatch, not control activation, because trust-gated \
+ controls may ignore synthetic events. \
Refused for heuristic bindings."
.into(),
input_schema: json!({
@@ -905,7 +907,9 @@ impl BrowserClickTool {
"description": "\"trusted\" (default): Input.dispatchMouseEvent. \
It refuses rather than foregrounding a standalone browser. \
\"dom_event\": synthetic full-background DOM click, only \
- when explicitly requested."
+ when explicitly requested. Dispatch does not prove the \
+ control activated; refresh page state and verify the \
+ expected postcondition."
},
},
"required": ["target_id", "tab_id"],
@@ -1143,16 +1147,23 @@ impl Tool for BrowserClickTool {
.await
{
Ok(_) => ToolResult::text(format!(
- "dispatched DOM click on {} in {tab_id}",
+ "dispatched synthetic DOM click on {} in {tab_id}; application effect not \
+ verified (trust-gated controls may ignore untrusted events). Refresh page \
+ state and verify the expected postcondition",
ext_ref.as_deref().unwrap_or("?")
))
.with_structured(json!({
"status": "ok",
+ "effect": "unverifiable",
"route": "dom_event",
"target_id": target_id,
"tab_id": tab_id,
"ref": ext_ref,
"frame": frame_kind,
+ "escalation": {
+ "recommended": "page",
+ "reason": "synthetic DOM dispatch cannot prove control activation; refresh page state and verify the expected postcondition",
+ },
})),
Err(e) => ToolResult::error(format!("DOM click failed: {e}")),
};
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs
index 7d2fb74340..1c7b507f0c 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs
@@ -15,6 +15,7 @@ use base64::{engine::general_purpose::STANDARD as BASE64, Engine as _};
use image::{DynamicImage, ImageFormat, Rgba, RgbaImage};
use serde_json::{json, Value};
+use crate::action_record::ActionExecutionRecord;
use crate::protocol::{Content, ToolResult};
use crate::tool::Tool;
@@ -2004,13 +2005,40 @@ async fn trusted_click_refuses_when_standalone_background_posture_is_unavailable
);
assert!(recorded_calls(&f, "Input.dispatchMouseEvent").is_empty());
+ let synthetic_args = json!({
+ "target_id": target, "tab_id": tab, "ref": main_ref,
+ "input_route": "dom_event", "session": SESSION
+ });
let synthetic = BrowserClickTool::new(f.engine.clone())
- .invoke(json!({
- "target_id": target, "tab_id": tab, "ref": main_ref,
- "input_route": "dom_event", "session": SESSION
- }))
+ .invoke(synthetic_args.clone())
.await;
assert_eq!(structured(&synthetic)["status"], "ok");
+ assert_eq!(structured(&synthetic)["effect"], "unverifiable");
+ assert_eq!(structured(&synthetic)["escalation"]["recommended"], "page");
+ assert!(synthetic.content.iter().any(|content| matches!(
+ content,
+ crate::protocol::Content::Text { text, .. }
+ if text.contains("application effect not verified")
+ && text.contains("trust-gated controls")
+ )));
+ let public = ActionExecutionRecord::from_legacy(
+ "browser_click",
+ &synthetic_args,
+ structured(&synthetic),
+ )
+ .expect("browser click action record")
+ .public_result()
+ .expect("public browser click result");
+ let public = serde_json::to_value(public).expect("serialize public result");
+ assert_eq!(public["effect"], "unverifiable", "{public}");
+ assert_eq!(public["route"], "dom", "{public}");
+ assert_eq!(public["delivery"]["mode"], "background", "{public}");
+ assert_eq!(public["escalation"]["target"], "page", "{public}");
+ assert_eq!(
+ public["escalation"]["reason"], "effect_unconfirmed",
+ "{public}"
+ );
+ assert!(public.get("status").is_none(), "{public}");
assert!(!recorded_calls(&f, "Runtime.callFunctionOn").is_empty());
assert!(recorded_calls(&f, "Page.bringToFront").is_empty());
assert!(recorded_calls(&f, "Target.activateTarget").is_empty());
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
index 4a6c4d0605..a9703be2fc 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
@@ -45,6 +45,31 @@ fn standalone_fixture_html() -> String {
)
}
+/// Synthetic control that records event delivery but deliberately rejects the
+/// application action unless the browser marks the click as trusted.
+fn standalone_trust_gated_click_html() -> String {
+ standalone_fixture_html().replace(
+ "</body>",
+ r#"<fieldset>
+ <legend>trust-gated click</legend>
+ <button id="standalone-trust-gated" data-cua-id="standalone-trust-gated">
+ Activate trust-gated control
+ </button>
+ <span id="standalone-trust-gated-state" data-cua-id="standalone-trust-gated-state">
+ activation=idle
+ </span>
+</fieldset>
+<script>
+ document.getElementById('standalone-trust-gated').addEventListener('click', event => {
+ document.getElementById('standalone-trust-gated-state').textContent = event.isTrusted
+ ? 'activation=accepted-trusted'
+ : 'activation=ignored-untrusted';
+ });
+</script>
+</body>"#,
+ )
+}
+
#[cfg(target_os = "macos")]
fn standalone_generic_type_text_html() -> String {
standalone_fixture_html().replace(
@@ -1579,6 +1604,68 @@ fn run_roundtrip(spec: &BrowserSpec) {
});
}
+fn run_trust_gated_dom_click(spec: &BrowserSpec) {
+ let scenario = format!(
+ "{}-{}-standalone-trust-gated-dom-click",
+ std::env::consts::OS,
+ spec.name
+ );
+ execute_case(case(&spec.name, "trust_gated_dom_click"), |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_trust_gated_click_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-trust-gated-dom-click-{}", fixture.pid);
+ let (target, tab, snapshot) = bind(fixture, &session);
+ let click_ref = ref_by_label(&snapshot, "id=standalone-trust-gated");
+ let click = fixture.driver.call(
+ "browser_click",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": click_ref,
+ "input_route": "dom_event",
+ "session": session,
+ }),
+ );
+
+ assert_eq!(click.action_effect(), Some("unverifiable"), "{}", click.raw);
+ assert_eq!(click.action_route(), Some("dom"), "{}", click.raw);
+ assert_eq!(
+ click.action_delivery_mode(),
+ Some("background"),
+ "{}",
+ click.raw
+ );
+ assert_eq!(
+ click.structured()["escalation"]["target"],
+ "page",
+ "{}",
+ click.raw
+ );
+ assert_eq!(
+ click.structured()["escalation"]["reason"],
+ "effect_unconfirmed",
+ "{}",
+ click.raw
+ );
+ assert!(
+ click.text().contains("application effect not verified")
+ && click.text().contains("trust-gated controls"),
+ "{}",
+ click.raw
+ );
+ wait_for_text(
+ &fixture.server,
+ "standalone-trust-gated-state",
+ "activation=ignored-untrusted",
+ );
+
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ });
+}
+
fn run_semantic_state(spec: &BrowserSpec) {
let scenario = format!(
"{}-{}-standalone-semantic-state",
@@ -4544,6 +4631,10 @@ macro_rules! standalone_browser_test {
}
standalone_browser_test!(standalone_browser_roundtrip, run_roundtrip);
+standalone_browser_test!(
+ standalone_browser_trust_gated_dom_click,
+ run_trust_gated_dom_click
+);
standalone_browser_test!(standalone_browser_semantic_state, run_semantic_state);
standalone_browser_test!(standalone_browser_background_type, run_background_type);
standalone_browser_test!(standalone_browser_type_replace, run_type_replace);
diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.sh
index ddaa128434..92a24f4245 100755
--- a/scripts/ci/run-rust-standalone-browser-e2e.sh
+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh
@@ -130,6 +130,7 @@ else
standalone_browser_roundtrip
standalone_browser_semantic_state
standalone_browser_stale_ref
+ standalone_browser_trust_gated_dom_click
standalone_browser_trusted_click
standalone_browser_upload
standalone_browser_window_collision
diff --git a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1 b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
index cf26ab1713..336ccd2d29 100644
--- a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
+++ b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
@@ -132,6 +132,7 @@ $tests = @(
"standalone_browser_roundtrip",
"standalone_browser_semantic_state",
"standalone_browser_stale_ref",
+ "standalone_browser_trust_gated_dom_click",
"standalone_browser_trusted_click",
"standalone_browser_upload",
"standalone_browser_window_collision"