Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): add browser telemetry contract

trycua-release[bot]fa138903f779
diff --git a/docs/content/docs/reference/cua-driver/browser-profile-attachment.mdx b/docs/content/docs/reference/cua-driver/browser-profile-attachment.mdx
index 281ce54dd7..a0a536cb2d 100644
--- a/docs/content/docs/reference/cua-driver/browser-profile-attachment.mdx
+++ b/docs/content/docs/reference/cua-driver/browser-profile-attachment.mdx
@@ -93,6 +93,8 @@ Consent turns persist redacted request metadata and the structured outcome.
They suppress browser screenshots and accessibility snapshots. Public results,
recordings, and telemetry must not contain profile paths or hashes, endpoint
URLs or ports, approval tokens, page content, tab URLs, cookies, or storage.
+The content-free browser operation and refusal fields are documented in
+[Telemetry and privacy](/reference/cua-driver/telemetry#structured-browser-refusals).
## Platform scope
diff --git a/docs/content/docs/reference/cua-driver/telemetry.mdx b/docs/content/docs/reference/cua-driver/telemetry.mdx
index 0e186bb76c..ead0531936 100644
--- a/docs/content/docs/reference/cua-driver/telemetry.mdx
+++ b/docs/content/docs/reference/cua-driver/telemetry.mdx
@@ -82,9 +82,9 @@ The client does not send an IP address as an event property. PostHog derives a c
| `cua_driver_cli_completed` | Fixed `command`, allowlisted tool for `call`, bounded `operation`, bounded MCP `client_kind`, `success`, `exit_class`, and `duration_bucket` |
| `cua_driver_mcp_startup_completed` | execution path, bounded daemon state, success, duration bucket, and `execution_mode` |
| `cua_driver_mcp_session_started` | normalized MCP client and protocol, capability booleans, optional reported agent context, and `execution_mode` |
-| `cua_driver_mcp_tool_completed` | allowlisted tool, bounded compound-tool `operation`, success, error class, duration bucket, output shape and size buckets, and `execution_mode` |
+| `cua_driver_mcp_tool_completed` | allowlisted tool, bounded `operation`, protocol success, error class, structured-refusal code, duration bucket, output shape and size buckets, and `execution_mode` |
| `cua_driver_agent_session_started` | declaration kind, revival flag, concurrent-session bucket, entry transport, and `execution_mode` |
-| `cua_driver_agent_session_ended` | end reason, duration and count buckets, success flags, feature-family flags, multi-transport flag, and `execution_mode` |
+| `cua_driver_agent_session_ended` | end reason, duration and count buckets, browser-refusal bucket, success flags, feature-family flags, multi-transport flag, and `execution_mode` |
| `cua_driver_permissions_gate_started` | missing-permission booleans |
| `cua_driver_permissions_gate_dismissed` | missing-permission booleans and duration bucket |
| `cua_driver_permissions_gate_completed` | missing-permission booleans, panel and dismissal flags, fixed resolution, and duration bucket |
@@ -122,17 +122,55 @@ The start event is emitted for a successful `start_session` declaration or the f
The end event is emitted for `end_session` and idle eviction. It contains aggregate counters and booleans. When the platform has a cursor entry for that session, it also includes bounded cursor categories: enabled state, built-in/default/custom icon class, automatic/custom color source, label presence, motion customization, and an active-cursor count bucket. `cursor_outcome_observed=false` distinguishes sessions with no readable cursor entry from a disabled or default cursor. Cua Driver uses the caller session string only as a process-local map key. The string, a hash of it, and a replacement join token are absent from telemetry payloads.
-Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, and state-changing `page` operations. State reads, screenshots, health checks, update checks, and session lifecycle tools do not count as computer actions.
+Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful `browser_navigate`, `browser_click`, and `browser_type` calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` is a read, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect.
+
+The end event includes `used_browser` and a bounded `browser_refusal_count_bucket`. It does not include browser targets, pages, profiles, or per-site information.
For finite CLI commands, `operation` distinguishes only reviewed verbs for recording, permissions, config, autostart, skills, and update. `client_kind` is populated only for `mcp-config`. Unknown values become `other`; commands without a meaningful sub-operation use `not_applicable`.
-For the compound `page` tool, `operation` is one of `execute_javascript`, `get_text`, `query_dom`, `click_element`, `insert_text`, `type_keystrokes`, `enable_javascript_apple_events`, or `other`. Other tools use `not_applicable`.
+For the compound `page` tool, `operation` is one of `execute_javascript`, `get_text`, `query_dom`, `click_element`, `insert_text`, `type_keystrokes`, `enable_javascript_apple_events`, or `other`.
+
+Typed browser tools use these reviewed operation values:
+
+| Tool | `operation` values |
+| --- | --- |
+| `get_browser_state` | `browser_bind`, `browser_snapshot_dom_refs_v1`, `browser_snapshot_semantic_v2`, or `other` |
+| `browser_prepare` | `browser_prepare_isolated`, `browser_prepare_existing_profile`, or `other` |
+| `browser_click` | `browser_click_trusted`, `browser_click_dom_event`, or `other` |
+| `browser_type` | `browser_type_insert_text`, `browser_type_keystrokes`, or `other` |
+| `browser_navigate` | `not_applicable`; the tool name already identifies the operation |
+
+### Structured browser refusals
+
+A browser refusal is a successful MCP exchange with a behavioral result of `refused`. The tool-completion event therefore keeps `success=true` and `error_class=none`, while `refusal_code` records one closed, content-free code. The allowed values are:
+
+- `none`
+- `browser_route_unavailable`
+- `browser_requires_setup`
+- `browser_binding_ambiguous`
+- `browser_binding_stale`
+- `browser_wrong_target_refused`
+- `browser_tab_required`
+- `browser_tab_not_found`
+- `browser_ref_stale`
+- `browser_input_trust_unavailable`
+- `browser_endpoint_owner_mismatch`
+- `browser_consent_required`
+- `browser_consent_revoked`
+- `browser_reconnect_exhausted`
+- `browser_input_incomplete`
+- `browser_action_unavailable`
+- `other` for an unrecognized future refusal until it is reviewed for telemetry
+
+`browser_consent_revoked` indicates only that the browser consent flow was dismissed or denied. It does not include who acted, the browser profile, the prompt contents, or any associated page state.
## Data that is never collected
-Routine telemetry excludes task text, prompts, tool arguments, tool response bodies, typed text, screenshots, accessibility trees, window titles, application names, file paths, URLs, arbitrary MCP metadata, raw cursor IDs, labels, colors, icon paths, numeric motion values, and raw error messages.
+Routine telemetry excludes task text, prompts, tool arguments, tool response bodies, typed text, screenshots, accessibility trees, window titles, application names, file paths, URLs, arbitrary MCP metadata, raw cursor IDs, labels, colors, icon paths, numeric motion values, and raw error messages. Browser telemetry also excludes target, tab, frame, ref, process, window, and session identifiers; profile names and paths; queries and coordinates; approval tokens; endpoint URLs and ports; refusal messages and details; and requested or delivered text lengths.
+
+Tool-completion events retain only coarse result shape: text, image, mixed, empty, or unknown; a size bucket; a duration bucket; a fixed error class; and, for reviewed structured browser refusals, the fixed code above. The content itself never crosses the telemetry observer boundary. A proxy and daemon negotiate one completion-event owner. Mixed-version pairs retain proxy ownership, which prevents duplicate events during upgrades.
-Tool-completion events retain only coarse result shape: text, image, mixed, empty, or unknown; a size bucket; a duration bucket; and a fixed error class. The content itself never crosses the telemetry observer boundary. A proxy and daemon negotiate one completion-event owner. Mixed-version pairs retain proxy ownership, which prevents duplicate events during upgrades.
+A CLI `call` handled in-process reports `transport=cli`. When the same call is delegated to a running daemon, the daemon owns the completion event and reports `transport=daemon`; the event is still emitted only once.
## Region and deletion controls
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
index 32dfc63e3f..b9ff172a1a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
@@ -126,6 +126,104 @@ impl ToolErrorClass {
}
}
+/// Fixed structured-refusal codes that may cross the telemetry observer seam.
+///
+/// This vocabulary deliberately mirrors only reviewed, content-free browser
+/// refusal codes. Unknown future wire values become [`Self::Other`] until they
+/// are explicitly adopted here; refusal messages and details are never read.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ToolRefusalCode {
+ None,
+ BrowserRouteUnavailable,
+ BrowserRequiresSetup,
+ BrowserBindingAmbiguous,
+ BrowserBindingStale,
+ BrowserWrongTargetRefused,
+ BrowserTabRequired,
+ BrowserTabNotFound,
+ BrowserRefStale,
+ BrowserInputTrustUnavailable,
+ BrowserEndpointOwnerMismatch,
+ BrowserConsentRequired,
+ BrowserConsentRevoked,
+ BrowserReconnectExhausted,
+ BrowserInputIncomplete,
+ BrowserActionUnavailable,
+ Other,
+}
+
+impl ToolRefusalCode {
+ pub const fn as_str(self) -> &'static str {
+ match self {
+ Self::None => "none",
+ Self::BrowserRouteUnavailable => "browser_route_unavailable",
+ Self::BrowserRequiresSetup => "browser_requires_setup",
+ Self::BrowserBindingAmbiguous => "browser_binding_ambiguous",
+ Self::BrowserBindingStale => "browser_binding_stale",
+ Self::BrowserWrongTargetRefused => "browser_wrong_target_refused",
+ Self::BrowserTabRequired => "browser_tab_required",
+ Self::BrowserTabNotFound => "browser_tab_not_found",
+ Self::BrowserRefStale => "browser_ref_stale",
+ Self::BrowserInputTrustUnavailable => "browser_input_trust_unavailable",
+ Self::BrowserEndpointOwnerMismatch => "browser_endpoint_owner_mismatch",
+ Self::BrowserConsentRequired => "browser_consent_required",
+ Self::BrowserConsentRevoked => "browser_consent_revoked",
+ Self::BrowserReconnectExhausted => "browser_reconnect_exhausted",
+ Self::BrowserInputIncomplete => "browser_input_incomplete",
+ Self::BrowserActionUnavailable => "browser_action_unavailable",
+ Self::Other => "other",
+ }
+ }
+
+ pub const fn is_refusal(self) -> bool {
+ !matches!(self, Self::None)
+ }
+
+ fn from_wire_code(code: Option<&str>) -> Self {
+ match code {
+ Some("browser_route_unavailable") => Self::BrowserRouteUnavailable,
+ Some("browser_requires_setup") => Self::BrowserRequiresSetup,
+ Some("browser_binding_ambiguous") => Self::BrowserBindingAmbiguous,
+ Some("browser_binding_stale") => Self::BrowserBindingStale,
+ Some("browser_wrong_target_refused") => Self::BrowserWrongTargetRefused,
+ Some("browser_tab_required") => Self::BrowserTabRequired,
+ Some("browser_tab_not_found") => Self::BrowserTabNotFound,
+ Some("browser_ref_stale") => Self::BrowserRefStale,
+ Some("browser_input_trust_unavailable") => Self::BrowserInputTrustUnavailable,
+ Some("browser_endpoint_owner_mismatch") => Self::BrowserEndpointOwnerMismatch,
+ Some("browser_consent_required") => Self::BrowserConsentRequired,
+ Some("browser_consent_revoked") => Self::BrowserConsentRevoked,
+ Some("browser_reconnect_exhausted") => Self::BrowserReconnectExhausted,
+ Some("browser_input_incomplete") => Self::BrowserInputIncomplete,
+ Some("browser_action_unavailable") => Self::BrowserActionUnavailable,
+ Some(_) | None => Self::Other,
+ }
+ }
+}
+
+impl From<crate::browser::refusal::BrowserRefusalCode> for ToolRefusalCode {
+ fn from(code: crate::browser::refusal::BrowserRefusalCode) -> Self {
+ use crate::browser::refusal::BrowserRefusalCode;
+ match code {
+ BrowserRefusalCode::BrowserRouteUnavailable => Self::BrowserRouteUnavailable,
+ BrowserRefusalCode::BrowserRequiresSetup => Self::BrowserRequiresSetup,
+ BrowserRefusalCode::BrowserBindingAmbiguous => Self::BrowserBindingAmbiguous,
+ BrowserRefusalCode::BrowserBindingStale => Self::BrowserBindingStale,
+ BrowserRefusalCode::BrowserWrongTargetRefused => Self::BrowserWrongTargetRefused,
+ BrowserRefusalCode::BrowserTabRequired => Self::BrowserTabRequired,
+ BrowserRefusalCode::BrowserTabNotFound => Self::BrowserTabNotFound,
+ BrowserRefusalCode::BrowserRefStale => Self::BrowserRefStale,
+ BrowserRefusalCode::BrowserInputTrustUnavailable => Self::BrowserInputTrustUnavailable,
+ BrowserRefusalCode::BrowserEndpointOwnerMismatch => Self::BrowserEndpointOwnerMismatch,
+ BrowserRefusalCode::BrowserConsentRequired => Self::BrowserConsentRequired,
+ BrowserRefusalCode::BrowserConsentRevoked => Self::BrowserConsentRevoked,
+ BrowserRefusalCode::BrowserReconnectExhausted => Self::BrowserReconnectExhausted,
+ BrowserRefusalCode::BrowserInputIncomplete => Self::BrowserInputIncomplete,
+ BrowserRefusalCode::BrowserActionUnavailable => Self::BrowserActionUnavailable,
+ }
+ }
+}
+
/// Privacy-bounded completion record for a known tool call on any transport.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ToolCompletionObservation {
@@ -135,6 +233,7 @@ pub struct ToolCompletionObservation {
pub operation: ToolOperation,
pub success: bool,
pub error_class: ToolErrorClass,
+ pub refusal_code: ToolRefusalCode,
pub duration_bucket: DurationBucket,
pub output_type: OutputType,
pub output_size_bucket: OutputSizeBucket,
@@ -153,6 +252,15 @@ pub enum ToolOperation {
InsertText,
TypeKeystrokes,
EnableJavascriptAppleEvents,
+ BrowserBind,
+ BrowserSnapshotDomRefsV1,
+ BrowserSnapshotSemanticV2,
+ BrowserClickTrusted,
+ BrowserClickDomEvent,
+ BrowserTypeInsertText,
+ BrowserTypeKeystrokes,
+ BrowserPrepareIsolated,
+ BrowserPrepareExistingProfile,
Other,
}
@@ -167,27 +275,77 @@ impl ToolOperation {
Self::InsertText => "insert_text",
Self::TypeKeystrokes => "type_keystrokes",
Self::EnableJavascriptAppleEvents => "enable_javascript_apple_events",
+ Self::BrowserBind => "browser_bind",
+ Self::BrowserSnapshotDomRefsV1 => "browser_snapshot_dom_refs_v1",
+ Self::BrowserSnapshotSemanticV2 => "browser_snapshot_semantic_v2",
+ Self::BrowserClickTrusted => "browser_click_trusted",
+ Self::BrowserClickDomEvent => "browser_click_dom_event",
+ Self::BrowserTypeInsertText => "browser_type_insert_text",
+ Self::BrowserTypeKeystrokes => "browser_type_keystrokes",
+ Self::BrowserPrepareIsolated => "browser_prepare_isolated",
+ Self::BrowserPrepareExistingProfile => "browser_prepare_existing_profile",
Self::Other => "other",
}
}
}
pub fn tool_operation(tool_name: &str, args: Option<&serde_json::Value>) -> ToolOperation {
- if tool_name != "page" {
- return ToolOperation::NotApplicable;
- }
- match args
- .and_then(|value| value.get("action"))
- .and_then(serde_json::Value::as_str)
- {
- Some("execute_javascript") => ToolOperation::ExecuteJavascript,
- Some("get_text") => ToolOperation::GetText,
- Some("query_dom") => ToolOperation::QueryDom,
- Some("click_element") => ToolOperation::ClickElement,
- Some("insert_text") => ToolOperation::InsertText,
- Some("type_keystrokes") => ToolOperation::TypeKeystrokes,
- Some("enable_javascript_apple_events") => ToolOperation::EnableJavascriptAppleEvents,
- _ => ToolOperation::Other,
+ let string_arg = |key: &str| {
+ args.and_then(|value| value.get(key))
+ .and_then(serde_json::Value::as_str)
+ };
+ match tool_name {
+ "page" => match string_arg("action") {
+ Some("execute_javascript") => ToolOperation::ExecuteJavascript,
+ Some("get_text") => ToolOperation::GetText,
+ Some("query_dom") => ToolOperation::QueryDom,
+ Some("click_element") => ToolOperation::ClickElement,
+ Some("insert_text") => ToolOperation::InsertText,
+ Some("type_keystrokes") => ToolOperation::TypeKeystrokes,
+ Some("enable_javascript_apple_events") => ToolOperation::EnableJavascriptAppleEvents,
+ _ => ToolOperation::Other,
+ },
+ "get_browser_state" => {
+ if string_arg("target_id").is_some() {
+ match string_arg("snapshot_format") {
+ Some("semantic_v2") => ToolOperation::BrowserSnapshotSemanticV2,
+ None | Some("dom_refs_v1") => ToolOperation::BrowserSnapshotDomRefsV1,
+ Some(_) => ToolOperation::Other,
+ }
+ } else if args.and_then(|value| value.get("pid")).is_some()
+ || args.and_then(|value| value.get("window_id")).is_some()
+ {
+ ToolOperation::BrowserBind
+ } else {
+ ToolOperation::Other
+ }
+ }
+ "browser_click" => match string_arg("input_route") {
+ None | Some("trusted") => ToolOperation::BrowserClickTrusted,
+ Some("dom_event") => ToolOperation::BrowserClickDomEvent,
+ Some(_) => ToolOperation::Other,
+ },
+ "browser_type" => match string_arg("mode") {
+ None | Some("insert_text") => ToolOperation::BrowserTypeInsertText,
+ Some("keystrokes") => ToolOperation::BrowserTypeKeystrokes,
+ Some(_) => ToolOperation::Other,
+ },
+ "browser_prepare" => {
+ let strategy = args
+ .and_then(|value| value.pointer("/strategy/kind"))
+ .and_then(serde_json::Value::as_str);
+ let profile_mode = args
+ .and_then(|value| value.pointer("/profile/mode"))
+ .and_then(serde_json::Value::as_str);
+ match (strategy, profile_mode) {
+ (Some("existing_profile"), _) => ToolOperation::BrowserPrepareExistingProfile,
+ (None, Some("isolated_new" | "isolated_named")) => {
+ ToolOperation::BrowserPrepareIsolated
+ }
+ _ => ToolOperation::Other,
+ }
+ }
+ _ => ToolOperation::NotApplicable,
}
}
@@ -461,6 +619,11 @@ fn classify_tool_completion(
} else {
structured_error_class(result)
};
+ let refusal_code = if success {
+ structured_refusal_code(&timer.tool_name, result)
+ } else {
+ ToolRefusalCode::None
+ };
let output_type = classify_output_type(result);
let output_size_bucket = result
@@ -473,12 +636,39 @@ fn classify_tool_completion(
operation: timer.operation,
success,
error_class,
+ refusal_code,
duration_bucket,
output_type,
output_size_bucket,
}
}
+fn structured_refusal_code(tool_name: &str, result: Option<&serde_json::Value>) -> ToolRefusalCode {
+ if !matches!(
+ tool_name,
+ "get_browser_state"
+ | "browser_prepare"
+ | "browser_navigate"
+ | "browser_click"
+ | "browser_type"
+ ) {
+ return ToolRefusalCode::None;
+ }
+ let structured = result.and_then(|value| value.get("structuredContent"));
+ if structured
+ .and_then(|value| value.get("status"))
+ .and_then(serde_json::Value::as_str)
+ != Some("refused")
+ {
+ return ToolRefusalCode::None;
+ }
+ ToolRefusalCode::from_wire_code(
+ structured
+ .and_then(|value| value.pointer("/refusal/code"))
+ .and_then(serde_json::Value::as_str),
+ )
+}
+
fn serialized_size_without_retaining(value: &serde_json::Value) -> Option<usize> {
struct ByteCounter(usize);
impl std::io::Write for ByteCounter {
@@ -715,6 +905,176 @@ mod observation_tests {
);
}
+ #[test]
+ fn browser_operations_are_closed_and_retain_no_argument_content() {
+ for (tool_name, args, expected) in [
+ (
+ "get_browser_state",
+ serde_json::json!({"pid": 42, "window_id": 7, "query": "private query"}),
+ ToolOperation::BrowserBind,
+ ),
+ (
+ "get_browser_state",
+ serde_json::json!({
+ "target_id": "private-target",
+ "snapshot_format": "dom_refs_v1"
+ }),
+ ToolOperation::BrowserSnapshotDomRefsV1,
+ ),
+ (
+ "get_browser_state",
+ serde_json::json!({
+ "target_id": "private-target",
+ "snapshot_format": "semantic_v2"
+ }),
+ ToolOperation::BrowserSnapshotSemanticV2,
+ ),
+ (
+ "browser_click",
+ serde_json::json!({"input_route": "trusted", "ref": "private-ref"}),
+ ToolOperation::BrowserClickTrusted,
+ ),
+ (
+ "browser_click",
+ serde_json::json!({"input_route": "dom_event", "ref": "private-ref"}),
+ ToolOperation::BrowserClickDomEvent,
+ ),
+ (
+ "browser_type",
+ serde_json::json!({"mode": "insert_text", "text": "private typed text"}),
+ ToolOperation::BrowserTypeInsertText,
+ ),
+ (
+ "browser_type",
+ serde_json::json!({"mode": "keystrokes", "text": "private typed text"}),
+ ToolOperation::BrowserTypeKeystrokes,
+ ),
+ (
+ "browser_prepare",
+ serde_json::json!({
+ "profile": {"mode": "isolated_named", "name": "private-profile"}
+ }),
+ ToolOperation::BrowserPrepareIsolated,
+ ),
+ (
+ "browser_prepare",
+ serde_json::json!({
+ "strategy": {"kind": "existing_profile"},
+ "approval_token": "private-token"
+ }),
+ ToolOperation::BrowserPrepareExistingProfile,
+ ),
+ ] {
+ let operation = tool_operation(tool_name, Some(&args));
+ assert_eq!(operation, expected, "tool={tool_name}");
+ let debug = format!("{operation:?}");
+ for forbidden in [
+ "private query",
+ "private-target",
+ "private-ref",
+ "private typed text",
+ "private-profile",
+ "private-token",
+ ] {
+ assert!(!debug.contains(forbidden), "observer leaked {forbidden}");
+ }
+ }
+ assert_eq!(
+ tool_operation(
+ "browser_click",
+ Some(&serde_json::json!({"input_route": "private-route"})),
+ ),
+ ToolOperation::Other
+ );
+ assert_eq!(
+ tool_operation(
+ "browser_navigate",
+ Some(&serde_json::json!({"url": "https://private.example"})),
+ ),
+ ToolOperation::NotApplicable
+ );
+ }
+
+ #[test]
+ fn browser_refusals_preserve_protocol_success_without_retaining_content() {
+ let response = Response::ok(
+ serde_json::json!(1),
+ serde_json::json!({
+ "content": [{"type":"text", "text":"private refusal prose"}],
+ "structuredContent": {
+ "status": "refused",
+ "refusal": {
+ "code": "browser_input_trust_unavailable",
+ "message": "private refusal message",
+ "detail": {"candidate": "private tab id"}
+ }
+ }
+ }),
+ );
+ let observation = ToolObservationTimer::start_with_operation(
+ "browser_click".to_owned(),
+ ToolOperation::BrowserClickTrusted,
+ true,
+ true,
+ StdioExecutionPath::InProcess,
+ )
+ .finish(&response);
+ assert!(observation.success);
+ assert_eq!(observation.error_class, ToolErrorClass::None);
+ assert_eq!(
+ observation.refusal_code,
+ ToolRefusalCode::BrowserInputTrustUnavailable
+ );
+ let debug = format!("{observation:?}");
+ for forbidden in ["private refusal prose", "private refusal message", "private tab id"] {
+ assert!(!debug.contains(forbidden), "observer leaked {forbidden}");
+ }
+ }
+
+ #[test]
+ fn browser_refusal_allowlist_matches_the_public_contract() {
+ use crate::browser::refusal::BrowserRefusalCode;
+ for code in [
+ BrowserRefusalCode::BrowserRouteUnavailable,
+ BrowserRefusalCode::BrowserRequiresSetup,
+ BrowserRefusalCode::BrowserBindingAmbiguous,
+ BrowserRefusalCode::BrowserBindingStale,
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ BrowserRefusalCode::BrowserTabRequired,
+ BrowserRefusalCode::BrowserTabNotFound,
+ BrowserRefusalCode::BrowserRefStale,
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ BrowserRefusalCode::BrowserEndpointOwnerMismatch,
+ BrowserRefusalCode::BrowserConsentRequired,
+ BrowserRefusalCode::BrowserConsentRevoked,
+ BrowserRefusalCode::BrowserReconnectExhausted,
+ BrowserRefusalCode::BrowserInputIncomplete,
+ BrowserRefusalCode::BrowserActionUnavailable,
+ ] {
+ assert_eq!(ToolRefusalCode::from(code).as_str(), code.as_str());
+ }
+ let unknown = Response::ok(
+ serde_json::json!(1),
+ serde_json::json!({
+ "structuredContent": {
+ "status": "refused",
+ "refusal": {"code": "private_future_refusal"}
+ }
+ }),
+ );
+ assert_eq!(
+ ToolObservationTimer::start(
+ "browser_click".to_owned(),
+ true,
+ true,
+ StdioExecutionPath::InProcess,
+ )
+ .finish(&unknown)
+ .refusal_code,
+ ToolRefusalCode::Other
+ );
+ }
+
#[test]
fn invalid_params_and_unknown_tool_are_distinct() {
let invalid = Response::error(serde_json::json!(1), -32602, "private invalid params");
@@ -808,5 +1168,13 @@ mod observation_tests {
assert_eq!(OutputType::Empty.as_str(), "empty");
assert_eq!(OutputSizeBucket::MiB1OrMore.as_str(), "gte_1mib");
assert_eq!(ToolErrorClass::TransportError.as_str(), "transport_error");
+ assert_eq!(
+ ToolRefusalCode::BrowserConsentRevoked.as_str(),
+ "browser_consent_revoked"
+ );
+ assert_eq!(
+ ToolOperation::BrowserSnapshotSemanticV2.as_str(),
+ "browser_snapshot_semantic_v2"
+ );
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
index b36680b37c..3e613420eb 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
@@ -229,7 +229,12 @@ fn is_computer_action(tool_name: &str, args: &serde_json::Value) -> bool {
|| capability.starts_with("input.keyboard.")
|| matches!(
capability.as_str(),
- "app.launch" | "app.kill" | "window.activate"
+ "app.launch"
+ | "app.kill"
+ | "window.activate"
+ | "browser.navigate"
+ | "browser.input.click"
+ | "browser.input.type"
)
})
}
@@ -470,6 +475,17 @@ mod tests {
.clone()
}
+ #[test]
+ fn browser_mutations_are_computer_actions_but_reads_and_prepare_are_not() {
+ let args = serde_json::json!({"session": "test-session"});
+ for tool_name in ["browser_navigate", "browser_click", "browser_type"] {
+ assert!(is_computer_action(tool_name, &args), "tool={tool_name}");
+ }
+ for tool_name in ["get_browser_state", "browser_prepare"] {
+ assert!(!is_computer_action(tool_name, &args), "tool={tool_name}");
+ }
+ }
+
#[test]
fn fire_session_end_is_idempotent_per_id() {
// Distinct, test-local ids so we don't collide with other tests that
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
index ae90ee210f..a636d86c70 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
@@ -1804,7 +1804,7 @@ mod telemetry_routing_tests {
fn request(origin: Option<ToolObservationOrigin>) -> DaemonRequest {
DaemonRequest {
method: "call".into(),
- name: Some("click".into()),
+ name: Some("browser_click".into()),
args: Some(serde_json::json!({})),
session_id: Some("bounded-session".into()),
observation_origin: origin,
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
index 7eb0605dd5..6c020bed44 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
@@ -251,6 +251,7 @@ pub fn inspect_event(event_name: &str) -> Result<Value, String> {
("operation", Value::String("not_applicable".into())),
("success", Value::Bool(true)),
("error_class", Value::String("none".into())),
+ ("refusal_code", Value::String("none".into())),
("duration_bucket", Value::String("lt_10ms".into())),
("output_type", Value::String("empty".into())),
("output_size_bucket", Value::String("0".into())),
@@ -276,9 +277,11 @@ pub fn inspect_event(event_name: &str) -> Result<Value, String> {
("tool_count_bucket", Value::String("1_4".into())),
("computer_action_count_bucket", Value::String("1_4".into())),
("error_count_bucket", Value::String("0".into())),
+ ("browser_refusal_count_bucket", Value::String("0".into())),
("had_successful_tool", Value::Bool(true)),
("had_successful_computer_action", Value::Bool(true)),
("used_page", Value::Bool(false)),
+ ("used_browser", Value::Bool(true)),
("used_cursor_tools", Value::Bool(false)),
("used_recording", Value::Bool(false)),
("used_config_write", Value::Bool(false)),
@@ -371,9 +374,11 @@ struct AgentSessionState {
tool_count: u64,
computer_action_count: u64,
error_count: u64,
+ browser_refusal_count: u64,
had_successful_tool: bool,
had_successful_computer_action: bool,
used_page: bool,
+ used_browser: bool,
used_cursor_tools: bool,
used_recording: bool,
used_config_write: bool,
@@ -388,9 +393,11 @@ impl AgentSessionState {
tool_count: 0,
computer_action_count: 0,
error_count: 0,
+ browser_refusal_count: 0,
had_successful_tool: false,
had_successful_computer_action: false,
used_page: false,
+ used_browser: false,
used_cursor_tools: false,
used_recording: false,
used_config_write: false,
@@ -408,14 +415,28 @@ impl AgentSessionState {
if matches!(tool_name, "start_session" | "end_session") {
return;
}
+ let used_browser = matches!(
+ tool_name,
+ "get_browser_state"
+ | "browser_prepare"
+ | "browser_navigate"
+ | "browser_click"
+ | "browser_type"
+ );
+ let refused = outcome.refusal_code.is_refusal();
+ let completed_computer_action = computer_action && !refused;
self.tool_count = self.tool_count.saturating_add(1);
self.computer_action_count = self
.computer_action_count
- .saturating_add(u64::from(computer_action));
+ .saturating_add(u64::from(completed_computer_action));
self.error_count = self.error_count.saturating_add(u64::from(!outcome.success));
+ self.browser_refusal_count = self
+ .browser_refusal_count
+ .saturating_add(u64::from(used_browser && refused));
self.had_successful_tool |= outcome.success;
- self.had_successful_computer_action |= outcome.success && computer_action;
+ self.had_successful_computer_action |= outcome.success && completed_computer_action;
self.used_page |= tool_name == "page";
+ self.used_browser |= used_browser;
self.used_cursor_tools |= matches!(
tool_name,
"set_agent_cursor_enabled"
@@ -469,9 +490,11 @@ impl AgentSessionState {
("tool_count_bucket", Value::String(agent_session_count_bucket(self.tool_count).into())),
("computer_action_count_bucket", Value::String(agent_session_count_bucket(self.computer_action_count).into())),
("error_count_bucket", Value::String(agent_session_error_bucket(self.error_count).into())),
+ ("browser_refusal_count_bucket", Value::String(agent_session_error_bucket(self.browser_refusal_count).into())),
("had_successful_tool", Value::Bool(self.had_successful_tool)),
("had_successful_computer_action", Value::Bool(self.had_successful_computer_action)),
("used_page", Value::Bool(self.used_page)),
+ ("used_browser", Value::Bool(self.used_browser)),
("used_cursor_tools", Value::Bool(self.used_cursor_tools)),
("used_recording", Value::Bool(self.used_recording)),
("used_config_write", Value::Bool(self.used_config_write)),
@@ -752,6 +775,7 @@ fn tool_completion_properties(
("operation", Value::String(outcome.operation.as_str().into())),
("success", Value::Bool(outcome.success)),
("error_class", Value::String(error_class.into())),
+ ("refusal_code", Value::String(outcome.refusal_code.as_str().into())),
("duration_bucket", Value::String(duration_bucket.into())),
("output_type", Value::String(output_type.into())),
("output_size_bucket", Value::String(output_size_bucket.into())),
@@ -2281,6 +2305,7 @@ mod tests {
let tool = inspect_event(event::MCP_TOOL_COMPLETED).unwrap();
assert_eq!(tool["properties"]["duration_bucket"], "lt_10ms");
assert_eq!(tool["properties"]["operation"], "not_applicable");
+ assert_eq!(tool["properties"]["refusal_code"], "none");
assert!(matches!(
tool["properties"]["execution_mode"].as_str(),
Some("embedded" | "standalone")
@@ -2403,6 +2428,7 @@ mod tests {
operation: cua_driver_core::server::ToolOperation::NotApplicable,
success: true,
error_class: ToolErrorClass::None,
+ refusal_code: cua_driver_core::server::ToolRefusalCode::None,
duration_bucket: DurationBucket::Under10Ms,
output_type: OutputType::Text,
output_size_bucket: OutputSizeBucket::Under1KiB,
@@ -2416,6 +2442,7 @@ mod tests {
operation: cua_driver_core::server::ToolOperation::QueryDom,
success: false,
error_class: ToolErrorClass::InternalError,
+ refusal_code: cua_driver_core::server::ToolRefusalCode::None,
duration_bucket: DurationBucket::Ms50To249,
output_type: OutputType::Empty,
output_size_bucket: OutputSizeBucket::Empty,
@@ -2440,6 +2467,8 @@ mod tests {
assert_eq!(properties["had_successful_tool"], true);
assert_eq!(properties["had_successful_computer_action"], true);
assert_eq!(properties["used_page"], true);
+ assert_eq!(properties["used_browser"], false);
+ assert_eq!(properties["browser_refusal_count_bucket"], "0");
assert_eq!(properties["cursor_style"], "custom_icon");
assert_eq!(properties["cursor_color_source"], "custom");
assert_eq!(properties["cursor_label_set"], true);
@@ -2462,6 +2491,46 @@ mod tests {
}
}
+ #[test]
+ fn browser_refusal_is_bounded_and_not_a_completed_computer_action() {
+ use cua_driver_core::server::{
+ DurationBucket, OutputSizeBucket, OutputType, ToolCompletionObservation,
+ ToolErrorClass, ToolOperation, ToolRefusalCode,
+ };
+
+ let outcome = ToolCompletionObservation {
+ tool_name: "browser_click".into(),
+ operation: ToolOperation::BrowserClickTrusted,
+ success: true,
+ error_class: ToolErrorClass::None,
+ refusal_code: ToolRefusalCode::BrowserInputTrustUnavailable,
+ duration_bucket: DurationBucket::Ms50To249,
+ output_type: OutputType::Text,
+ output_size_bucket: OutputSizeBucket::Under1KiB,
+ };
+ let event_properties = tool_completion_properties(outcome.clone());
+ assert_eq!(event_properties["success"], true);
+ assert_eq!(event_properties["error_class"], "none");
+ assert_eq!(
+ event_properties["refusal_code"],
+ "browser_input_trust_unavailable"
+ );
+ assert_eq!(event_properties["operation"], "browser_click_trusted");
+
+ let mut state = AgentSessionState::new(Transport::McpStdio);
+ state.observe(Transport::McpStdio, true, &outcome);
+ let session_properties = state.ended_properties(
+ cua_driver_core::session::SessionEndReason::Explicit,
+ None,
+ );
+ assert_eq!(session_properties["computer_action_count_bucket"], "0");
+ assert_eq!(session_properties["error_count_bucket"], "0");
+ assert_eq!(session_properties["browser_refusal_count_bucket"], "1");
+ assert_eq!(session_properties["had_successful_tool"], true);
+ assert_eq!(session_properties["had_successful_computer_action"], false);
+ assert_eq!(session_properties["used_browser"], true);
+ }
+
#[test]
fn agent_session_buckets_have_fixed_boundaries() {
for (count, expected) in [
@@ -2500,6 +2569,10 @@ mod tests {
use cua_driver_core::session::{
SessionDeclaration, SessionObserver, SessionStartObservation, SessionTransport,
};
+ use cua_driver_core::server::{
+ DurationBucket, OutputSizeBucket, OutputType, ToolCompletionObservation,
+ ToolErrorClass, ToolOperation, ToolRefusalCode,
+ };
let _guard = ENV_LOCK.lock().unwrap();
with_isolated_home(|_| {
@@ -2513,6 +2586,19 @@ mod tests {
transport: SessionTransport::McpStdio,
},
);
+ capture_tool_completed(
+ ToolCompletionObservation {
+ tool_name: "browser_prepare".into(),
+ operation: ToolOperation::BrowserPrepareExistingProfile,
+ success: true,
+ error_class: ToolErrorClass::None,
+ refusal_code: ToolRefusalCode::BrowserConsentRevoked,
+ duration_bucket: DurationBucket::Ms50To249,
+ output_type: OutputType::Text,
+ output_size_bucket: OutputSizeBucket::Under1KiB,
+ },
+ Transport::McpStdio,
+ );
assert!(!AGENT_SESSIONS
.get_or_init(|| Mutex::new(HashMap::new()))
.lock()
@@ -2534,6 +2620,7 @@ mod tests {
operation: cua_driver_core::server::ToolOperation::NotApplicable,
success: true,
error_class: ToolErrorClass::None,
+ refusal_code: cua_driver_core::server::ToolRefusalCode::None,
duration_bucket: DurationBucket::Under10Ms,
output_type: OutputType::Text,
output_size_bucket: OutputSizeBucket::Under1KiB,
@@ -2565,6 +2652,7 @@ mod tests {
operation: ToolOperation::InsertText,
success: true,
error_class: ToolErrorClass::None,
+ refusal_code: cua_driver_core::server::ToolRefusalCode::None,
duration_bucket: DurationBucket::Under10Ms,
output_type: OutputType::Text,
output_size_bucket: OutputSizeBucket::Under1KiB,