Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): complete the browser action surface (#2323)
Francesco Bonacci01a9505aeac5
diff --git a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdxindex 85cc2e02e0..8969a0a2d9 100644--- a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx+++ b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx@@ -18,6 +18,12 @@ Agents still begin with `list_apps` and `list_windows`. They select a concretemints opaque target and tab capabilities. Raw DevTools target identifiers arenot part of the public contract.+Selected-tab state is deliberately tri-state. `active: true` or `false` is+reported only when the native window title uniquely identifies one DevTools+tab. Duplicate titles, empty titles, and other ambiguity produce `active:+null` for every candidate instead of treating DevTools list order as native+selection evidence. A caller can still target an explicit returned tab id.+This preserves one targeting model across native and web content. It alsoprevents a browser helper from choosing the first tab, first window, or firstprocess match when several candidates exist.@@ -54,13 +60,24 @@ default click uses `Input.dispatchMouseEvent`, but Chromium's standalone windowis known to activate on that route on macOS and Linux. Cua Driver returns`browser_input_trust_unavailable` there before dispatch. Standalone Chrome andEdge on Windows, and the bounded embedded Electron route, have passing trusted-background evidence.+background evidence. The same trust distinction applies to hover,+right-click, double-click, scroll, and drag through `browser_pointer`.A caller must explicitly request `input_route: "dom_event"` to invoke anelement's DOM click behavior. That route is synthetic even when it preservesfull-background posture. The driver does not silently change trust models tomake a call appear successful.+Page-owned JavaScript dialogs are modeled as short-lived capabilities rather+than native-window guesses. Inspection returns the kind and an opaque dialog+generation; accept or dismiss succeeds only while that exact dialog remains+current. The page's initial creation of a Chromium native modal may activate+the browser; after occlusion is re-established, inspection and resolution do+not require another activation. File assignment bypasses native pickers through an exact live file+input ref. Downloads additionally cross an open-world filesystem boundary, so+they require host approval, a canonical destination directory, exact event+correlation, and path-free output.+## Capabilities have a lifetimeTarget ids, tab ids, and page refs belong to a named driver session. Page refsdiff --git a/docs/content/docs/concepts/how-cua-driver-is-validated.mdx b/docs/content/docs/concepts/how-cua-driver-is-validated.mdxindex 6dedc418a1..e9e0e78d10 100644--- a/docs/content/docs/concepts/how-cua-driver-is-validated.mdx+++ b/docs/content/docs/concepts/how-cua-driver-is-validated.mdx@@ -41,6 +41,15 @@ keys, scrolling, dragging, child windows, controls, and editor behavior. Thismakes a failure reproducible without depending on the changing state of aninstalled third-party application.+The optional standalone-browser lane launches installed Chrome, Edge, or+Chromium with a fresh repo-owned profile and one exact source-built driver. Its+rows cover binding and ambiguous active-tab evidence, semantic snapshots,+frames, inactive tabs, JavaScript dialogs, file assignment, extended pointer+actions, and approval-scoped downloads. The target stays fully occluded behind+the foreground sentinel while the fixture journal proves page state. This lane+is the release evidence for typed browser behavior; CDP acknowledgements alone+do not count as delivery.+## One catalog describes each behavior cellThe Rust catalog records the dimensions that affect delivery:diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdxindex aa6423a1be..c23e70cded 100644--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx@@ -154,9 +154,13 @@ get_browser_state({})```-Keep the returned opaque `target_id` and the `tab_id` whose `active` field is-`true`. Mutation is available only when `binding_quality` is `exact`. A-heuristic or ambiguous match is refused.+Keep the returned opaque `target_id`. Use the `tab_id` whose `active` field is+`true` only when selection is uniquely proven. `active: false` is proven+unselected; `active: null` means the native window cannot distinguish tabs,+such as two tabs with the same title. In that case, select an explicit tab by+its returned metadata and never infer selection from list order. Mutation is+available only when `binding_quality` is `exact`. A heuristic or ambiguous+window match is refused.## Snapshot the tab@@ -249,6 +253,61 @@ browser_type({The default `insert_text` mode is efficient for ordinary text. Use`mode: "keystrokes"` when the page depends on per-character keyboard events.+## Use extended pointer actions++`browser_pointer` supports `hover`, `right_click`, `double_click`, `scroll`,+and `drag`. Use the trusted route when genuine browser input semantics are+required and the platform reports that it can preserve posture. Use the+explicit synthetic route with current refs when DOM event semantics are+acceptable:++```jsonc+browser_pointer({+ "target_id": "<target_id>",+ "tab_id": "<tab_id>",+ "ref": "p3:4",+ "action": "scroll",+ "input_route": "dom_event",+ "delta_y": 240,+ "session": "research-1"+})+```++For drag, pass `destination_ref` from the same exact frame. The driver refuses+mixed-frame or stale destinations instead of translating them approximately.++## Handle a page-owned dialog++Prime and inspect the exact tab with `browser_dialog({action:"inspect"})`.+When `present` is true, pass the returned `dialog_id` to `accept` or `dismiss`.+Only an accepted prompt may include `prompt_text`. This tool does not handle+browser permission prompts, extension UI, native sheets, or file pickers.+Creating a Chromium native modal can activate its browser window; once you+restore the intended occlusion, inspecting and resolving that exact dialog do+not activate it again on Windows and macOS. Resolution defaults to+`delivery_mode:"background"`. Linux Chromium cannot resolve the native modal+while preserving background posture, so the driver returns+`browser_input_trust_unavailable` before dispatch. Retry with+`delivery_mode:"foreground"` only when foreground activation is acceptable.++## Assign files without a picker++Take a semantic snapshot and choose a file-input ref whose `actions` contains+`upload`. Call `browser_set_input_files` with absolute paths to direct regular+files. Symlinks, directories, missing paths, and more than 32 files are+refused. Verify the page's uploaded-file state afterward; the tool response+contains only a count and never echoes the local paths.++## Download into an approved directory++Choose a current ref that activates the download and call `browser_download`+through an MCP host that presents its destructive-tool approval. Pass an+existing canonical absolute `destination_root`. The tool temporarily scopes+Chromium's browser-wide download behavior, correlates the exact frame and+opaque download id, restores the default on every outcome, and returns only+the final byte count and opaque id. It does not return a filename, URL, or+local path. A direct raw call without host approval is refused.+## Navigate```jsoncdiff --git a/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdxindex 131525af32..74fa8ced16 100644--- a/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx+++ b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx@@ -97,9 +97,11 @@ Entries in `refs` declare their supported action kinds:}```-`browser_click` and `browser_type` reject a semantic ref when the requested-action is absent from `actions`, returning `browser_action_unavailable` before-delivery. Legacy `dom_refs_v1` refs retain their existing behavior.+The closed action vocabulary is `click`, `type`, and `upload`.+`browser_click`, `browser_type`, `browser_set_input_files`, and+`browser_download` reject a semantic ref when the required action is absent,+returning `browser_action_unavailable` before delivery. A file input declares+`upload`, not `type`. Legacy `dom_refs_v1` refs retain their existing behavior.## Content refsdiff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex cc7ce61231..726273d5ef 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposesimport { Callout } from 'fumadocs-ui/components/callout';-`cua-driver` exposes 43 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.+`cua-driver` exposes 47 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`.@@ -425,7 +425,7 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number- `y` (number, optional): Window-local screenshot Y. See `x`.```json-{"direction":"down"}+{"direction":"up"}```### `move_cursor`@@ -493,7 +493,7 @@ Actions:- `window_id` (integer, optional): Target window ID from list_windows.```json-{"action":"get_text"}+{"action":"execute_javascript"}```## Recording tools@@ -827,3 +827,77 @@ Type text into an exactly-bound tab via the Input domain. mode="insert_text" (de```json{"ref":"example","tab_id":"example","target_id":"example","text":"hello"}```++### `browser_dialog`++Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.++**Arguments:**++- `action` (string, required)+- `delivery_mode` (string, optional): Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only.+- `dialog_id` (string, optional): Opaque current dialog generation returned by action=inspect.+- `prompt_text` (string, optional): Sensitive response text, valid only when accepting a prompt dialog.+- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.+- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).+- `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).++```json+{"action":"inspect","tab_id":"example","target_id":"example"}+```++### `browser_set_input_files`++Assign one or more explicit absolute local files to an exact live <input type=file> ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.++**Arguments:**++- `files` (array of string, required)+- `ref` (string, required): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.+- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.+- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).+- `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).++```json+{"files":["example"],"ref":"example","tab_id":"example","target_id":"example"}+```++### `browser_download`++Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.++**Arguments:**++- `destination_root` (string, required): Absolute, existing, canonical directory approved to receive the download.+- `ref` (string, required): Live page ref whose activation initiates the download.+- `session` (string, required): Explicit caller session owning the browser capabilities.+- `tab_id` (string, required): Opaque exact tab id from get_browser_state.+- `target_id` (string, required): Opaque exact browser target id from get_browser_state.++```json+{"destination_root":"example","ref":"example","session":"example","tab_id":"example","target_id":"example"}+```++### `browser_pointer`++Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.++**Arguments:**++- `action` (string, required)+- `delta_x` (number, optional): Horizontal scroll delta in CSS pixels.+- `delta_y` (number, optional): Vertical scroll delta in CSS pixels.+- `destination_ref` (string, optional): Drag destination page ref in the exact same frame.+- `input_route` (string, optional)+- `ref` (string, optional): Origin page ref. Alternative to x/y.+- `session` (string, required): Explicit caller session owning the browser capabilities.+- `tab_id` (string, required): Opaque tab id minted by get_browser_state.+- `target_id` (string, required): Opaque target id minted by get_browser_state.+- `to_x` (number, optional): Drag destination viewport x in CSS pixels.+- `to_y` (number, optional): Drag destination viewport y in CSS pixels.+- `x` (number, optional): Origin viewport x in CSS pixels.+- `y` (number, optional): Origin viewport y in CSS pixels.++```json+{"action":"hover","session":"example","tab_id":"example","target_id":"example"}+```diff --git a/docs/content/docs/reference/cua-driver/platform-support.mdx b/docs/content/docs/reference/cua-driver/platform-support.mdxindex c1e347f22d..24398dcbd7 100644--- a/docs/content/docs/reference/cua-driver/platform-support.mdx+++ b/docs/content/docs/reference/cua-driver/platform-support.mdx@@ -38,9 +38,9 @@ harnesses; an unlisted trust class or host shape is not implied.| Surface | Proven page routes | Explicit limit || --- | --- | --- |-| Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse |-| Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |-| Standalone Chrome on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |+| Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse |+| Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |+| Standalone Chrome and Edge on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, foreground JavaScript-dialog resolution, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input and background JavaScript-dialog resolution return `browser_input_trust_unavailable` before dispatch || Electron on Windows, macOS, X11, and validated Sway | Typed mutation while one proven native window maps to one CDP page | A second page or native window invalidates the bounded exact route || Tauri, WKWebView, WebKitGTK, and common split-process WebView2 | Browser identity plus a side-effect-free structured refusal | No exact native-host-to-engine relationship is currently proven || Safari and Firefox | Native discovery and native AX/PX fallbacks | No typed browser mutation engine is currently advertised |diff --git a/docs/content/docs/reference/cua-driver/telemetry.mdx b/docs/content/docs/reference/cua-driver/telemetry.mdxindex 4344e54bf6..506a0eee80 100644--- a/docs/content/docs/reference/cua-driver/telemetry.mdx+++ b/docs/content/docs/reference/cua-driver/telemetry.mdx@@ -123,7 +123,7 @@ The start event is emitted for a successful `start_session` declaration or the fThe end event is emitted for `end_session` and idle eviction. It contains aggregate counters and booleans. When the platform has a cursor entry for that session, it also includes bounded cursor categories: enabled state, built-in/default/custom icon class, automatic/custom color source, label presence, motion customization, and an active-cursor count bucket. `cursor_outcome_observed=false` distinguishes sessions with no readable cursor entry from a disabled or default cursor. Cua Driver uses the caller session string only as a process-local map key. The string, a hash of it, and a replacement join token are absent from telemetry payloads.-Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful `browser_navigate`, `browser_click`, and `browser_type` calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` is a read, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect.+Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful browser navigation, click, type, pointer, file-assignment, download, or dialog-resolution calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` and `browser_dialog` inspection are reads for this classifier, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect.The end event includes `used_browser` and a bounded `browser_refusal_count_bucket`. It does not include browser targets, pages, profiles, or per-site information.@@ -141,6 +141,10 @@ Typed browser tools use these reviewed operation values:| `browser_prepare` | `browser_prepare_isolated`, `browser_prepare_existing_profile`, or `other` || `browser_click` | `browser_click_trusted`, `browser_click_dom_event`, or `other` || `browser_type` | `browser_type_insert_text`, `browser_type_keystrokes`, or `other` |+| `browser_dialog` | `browser_dialog_inspect`, `browser_dialog_accept`, `browser_dialog_dismiss`, or `other` |+| `browser_set_input_files` | `browser_set_input_files` |+| `browser_download` | `browser_download` |+| `browser_pointer` | Action-and-route values such as `browser_pointer_scroll_trusted` or `browser_pointer_drag_dom_event`; invalid combinations become `other` || `browser_navigate` | `not_applicable`; the tool name already identifies the operation |### Structured browser refusalsdiff --git a/libs/cua-driver/docs/browser-tool-implementation-journal.md b/libs/cua-driver/docs/browser-tool-implementation-journal.mdindex c3f68cd7a3..836a118187 100644--- a/libs/cua-driver/docs/browser-tool-implementation-journal.md+++ b/libs/cua-driver/docs/browser-tool-implementation-journal.md@@ -6,9 +6,8 @@ URLs, profile paths, and raw DevTools endpoint identifiers.## Definition of done-- Add a capability-aware browser API with five typed tools:- `get_browser_state`, `browser_prepare`, `browser_navigate`, `browser_click`,- and `browser_type`.+- Add a capability-aware browser API whose typed tools share one exact target,+ tab, frame, and ref model.- Keep `get_browser_state` read-only and make setup explicit.- Bind native `(pid, window_id)` targets to browser tabs exactly or refuse allmutation. Never promote a heuristic match into an action route.@@ -719,3 +718,41 @@ link checks. A source-installed macOS product-path replay passed the real Chromesemantic row with the TCC-authorized daemon, foreground sentinel, fixturejournal, exact DOM click, re-snapshot, and typed-value oracle. Windows and LinuxX11 exact-SHA workflow evidence is the remaining release-acceptance gate.++## 2026-07-18: browser completeness and adversarial tab state++The bind response no longer labels a representative DevTools target as the+active native tab. A unique exact native-title match produces one `true` and+the remaining tabs `false`; duplicate, empty, substring-only, or otherwise+ambiguous titles produce `null` for every tab. The standalone harness adds a+same-title adversarial row so future list-order regressions are externally+visible.++The typed surface now includes page-owned JavaScript dialog inspection and+resolution, exact file-input assignment, extended pointer actions, and+approval-scoped downloads. All mutations retain the existing exact binding,+tab lock, reconnect generation, ref/frame identity, and fail-closed posture.+Synthetic pointer actions are explicit `dom_event` requests; trusted pointer+actions never fall back after a refusal. Downloads serialize Chromium's+browser-wide setting, subscribe before triggering, correlate exact frame and+opaque id events, restore defaults on every outcome, and remove only a proven+direct partial file after failure.++The real-browser harness adds separate rows for dialogs, upload,+hover/right-click/double-click/scroll/drag, and downloads. A fixed loopback+attachment keeps download evidence deterministic. Background rows use the+foreground sentinel and fixture-owned state rather than treating a CDP reply+as delivery. Linux Chromium cannot resolve a native JavaScript modal without+changing foreground posture, so its matrix records a no-dispatch background+refusal and a separate explicit foreground delivery row. Windows and macOS+retain strict background dialog resolution. Trajectory recording redacts+prompt text, upload paths, download directories, and internal approval+markers; product telemetry retains only fixed operation and refusal+categories.++The dialog design adapts the exact-target, no-replay security model introduced+in PR #2166. Its contributor remains credited through a parsed+`Co-authored-by` trailer in the landing history and `Salvaged from #2166` in+the pull request body even though this implementation supersedes that earlier+code shape. Issue authors remain reporter-attributed when the landing PR+closes their linked issues.diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.mdindex 73646a4848..21592797a1 100644--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md@@ -20,7 +20,8 @@ list_windows or launch_appget_browser_state(pid, window_id, session) # bindget_browser_state(target_id, tab_id, session,snapshot_format=semantic_v2) # snapshot-browser_navigate / browser_click / browser_type+browser_navigate / browser_click / browser_type / browser_pointer+browser_dialog / browser_set_input_files / browser_downloadget_browser_state(target_id, tab_id, session,snapshot_format=semantic_v2) # verify and refresh refsend_session@@ -119,7 +120,10 @@ Never:## 3. Snapshot the selected tab-Choose a returned `tab_id`, then request the page snapshot:+Choose a returned `tab_id`, then request the page snapshot. `active` is+tri-state: `true` is a uniquely proven selected tab, `false` is a proven+unselected tab, and `null` means native evidence cannot distinguish the+selection. Never guess from list order when all tabs are `null`.```bashcua-driver get_browser_state \@@ -240,14 +244,61 @@ ownership, and reports requested versus delivered characters. Snapshot againto verify application state rather than treating transport completion as thetask result.+### Extended pointer actions++Use `browser_pointer` for `hover`, `right_click`, `double_click`, `scroll`, and+`drag`. It uses the same `trusted` versus explicit `dom_event` distinction as+`browser_click`. The synthetic route requires a current ref; drag also requires+`destination_ref` in the same proven frame. Coordinate origins and destinations+are available only where the trusted route can preserve the requested posture.++```bash+cua-driver browser_pointer \+ '{"target_id":"<target>","tab_id":"<tab>","ref":"p5:2",+ "action":"scroll","input_route":"dom_event","delta_y":240,+ "session":"browser-run-1"}'+```++### JavaScript dialogs++`browser_dialog` handles only page-owned `alert`, `confirm`, `prompt`, and+`beforeunload` dialogs. First inspect the exact tab, then accept or dismiss the+returned opaque `dialog_id`. A prompt response is allowed only with+`action:"accept"` on a current prompt. Browser permission UI and native dialogs+remain outside this tool. Creating Chromium's native modal can activate the+browser; after the caller restores occlusion, inspecting and resolving the+exact page-owned dialog do not require another activation on Windows and+macOS. Resolution defaults to `delivery_mode:"background"`. Linux Chromium+cannot resolve its native modal while preserving background posture, so the+driver refuses that mode before dispatch; retry explicitly with+`delivery_mode:"foreground"` when foreground activation is acceptable.++### File inputs++Use a current semantic ref whose `actions` contains `upload`, then call+`browser_set_input_files` with one to 32 absolute regular-file paths. The tool+rejects symlinks and directories, bypasses the native file picker, and returns+only the assigned file count. Paths are redacted from trajectory arguments.++### Downloads++`browser_download` activates one exact ref under a destructive MCP-host+approval and saves the result under an existing canonical absolute+`destination_root`. It correlates browser download events to the exact frame,+serializes Chromium's browser-wide download setting, restores that setting on+every outcome, and returns only an opaque download id and byte count. It never+returns the source URL, filename, or destination path. Direct raw calls without+the host approval proof are refused.+## Browser chrome and native fallbacksThe browser tools operate on page content, not the surrounding native UI. Usethe normal native loop for:- tabs, address bar, menus, bookmarks, and extension UI;-- permission prompts and remote-debugging consent UI;-- downloads, save dialogs, authentication sheets, and file pickers;+- permission prompts, remote-debugging consent UI, and authentication sheets;+- native save dialogs and file pickers that are not represented by an exact+ page ref;- WebView2, WKWebView, WebKitGTK, Tauri, or Electron surfaces that cannot beexactly correlated to a page target;- Safari and Firefox, whose typed mutation engines are not yet supported.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rsindex a445ab9c62..252818b903 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs@@ -86,6 +86,35 @@ fn title_matches(native_title: &str, candidate_title: &str) -> bool {!candidate_title.is_empty() && native_title.contains(candidate_title)}+/// Prove the selected page in one already-correlated browser window without+/// activating a target. Chromium exposes the native window title but no+/// browser-level selected-tab bit. A unique title match is therefore proof;+/// duplicate, empty, absent, or substring-colliding titles remain unknown.+/// Embedded single-page endpoints are selected by cardinality.+pub fn selected_tab_target_id<'a>(+ native_title: &str,+ candidates: &'a [CdpWindowCandidate],+ cdp_window_id: Option<i64>,+) -> Option<&'a str> {+ let tabs = candidates+ .iter()+ .filter(|candidate| match cdp_window_id {+ Some(window_id) => candidate.cdp_window_id == Some(window_id),+ None => candidate.cdp_window_id.is_none(),+ })+ .collect::<Vec<_>>();++ if cdp_window_id.is_none() {+ return (tabs.len() == 1).then(|| tabs[0].cdp_target_id.as_str());+ }++ let title_hits = tabs+ .into_iter()+ .filter(|candidate| title_matches(native_title, &candidate.title))+ .collect::<Vec<_>>();+ (title_hits.len() == 1).then(|| title_hits[0].cdp_target_id.as_str())+}+/// Reduce page targets to one representative per proven CDP window./// The representative prefers a tab title displayed by the native/// window, but its target id is only a handle for the selected window;@@ -328,6 +357,45 @@ mod tests {));}+ #[test]+ fn selected_tab_requires_one_unique_native_title_match() {+ let tabs = [cand("first", "Checkout", B), cand("second", "Cart", B)];+ assert_eq!(+ selected_tab_target_id("Cart - Chrome", &tabs, Some(1)),+ Some("second")+ );++ let duplicate_titles = [cand("first", "Checkout", B), cand("second", "Checkout", B)];+ assert_eq!(+ selected_tab_target_id("Checkout - Chrome", &duplicate_titles, Some(1)),+ None+ );++ let substring_collision = [cand("first", "Mail", B), cand("second", "Mail Inbox", B)];+ assert_eq!(+ selected_tab_target_id("Mail Inbox - Chrome", &substring_collision, Some(1)),+ None+ );++ let empty_titles = [cand("first", "", B), cand("second", "", B)];+ assert_eq!(+ selected_tab_target_id("Chrome", &empty_titles, Some(1)),+ None+ );+ }++ #[test]+ fn embedded_single_page_is_selected_by_cardinality_only() {+ let one = [embedded_cand("only", "Fixture")];+ assert_eq!(selected_tab_target_id("Fixture", &one, None), Some("only"));++ let two = [+ embedded_cand("first", "Fixture"),+ embedded_cand("second", "Fixture"),+ ];+ assert_eq!(selected_tab_target_id("Fixture", &two, None), None);+ }+#[test]fn unique_title_disambiguates_distinct_maximized_windows_with_shared_bounds() {let n = native("Docs - Chrome", B);diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rsindex 122438242c..793a173feb 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs@@ -61,6 +61,15 @@ pub struct CdpEvent {pub params: Value,}+/// Bounded state for one page-owned JavaScript dialog. Message text and URLs+/// are deliberately not retained.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct CdpDialogState {+ pub generation: u64,+ pub kind: String,+ pub session_id: String,+}+/// What the reader routed back for one in-flight call.enum CallOutcome {Result(Value),@@ -71,6 +80,9 @@ enum CallOutcome {struct Demux {pending: StdMutex<HashMap<u64, oneshot::Sender<CallOutcome>>>,subscribers: StdMutex<Vec<mpsc::UnboundedSender<CdpEvent>>>,+ session_targets: StdMutex<HashMap<String, String>>,+ dialogs: StdMutex<HashMap<String, CdpDialogState>>,+ next_dialog_generation: AtomicU64,closed: AtomicBool,}@@ -81,6 +93,8 @@ impl Demux {// error, which `call` reports as a closed socket.self.pending.lock().unwrap().clear();self.subscribers.lock().unwrap().clear();+ self.session_targets.lock().unwrap().clear();+ self.dialogs.lock().unwrap().clear();}}@@ -126,6 +140,43 @@ async fn read_loop(mut read: SplitStream<WsStream>, demux: Arc<Demux>) {.map(str::to_owned),params: v.get("params").cloned().unwrap_or(Value::Null),};+ if let Some(session_id) = event.session_id.as_deref() {+ let target_id = demux+ .session_targets+ .lock()+ .unwrap()+ .get(session_id)+ .cloned();+ match event.method.as_str() {+ "Page.javascriptDialogOpening" => {+ let kind = match event.params.get("type").and_then(Value::as_str) {+ Some("alert") => "alert",+ Some("confirm") => "confirm",+ Some("prompt") => "prompt",+ Some("beforeunload") => "beforeunload",+ _ => "other",+ };+ let generation =+ demux.next_dialog_generation.fetch_add(1, Ordering::Relaxed);+ if let Some(target_id) = target_id {+ demux.dialogs.lock().unwrap().insert(+ target_id,+ CdpDialogState {+ generation,+ kind: kind.to_owned(),+ session_id: session_id.to_owned(),+ },+ );+ }+ }+ "Page.javascriptDialogClosed" => {+ if let Some(target_id) = target_id {+ demux.dialogs.lock().unwrap().remove(&target_id);+ }+ }+ _ => {}+ }+ }demux.subscribers.lock()@@ -162,6 +213,9 @@ impl CdpConnection {let demux = Arc::new(Demux {pending: StdMutex::new(HashMap::new()),subscribers: StdMutex::new(Vec::new()),+ session_targets: StdMutex::new(HashMap::new()),+ dialogs: StdMutex::new(HashMap::new()),+ next_dialog_generation: AtomicU64::new(1),closed: AtomicBool::new(false),});let reader = tokio::spawn(read_loop(read, demux.clone()));@@ -190,6 +244,42 @@ impl CdpConnection {rx}+ /// Associate the one Page-enabled dialog session with its exact target.+ /// Later calls may use fresh attachment sessions, but dialog events keep+ /// arriving on this bounded, persistent event session.+ pub fn register_dialog_session(&self, session_id: &str, target_id: &str) {+ let mut sessions = self.demux.session_targets.lock().unwrap();+ sessions.retain(|_, mapped_target| mapped_target != target_id);+ sessions.insert(session_id.to_owned(), target_id.to_owned());+ }++ pub fn unregister_dialog_session(&self, session_id: &str, target_id: &str) {+ let mut sessions = self.demux.session_targets.lock().unwrap();+ if sessions.get(session_id).map(String::as_str) == Some(target_id) {+ sessions.remove(session_id);+ }+ }++ pub fn has_dialog_session(&self, target_id: &str) -> bool {+ self.demux+ .session_targets+ .lock()+ .unwrap()+ .values()+ .any(|mapped_target| mapped_target == target_id)+ }++ pub fn dialog_state(&self, target_id: &str) -> Option<CdpDialogState> {+ self.demux.dialogs.lock().unwrap().get(target_id).cloned()+ }++ pub fn clear_dialog_state(&self, target_id: &str, generation: u64) {+ let mut dialogs = self.demux.dialogs.lock().unwrap();+ if dialogs.get(target_id).map(|state| state.generation) == Some(generation) {+ dialogs.remove(target_id);+ }+ }+/// Issue one CDP command and await its `id`-matched reply./// `session_id` targets an attached (flattened) target session./// Returns the `result` object, or an error carrying the CDP@@ -589,6 +679,41 @@ mod tests {assert!(events.try_recv().is_err(), "no phantom events");}+ #[tokio::test]+ async fn javascript_dialog_journal_is_bounded_and_generation_checked() {+ let server = MockCdpServer::start(StdArc::new(|call| match call.method.as_str() {+ "Dialog.open" => MockReply::ok(json!({})).with_events(vec![MockEvent {+ method: "Page.javascriptDialogOpening".into(),+ session_id: call.session_id.clone(),+ params: json!({+ "type": "prompt",+ "message": "private dialog text",+ "url": "https://private.example"+ }),+ }]),+ "Dialog.close" => MockReply::ok(json!({})).with_events(vec![MockEvent {+ method: "Page.javascriptDialogClosed".into(),+ session_id: call.session_id.clone(),+ params: json!({}),+ }]),+ _ => MockReply::ok(json!({})),+ }))+ .await;+ let conn = CdpConnection::connect(&server.ws_url()).await.unwrap();+ conn.register_dialog_session("tab-session-a", "page-target");+ conn.call(Some("tab-session-a"), "Dialog.open", json!({}))+ .await+ .unwrap();+ let dialog = conn.dialog_state("page-target").expect("dialog journaled");+ assert_eq!(dialog.kind, "prompt");+ assert_eq!(dialog.session_id, "tab-session-a");+ assert!(!format!("{dialog:?}").contains("private"));+ conn.call(Some("tab-session-a"), "Dialog.close", json!({}))+ .await+ .unwrap();+ assert!(conn.dialog_state("page-target").is_none());+ }+#[tokio::test]async fn out_of_order_replies_route_to_the_right_callers() {// Raw server: read two requests, reply to the SECOND first.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rsnew file mode 100644index 0000000000..46871f5218--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rs@@ -0,0 +1,593 @@+//! Approval-gated downloads for exact browser tab and element capabilities.+//!+//! Chromium's download behavior is browser-wide, so this tool enables it only+//! for the bounded interval around one exact ref mutation and restores the+//! default before returning. Output deliberately omits URLs, filenames, and+//! local paths.++use std::path::{Component, Path, PathBuf};+use std::sync::{Arc, OnceLock};+use std::time::Duration;++use async_trait::async_trait;+use serde_json::{json, Value};++use crate::protocol::ToolResult;+use crate::tool::{Tool, ToolDef};+use crate::tool_args::ArgsExt;++use super::cdp_ws::CdpConnection;+use super::engine::BrowserEngine;+use super::refusal::{BrowserRefusal, BrowserRefusalCode};+use super::store::BrowserActionKind;++/// Injected by an MCP host only after its destructive-tool approval flow.+/// It is intentionally absent from the public input schema.+pub const MCP_HOST_DOWNLOAD_APPROVAL_ARG: &str = "_cua_browser_download_mcp_host_approved";++const DOWNLOAD_TIMEOUT: Duration = Duration::from_secs(30);++fn browser_download_gate() -> &'static tokio::sync::Mutex<()> {+ static GATE: OnceLock<tokio::sync::Mutex<()>> = OnceLock::new();+ GATE.get_or_init(|| tokio::sync::Mutex::new(()))+}++pub struct BrowserDownloadTool {+ def: ToolDef,+ engine: Arc<BrowserEngine>,+}++impl BrowserDownloadTool {+ pub fn new(engine: Arc<BrowserEngine>) -> Self {+ Self {+ def: ToolDef {+ name: "browser_download".into(),+ description: "Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.".into(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "session": {+ "type": "string",+ "description": "Explicit caller session owning the browser capabilities."+ },+ "target_id": {+ "type": "string",+ "description": "Opaque exact browser target id from get_browser_state."+ },+ "tab_id": {+ "type": "string",+ "description": "Opaque exact tab id from get_browser_state."+ },+ "ref": {+ "type": "string",+ "description": "Live page ref whose activation initiates the download."+ },+ "destination_root": {+ "type": "string",+ "description": "Absolute, existing, canonical directory approved to receive the download."+ }+ },+ "required": ["session", "target_id", "tab_id", "ref", "destination_root"],+ "additionalProperties": true+ }),+ read_only: false,+ destructive: true,+ idempotent: false,+ open_world: true,+ },+ engine,+ }+ }+}++fn consent_required() -> ToolResult {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserConsentRequired,+ "browser_download requires approval through the MCP host's destructive-tool confirmation flow",+ )+ .to_tool_result()+}++fn explicit_session(args: &Value) -> Result<String, ToolResult> {+ let session = args.require_str("session")?;+ if session.trim().is_empty() || session == "default" {+ return Err(ToolResult::error(+ "browser_download requires an explicit non-default session",+ ));+ }+ Ok(session)+}++fn validate_destination_root(raw: &str) -> Result<PathBuf, ToolResult> {+ let path = Path::new(raw);+ if !path.is_absolute() {+ return Err(ToolResult::error("destination_root must be absolute"));+ }++ let metadata = std::fs::symlink_metadata(path)+ .map_err(|_| ToolResult::error("destination_root must be an existing directory"))?;+ if metadata.file_type().is_symlink() {+ return Err(ToolResult::error(+ "destination_root must name the directory directly, not a symlink",+ ));+ }+ if !metadata.is_dir() {+ return Err(ToolResult::error(+ "destination_root must be an existing directory",+ ));+ }++ let canonical = std::fs::canonicalize(path)+ .map_err(|_| ToolResult::error("destination_root could not be canonicalized"))?;+ if canonical != path {+ return Err(ToolResult::error(+ "destination_root must already be in canonical form",+ ));+ }+ Ok(canonical)+}++fn download_path_for_cdp(path: &Path) -> String {+ let raw = path.to_string_lossy();+ #[cfg(target_os = "windows")]+ {+ // `canonicalize` deliberately gives containment checks a verbatim+ // Windows path. Chromium's download API does not accept that transport+ // spelling, so remove only the namespace prefix after validation.+ if let Some(unc) = raw.strip_prefix(r"\\?\UNC\") {+ return format!(r"\\{unc}");+ }+ if let Some(local) = raw.strip_prefix(r"\\?\") {+ return local.to_owned();+ }+ }+ raw.into_owned()+}++fn main_frame_id(frame_tree: &Value) -> Option<&str> {+ frame_tree+ .get("frameTree")?+ .get("frame")?+ .get("id")?+ .as_str()+}++fn is_safe_guid(guid: &str) -> bool {+ let mut components = Path::new(guid).components();+ matches!(components.next(), Some(Component::Normal(_))) && components.next().is_none()+}++fn completed_download_size(root: &Path, guid: &str) -> Result<u64, BrowserRefusal> {+ if !is_safe_guid(guid) {+ return Err(BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the browser returned an invalid opaque download id",+ ));+ }+ let file = root.join(guid);+ let metadata = std::fs::symlink_metadata(&file).map_err(|_| {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the completed download could not be proven inside the approved directory",+ )+ })?;+ if metadata.file_type().is_symlink() || !metadata.is_file() {+ return Err(BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the completed download is not a direct regular file",+ ));+ }+ let canonical = std::fs::canonicalize(&file).map_err(|_| {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the completed download could not be canonicalized",+ )+ })?;+ if canonical.parent() != Some(root) {+ return Err(BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the completed download escaped the approved directory",+ ));+ }+ Ok(metadata.len())+}++async fn reset_download_behavior(conn: &CdpConnection) {+ let _ = conn+ .call(+ None,+ "Browser.setDownloadBehavior",+ json!({ "behavior": "default", "eventsEnabled": false }),+ )+ .await;+}++enum DownloadEventOutcome {+ Completed { guid: String },+ Canceled { guid: String },+ EventStreamClosed,+}++async fn await_download(+ events: &mut tokio::sync::mpsc::UnboundedReceiver<super::cdp_ws::CdpEvent>,+ expected_frame_id: &str,+ observed_guid: &mut Option<String>,+) -> DownloadEventOutcome {+ while let Some(event) = events.recv().await {+ match event.method.as_str() {+ "Browser.downloadWillBegin" if observed_guid.is_none() => {+ if event.params.get("frameId").and_then(Value::as_str) != Some(expected_frame_id) {+ continue;+ }+ *observed_guid = event+ .params+ .get("guid")+ .and_then(Value::as_str)+ .map(str::to_owned);+ }+ "Browser.downloadProgress" => {+ let Some(guid) = observed_guid.as_deref() else {+ continue;+ };+ if event.params.get("guid").and_then(Value::as_str) != Some(guid) {+ continue;+ }+ match event.params.get("state").and_then(Value::as_str) {+ Some("completed") => {+ return DownloadEventOutcome::Completed {+ guid: guid.to_owned(),+ }+ }+ Some("canceled") => {+ return DownloadEventOutcome::Canceled {+ guid: guid.to_owned(),+ }+ }+ _ => {}+ }+ }+ _ => {}+ }+ }+ DownloadEventOutcome::EventStreamClosed+}++fn remove_proven_partial(root: &Path, guid: Option<&str>) {+ let Some(guid) = guid.filter(|guid| is_safe_guid(guid)) else {+ return;+ };+ let path = root.join(guid);+ let Ok(metadata) = std::fs::symlink_metadata(&path) else {+ return;+ };+ if metadata.file_type().is_symlink() || !metadata.is_file() {+ return;+ }+ let Ok(canonical) = std::fs::canonicalize(&path) else {+ return;+ };+ if canonical.parent() == Some(root) {+ let _ = std::fs::remove_file(canonical);+ }+}++#[async_trait]+impl Tool for BrowserDownloadTool {+ fn def(&self) -> &ToolDef {+ &self.def+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let (session, target_id, tab_id, ext_ref, destination_raw) = match (+ explicit_session(&args),+ args.require_str("target_id"),+ args.require_str("tab_id"),+ args.require_str("ref"),+ args.require_str("destination_root"),+ ) {+ (Ok(session), Ok(target), Ok(tab), Ok(ext_ref), Ok(destination)) => {+ (session, target, tab, ext_ref, destination)+ }+ (Err(error), _, _, _, _)+ | (_, Err(error), _, _, _)+ | (_, _, Err(error), _, _)+ | (_, _, _, Err(error), _)+ | (_, _, _, _, Err(error)) => return error,+ };++ if args+ .get(MCP_HOST_DOWNLOAD_APPROVAL_ARG)+ .and_then(Value::as_bool)+ != Some(true)+ {+ return consent_required();+ }+ let destination_root = match validate_destination_root(&destination_raw) {+ Ok(root) => root,+ Err(error) => return error,+ };++ let _mutation = match self+ .engine+ .lock_mutation(&session, &target_id, &tab_id)+ .await+ {+ Ok(guard) => guard,+ Err(refusal) => return refusal.to_tool_result(),+ };+ // Download routing is browser-wide rather than tab-scoped. Serialize+ // these bounded mutations process-wide so two exact tabs cannot race+ // to replace each other's approved destination.+ let _download_mutation = browser_download_gate().lock().await;+ let validated = match self+ .engine+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))+ .await+ {+ Ok(validated) => validated,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let entry = match self+ .engine+ .store+ .resolve_ref(&session, &target_id, &tab_id, &ext_ref)+ {+ Ok(entry) => entry,+ Err(refusal) => return refusal.to_tool_result(),+ };+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Click) {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the semantic ref does not declare an activation action",+ )+ .to_tool_result();+ }+ let ref_session = match self+ .engine+ .frame_session_for_mutation(&session, &target_id, &tab_id, &validated, &entry.frame)+ .await+ {+ Ok(session) => session,+ Err(refusal) => return refusal.to_tool_result(),+ };++ let frame_tree = match validated+ .conn+ .call(Some(&validated.cdp_session), "Page.getFrameTree", json!({}))+ .await+ {+ Ok(tree) => tree,+ Err(_) => {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRouteUnavailable,+ "the exact tab's main frame could not be proven",+ )+ .to_tool_result()+ }+ };+ let Some(main_frame_id) = main_frame_id(&frame_tree).map(str::to_owned) else {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRouteUnavailable,+ "Page.getFrameTree did not identify the exact tab's main frame",+ )+ .to_tool_result();+ };+ let expected_frame_id = entry+ .frame+ .identity+ .as_ref()+ .map(|identity| identity.frame_id.clone())+ .unwrap_or(main_frame_id);++ let resolved = match validated+ .conn+ .call(+ Some(&ref_session),+ "DOM.resolveNode",+ json!({ "backendNodeId": entry.backend_node_id }),+ )+ .await+ {+ Ok(resolved) => resolved,+ Err(_) => {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRefStale,+ "the download ref no longer resolves in the live page",+ )+ .to_tool_result()+ }+ };+ let Some(object_id) = resolved+ .get("object")+ .and_then(|object| object.get("objectId"))+ .and_then(Value::as_str)+ .map(str::to_owned)+ else {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRefStale,+ "the download ref has no live page object",+ )+ .to_tool_result();+ };++ // Browser events are endpoint-wide. Subscribe before enabling and+ // triggering so no matching event can be lost between command replies.+ let mut events = validated.conn.subscribe();+ if validated+ .conn+ .call(+ None,+ "Browser.setDownloadBehavior",+ json!({+ "behavior": "allowAndName",+ "downloadPath": download_path_for_cdp(&destination_root),+ "eventsEnabled": true+ }),+ )+ .await+ .is_err()+ {+ reset_download_behavior(&validated.conn).await;+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the browser could not configure the approved download",+ )+ .to_tool_result();+ }++ let trigger = validated+ .conn+ .call(+ Some(&ref_session),+ "Runtime.callFunctionOn",+ json!({+ "objectId": object_id,+ "functionDeclaration": "function() { this.click(); }",+ "userGesture": true,+ "awaitPromise": false+ }),+ )+ .await;+ let trigger_failed = trigger+ .as_ref()+ .map_or(true, |result| result.get("exceptionDetails").is_some());+ if trigger_failed {+ reset_download_behavior(&validated.conn).await;+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRefStale,+ "the exact download ref could not be activated",+ )+ .to_tool_result();+ }++ let mut observed_guid = None;+ let outcome = tokio::time::timeout(+ DOWNLOAD_TIMEOUT,+ await_download(&mut events, &expected_frame_id, &mut observed_guid),+ )+ .await;+ reset_download_behavior(&validated.conn).await;++ match outcome {+ Ok(DownloadEventOutcome::Completed { guid }) => {+ match completed_download_size(&destination_root, &guid) {+ Ok(bytes) => ToolResult::text("browser download completed").with_structured(+ json!({ "status": "completed", "download_id": guid, "bytes": bytes }),+ ),+ Err(refusal) => refusal.to_tool_result(),+ }+ }+ Ok(DownloadEventOutcome::Canceled { guid }) => {+ remove_proven_partial(&destination_root, Some(&guid));+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the approved browser download was canceled",+ )+ .to_tool_result()+ }+ Ok(DownloadEventOutcome::EventStreamClosed) => {+ remove_proven_partial(&destination_root, observed_guid.as_deref());+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the browser connection closed before the approved download completed",+ )+ .to_tool_result()+ }+ Err(_) => {+ remove_proven_partial(&destination_root, observed_guid.as_deref());+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the approved browser download did not complete within 30 seconds",+ )+ .to_tool_result()+ }+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn destination_must_be_absolute_existing_canonical_directory() {+ assert!(validate_destination_root("relative").is_err());+ let parent = tempfile::tempdir().unwrap();+ let missing = parent.path().join("does-not-exist");+ assert!(validate_destination_root(missing.to_str().unwrap()).is_err());++ let root = tempfile::tempdir().unwrap();+ let canonical = std::fs::canonicalize(root.path()).unwrap();+ assert_eq!(+ validate_destination_root(canonical.to_str().unwrap()).unwrap(),+ canonical+ );+ }++ #[test]+ fn destination_rejects_regular_files() {+ let root = tempfile::tempdir().unwrap();+ let file = root.path().join("not-a-directory");+ std::fs::write(&file, b"x").unwrap();+ assert!(validate_destination_root(file.to_str().unwrap()).is_err());+ }++ #[cfg(unix)]+ #[test]+ fn destination_rejects_symlink_root() {+ use std::os::unix::fs::symlink;++ let parent = tempfile::tempdir().unwrap();+ let real = parent.path().join("real");+ let link = parent.path().join("link");+ std::fs::create_dir(&real).unwrap();+ symlink(&real, &link).unwrap();+ assert!(validate_destination_root(link.to_str().unwrap()).is_err());+ }++ #[test]+ fn completion_proof_accepts_only_direct_regular_guid_file() {+ let root = tempfile::tempdir().unwrap();+ let canonical = std::fs::canonicalize(root.path()).unwrap();+ std::fs::write(canonical.join("opaque-guid"), b"abc").unwrap();+ assert_eq!(+ completed_download_size(&canonical, "opaque-guid").unwrap(),+ 3+ );+ assert!(completed_download_size(&canonical, "../escape").is_err());+ assert!(completed_download_size(&canonical, "missing").is_err());+ }++ #[test]+ fn partial_cleanup_is_confined_to_a_direct_guid_file() {+ let root = tempfile::tempdir().unwrap();+ let canonical = std::fs::canonicalize(root.path()).unwrap();+ let partial = canonical.join("opaque-guid");+ std::fs::write(&partial, b"partial").unwrap();++ remove_proven_partial(&canonical, Some("../escape"));+ assert!(partial.exists());++ remove_proven_partial(&canonical, Some("opaque-guid"));+ assert!(!partial.exists());+ }++ #[test]+ fn cdp_download_path_preserves_ordinary_paths() {+ let path = Path::new("/tmp/approved-downloads");+ assert_eq!(download_path_for_cdp(path), path.to_string_lossy());+ }++ #[cfg(target_os = "windows")]+ #[test]+ fn cdp_download_path_removes_only_windows_verbatim_prefixes() {+ assert_eq!(+ download_path_for_cdp(Path::new(r"\\?\C:\approved\downloads")),+ r"C:\approved\downloads"+ );+ assert_eq!(+ download_path_for_cdp(Path::new(r"\\?\UNC\server\share\downloads")),+ r"\\server\share\downloads"+ );+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rsindex bad656150a..4cb7ac2ce5 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs@@ -31,7 +31,10 @@ use uuid::Uuid;use crate::session::register_session_end_hook;-use super::binding::{cardinality_exact_candidate, correlate, BindingOutcome, CdpWindowCandidate};+use super::binding::{+ cardinality_exact_candidate, correlate, selected_tab_target_id, BindingOutcome,+ CdpWindowCandidate,+};use super::cdp_ws::{CdpConnection, CdpPool};use super::grant::{ExistingProfileGrant, ExistingProfileGrants, GrantLookup};use super::mutation::{MutationGates, MutationKey};@@ -952,7 +955,11 @@ impl BrowserEngine {}};- // Tabs = page targets living in the bound CDP window.+ // Tabs = page targets living in the bound CDP window. Selection is a+ // separate proof from native-window correlation: a representative CDP+ // target is only a window handle and must never be reported as active.+ let selected_cdp_target_id =+ selected_tab_target_id(&native.title, &candidates, candidate.cdp_window_id);let mut tabs = HashMap::new();for c in candidates.iter().filter(|c| match candidate.cdp_window_id {Some(window_id) => c.cdp_window_id == Some(window_id),@@ -966,6 +973,7 @@ impl BrowserEngine {cdp_target_id: c.cdp_target_id.clone(),title: c.title.clone(),url: c.url.clone(),+ active: selected_cdp_target_id.map(|selected| selected == c.cdp_target_id),generation: grant.as_ref().map_or(0, |grant| grant.generation),snapshots: HashMap::new(),},diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rsindex ea899520c4..cbd3ccb64d 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs@@ -1,8 +1,7 @@//! Platform-agnostic browser-tool v1 core.//!-//! Five typed tools — `get_browser_state` (strictly read-only),-//! `browser_prepare`, `browser_navigate`, `browser_click`,-//! `browser_type` — over an exact-or-refused binding model:+//! Typed browser inspection, preparation, navigation, input, dialog, upload,+//! and download tools over an exact-or-refused binding model://!//! - The native entrypoint is `pid + window_id`. Browser target ids,//! tab ids, and page refs (`p<snapshot>:<index>`) are opaque,@@ -39,12 +38,14 @@pub mod approval;pub mod binding;pub mod cdp_ws;+pub mod download;pub mod engine;mod grant;#[cfg(test)]pub(crate) mod mock_cdp;mod mutation;pub mod platform;+pub mod pointer;mod prepare;mod reconnect;pub mod refusal;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rsnew file mode 100644index 0000000000..d8e08507ad--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rs@@ -0,0 +1,871 @@+//! Extended browser pointer actions.+//!+//! This module deliberately shares the exact-or-refused posture of the+//! original browser click tool. Every mutation is serialized by the real CDP+//! tab, then revalidates the native window, endpoint, tab, and (for refs) frame+//! identity before dispatch. It never activates a target or brings a page to+//! the foreground.++use std::sync::Arc;+use std::time::Duration;++use async_trait::async_trait;+use serde_json::{json, Value};++use crate::protocol::ToolResult;+use crate::tool::{Tool, ToolDef};+use crate::tool_args::ArgsExt;++use super::cdp_ws::CdpConnection;+use super::engine::{BrowserEngine, ValidatedTab};+use super::refusal::{BrowserRefusal, BrowserRefusalCode};+use super::store::{FrameKind, FrameRef};++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum PointerAction {+ Hover,+ RightClick,+ DoubleClick,+ Scroll,+ Drag,+}++impl PointerAction {+ fn parse(value: &str) -> Result<Self, String> {+ match value {+ "hover" => Ok(Self::Hover),+ "right_click" => Ok(Self::RightClick),+ "double_click" => Ok(Self::DoubleClick),+ "scroll" => Ok(Self::Scroll),+ "drag" => Ok(Self::Drag),+ _ => Err(format!(+ "action must be hover, right_click, double_click, scroll, or drag, got {value:?}"+ )),+ }+ }++ fn as_str(self) -> &'static str {+ match self {+ Self::Hover => "hover",+ Self::RightClick => "right_click",+ Self::DoubleClick => "double_click",+ Self::Scroll => "scroll",+ Self::Drag => "drag",+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum InputRoute {+ Trusted,+ DomEvent,+}++impl InputRoute {+ fn parse(value: &str) -> Result<Self, String> {+ match value {+ "trusted" => Ok(Self::Trusted),+ "dom_event" => Ok(Self::DomEvent),+ _ => Err(format!(+ "input_route must be \"trusted\" or \"dom_event\", got {value:?}"+ )),+ }+ }++ fn as_str(self) -> &'static str {+ match self {+ Self::Trusted => "trusted",+ Self::DomEvent => "dom_event",+ }+ }+}++#[derive(Debug, Clone, PartialEq)]+enum Location {+ Ref(String),+ Coordinates(f64, f64),+}++#[derive(Debug, Clone, PartialEq)]+struct PointerRequest {+ action: PointerAction,+ route: InputRoute,+ origin: Location,+ destination: Option<Location>,+ delta_x: f64,+ delta_y: f64,+}++fn explicit_session(args: &Value) -> Result<String, ToolResult> {+ let session = args+ .opt_str("session")+ .or_else(|| args.opt_str("_session_id"))+ .unwrap_or_else(|| "default".into());+ if session.is_empty() || session == "default" {+ return Err(ToolResult::error(+ "Browser targets and page refs are session-scoped capabilities - declare an explicit session (start_session) and pass its id on this call.",+ ));+ }+ Ok(session)+}++fn finite_pair(args: &Value, x_name: &str, y_name: &str) -> Result<Option<(f64, f64)>, String> {+ match (args.opt_f64(x_name), args.opt_f64(y_name)) {+ (None, None) => Ok(None),+ (Some(x), Some(y)) if x.is_finite() && y.is_finite() => Ok(Some((x, y))),+ (Some(_), Some(_)) => Err(format!("{x_name} and {y_name} must be finite numbers")),+ _ => Err(format!("pass both {x_name} and {y_name}, or neither")),+ }+}++fn one_location(+ args: &Value,+ ref_name: &str,+ x_name: &str,+ y_name: &str,+) -> Result<Option<Location>, String> {+ let reference = args.opt_str(ref_name);+ let coords = finite_pair(args, x_name, y_name)?;+ match (reference, coords) {+ (Some(_), Some(_)) => Err(format!(+ "pass either {ref_name} or {x_name}/{y_name}, not both"+ )),+ (Some(reference), None) => Ok(Some(Location::Ref(reference))),+ (None, Some((x, y))) => Ok(Some(Location::Coordinates(x, y))),+ (None, None) => Ok(None),+ }+}++fn parse_request(args: &Value) -> Result<PointerRequest, String> {+ let action = PointerAction::parse(+ args.get("action")+ .and_then(Value::as_str)+ .ok_or_else(|| "missing required string field: action".to_owned())?,+ )?;+ let route = InputRoute::parse(+ args.get("input_route")+ .and_then(Value::as_str)+ .unwrap_or("trusted"),+ )?;+ let origin = one_location(args, "ref", "x", "y")?+ .ok_or_else(|| "browser_pointer needs a ref or x/y coordinates".to_owned())?;+ let destination = one_location(args, "destination_ref", "to_x", "to_y")?;++ if route == InputRoute::DomEvent && !matches!(origin, Location::Ref(_)) {+ return Err("input_route=dom_event requires a ref".into());+ }+ if action == PointerAction::Drag && destination.is_none() {+ return Err("action=drag requires destination_ref or to_x/to_y".into());+ }+ if action != PointerAction::Drag && destination.is_some() {+ return Err("destination_ref and to_x/to_y are valid only for action=drag".into());+ }++ let delta_x = args.opt_f64("delta_x").unwrap_or(0.0);+ let delta_y = args.opt_f64("delta_y").unwrap_or(0.0);+ if !delta_x.is_finite() || !delta_y.is_finite() {+ return Err("delta_x and delta_y must be finite numbers".into());+ }+ if action == PointerAction::Scroll && delta_x == 0.0 && delta_y == 0.0 {+ return Err("action=scroll requires a non-zero delta_x or delta_y".into());+ }+ if action != PointerAction::Scroll+ && (args.get("delta_x").is_some() || args.get("delta_y").is_some())+ {+ return Err("delta_x and delta_y are valid only for action=scroll".into());+ }++ Ok(PointerRequest {+ action,+ route,+ origin,+ destination,+ delta_x,+ delta_y,+ })+}++struct ResolvedRef {+ external: String,+ backend_node_id: i64,+ frame: FrameRef,+ cdp_session: String,+}++fn same_exact_frame(left: &FrameRef, right: &FrameRef) -> bool {+ left.kind == right.kind+ && left.oopif_target_id == right.oopif_target_id+ && left.identity == right.identity+}++fn stale(message: impl Into<String>) -> ToolResult {+ BrowserRefusal::new(BrowserRefusalCode::BrowserRefStale, message).to_tool_result()+}++async fn resolve_object(+ conn: &CdpConnection,+ cdp_session: &str,+ backend_node_id: i64,+) -> Result<String, ToolResult> {+ let resolved = conn+ .call(+ Some(cdp_session),+ "DOM.resolveNode",+ json!({ "backendNodeId": backend_node_id }),+ )+ .await+ .map_err(|_| stale("the ref's node no longer resolves in the live page"))?;+ resolved+ .get("object")+ .and_then(|object| object.get("objectId"))+ .and_then(Value::as_str)+ .map(str::to_owned)+ .ok_or_else(|| stale("the ref's node has no live object in the page"))+}++async fn point_for_ref(+ conn: &CdpConnection,+ cdp_session: &str,+ backend_node_id: i64,+) -> Result<(f64, f64), ToolResult> {+ let _ = conn+ .call(+ Some(cdp_session),+ "DOM.scrollIntoViewIfNeeded",+ json!({ "backendNodeId": backend_node_id }),+ )+ .await;+ let model = conn+ .call(+ Some(cdp_session),+ "DOM.getBoxModel",+ json!({ "backendNodeId": backend_node_id }),+ )+ .await+ .map_err(|_| stale("the ref's node has no live layout box"))?;+ quad_center(&model).ok_or_else(|| stale("the ref's node returned an unusable layout box"))+}++fn quad_center(box_model: &Value) -> Option<(f64, f64)> {+ let quad = box_model.get("model")?.get("content")?.as_array()?;+ if quad.len() < 8 {+ return None;+ }+ let values = quad+ .iter()+ .take(8)+ .map(Value::as_f64)+ .collect::<Option<Vec<_>>>()?;+ Some((+ (values[0] + values[2] + values[4] + values[6]) / 4.0,+ (values[1] + values[3] + values[5] + values[7]) / 4.0,+ ))+}++pub struct BrowserPointerTool {+ def: ToolDef,+ engine: Arc<BrowserEngine>,+}++impl BrowserPointerTool {+ pub fn new(engine: Arc<BrowserEngine>) -> Self {+ Self {+ def: ToolDef {+ name: "browser_pointer".into(),+ description: "Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.".into(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "target_id": { "type": "string", "description": "Opaque target id minted by get_browser_state." },+ "tab_id": { "type": "string", "description": "Opaque tab id minted by get_browser_state." },+ "session": { "type": "string", "description": "Explicit caller session owning the browser capabilities." },+ "action": { "type": "string", "enum": ["hover", "right_click", "double_click", "scroll", "drag"] },+ "input_route": { "type": "string", "enum": ["trusted", "dom_event"], "default": "trusted" },+ "ref": { "type": "string", "description": "Origin page ref. Alternative to x/y." },+ "x": { "type": "number", "description": "Origin viewport x in CSS pixels." },+ "y": { "type": "number", "description": "Origin viewport y in CSS pixels." },+ "destination_ref": { "type": "string", "description": "Drag destination page ref in the exact same frame." },+ "to_x": { "type": "number", "description": "Drag destination viewport x in CSS pixels." },+ "to_y": { "type": "number", "description": "Drag destination viewport y in CSS pixels." },+ "delta_x": { "type": "number", "description": "Horizontal scroll delta in CSS pixels." },+ "delta_y": { "type": "number", "description": "Vertical scroll delta in CSS pixels." }+ },+ "required": ["target_id", "tab_id", "session", "action"],+ "additionalProperties": true+ }),+ read_only: false,+ destructive: false,+ idempotent: false,+ open_world: false,+ },+ engine,+ }+ }++ async fn resolve_ref(+ &self,+ session: &str,+ target_id: &str,+ tab_id: &str,+ validated: &ValidatedTab,+ external: &str,+ ) -> Result<ResolvedRef, ToolResult> {+ let entry = self+ .engine+ .store+ .resolve_ref(session, target_id, tab_id, external)+ .map_err(|refusal| refusal.to_tool_result())?;+ let cdp_session = self+ .engine+ .frame_session_for_mutation(session, target_id, tab_id, validated, &entry.frame)+ .await+ .map_err(|refusal| refusal.to_tool_result())?;+ Ok(ResolvedRef {+ external: external.to_owned(),+ backend_node_id: entry.backend_node_id,+ frame: entry.frame,+ cdp_session,+ })+ }++ fn trusted_background_refusal(&self, validated: &ValidatedTab) -> Option<ToolResult> {+ if validated.record.cdp_window_id.is_some() {+ if let Some(limitation) = self+ .engine+ .platform+ .standalone_trusted_input_background_limitation()+ {+ return Some(+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ format!(+ "{limitation}; use input_route=\"dom_event\" with refs to explicitly request synthetic full-background pointer delivery"+ ),+ )+ .to_tool_result(),+ );+ }+ }+ None+ }++ async fn dom_event(+ &self,+ request: &PointerRequest,+ validated: &ValidatedTab,+ origin: &ResolvedRef,+ destination: Option<&ResolvedRef>,+ ) -> ToolResult {+ let conn = &validated.conn;+ let object_id =+ match resolve_object(conn, &origin.cdp_session, origin.backend_node_id).await {+ Ok(id) => id,+ Err(result) => return result,+ };++ let (function, arguments) = match request.action {+ PointerAction::Hover => (+ "function() { const o={bubbles:true,composed:true,view:this.ownerDocument.defaultView}; this.dispatchEvent(new PointerEvent('pointerover',o)); this.dispatchEvent(new PointerEvent('pointerenter',{...o,bubbles:false})); this.dispatchEvent(new MouseEvent('mouseover',o)); this.dispatchEvent(new MouseEvent('mouseenter',{...o,bubbles:false})); return true; }",+ json!([]),+ ),+ PointerAction::RightClick => (+ "function() { const o={bubbles:true,cancelable:true,composed:true,button:2,buttons:2,view:this.ownerDocument.defaultView}; this.dispatchEvent(new PointerEvent('pointerdown',o)); this.dispatchEvent(new MouseEvent('mousedown',o)); this.dispatchEvent(new MouseEvent('mouseup',{...o,buttons:0})); this.dispatchEvent(new PointerEvent('pointerup',{...o,buttons:0})); this.dispatchEvent(new MouseEvent('contextmenu',{...o,buttons:0})); return true; }",+ json!([]),+ ),+ PointerAction::DoubleClick => (+ "function() { const o={bubbles:true,cancelable:true,composed:true,button:0,view:this.ownerDocument.defaultView}; for (let detail=1; detail<=2; detail++) { this.dispatchEvent(new PointerEvent('pointerdown',{...o,buttons:1,detail})); this.dispatchEvent(new MouseEvent('mousedown',{...o,buttons:1,detail})); this.dispatchEvent(new MouseEvent('mouseup',{...o,buttons:0,detail})); this.dispatchEvent(new PointerEvent('pointerup',{...o,buttons:0,detail})); this.dispatchEvent(new MouseEvent('click',{...o,buttons:0,detail})); } this.dispatchEvent(new MouseEvent('dblclick',{...o,buttons:0,detail:2})); return true; }",+ json!([]),+ ),+ PointerAction::Scroll => (+ "function(dx,dy) { const o={deltaX:dx,deltaY:dy,bubbles:true,cancelable:true,composed:true,view:this.ownerDocument.defaultView}; this.dispatchEvent(new WheelEvent('wheel',o)); let n=this; while (n && n !== this.ownerDocument.documentElement) { const s=this.ownerDocument.defaultView.getComputedStyle(n); if (/(auto|scroll)/.test(s.overflow+s.overflowX+s.overflowY)) break; n=n.parentElement; } const target=n || this.ownerDocument.scrollingElement || this.ownerDocument.documentElement; const beforeX=target.scrollLeft; const beforeY=target.scrollTop; target.scrollBy(dx,dy); const changed=target.scrollLeft !== beforeX || target.scrollTop !== beforeY; if (changed) target.dispatchEvent(new Event('scroll')); return changed; }",+ json!([{ "value": request.delta_x }, { "value": request.delta_y }]),+ ),+ PointerAction::Drag => {+ let destination_argument = if let Some(destination) = destination {+ let object_id = match resolve_object(+ conn,+ &destination.cdp_session,+ destination.backend_node_id,+ )+ .await+ {+ Ok(id) => id,+ Err(result) => return result,+ };+ json!([{ "objectId": object_id }, { "value": Value::Null }, { "value": Value::Null }])+ } else if let Some(Location::Coordinates(x, y)) = &request.destination {+ if origin.frame.kind != FrameKind::Main {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "coordinate drag destinations are only provably in the same frame for a main-frame origin; use destination_ref for iframe or OOPIF drag",+ )+ .to_tool_result();+ }+ json!([{ "value": Value::Null }, { "value": x }, { "value": y }])+ } else {+ unreachable!("drag destination validated")+ };+ (+ "function(destination,x,y) { const doc=this.ownerDocument; const dest=destination || doc.elementFromPoint(x,y); if (!dest || dest.ownerDocument !== doc) return false; let data=null; try { data=new DataTransfer(); } catch (_) {} const common={bubbles:true,cancelable:true,composed:true}; this.dispatchEvent(new PointerEvent('pointerdown',{...common,button:0,buttons:1})); this.dispatchEvent(new MouseEvent('mousedown',{...common,button:0,buttons:1})); this.dispatchEvent(new DragEvent('dragstart',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('dragenter',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('dragover',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('drop',{...common,dataTransfer:data})); this.dispatchEvent(new DragEvent('dragend',{...common,dataTransfer:data})); dest.dispatchEvent(new MouseEvent('mouseup',{...common,button:0,buttons:0})); dest.dispatchEvent(new PointerEvent('pointerup',{...common,button:0,buttons:0})); return true; }",+ destination_argument,+ )+ }+ };++ match conn+ .call(+ Some(&origin.cdp_session),+ "Runtime.callFunctionOn",+ json!({+ "objectId": object_id,+ "functionDeclaration": function,+ "arguments": arguments,+ "returnByValue": true,+ }),+ )+ .await+ {+ Ok(value)+ if value.get("exceptionDetails").is_none()+ && (!matches!(request.action, PointerAction::Scroll | PointerAction::Drag)+ || value.pointer("/result/value").and_then(Value::as_bool)+ != Some(false)) =>+ {+ ToolResult::text(format!(+ "dispatched synthetic {} in {}",+ request.action.as_str(),+ validated.tab.tab_id+ ))+ .with_structured(self.success_json(+ request,+ validated,+ origin,+ destination,+ None,+ None,+ ))+ }+ Ok(value) if value.get("exceptionDetails").is_some() => BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!(+ "synthetic {} raised a page-side exception and delivery was not proven",+ request.action.as_str()+ ),+ )+ .to_tool_result(),+ Ok(_) if request.action == PointerAction::Drag => BrowserRefusal::new(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "the drag destination did not resolve in the origin ref's exact document",+ )+ .to_tool_result(),+ Ok(_) => BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the synthetic scroll target did not move, so delivery was not proven",+ )+ .to_tool_result(),+ Err(error) => ToolResult::error(format!(+ "synthetic {} failed: {error}",+ request.action.as_str()+ )),+ }+ }++ fn success_json(+ &self,+ request: &PointerRequest,+ validated: &ValidatedTab,+ origin: &ResolvedRef,+ destination: Option<&ResolvedRef>,+ origin_point: Option<(f64, f64)>,+ destination_point: Option<(f64, f64)>,+ ) -> Value {+ let external_ref = (!origin.external.is_empty()).then_some(origin.external.as_str());+ json!({+ "status": "ok",+ "action": request.action.as_str(),+ "route": request.route.as_str(),+ "target_id": validated.record.target_id,+ "tab_id": validated.tab.tab_id,+ "ref": external_ref,+ "frame": origin.frame.kind.as_str(),+ "destination_ref": destination.map(|value| value.external.as_str()),+ "x": origin_point.map(|point| point.0),+ "y": origin_point.map(|point| point.1),+ "to_x": destination_point.map(|point| point.0),+ "to_y": destination_point.map(|point| point.1),+ "delta_x": (request.action == PointerAction::Scroll).then_some(request.delta_x),+ "delta_y": (request.action == PointerAction::Scroll).then_some(request.delta_y),+ })+ }++ async fn trusted(+ &self,+ request: &PointerRequest,+ validated: &ValidatedTab,+ origin_ref: Option<&ResolvedRef>,+ destination_ref: Option<&ResolvedRef>,+ ) -> ToolResult {+ let conn = &validated.conn;+ let cdp_session = origin_ref+ .map(|reference| reference.cdp_session.as_str())+ .unwrap_or(validated.cdp_session.as_str());+ let origin = match (&request.origin, origin_ref) {+ (Location::Coordinates(x, y), None) => (*x, *y),+ (Location::Ref(_), Some(reference)) => {+ match point_for_ref(conn, &reference.cdp_session, reference.backend_node_id).await {+ Ok(point) => point,+ Err(result) => return result,+ }+ }+ _ => unreachable!("origin resolution matches request"),+ };+ let destination = match (&request.destination, destination_ref) {+ (Some(Location::Coordinates(x, y)), None) => Some((*x, *y)),+ (Some(Location::Ref(_)), Some(reference)) => {+ match point_for_ref(conn, &reference.cdp_session, reference.backend_node_id).await {+ Ok(point) => Some(point),+ Err(result) => return result,+ }+ }+ (None, None) => None,+ _ => unreachable!("destination resolution matches request"),+ };++ if let Err(error) = conn+ .call(+ Some(cdp_session),+ "Emulation.setFocusEmulationEnabled",+ json!({ "enabled": true }),+ )+ .await+ {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ format!("the target tab could not enter CDP focus emulation: {error}"),+ )+ .to_tool_result();+ }+ tokio::time::sleep(Duration::from_millis(25)).await;++ let delivery = self+ .dispatch_trusted(conn, cdp_session, request, origin, destination)+ .await;+ let cleanup = conn+ .call(+ Some(cdp_session),+ "Emulation.setFocusEmulationEnabled",+ json!({ "enabled": false }),+ )+ .await;+ if let Err(error) = delivery {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ format!(+ "trusted {} failed ({error}); no synthetic fallback was attempted",+ request.action.as_str()+ ),+ )+ .to_tool_result();+ }+ if let Err(error) = cleanup {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ format!(+ "trusted {} was acknowledged but focus emulation could not be restored ({error}); delivery is unknown and must not be retried automatically",+ request.action.as_str()+ ),+ )+ .with_detail(json!({ "delivery": "unknown", "retryable": false }))+ .to_tool_result();+ }++ let synthetic_origin = ResolvedRef {+ external: match &request.origin {+ Location::Ref(reference) => reference.clone(),+ Location::Coordinates(_, _) => String::new(),+ },+ backend_node_id: origin_ref.map_or(0, |reference| reference.backend_node_id),+ frame: origin_ref+ .map(|reference| reference.frame.clone())+ .unwrap_or_else(FrameRef::main_unproven),+ cdp_session: cdp_session.to_owned(),+ };+ ToolResult::text(format!(+ "dispatched trusted {} in {}",+ request.action.as_str(),+ validated.tab.tab_id+ ))+ .with_structured(self.success_json(+ request,+ validated,+ &synthetic_origin,+ destination_ref,+ Some(origin),+ destination,+ ))+ }++ async fn dispatch_trusted(+ &self,+ conn: &CdpConnection,+ cdp_session: &str,+ request: &PointerRequest,+ origin: (f64, f64),+ destination: Option<(f64, f64)>,+ ) -> anyhow::Result<()> {+ let call = |params: Value| conn.call(Some(cdp_session), "Input.dispatchMouseEvent", params);+ match request.action {+ PointerAction::Hover => {+ call(+ json!({ "type": "mouseMoved", "x": origin.0, "y": origin.1, "button": "none" }),+ )+ .await?;+ }+ PointerAction::RightClick => {+ for kind in ["mousePressed", "mouseReleased"] {+ call(json!({ "type": kind, "x": origin.0, "y": origin.1, "button": "right", "clickCount": 1 })).await?;+ }+ }+ PointerAction::DoubleClick => {+ for click_count in [1, 2] {+ for kind in ["mousePressed", "mouseReleased"] {+ call(json!({ "type": kind, "x": origin.0, "y": origin.1, "button": "left", "clickCount": click_count })).await?;+ }+ }+ }+ PointerAction::Scroll => {+ call(json!({ "type": "mouseWheel", "x": origin.0, "y": origin.1, "deltaX": request.delta_x, "deltaY": request.delta_y })).await?;+ }+ PointerAction::Drag => {+ let destination = destination.expect("drag destination validated");+ call(+ json!({ "type": "mouseMoved", "x": origin.0, "y": origin.1, "button": "none" }),+ )+ .await?;+ call(json!({ "type": "mousePressed", "x": origin.0, "y": origin.1, "button": "left", "buttons": 1, "clickCount": 1 })).await?;+ for step in 1..=8 {+ let progress = f64::from(step) / 8.0;+ let x = origin.0 + (destination.0 - origin.0) * progress;+ let y = origin.1 + (destination.1 - origin.1) * progress;+ call(json!({ "type": "mouseMoved", "x": x, "y": y, "button": "left", "buttons": 1 })).await?;+ }+ call(json!({ "type": "mouseReleased", "x": destination.0, "y": destination.1, "button": "left", "buttons": 0, "clickCount": 1 })).await?;+ }+ }+ Ok(())+ }+}++#[async_trait]+impl Tool for BrowserPointerTool {+ fn def(&self) -> &ToolDef {+ &self.def+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let target_id = match args.require_str("target_id") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let tab_id = match args.require_str("tab_id") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let session = match explicit_session(&args) {+ Ok(value) => value,+ Err(error) => return error,+ };+ let request = match parse_request(&args) {+ Ok(value) => value,+ Err(error) => return ToolResult::error(error),+ };++ let _mutation = match self+ .engine+ .lock_mutation(&session, &target_id, &tab_id)+ .await+ {+ Ok(guard) => guard,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let validated = match self+ .engine+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))+ .await+ {+ Ok(value) => value,+ Err(refusal) => return refusal.to_tool_result(),+ };+ if request.route == InputRoute::Trusted {+ if let Some(refusal) = self.trusted_background_refusal(&validated) {+ return refusal;+ }+ }++ let origin_ref = match &request.origin {+ Location::Ref(external) => match self+ .resolve_ref(&session, &target_id, &tab_id, &validated, external)+ .await+ {+ Ok(reference) => Some(reference),+ Err(result) => return result,+ },+ Location::Coordinates(_, _) => None,+ };+ let destination_ref = match &request.destination {+ Some(Location::Ref(external)) => match self+ .resolve_ref(&session, &target_id, &tab_id, &validated, external)+ .await+ {+ Ok(reference) => Some(reference),+ Err(result) => return result,+ },+ _ => None,+ };++ if let (Some(origin), Some(destination)) = (&origin_ref, &destination_ref) {+ if !same_exact_frame(&origin.frame, &destination.frame)+ || origin.cdp_session != destination.cdp_session+ {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "drag origin and destination refs must belong to the exact same live frame",+ )+ .to_tool_result();+ }+ }+ if request.action == PointerAction::Drag+ && origin_ref.is_none()+ && destination_ref.is_some()+ {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "a coordinate drag origin cannot prove it shares a frame with destination_ref; use two refs or two coordinate pairs",+ )+ .to_tool_result();+ }+ if request.action == PointerAction::Drag+ && matches!(&request.destination, Some(Location::Coordinates(_, _)))+ && origin_ref+ .as_ref()+ .is_some_and(|reference| reference.frame.kind != FrameKind::Main)+ {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "a viewport-coordinate drag destination is only in the same provable frame as a main-frame ref; use destination_ref for iframe or OOPIF drag",+ )+ .to_tool_result();+ }++ match request.route {+ InputRoute::DomEvent => {+ self.dom_event(+ &request,+ &validated,+ origin_ref.as_ref().expect("dom route requires ref"),+ destination_ref.as_ref(),+ )+ .await+ }+ InputRoute::Trusted => {+ self.trusted(+ &request,+ &validated,+ origin_ref.as_ref(),+ destination_ref.as_ref(),+ )+ .await+ }+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn parses_every_action_and_coordinate_origin() {+ for action in ["hover", "right_click", "double_click"] {+ let parsed = parse_request(&json!({+ "action": action,+ "x": 12.5,+ "y": 8,+ }))+ .unwrap();+ assert_eq!(parsed.origin, Location::Coordinates(12.5, 8.0));+ assert_eq!(parsed.route, InputRoute::Trusted);+ }+ }++ #[test]+ fn dom_event_requires_a_ref() {+ let error = parse_request(&json!({+ "action": "hover",+ "input_route": "dom_event",+ "x": 1,+ "y": 2,+ }))+ .unwrap_err();+ assert!(error.contains("requires a ref"), "{error}");+ }++ #[test]+ fn drag_requires_exactly_one_destination_shape() {+ assert!(parse_request(&json!({ "action": "drag", "ref": "p1:0" }))+ .unwrap_err()+ .contains("requires destination"));+ assert!(parse_request(&json!({+ "action": "drag",+ "ref": "p1:0",+ "destination_ref": "p1:1",+ "to_x": 3,+ "to_y": 4,+ }))+ .unwrap_err()+ .contains("not both"));+ }++ #[test]+ fn scroll_requires_a_nonzero_finite_delta() {+ assert!(parse_request(&json!({ "action": "scroll", "ref": "p1:0" }))+ .unwrap_err()+ .contains("non-zero"));+ let parsed = parse_request(&json!({+ "action": "scroll",+ "ref": "p1:0",+ "delta_y": 240,+ }))+ .unwrap();+ assert_eq!(parsed.delta_y, 240.0);+ }++ #[test]+ fn quad_center_rejects_malformed_models() {+ assert_eq!(+ quad_center(&json!({ "model": { "content": [0, 0, 10, 0, 10, 20, 0, 20] } })),+ Some((5.0, 10.0))+ );+ assert_eq!(+ quad_center(&json!({ "model": { "content": [0, 0] } })),+ None+ );+ }++ #[test]+ fn exact_frame_comparison_includes_document_identity() {+ let first = FrameRef::main_unproven();+ let second = FrameRef::main_unproven();+ assert!(same_exact_frame(&first, &second));++ let navigated = FrameRef {+ kind: FrameKind::Main,+ oopif_target_id: None,+ identity: Some(super::super::store::FrameIdentity {+ frame_id: "main".into(),+ loader_id: "new-loader".into(),+ }),+ };+ assert!(!same_exact_frame(&first, &navigated));+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rsindex 956dc47a75..b2348ef831 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs@@ -773,7 +773,18 @@ fn action_kinds(.get("contenteditable").is_some_and(|value| value.is_empty() || value.eq_ignore_ascii_case("true"))});- if matches!(role, "textbox" | "searchbox") || matches!(tag, "input" | "textarea") || editable {+ let file_input = tag == "input"+ && dom.is_some_and(|meta| {+ meta.attrs+ .get("type")+ .is_some_and(|value| value.eq_ignore_ascii_case("file"))+ });+ if file_input {+ actions.push(BrowserActionKind::Upload);+ } else if matches!(role, "textbox" | "searchbox")+ || matches!(tag, "input" | "textarea")+ || editable+ {actions.push(BrowserActionKind::Type);}if actions.is_empty()@@ -1073,6 +1084,22 @@ mod tests {use serde_json::json;use super::*;++ #[test]+ fn file_inputs_expose_upload_instead_of_text_typing() {+ let dom = DomMeta {+ tag: "input".into(),+ attrs: HashMap::from([("type".into(), "file".into())]),+ order: 0,+ css_hidden: false,+ parent_backend_node_id: None,+ frame_id: None,+ };+ assert_eq!(+ action_kinds("textbox", Some(&dom), &BTreeMap::new(), None),+ vec![BrowserActionKind::Upload]+ );+ }use crate::browser::store::FrameRef;fn frame() -> FrameRef {diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rsindex 0b47f26ddf..7bc9740510 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs@@ -32,6 +32,7 @@ use super::types::{BindingQuality, ProcessFingerprint, Rect};pub enum BrowserActionKind {Click,Type,+ Upload,}impl BrowserActionKind {@@ -39,6 +40,7 @@ impl BrowserActionKind {match self {Self::Click => "click",Self::Type => "type",+ Self::Upload => "upload",}}}@@ -187,6 +189,10 @@ pub struct TabRecord {pub cdp_target_id: String,pub title: String,pub url: String,+ /// Native-window selection proof captured at bind time. `None` means the+ /// selected tab could not be proven without activating or foregrounding a+ /// page, so the public `active` field must be JSON null.+ pub active: Option<bool>,pub generation: u64,pub snapshots: HashMap<u64, SnapshotRecord>,}@@ -509,6 +515,7 @@ mod tests {cdp_target_id: "CDP1".into(),title: "Example".into(),url: "https://example.test".into(),+ active: Some(true),generation: 0,snapshots: HashMap::from([(snap_id,diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rsindex 64b1eb7741..fcbe2bcad3 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs@@ -1,5 +1,4 @@-//! The five browser tools: get_browser_state, browser_prepare,-//! browser_navigate, browser_click, browser_type.+//! Browser state, preparation, navigation, input, and page-owned dialog tools.//!//! Schemas, annotations, and exact-or-refused semantics live here; OS//! identity comes from the [`BrowserPlatform`] adapter; CDP mechanics@@ -16,13 +15,15 @@ use crate::tool::{Tool, ToolDef, ToolRegistry};use crate::tool_args::ArgsExt;use super::approval::MCP_HOST_APPROVAL_ARG;+use super::download::BrowserDownloadTool;use super::engine::BrowserEngine;use super::platform::{PrepareAuthorization, PrepareProfile, PrepareRequest, PrepareStrategy};+use super::pointer::BrowserPointerTool;use super::refusal::{BrowserRefusal, BrowserRefusalCode};use super::store::BrowserActionKind;use super::types::BindingQuality;-/// Register all five browser tools against one shared engine. Platform+/// Register the complete browser surface against one shared engine. Platform/// crates call this from their `register_all` after constructing the/// engine with their adapter.pub fn register_browser_tools(engine: &Arc<BrowserEngine>, registry: &mut ToolRegistry) {@@ -31,6 +32,10 @@ pub fn register_browser_tools(engine: &Arc<BrowserEngine>, registry: &mut ToolReregistry.register(Box::new(BrowserNavigateTool::new(engine.clone())));registry.register(Box::new(BrowserClickTool::new(engine.clone())));registry.register(Box::new(BrowserTypeTool::new(engine.clone())));+ registry.register(Box::new(BrowserDialogTool::new(engine.clone())));+ registry.register(Box::new(BrowserSetInputFilesTool::new(engine.clone())));+ registry.register(Box::new(BrowserDownloadTool::new(engine.clone())));+ registry.register(Box::new(BrowserPointerTool::new(engine.clone())));}// ── Shared helpers ───────────────────────────────────────────────────────────@@ -342,7 +347,7 @@ impl Tool for GetBrowserStateTool {"tab_id": t.tab_id,"title": t.title,"url": t.url,- "active": t.cdp_target_id == record.cdp_target_id,+ "active": t.active,})}).collect();@@ -1405,6 +1410,386 @@ impl Tool for BrowserTypeTool {}}+// ── browser_dialog ──────────────────────────────────────────────────────────++pub struct BrowserDialogTool {+ def: ToolDef,+ engine: Arc<BrowserEngine>,+}++impl BrowserDialogTool {+ pub fn new(engine: Arc<BrowserEngine>) -> Self {+ Self {+ def: ToolDef {+ name: "browser_dialog".into(),+ description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.".into(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "target_id": schema_target_id(),+ "tab_id": schema_tab_id(),+ "session": schema_session(),+ "action": { "type": "string", "enum": ["inspect", "accept", "dismiss"] },+ "dialog_id": { "type": "string", "description": "Opaque current dialog generation returned by action=inspect." },+ "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." },+ "delivery_mode": {+ "type": "string",+ "enum": ["background", "foreground"],+ "default": "background",+ "description": "Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only."+ }+ },+ "required": ["target_id", "tab_id", "action"],+ "additionalProperties": true+ }),+ read_only: false,+ destructive: false,+ idempotent: false,+ open_world: false,+ },+ engine,+ }+ }+}++#[async_trait]+impl Tool for BrowserDialogTool {+ fn def(&self) -> &ToolDef {+ &self.def+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let (target_id, tab_id, action) = match (+ args.require_str("target_id"),+ args.require_str("tab_id"),+ args.require_str("action"),+ ) {+ (Ok(target), Ok(tab), Ok(action)) => (target, tab, action),+ (Err(error), _, _) | (_, Err(error), _) | (_, _, Err(error)) => return error,+ };+ if !matches!(action.as_str(), "inspect" | "accept" | "dismiss") {+ return ToolResult::error("action must be inspect, accept, or dismiss");+ }+ let delivery_mode = args+ .opt_str("delivery_mode")+ .unwrap_or_else(|| "background".into());+ if !matches!(delivery_mode.as_str(), "background" | "foreground") {+ return ToolResult::error("delivery_mode must be background or foreground");+ }+ let session = match require_explicit_session(&args) {+ Ok(session) => session,+ Err(error) => return error,+ };+ let _mutation = match self+ .engine+ .lock_mutation(&session, &target_id, &tab_id)+ .await+ {+ Ok(guard) => guard,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let validated = match self+ .engine+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))+ .await+ {+ Ok(validated) => validated,+ Err(refusal) => return refusal.to_tool_result(),+ };+ if cfg!(target_os = "linux") && action != "inspect" && delivery_mode == "background" {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserInputTrustUnavailable,+ "Chromium's native JavaScript modal cannot be resolved on Linux without changing foreground posture; retry with delivery_mode=\"foreground\"",+ )+ .with_detail(json!({ "supported_delivery_mode": "foreground" }))+ .to_tool_result();+ }+ let conn = &validated.conn;+ let cdp_session = validated.cdp_session.as_str();+ let cdp_target_id = validated.tab.cdp_target_id.as_str();+ if conn.dialog_state(cdp_target_id).is_none() && !conn.has_dialog_session(cdp_target_id) {+ // Register before Page.enable so an opening event delivered before+ // the command reply is still attributed to the exact target.+ conn.register_dialog_session(cdp_session, cdp_target_id);+ if let Err(error) = conn.call(Some(cdp_session), "Page.enable", json!({})).await {+ if conn.dialog_state(cdp_target_id).is_none() {+ conn.unregister_dialog_session(cdp_session, cdp_target_id);+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!("the exact tab does not expose JavaScript dialog events: {error}"),+ )+ .to_tool_result();+ }+ }+ tokio::task::yield_now().await;+ }++ let Some(dialog) = conn.dialog_state(cdp_target_id) else {+ return if action == "inspect" {+ ToolResult::text(format!(+ "no page-owned JavaScript dialog is open in {tab_id}"+ ))+ .with_structured(json!({+ "status": "ok", "target_id": target_id, "tab_id": tab_id,+ "present": false+ }))+ } else {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "no current page-owned JavaScript dialog exists on the exact tab",+ )+ .to_tool_result()+ };+ };+ let dialog_id = format!("dialog-{}", dialog.generation);+ if action == "inspect" {+ return ToolResult::text(format!("{} dialog is open in {tab_id}", dialog.kind))+ .with_structured(json!({+ "status": "ok", "target_id": target_id, "tab_id": tab_id,+ "present": true, "dialog_id": dialog_id, "kind": dialog.kind+ }));+ }++ let supplied_id = match args.require_str("dialog_id") {+ Ok(id) => id,+ Err(error) => return error,+ };+ if supplied_id != dialog_id {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the dialog capability is stale or belongs to another dialog",+ )+ .to_tool_result();+ }+ let prompt_text = args.opt_str("prompt_text");+ if prompt_text.is_some() && (action != "accept" || dialog.kind != "prompt") {+ return ToolResult::error("prompt_text is valid only when accepting a prompt dialog");+ }+ let mut params = json!({ "accept": action == "accept" });+ if let Some(text) = prompt_text {+ params["promptText"] = Value::String(text);+ }+ match conn+ .call(+ Some(&dialog.session_id),+ "Page.handleJavaScriptDialog",+ params,+ )+ .await+ {+ Ok(_) => {+ conn.clear_dialog_state(cdp_target_id, dialog.generation);+ ToolResult::text(format!("{action}ed {} dialog in {tab_id}", dialog.kind))+ .with_structured(json!({+ "status": "ok", "target_id": target_id, "tab_id": tab_id,+ "dialog_id": dialog_id, "kind": dialog.kind, "action": action+ }))+ }+ Err(error) => BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!("the exact JavaScript dialog could not be resolved: {error}"),+ )+ .to_tool_result(),+ }+ }+}++// ── browser_set_input_files ─────────────────────────────────────────────────++pub struct BrowserSetInputFilesTool {+ def: ToolDef,+ engine: Arc<BrowserEngine>,+}++impl BrowserSetInputFilesTool {+ pub fn new(engine: Arc<BrowserEngine>) -> Self {+ Self {+ def: ToolDef {+ name: "browser_set_input_files".into(),+ description: "Assign one or more explicit absolute local files to an exact live <input type=file> ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.".into(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "target_id": schema_target_id(),+ "tab_id": schema_tab_id(),+ "session": schema_session(),+ "ref": schema_ref(),+ "files": {+ "type": "array", "minItems": 1, "maxItems": 32,+ "items": { "type": "string", "description": "Absolute path to one local regular file." }+ }+ },+ "required": ["target_id", "tab_id", "ref", "files"],+ "additionalProperties": true+ }),+ read_only: false,+ destructive: false,+ idempotent: false,+ open_world: true,+ },+ engine,+ }+ }+}++fn validated_upload_paths(args: &Value) -> Result<Vec<String>, ToolResult> {+ let Some(files) = args.get("files").and_then(Value::as_array) else {+ return Err(ToolResult::error(+ "files must be a non-empty array of absolute paths",+ ));+ };+ if files.is_empty() || files.len() > 32 {+ return Err(ToolResult::error(+ "files must contain between 1 and 32 paths",+ ));+ }+ files+ .iter()+ .map(|value| {+ let raw = value+ .as_str()+ .ok_or_else(|| ToolResult::error("every files entry must be a string"))?;+ let path = std::path::Path::new(raw);+ if !path.is_absolute() {+ return Err(ToolResult::error("every upload path must be absolute"));+ }+ let metadata = std::fs::symlink_metadata(path)+ .map_err(|_| ToolResult::error("an approved upload path does not exist"))?;+ if metadata.file_type().is_symlink() || !metadata.is_file() {+ return Err(ToolResult::error(+ "upload paths must name regular files directly, not links or directories",+ ));+ }+ std::fs::canonicalize(path)+ .map(|path| path.to_string_lossy().into_owned())+ .map_err(|_| ToolResult::error("an upload path could not be canonicalized"))+ })+ .collect()+}++#[async_trait]+impl Tool for BrowserSetInputFilesTool {+ fn def(&self) -> &ToolDef {+ &self.def+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let (target_id, tab_id, ext_ref) = match (+ args.require_str("target_id"),+ args.require_str("tab_id"),+ args.require_str("ref"),+ ) {+ (Ok(target), Ok(tab), Ok(ext_ref)) => (target, tab, ext_ref),+ (Err(error), _, _) | (_, Err(error), _) | (_, _, Err(error)) => return error,+ };+ let files = match validated_upload_paths(&args) {+ Ok(files) => files,+ Err(error) => return error,+ };+ let session = match require_explicit_session(&args) {+ Ok(session) => session,+ Err(error) => return error,+ };+ let _mutation = match self+ .engine+ .lock_mutation(&session, &target_id, &tab_id)+ .await+ {+ Ok(guard) => guard,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let validated = match self+ .engine+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))+ .await+ {+ Ok(validated) => validated,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let entry = match self+ .engine+ .store+ .resolve_ref(&session, &target_id, &tab_id, &ext_ref)+ {+ Ok(entry) => entry,+ Err(refusal) => return refusal.to_tool_result(),+ };+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Upload) {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the semantic ref is not a file-upload control",+ )+ .to_tool_result();+ }+ let cdp_session = match self+ .engine+ .frame_session_for_mutation(&session, &target_id, &tab_id, &validated, &entry.frame)+ .await+ {+ Ok(session) => session,+ Err(refusal) => return refusal.to_tool_result(),+ };+ let described = match validated+ .conn+ .call(+ Some(&cdp_session),+ "DOM.describeNode",+ json!({ "backendNodeId": entry.backend_node_id }),+ )+ .await+ {+ Ok(node) => node,+ Err(_) => {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserRefStale,+ "the upload ref no longer resolves in the live page",+ )+ .to_tool_result()+ }+ };+ let node = &described["node"];+ let attributes = node["attributes"].as_array().cloned().unwrap_or_default();+ let is_file_input = node["nodeName"]+ .as_str()+ .is_some_and(|name| name.eq_ignore_ascii_case("input"))+ && attributes.chunks_exact(2).any(|pair| {+ pair[0]+ .as_str()+ .is_some_and(|name| name.eq_ignore_ascii_case("type"))+ && pair[1]+ .as_str()+ .is_some_and(|value| value.eq_ignore_ascii_case("file"))+ });+ if !is_file_input {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ "the live ref is not an input[type=file] element",+ )+ .to_tool_result();+ }+ match validated+ .conn+ .call(+ Some(&cdp_session),+ "DOM.setFileInputFiles",+ json!({ "backendNodeId": entry.backend_node_id, "files": files }),+ )+ .await+ {+ Ok(_) => ToolResult::text(format!("assigned {} file(s) in {tab_id}", files.len()))+ .with_structured(json!({+ "status": "ok", "target_id": target_id, "tab_id": tab_id,+ "ref": ext_ref, "frame": entry.frame.kind.as_str(), "file_count": files.len()+ })),+ Err(error) => BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!("the browser refused the exact file input assignment: {error}"),+ )+ .to_tool_result(),+ }+ }+}+#[cfg(test)]mod tests {use super::*;@@ -1541,6 +1926,12 @@ mod tests {assert!(!prepare_properties.contains_key("allow_restart"));assert!(!prepare_properties.contains_key(MCP_HOST_APPROVAL_ARG));+ let dialog = BrowserDialogTool::new(e.clone());+ assert_eq!(+ dialog.def().input_schema["properties"]["delivery_mode"]["default"],+ "background"+ );+for (def, name) in [(BrowserPrepareTool::new(e.clone()).def().clone(),@@ -1558,18 +1949,37 @@ mod tests {BrowserTypeTool::new(e.clone()).def().clone(),"browser_type",),+ (+ BrowserDialogTool::new(e.clone()).def().clone(),+ "browser_dialog",+ ),+ (+ BrowserSetInputFilesTool::new(e.clone()).def().clone(),+ "browser_set_input_files",+ ),+ (+ BrowserDownloadTool::new(e.clone()).def().clone(),+ "browser_download",+ ),+ (+ BrowserPointerTool::new(e.clone()).def().clone(),+ "browser_pointer",+ ),] {assert_eq!(def.name, name);assert!(!def.read_only, "{name} is a mutation");assert!(def.input_schema["properties"].is_object(), "{name} schema");}- // Navigate reaches the open web; the input tools do not.+ // Navigate and download reach the open web; ordinary input does not.assert!(BrowserNavigateTool::new(e.clone()).def().open_world);- assert!(!BrowserClickTool::new(e).def().open_world);+ assert!(BrowserDownloadTool::new(e.clone()).def().open_world);+ assert!(BrowserDownloadTool::new(e.clone()).def().destructive);+ assert!(!BrowserClickTool::new(e.clone()).def().open_world);+ assert!(!BrowserPointerTool::new(e).def().open_world);}#[test]- fn registration_registers_all_five() {+ fn registration_registers_all_browser_tools() {let e = engine();let mut registry = ToolRegistry::new();register_browser_tools(&e, &mut registry);@@ -1581,11 +1991,27 @@ mod tests {"browser_prepare","browser_navigate","browser_click",- "browser_type"+ "browser_type",+ "browser_dialog",+ "browser_set_input_files",+ "browser_download",+ "browser_pointer"]);}+ #[test]+ fn upload_paths_are_absolute_regular_files_and_outputs_need_no_path() {+ let path = std::env::temp_dir().join(format!("cua-upload-{}.txt", uuid::Uuid::new_v4()));+ std::fs::write(&path, b"fixture").unwrap();+ let args = json!({ "files": [path.to_string_lossy()] });+ let validated = validated_upload_paths(&args).unwrap();+ assert_eq!(validated.len(), 1);+ assert!(std::path::Path::new(&validated[0]).is_absolute());+ assert!(validated_upload_paths(&json!({ "files": ["relative.txt"] })).is_err());+ std::fs::remove_file(path).unwrap();+ }+#[tokio::test]async fn bind_requires_an_explicit_session() {let e = engine();@@ -1781,6 +2207,7 @@ mod tests {cdp_target_id: "CDPX".into(),title: "Mock".into(),url: "https://example.test".into(),+ active: Some(true),generation: 0,snapshots: HashMap::new(),},@@ -1837,6 +2264,7 @@ mod tests {cdp_target_id: "CDPX".into(),title: "Mock".into(),url: "https://example.test".into(),+ active: Some(true),generation: 0,snapshots: HashMap::new(),},diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rsindex 3f9bf8e818..60c3161ad1 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs@@ -265,6 +265,21 @@ pub enum ToolOperation {BrowserTypeKeystrokes,BrowserPrepareIsolated,BrowserPrepareExistingProfile,+ BrowserDialogInspect,+ BrowserDialogAccept,+ BrowserDialogDismiss,+ BrowserSetInputFiles,+ BrowserDownload,+ BrowserPointerHoverTrusted,+ BrowserPointerHoverDomEvent,+ BrowserPointerRightClickTrusted,+ BrowserPointerRightClickDomEvent,+ BrowserPointerDoubleClickTrusted,+ BrowserPointerDoubleClickDomEvent,+ BrowserPointerScrollTrusted,+ BrowserPointerScrollDomEvent,+ BrowserPointerDragTrusted,+ BrowserPointerDragDomEvent,Other,}@@ -288,6 +303,21 @@ impl ToolOperation {Self::BrowserTypeKeystrokes => "browser_type_keystrokes",Self::BrowserPrepareIsolated => "browser_prepare_isolated",Self::BrowserPrepareExistingProfile => "browser_prepare_existing_profile",+ Self::BrowserDialogInspect => "browser_dialog_inspect",+ Self::BrowserDialogAccept => "browser_dialog_accept",+ Self::BrowserDialogDismiss => "browser_dialog_dismiss",+ Self::BrowserSetInputFiles => "browser_set_input_files",+ Self::BrowserDownload => "browser_download",+ Self::BrowserPointerHoverTrusted => "browser_pointer_hover_trusted",+ Self::BrowserPointerHoverDomEvent => "browser_pointer_hover_dom_event",+ Self::BrowserPointerRightClickTrusted => "browser_pointer_right_click_trusted",+ Self::BrowserPointerRightClickDomEvent => "browser_pointer_right_click_dom_event",+ Self::BrowserPointerDoubleClickTrusted => "browser_pointer_double_click_trusted",+ Self::BrowserPointerDoubleClickDomEvent => "browser_pointer_double_click_dom_event",+ Self::BrowserPointerScrollTrusted => "browser_pointer_scroll_trusted",+ Self::BrowserPointerScrollDomEvent => "browser_pointer_scroll_dom_event",+ Self::BrowserPointerDragTrusted => "browser_pointer_drag_trusted",+ Self::BrowserPointerDragDomEvent => "browser_pointer_drag_dom_event",Self::Other => "other",}}@@ -349,6 +379,35 @@ pub fn tool_operation(tool_name: &str, args: Option<&serde_json::Value>) -> Tool_ => ToolOperation::Other,}}+ "browser_dialog" => match string_arg("action") {+ Some("inspect") => ToolOperation::BrowserDialogInspect,+ Some("accept") => ToolOperation::BrowserDialogAccept,+ Some("dismiss") => ToolOperation::BrowserDialogDismiss,+ _ => ToolOperation::Other,+ },+ "browser_set_input_files" => ToolOperation::BrowserSetInputFiles,+ "browser_download" => ToolOperation::BrowserDownload,+ "browser_pointer" => match (string_arg("action"), string_arg("input_route")) {+ (Some("hover"), None | Some("trusted")) => ToolOperation::BrowserPointerHoverTrusted,+ (Some("hover"), Some("dom_event")) => ToolOperation::BrowserPointerHoverDomEvent,+ (Some("right_click"), None | Some("trusted")) => {+ ToolOperation::BrowserPointerRightClickTrusted+ }+ (Some("right_click"), Some("dom_event")) => {+ ToolOperation::BrowserPointerRightClickDomEvent+ }+ (Some("double_click"), None | Some("trusted")) => {+ ToolOperation::BrowserPointerDoubleClickTrusted+ }+ (Some("double_click"), Some("dom_event")) => {+ ToolOperation::BrowserPointerDoubleClickDomEvent+ }+ (Some("scroll"), None | Some("trusted")) => ToolOperation::BrowserPointerScrollTrusted,+ (Some("scroll"), Some("dom_event")) => ToolOperation::BrowserPointerScrollDomEvent,+ (Some("drag"), None | Some("trusted")) => ToolOperation::BrowserPointerDragTrusted,+ (Some("drag"), Some("dom_event")) => ToolOperation::BrowserPointerDragDomEvent,+ _ => ToolOperation::Other,+ },_ => ToolOperation::NotApplicable,}}@@ -368,6 +427,18 @@ pub fn is_computer_action(tool_name: &str, operation: ToolOperation) -> bool {| ToolOperation::EnableJavascriptAppleEvents);}+ if tool_name == "browser_dialog" {+ return matches!(+ operation,+ ToolOperation::BrowserDialogAccept | ToolOperation::BrowserDialogDismiss+ );+ }+ if matches!(+ tool_name,+ "browser_set_input_files" | "browser_download" | "browser_pointer"+ ) {+ return true;+ }crate::tool::default_capabilities_for(tool_name).iter().any(|capability| {@@ -688,6 +759,10 @@ fn structured_refusal_code(tool_name: &str, result: Option<&serde_json::Value>)| "browser_navigate"| "browser_click"| "browser_type"+ | "browser_dialog"+ | "browser_set_input_files"+ | "browser_download"+ | "browser_pointer") {return ToolRefusalCode::None;}@@ -856,6 +931,14 @@ pub async fn handle_request();}}+ if call.name == "browser_download" {+ if let Some(arguments) = call.args.as_object_mut() {+ arguments.insert(+ crate::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG.to_owned(),+ serde_json::Value::Bool(true),+ );+ }+ }let result = registry.invoke(&call.name, call.args).await;match serde_json::to_value(result) {@@ -911,7 +994,11 @@ mod observation_tests {#[test]fn page_operation_is_closed_and_retains_no_argument_content() {for (action, expected, computer_action) in [- ("execute_javascript", ToolOperation::ExecuteJavascript, false),+ (+ "execute_javascript",+ ToolOperation::ExecuteJavascript,+ false,+ ),("get_text", ToolOperation::GetText, false),("query_dom", ToolOperation::QueryDom, false),("click_element", ToolOperation::ClickElement, true),@@ -1008,6 +1095,26 @@ mod observation_tests {}),ToolOperation::BrowserPrepareExistingProfile,),+ (+ "browser_dialog",+ serde_json::json!({"action": "accept", "prompt_text": "private prompt"}),+ ToolOperation::BrowserDialogAccept,+ ),+ (+ "browser_set_input_files",+ serde_json::json!({"files": ["/private/path"]}),+ ToolOperation::BrowserSetInputFiles,+ ),+ (+ "browser_download",+ serde_json::json!({"destination_root": "/private/destination"}),+ ToolOperation::BrowserDownload,+ ),+ (+ "browser_pointer",+ serde_json::json!({"action": "drag", "input_route": "dom_event", "ref": "private-ref"}),+ ToolOperation::BrowserPointerDragDomEvent,+ ),] {let operation = tool_operation(tool_name, Some(&args));assert_eq!(operation, expected, "tool={tool_name}");@@ -1019,6 +1126,9 @@ mod observation_tests {"private typed text","private-profile","private-token",+ "private prompt",+ "/private/path",+ "/private/destination",] {assert!(!debug.contains(forbidden), "observer leaked {forbidden}");}@@ -1070,7 +1180,11 @@ mod observation_tests {ToolRefusalCode::BrowserInputTrustUnavailable);let debug = format!("{observation:?}");- for forbidden in ["private refusal prose", "private refusal message", "private tab id"] {+ for forbidden in [+ "private refusal prose",+ "private refusal message",+ "private tab id",+ ] {assert!(!debug.contains(forbidden), "observer leaked {forbidden}");}}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rsindex 67870568dc..f0e6bf29d0 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs@@ -452,7 +452,14 @@ mod tests {#[test]fn browser_mutations_are_computer_actions_but_reads_and_prepare_are_not() {let args = serde_json::json!({"session": "test-session"});- for tool_name in ["browser_navigate", "browser_click", "browser_type"] {+ for tool_name in [+ "browser_navigate",+ "browser_click",+ "browser_type",+ "browser_set_input_files",+ "browser_download",+ "browser_pointer",+ ] {let operation = crate::server::tool_operation(tool_name, Some(&args));assert!(crate::server::is_computer_action(tool_name, operation),@@ -466,6 +473,20 @@ mod tests {"tool={tool_name}");}+ for action in ["accept", "dismiss"] {+ let args = serde_json::json!({"action": action});+ let operation = crate::server::tool_operation("browser_dialog", Some(&args));+ assert!(crate::server::is_computer_action(+ "browser_dialog",+ operation+ ));+ }+ let inspect = serde_json::json!({"action": "inspect"});+ let operation = crate::server::tool_operation("browser_dialog", Some(&inspect));+ assert!(!crate::server::is_computer_action(+ "browser_dialog",+ operation+ ));}#[test]diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rsindex aa593ca324..da0299c091 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs@@ -103,7 +103,8 @@ impl ToolDef {/// `recording.replay`, `recording.install_dependency`/// - `page.action`/// - `browser.state`, `browser.prepare`, `browser.navigate`,-/// `browser.input.click`, `browser.input.type`+/// `browser.input.click`, `browser.input.type`, `browser.input.files`,+/// `browser.dialog`/// - `driver.update_check`, `driver.probe`////// Tools with no entry get `[]` — that's fine, it just means@@ -265,6 +266,10 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {"browser_navigate" => &["browser.navigate"],"browser_click" => &["browser.input.click"],"browser_type" => &["browser.input.type"],+ "browser_dialog" => &["browser.dialog"],+ "browser_set_input_files" => &["browser.input.files"],+ "browser_download" => &["browser.download"],+ "browser_pointer" => &["browser.input.pointer"],// ── driver self-service ──────────────────────────────────────"check_for_update" => &["driver.update_check"],@@ -482,19 +487,48 @@ fn is_existing_profile_prepare(tool_name: &str, args: &Value) -> bool {}fn recording_args_for(tool_name: &str, args: &Value) -> Value {- if tool_name != "browser_prepare" {- return args.clone();- }let mut redacted = args.clone();if let Some(arguments) = redacted.as_object_mut() {- if arguments.contains_key("approval_token") {- arguments.insert(- "approval_token".to_owned(),- Value::String("[redacted]".to_owned()),- );+ match tool_name {+ "browser_prepare" => {+ if arguments.contains_key("approval_token") {+ arguments.insert(+ "approval_token".to_owned(),+ Value::String("[redacted]".to_owned()),+ );+ }+ arguments.remove("_cua_browser_prepare_mcp_host_approved");+ arguments.remove("_transport_session_id");+ }+ "browser_dialog" => {+ if arguments.contains_key("prompt_text") {+ arguments.insert(+ "prompt_text".to_owned(),+ Value::String("[redacted]".to_owned()),+ );+ }+ }+ "browser_set_input_files" => {+ if let Some(count) = arguments+ .get("files")+ .and_then(Value::as_array)+ .map(Vec::len)+ {+ arguments.insert("files".to_owned(), serde_json::json!({ "count": count }));+ }+ }+ "browser_download" => {+ arguments.remove("_cua_browser_download_mcp_host_approved");+ if arguments.contains_key("destination_root") {+ arguments.insert(+ "destination_root".to_owned(),+ Value::String("[redacted]".to_owned()),+ );+ }+ }+ "browser_pointer" => {}+ _ => {}}- arguments.remove("_cua_browser_prepare_mcp_host_approved");- arguments.remove("_transport_session_id");}redacted}@@ -584,6 +618,46 @@ mod capability_tests {assert_eq!(recorded["strategy"]["kind"], "existing_profile");}+ #[test]+ fn browser_sensitive_recording_args_are_path_and_text_free() {+ let dialog = recording_args_for(+ "browser_dialog",+ &serde_json::json!({"action": "accept", "prompt_text": "private reply"}),+ );+ assert_eq!(dialog["prompt_text"], "[redacted]");++ let upload = recording_args_for(+ "browser_set_input_files",+ &serde_json::json!({"files": ["/private/one", "/private/two"]}),+ );+ assert_eq!(upload["files"], serde_json::json!({"count": 2}));++ let download = recording_args_for(+ "browser_download",+ &serde_json::json!({+ "destination_root": "/private/destination",+ "_cua_browser_download_mcp_host_approved": true,+ }),+ );+ assert_eq!(download["destination_root"], "[redacted]");+ assert!(download+ .get("_cua_browser_download_mcp_host_approved")+ .is_none());++ let serialized = serde_json::json!([dialog, upload, download]).to_string();+ for forbidden in [+ "private reply",+ "/private/one",+ "/private/two",+ "/private/destination",+ ] {+ assert!(+ !serialized.contains(forbidden),+ "recording leaked {forbidden}"+ );+ }+ }+#[test]fn existing_profile_prepare_is_a_private_consent_turn() {assert!(is_existing_profile_prepare(@@ -667,6 +741,10 @@ mod capability_tests {"browser_navigate","browser_click","browser_type",+ "browser_dialog",+ "browser_set_input_files",+ "browser_download",+ "browser_pointer",];/// All capability tokens in the canonical vocabulary. Any token@@ -739,6 +817,10 @@ mod capability_tests {"browser.navigate","browser.input.click","browser.input.type",+ "browser.input.files",+ "browser.dialog",+ "browser.download",+ "browser.input.pointer",// driver self"driver.update_check","driver.probe",diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rsindex e597dc09d3..eb9532d3c2 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs@@ -13,6 +13,7 @@ use std::time::Duration;pub struct BrowserFixtureServer {page_url: String,journal_url: String,+ download_url: String,latest: Arc<Mutex<serde_json::Value>>,observed: Arc<Mutex<Vec<serde_json::Value>>>,stop: Arc<AtomicBool>,@@ -30,6 +31,7 @@ impl BrowserFixtureServer {.expect("read browser fixture server address");let page_url = format!("http://{address}/fixture");let journal_url = format!("http://{address}/state");+ let download_url = format!("http://{address}/download");let journal_script = format!("<script>window.__CUA_E2E_FIXTURE_JOURNAL_URL={};</script>",serde_json::to_string(&journal_url).expect("serialize fixture journal URL")@@ -73,6 +75,7 @@ impl BrowserFixtureServer {Self {page_url,journal_url,+ download_url,latest,observed,stop,@@ -100,6 +103,11 @@ impl BrowserFixtureServer {&self.journal_url}+ /// Fixed loopback attachment used by browser download E2E rows.+ pub fn download_url(&self) -> &str {+ &self.download_url+ }+pub fn contains(&self, marker: &str) -> bool {self.latest.lock()@@ -214,6 +222,14 @@ fn handle_connection();let _ = stream.write_all(headers.as_bytes());let _ = stream.write_all(page);+ } else if request_line.starts_with("GET /download ") {+ const DOWNLOAD: &[u8] = b"CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1\n";+ let headers = format!(+ "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\nContent-Disposition: attachment; filename=fixture.txt\r\nContent-Length: {}\r\nCache-Control: no-store\r\nConnection: close\r\n\r\n",+ DOWNLOAD.len()+ );+ let _ = stream.write_all(headers.as_bytes());+ let _ = stream.write_all(DOWNLOAD);} else if request_line.starts_with("POST /state ") {if let Some((body_start, content_len)) = body_range {if let Some(body) = request.get(body_start..body_start + content_len) {@@ -261,10 +277,12 @@ mod tests {#[test]fn serves_fixture_and_receives_external_state() {let server = BrowserFixtureServer::start("<html><head></head><body>marker</body></html>");- let address = address(server.page_url(), "/fixture");- let mut stream = TcpStream::connect(&address).expect("connect browser fixture");+ let fixture_address = address(server.page_url(), "/fixture");+ let mut stream = TcpStream::connect(&fixture_address).expect("connect browser fixture");stream- .write_all(format!("GET /fixture HTTP/1.1\r\nHost: {address}\r\n\r\n").as_bytes())+ .write_all(+ format!("GET /fixture HTTP/1.1\r\nHost: {fixture_address}\r\n\r\n").as_bytes(),+ ).expect("request browser fixture");let mut response = String::new();stream@@ -274,12 +292,27 @@ mod tests {assert!(response.contains("marker"));assert!(response.contains(server.journal_url()));+ let download_address = address(server.download_url(), "/download");+ let mut stream =+ TcpStream::connect(&download_address).expect("connect browser download fixture");+ stream+ .write_all(+ format!("GET /download HTTP/1.1\r\nHost: {download_address}\r\n\r\n").as_bytes(),+ )+ .expect("request browser download fixture");+ let mut response = String::new();+ stream+ .read_to_string(&mut response)+ .expect("read browser download fixture");+ assert!(response.contains("Content-Disposition: attachment"));+ assert!(response.contains("CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1"));+let body = r#"{"status":{"text":"last_action=left_click"}}"#;- let mut stream = TcpStream::connect(&address).expect("connect browser journal");+ let mut stream = TcpStream::connect(&fixture_address).expect("connect browser journal");stream.write_all(format!(- "POST /state HTTP/1.1\r\nHost: {address}\r\nContent-Length: {}\r\n\r\n{body}",+ "POST /state HTTP/1.1\r\nHost: {fixture_address}\r\nContent-Length: {}\r\n\r\n{body}",body.len()).as_bytes(),diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rsindex a636d86c70..45552e7bfd 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs@@ -31,7 +31,7 @@ fn is_active_proxy_session(session: Option<&str>) -> bool {session.is_some_and(|session| active_proxy_sessions().lock().unwrap().contains(session))}-fn inject_browser_prepare_approval(+fn inject_browser_approvals(tool_name: &str,args: &mut serde_json::Value,session: Option<&str>,@@ -44,6 +44,14 @@ fn inject_browser_prepare_approval();}}+ if tool_name == "browser_download" && is_active_proxy_session(session) {+ if let Some(arguments) = args.as_object_mut() {+ arguments.insert(+ cua_driver_core::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG.to_owned(),+ serde_json::Value::Bool(true),+ );+ }+ }}// ── Recording idle-TTL backstop (#1764) ─────────────────────────────────────────@@ -482,7 +490,7 @@ async fn invoke_daemon_tool(}}- inject_browser_prepare_approval(&tool_name, &mut args, req.session_id.as_deref());+ inject_browser_approvals(&tool_name, &mut args, req.session_id.as_deref());let session_context = observation_transport.and_then(|transport| {let transport = match transport {@@ -1843,8 +1851,9 @@ mod telemetry_routing_tests {#[cfg(test)]mod session_boundary_tests {- use super::{active_proxy_sessions, apply_session_identity, inject_browser_prepare_approval};+ use super::{active_proxy_sessions, apply_session_identity, inject_browser_approvals};use cua_driver_core::browser::approval::MCP_HOST_APPROVAL_ARG;+ use cua_driver_core::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG;use serde_json::json;#[test]@@ -1899,7 +1908,7 @@ mod session_boundary_tests {fn browser_prepare_approval_requires_a_live_proxy_session() {let session = "approval-boundary-test";let mut raw_args = json!({"pid": 42});- inject_browser_prepare_approval("browser_prepare", &mut raw_args, Some(session));+ inject_browser_approvals("browser_prepare", &mut raw_args, Some(session));assert!(raw_args.get(MCP_HOST_APPROVAL_ARG).is_none());active_proxy_sessions()@@ -1907,7 +1916,7 @@ mod session_boundary_tests {.unwrap().insert(session.to_owned());let mut proxy_args = json!({"pid": 42});- inject_browser_prepare_approval("browser_prepare", &mut proxy_args, Some(session));+ inject_browser_approvals("browser_prepare", &mut proxy_args, Some(session));active_proxy_sessions().lock().unwrap().remove(session);assert_eq!(proxy_args[MCP_HOST_APPROVAL_ARG], true);@@ -1916,8 +1925,21 @@ mod session_boundary_tests {.lock().unwrap().insert(session.to_owned());- inject_browser_prepare_approval("get_browser_state", &mut other_tool, Some(session));+ inject_browser_approvals("get_browser_state", &mut other_tool, Some(session));active_proxy_sessions().lock().unwrap().remove(session);assert!(other_tool.get(MCP_HOST_APPROVAL_ARG).is_none());++ let mut raw_download = json!({"destination_root": "/private/path"});+ inject_browser_approvals("browser_download", &mut raw_download, Some(session));+ assert!(raw_download.get(MCP_HOST_DOWNLOAD_APPROVAL_ARG).is_none());++ active_proxy_sessions()+ .lock()+ .unwrap()+ .insert(session.to_owned());+ let mut proxy_download = json!({"destination_root": "/private/path"});+ inject_browser_approvals("browser_download", &mut proxy_download, Some(session));+ active_proxy_sessions().lock().unwrap().remove(session);+ assert_eq!(proxy_download[MCP_HOST_DOWNLOAD_APPROVAL_ARG], true);}}diff --git a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rsindex b7335212f6..2d1e03a378 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs@@ -426,6 +426,10 @@ impl AgentSessionState {| "browser_navigate"| "browser_click"| "browser_type"+ | "browser_dialog"+ | "browser_set_input_files"+ | "browser_download"+ | "browser_pointer");let refused = outcome.refusal_code.is_refusal();let completed_computer_action = computer_action && !refused;@@ -2270,6 +2274,7 @@ mod tests {#[test]fn v3_payload_allows_server_geoip_without_sending_an_ip_or_client_timestamp() {+ let _guard = ENV_LOCK.lock().unwrap();let identity = InstallationIdentity { id: "test-id".into(), persisted: true };let payload = build_payload(event::MCP_TOOL_COMPLETED,@@ -2568,6 +2573,55 @@ mod tests {assert_eq!(session_properties["used_browser"], true);}+ #[test]+ fn every_browser_tool_marks_the_session_without_retaining_arguments() {+ use cua_driver_core::server::{+ DurationBucket, OutputSizeBucket, OutputType, ToolCompletionObservation,+ ToolErrorClass, ToolOperation, ToolRefusalCode,+ };++ for (tool_name, operation) in [+ ("browser_dialog", ToolOperation::BrowserDialogAccept),+ (+ "browser_set_input_files",+ ToolOperation::BrowserSetInputFiles,+ ),+ ("browser_download", ToolOperation::BrowserDownload),+ (+ "browser_pointer",+ ToolOperation::BrowserPointerDoubleClickDomEvent,+ ),+ ] {+ let mut state = AgentSessionState::new(Transport::McpStdio);+ state.observe(+ Transport::McpStdio,+ true,+ &ToolCompletionObservation {+ tool_name: tool_name.into(),+ operation,+ computer_action: true,+ success: true,+ error_class: ToolErrorClass::None,+ refusal_code: ToolRefusalCode::None,+ duration_bucket: DurationBucket::Under10Ms,+ output_type: OutputType::Text,+ output_size_bucket: OutputSizeBucket::Under1KiB,+ },+ );+ let properties =+ state.ended_properties(cua_driver_core::session::SessionEndReason::Explicit, None);+ assert_eq!(properties["used_browser"], true, "tool={tool_name}");+ assert_eq!(+ properties["had_successful_computer_action"], true,+ "tool={tool_name}"+ );+ let serialized = serde_json::to_string(&properties).unwrap();+ assert!(!serialized.contains("destination_root"));+ assert!(!serialized.contains("prompt_text"));+ assert!(!serialized.contains("file_paths"));+ }+ }+#[test]fn agent_session_buckets_have_fixed_boundaries() {for (count, expected) in [diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rsindex 3697cd733c..43701e107b 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs@@ -45,6 +45,55 @@ fn standalone_fixture_html() -> String {)}+fn standalone_browser_completeness_html() -> String {+ standalone_fixture_html().replace(+ "</body>",+ r#"<fieldset>+ <legend>browser completeness</legend>+ <button id="standalone-alert" data-cua-id="standalone-alert">Open alert</button>+ <button id="standalone-confirm" data-cua-id="standalone-confirm">Open confirm</button>+ <button id="standalone-prompt" data-cua-id="standalone-prompt">Open prompt</button>+ <span id="standalone-dialog-state" data-cua-id="standalone-dialog-state">dialog=none</span>+ <input id="standalone-upload" data-cua-id="standalone-upload" aria-label="standalone-upload" type="file" multiple>+ <span id="standalone-upload-state" data-cua-id="standalone-upload-state">upload=0</span>+ <a id="standalone-download" data-cua-id="standalone-download" aria-label="standalone-download" href="/download" download>Download fixture</a>+ <span id="standalone-hover-state" data-cua-id="standalone-hover-state">hover=false</span>+</fieldset>+<script>+ const dialogState = document.getElementById('standalone-dialog-state');+ document.getElementById('standalone-alert').addEventListener('click', () => {+ setTimeout(() => { alert('fixture alert'); dialogState.textContent = 'dialog=alert:accepted'; }, 0);+ });+ document.getElementById('standalone-confirm').addEventListener('click', () => {+ setTimeout(() => { dialogState.textContent = `dialog=confirm:${confirm('fixture confirm')}`; }, 0);+ });+ document.getElementById('standalone-prompt').addEventListener('click', () => {+ setTimeout(() => { dialogState.textContent = `dialog=prompt:${prompt('fixture prompt', '')}`; }, 0);+ });+ document.getElementById('standalone-upload').addEventListener('change', event => {+ const names = Array.from(event.target.files).map(file => file.name).join(',');+ document.getElementById('standalone-upload-state').textContent = `upload=${event.target.files.length}:${names}`;+ });+ document.getElementById('scroll-tall').setAttribute('tabindex', '0');+ document.getElementById('scroll-tall').setAttribute('role', 'region');+ document.getElementById('drag-source').setAttribute('tabindex', '0');+ document.getElementById('drag-source').setAttribute('role', 'button');+ document.getElementById('drag-source').setAttribute('draggable', 'true');+ document.getElementById('drop-target').setAttribute('tabindex', '0');+ document.getElementById('drop-target').setAttribute('role', 'button');+ document.getElementById('click-target').addEventListener('pointerover', () => {+ document.getElementById('standalone-hover-state').textContent = 'hover=true';+ });+ document.getElementById('drop-target').addEventListener('dragover', event => event.preventDefault());+ document.getElementById('drop-target').addEventListener('drop', event => {+ event.preventDefault();+ document.getElementById('drag-status').textContent = 'drag_status=dropped';+ });+</script>+</body>"#,+ )+}+fn standalone_semantic_fixture_html() -> String {let hidden = (0..320).map(|index| {@@ -685,13 +734,16 @@ fn wait_for_exact_browser_binding("session": session,}),);- if state.structured()["binding_quality"] == "exact" {+ let has_unique_active_tab = state.structured()["tabs"]+ .as_array()+ .is_some_and(|tabs| tabs.iter().filter(|tab| tab["active"] == true).count() == 1);+ if state.structured()["binding_quality"] == "exact" && has_unique_active_tab {return Some((window_id, state));}}if Instant::now() >= deadline {eprintln!(- "no exact browser window binding for prepared pid {pid}; last windows={}",+ "no exact browser window binding with one active tab for prepared pid {pid}; last windows={}",windows.raw);return None;@@ -982,6 +1034,20 @@ fn refusal_case(browser: &str, action: &str, code: RefusalCode) -> CaseSpec {case(browser, action).expecting_refusal(vec![code])}+fn foreground_page_case(browser: &str, action: &str) -> CaseSpec {+ CaseSpec::delivered(+ format!("{}-{browser}-standalone-{action}", std::env::consts::OS),+ browser,+ "standalone-chromium",+ action,+ Targeting::Page,+ Delivery::Foreground,+ Scope::Window,+ DriverRoute::Cdp,+ vec![OracleKind::FixtureState],+ )+}+fn run_with_background_oracles(fixture: &mut BrowserFixture,action: impl FnOnce(&mut BrowserFixture) -> Observation,@@ -2134,6 +2200,485 @@ fn run_multi_tab(spec: &BrowserSpec) {});}+fn run_same_title_tabs(spec: &BrowserSpec) {+ let scenario = format!(+ "{}-{}-standalone-same-title-tabs",+ std::env::consts::OS,+ spec.name+ );+ execute_case(+ case(&spec.name, "same_title_tab_selection_unknown"),+ |evidence| {+ let mut fixture = launch_browser(spec, &scenario);+ *evidence = recording_evidence(fixture.driver.recording_dir());+ let session = format!("standalone-same-title-{}", fixture.pid);+ let _ = bind(&mut fixture, &session);+ let second_server = BrowserFixtureServer::start(&standalone_fixture_html());+ let created = harness_cdp_call(+ fixture.cdp_port,+ "Target.createTarget",+ serde_json::json!({+ "url": second_server.page_url(),+ "newWindow": false,+ "background": true,+ }),+ );+ assert!(created["targetId"].is_string(), "{created}");+ wait_for_observed(&second_server, "WEB_HARNESS_MARKER_v1");+ bring_harness_page_to_front(fixture.cdp_port, &fixture.server.page_url());++ run_with_background_oracles(&mut fixture, |fixture| {+ let deadline = Instant::now() + Duration::from_secs(5);+ let rebound = loop {+ let state = fixture.driver.call(+ "get_browser_state",+ serde_json::json!({+ "pid": fixture.pid as i64,+ "window_id": fixture.window_id,+ "session": session,+ }),+ );+ if state.structured()["tabs"]+ .as_array()+ .is_some_and(|tabs| tabs.len() >= 2)+ {+ break state;+ }+ assert!(+ Instant::now() < deadline,+ "same-title bind failed: {}",+ state.raw+ );+ thread::sleep(Duration::from_millis(100));+ };+ assert_eq!(rebound.structured()["binding_quality"], "exact");+ let tabs = rebound.structured()["tabs"]+ .as_array()+ .expect("same-title tab list");+ assert!(tabs+ .iter()+ .any(|tab| tab["url"] == fixture.server.page_url()));+ assert!(tabs+ .iter()+ .any(|tab| tab["url"] == second_server.page_url()));+ assert!(+ tabs.iter().all(|tab| tab["active"].is_null()),+ "same-title selection must be unknown instead of guessed: {}",+ rebound.raw+ );+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ },+ );+}++fn run_dialogs(spec: &BrowserSpec) {+ let scenario = format!("{}-{}-standalone-dialogs", std::env::consts::OS, spec.name);+ let spec_case = if cfg!(target_os = "linux") {+ foreground_page_case(&spec.name, "browser_dialogs")+ } else {+ case(&spec.name, "browser_dialogs")+ };+ execute_case(spec_case, |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ let session = format!("standalone-dialogs-{}", fixture.pid);+ let (target_id, tab_id, snapshot) = bind(&mut fixture, &session);+ let primed = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "inspect",+ "session": session,+ }),+ );+ assert_eq!(primed.structured()["present"], false, "{}", primed.raw);++ let sentinel = if cfg!(target_os = "linux") {+ None+ } else {+ Some(ForegroundSentinel::launch(&mut fixture.driver))+ };+ let target = TargetWindow {+ pid: fixture.pid,+ native_id: fixture.window_id,+ };+ fixture.driver.start_behavior_recording();+ let mut passed_oracles = vec![OracleKind::FixtureState];++ for (button, kind, action, prompt_text, expected) in [+ (+ "id=standalone-alert",+ "alert",+ "accept",+ None,+ "dialog=alert:accepted",+ ),+ (+ "id=standalone-confirm",+ "confirm",+ "dismiss",+ None,+ "dialog=confirm:false",+ ),+ (+ "id=standalone-prompt",+ "prompt",+ "accept",+ Some("fixture response"),+ "dialog=prompt:fixture response",+ ),+ ] {+ // Opening a native browser modal can activate Chromium. Treat that+ // as fixture setup, then prove the typed inspect/resolve operation+ // itself preserves a fully occluded background posture.+ let clicked = fixture.driver.call(+ "browser_click",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "ref": ref_by_label(&snapshot, button),+ "input_route": "dom_event",+ "session": session,+ }),+ );+ assert_eq!(clicked.structured()["status"], "ok", "{}", clicked.raw);+ thread::sleep(Duration::from_millis(100));++ if cfg!(target_os = "linux") {+ let deadline = Instant::now() + Duration::from_secs(5);+ let current = loop {+ let current = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "inspect",+ "session": session,+ }),+ );+ if current.structured()["present"] == true {+ break current;+ }+ assert!(+ Instant::now() < deadline,+ "{kind} dialog was not observed: {}",+ current.raw+ );+ thread::sleep(Duration::from_millis(25));+ };+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);+ let dialog_id = current.structured()["dialog_id"]+ .as_str()+ .expect("opaque dialog id")+ .to_owned();+ let mut args = serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": action,+ "dialog_id": dialog_id,+ "delivery_mode": "foreground",+ "session": session,+ });+ if let Some(prompt_text) = prompt_text {+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());+ }+ let handled = fixture.driver.call("browser_dialog", args);+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);+ continue;+ }++ sentinel+ .as_ref()+ .expect("background dialog sentinel")+ .prepare_background_observation(&mut fixture.driver, target)+ .expect("occlude the open JavaScript dialog");++ let (_, passed) = sentinel+ .as_ref()+ .expect("background dialog sentinel")+ .observe_background(target, || {+ let deadline = Instant::now() + Duration::from_secs(5);+ let current = loop {+ let current = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "inspect",+ "session": session,+ }),+ );+ if current.structured()["present"] == true {+ break current;+ }+ assert!(+ Instant::now() < deadline,+ "{kind} dialog was not observed: {}",+ current.raw+ );+ thread::sleep(Duration::from_millis(25));+ };+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);+ let dialog_id = current.structured()["dialog_id"]+ .as_str()+ .expect("opaque dialog id")+ .to_owned();+ let mut args = serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": action,+ "dialog_id": dialog_id,+ "session": session,+ });+ if let Some(prompt_text) = prompt_text {+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());+ }+ let handled = fixture.driver.call("browser_dialog", args);+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ .expect("observe background JavaScript dialog resolution");+ passed_oracles.extend(passed);+ }++ Observation::delivered(passed_oracles, Evidence::default())+ });+}++#[cfg(target_os = "linux")]+fn run_dialog_background_refusal(spec: &BrowserSpec) {+ let scenario = format!(+ "{}-{}-standalone-dialog-background-refusal",+ std::env::consts::OS,+ spec.name+ );+ execute_case(+ refusal_case(+ &spec.name,+ "browser_dialogs_background",+ RefusalCode::BrowserInputTrustUnavailable,+ ),+ |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-dialog-refusal-{}", fixture.pid);+ let (target_id, tab_id, _) = bind(fixture, &session);+ let refused = fixture.driver.call(+ "browser_dialog",+ serde_json::json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "action": "accept",+ "dialog_id": "dialog-unavailable",+ "session": session,+ }),+ );+ assert_eq!(refused.structured()["status"], "refused", "{}", refused.raw);+ assert_eq!(+ refused.structured()["refusal"]["code"],+ "browser_input_trust_unavailable",+ "{}",+ refused.raw+ );+ wait_for_text(&fixture.server, "standalone-dialog-state", "dialog=none");+ Observation::refused(+ RefusalCode::BrowserInputTrustUnavailable,+ vec![OracleKind::FixtureState],+ refused.text(),+ Evidence::default(),+ )+ })+ },+ );+}++fn run_upload(spec: &BrowserSpec) {+ let scenario = format!("{}-{}-standalone-upload", std::env::consts::OS, spec.name);+ execute_case(case(&spec.name, "browser_file_upload"), |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-upload-{}", fixture.pid);+ let (target, tab, _) = bind(fixture, &session);+ let snapshot = fixture.driver.call(+ "get_browser_state",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "session": session,+ "snapshot_format": "semantic_v2",+ "query": "standalone-upload",+ }),+ );+ let upload_ref = semantic_ref_by_name(&snapshot, "standalone-upload", "upload");+ let directory = tempfile::tempdir().expect("create upload fixture directory");+ let path = directory.path().join("fixture-upload.txt");+ std::fs::write(&path, b"fixture upload payload").expect("write upload fixture");+ let canonical = std::fs::canonicalize(&path).expect("canonical upload fixture");+ let uploaded = fixture.driver.call(+ "browser_set_input_files",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": upload_ref,+ "files": [canonical],+ "session": session,+ }),+ );+ assert_eq!(uploaded.structured()["status"], "ok", "{}", uploaded.raw);+ assert_eq!(uploaded.structured()["file_count"], 1, "{}", uploaded.raw);+ let public = uploaded.raw.to_string();+ assert!(+ !public.contains(&path.display().to_string()),+ "{}",+ uploaded.raw+ );+ assert!(!public.contains("fixture-upload.txt"), "{}", uploaded.raw);+ wait_for_text(+ &fixture.server,+ "standalone-upload-state",+ "upload=1:fixture-upload.txt",+ );+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ });+}++fn run_pointer_actions(spec: &BrowserSpec) {+ let scenario = format!("{}-{}-standalone-pointer", std::env::consts::OS, spec.name);+ execute_case(case(&spec.name, "browser_pointer_actions"), |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-pointer-{}", fixture.pid);+ let (target, tab, snapshot) = bind(fixture, &session);+ let click_ref = ref_by_label(&snapshot, "id=click-target");++ for action in ["hover", "right_click", "double_click"] {+ let response = fixture.driver.call(+ "browser_pointer",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": click_ref,+ "action": action,+ "input_route": "dom_event",+ "session": session,+ }),+ );+ assert_eq!(response.structured()["status"], "ok", "{}", response.raw);+ }+ wait_for_text(&fixture.server, "standalone-hover-state", "hover=true");+ wait_for_text(+ &fixture.server,+ "lbl-last-action",+ "last_action=double_click",+ );++ let scrolled = fixture.driver.call(+ "browser_pointer",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": ref_by_label(&snapshot, "id=scroll-tall"),+ "action": "scroll",+ "input_route": "dom_event",+ "delta_y": 240,+ "session": session,+ }),+ );+ assert_eq!(scrolled.structured()["status"], "ok", "{}", scrolled.raw);+ let deadline = Instant::now() + Duration::from_secs(5);+ loop {+ if fixture+ .server+ .text("lbl-scroll-offset")+ .is_some_and(|value| value != "scroll_offset=0")+ {+ break;+ }+ assert!(Instant::now() < deadline, "scroll oracle did not change");+ thread::sleep(Duration::from_millis(25));+ }++ let dragged = fixture.driver.call(+ "browser_pointer",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": ref_by_label(&snapshot, "id=drag-source"),+ "destination_ref": ref_by_label(&snapshot, "id=drop-target"),+ "action": "drag",+ "input_route": "dom_event",+ "session": session,+ }),+ );+ assert_eq!(dragged.structured()["status"], "ok", "{}", dragged.raw);+ wait_for_text(&fixture.server, "drag-status", "drag_status=dropped");+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ });+}++fn run_download(spec: &BrowserSpec) {+ let scenario = format!("{}-{}-standalone-download", std::env::consts::OS, spec.name);+ execute_case(case(&spec.name, "browser_download"), |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-download-{}", fixture.pid);+ let (target, tab, snapshot) = bind(fixture, &session);+ let destination = tempfile::tempdir().expect("create approved download directory");+ let canonical =+ std::fs::canonicalize(destination.path()).expect("canonical download directory");+ let downloaded = fixture.driver.call(+ "browser_download",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": ref_by_label(&snapshot, "id=standalone-download"),+ "destination_root": canonical,+ "session": session,+ }),+ );+ assert_eq!(+ downloaded.structured()["status"],+ "completed",+ "{}",+ downloaded.raw+ );+ let entries = std::fs::read_dir(destination.path())+ .expect("read approved download directory")+ .collect::<Result<Vec<_>, _>>()+ .expect("enumerate approved download directory");+ assert_eq!(entries.len(), 1, "expected one completed download");+ let bytes = std::fs::read(entries[0].path()).expect("read completed download");+ assert_eq!(bytes, b"CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1\n");+ assert_eq!(+ downloaded.structured()["bytes"],+ bytes.len() as u64,+ "{}",+ downloaded.raw+ );+ let public = downloaded.raw.to_string();+ assert!(!public.contains(&destination.path().display().to_string()));+ assert!(!public.contains("fixture.txt"));+ assert!(!public.contains(fixture.server.download_url()));+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ });+}+fn run_two_window_collision(spec: &BrowserSpec) {let scenario = format!("{}-{}-standalone-two-window",@@ -2261,6 +2806,16 @@ standalone_browser_test!(standalone_browser_test!(standalone_browser_stale_ref, run_stale_ref);standalone_browser_test!(standalone_browser_frames, run_frame_roundtrip);standalone_browser_test!(standalone_browser_multi_tab, run_multi_tab);+standalone_browser_test!(standalone_browser_same_title_tabs, run_same_title_tabs);+standalone_browser_test!(standalone_browser_dialogs, run_dialogs);+#[cfg(target_os = "linux")]+standalone_browser_test!(+ standalone_browser_dialog_background_refusal,+ run_dialog_background_refusal+);+standalone_browser_test!(standalone_browser_upload, run_upload);+standalone_browser_test!(standalone_browser_pointer_actions, run_pointer_actions);+standalone_browser_test!(standalone_browser_download, run_download);standalone_browser_test!(standalone_browser_window_collision,run_two_window_collisiondiff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rsindex 063923db88..1494da8fa2 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs@@ -930,7 +930,19 @@ pub fn x11_activate_window_persistent(xid: u64) -> Result<Option<u64>> {prior.unwrap_or(0) as x11::xlib::Window,);unsafe {+ // Some WMs honor `_NET_ACTIVE_WINDOW` as raise-only. Persistent+ // activation must establish input focus too, just like the bounded+ // foreground rung above, or `bring_to_front` can report success while+ // keyboard focus remains on the previous window.+ let previous_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));+ x11::xlib::XSetInputFocus(+ display,+ xid as x11::xlib::Window,+ x11::xlib::RevertToParent,+ x11::xlib::CurrentTime,+ );x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);x11::xlib::XCloseDisplay(display);}Ok(prior)diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.shindex 8b784c2af3..3289edcd6e 100755--- a/scripts/ci/run-rust-standalone-browser-e2e.sh+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh@@ -90,15 +90,24 @@ fitests=(standalone_browser_background_type+ standalone_browser_dialogs+)+if [[ "${HOST_OS}" == Linux ]]; then+ tests+=(standalone_browser_dialog_background_refusal)+fi+tests+=(+ standalone_browser_downloadstandalone_browser_existing_profilestandalone_browser_existing_profile_setupstandalone_browser_framesstandalone_browser_multi_tab+ standalone_browser_pointer_actionsstandalone_browser_prepare_isolatedstandalone_browser_roundtripstandalone_browser_semantic_statestandalone_browser_stale_refstandalone_browser_trusted_click+ standalone_browser_uploadstandalone_browser_window_collision)failure_count=0diff --git a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1 b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1index 6cb111ac86..cc3417b43e 100644--- a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1+++ b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1@@ -117,15 +117,19 @@ if (-not (Test-Path $env:CUA_TEST_DRIVER_BIN)) {$tests = @("standalone_browser_background_type",+ "standalone_browser_dialogs",+ "standalone_browser_download","standalone_browser_existing_profile","standalone_browser_existing_profile_setup","standalone_browser_frames","standalone_browser_multi_tab",+ "standalone_browser_pointer_actions","standalone_browser_prepare_isolated","standalone_browser_roundtrip","standalone_browser_semantic_state","standalone_browser_stale_ref","standalone_browser_trusted_click",+ "standalone_browser_upload","standalone_browser_window_collision")$failureCount = 0diff --git a/scripts/docs-generators/cua-driver.ts b/scripts/docs-generators/cua-driver.tsindex 098486d0ce..59ae169cc3 100644--- a/scripts/docs-generators/cua-driver.ts+++ b/scripts/docs-generators/cua-driver.ts@@ -76,6 +76,8 @@ export interface MCPPropertyDoc {type: string;description: string;items?: { type: string };+ enum?: string[];+ minItems?: number;}export interface MCPDocumentation {@@ -773,6 +775,9 @@ function formatPropertyType(prop: MCPPropertyDoc): string {}function syntheticExampleValue(name: string, prop: MCPPropertyDoc): unknown {+ if (prop.enum?.length) {+ return prop.enum[0];+ }switch (prop.type) {case 'integer':if (name === 'pid') return 844;@@ -794,6 +799,7 @@ function syntheticExampleValue(name: string, prop: MCPPropertyDoc): unknown {case 'array':if (name === 'keys') return ['cmd', 'c'];if (name === 'modifiers') return ['cmd'];+ if (name === 'files' || (prop.minItems ?? 0) > 0) return ['example'];return [];case 'string':if (name === 'text') return 'hello';