Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): complete the browser action surface (#2323)

Francesco Bonacci01a9505aeac5
diff --git a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
index 85cc2e02e0..8969a0a2d9 100644
--- a/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
+++ b/docs/content/docs/concepts/browser-targeting-and-background-delivery.mdx
@@ -18,6 +18,12 @@ Agents still begin with `list_apps` and `list_windows`. They select a concrete
mints opaque target and tab capabilities. Raw DevTools target identifiers are
not part of the public contract.
+Selected-tab state is deliberately tri-state. `active: true` or `false` is
+reported only when the native window title uniquely identifies one DevTools
+tab. Duplicate titles, empty titles, and other ambiguity produce `active:
+null` for every candidate instead of treating DevTools list order as native
+selection evidence. A caller can still target an explicit returned tab id.
+
This preserves one targeting model across native and web content. It also
prevents a browser helper from choosing the first tab, first window, or first
process match when several candidates exist.
@@ -54,13 +60,24 @@ default click uses `Input.dispatchMouseEvent`, but Chromium's standalone window
is known to activate on that route on macOS and Linux. Cua Driver returns
`browser_input_trust_unavailable` there before dispatch. Standalone Chrome and
Edge on Windows, and the bounded embedded Electron route, have passing trusted
-background evidence.
+background evidence. The same trust distinction applies to hover,
+right-click, double-click, scroll, and drag through `browser_pointer`.
A caller must explicitly request `input_route: "dom_event"` to invoke an
element's DOM click behavior. That route is synthetic even when it preserves
full-background posture. The driver does not silently change trust models to
make a call appear successful.
+Page-owned JavaScript dialogs are modeled as short-lived capabilities rather
+than native-window guesses. Inspection returns the kind and an opaque dialog
+generation; accept or dismiss succeeds only while that exact dialog remains
+current. The page's initial creation of a Chromium native modal may activate
+the browser; after occlusion is re-established, inspection and resolution do
+not require another activation. File assignment bypasses native pickers through an exact live file
+input ref. Downloads additionally cross an open-world filesystem boundary, so
+they require host approval, a canonical destination directory, exact event
+correlation, and path-free output.
+
## Capabilities have a lifetime
Target ids, tab ids, and page refs belong to a named driver session. Page refs
diff --git a/docs/content/docs/concepts/how-cua-driver-is-validated.mdx b/docs/content/docs/concepts/how-cua-driver-is-validated.mdx
index 6dedc418a1..e9e0e78d10 100644
--- a/docs/content/docs/concepts/how-cua-driver-is-validated.mdx
+++ b/docs/content/docs/concepts/how-cua-driver-is-validated.mdx
@@ -41,6 +41,15 @@ keys, scrolling, dragging, child windows, controls, and editor behavior. This
makes a failure reproducible without depending on the changing state of an
installed third-party application.
+The optional standalone-browser lane launches installed Chrome, Edge, or
+Chromium with a fresh repo-owned profile and one exact source-built driver. Its
+rows cover binding and ambiguous active-tab evidence, semantic snapshots,
+frames, inactive tabs, JavaScript dialogs, file assignment, extended pointer
+actions, and approval-scoped downloads. The target stays fully occluded behind
+the foreground sentinel while the fixture journal proves page state. This lane
+is the release evidence for typed browser behavior; CDP acknowledgements alone
+do not count as delivery.
+
## One catalog describes each behavior cell
The Rust catalog records the dimensions that affect delivery:
diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
index aa6423a1be..c23e70cded 100644
--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx
@@ -154,9 +154,13 @@ get_browser_state({
})
```
-Keep the returned opaque `target_id` and the `tab_id` whose `active` field is
-`true`. Mutation is available only when `binding_quality` is `exact`. A
-heuristic or ambiguous match is refused.
+Keep the returned opaque `target_id`. Use the `tab_id` whose `active` field is
+`true` only when selection is uniquely proven. `active: false` is proven
+unselected; `active: null` means the native window cannot distinguish tabs,
+such as two tabs with the same title. In that case, select an explicit tab by
+its returned metadata and never infer selection from list order. Mutation is
+available only when `binding_quality` is `exact`. A heuristic or ambiguous
+window match is refused.
## Snapshot the tab
@@ -249,6 +253,61 @@ browser_type({
The default `insert_text` mode is efficient for ordinary text. Use
`mode: "keystrokes"` when the page depends on per-character keyboard events.
+## Use extended pointer actions
+
+`browser_pointer` supports `hover`, `right_click`, `double_click`, `scroll`,
+and `drag`. Use the trusted route when genuine browser input semantics are
+required and the platform reports that it can preserve posture. Use the
+explicit synthetic route with current refs when DOM event semantics are
+acceptable:
+
+```jsonc
+browser_pointer({
+ "target_id": "<target_id>",
+ "tab_id": "<tab_id>",
+ "ref": "p3:4",
+ "action": "scroll",
+ "input_route": "dom_event",
+ "delta_y": 240,
+ "session": "research-1"
+})
+```
+
+For drag, pass `destination_ref` from the same exact frame. The driver refuses
+mixed-frame or stale destinations instead of translating them approximately.
+
+## Handle a page-owned dialog
+
+Prime and inspect the exact tab with `browser_dialog({action:"inspect"})`.
+When `present` is true, pass the returned `dialog_id` to `accept` or `dismiss`.
+Only an accepted prompt may include `prompt_text`. This tool does not handle
+browser permission prompts, extension UI, native sheets, or file pickers.
+Creating a Chromium native modal can activate its browser window; once you
+restore the intended occlusion, inspecting and resolving that exact dialog do
+not activate it again on Windows and macOS. Resolution defaults to
+`delivery_mode:"background"`. Linux Chromium cannot resolve the native modal
+while preserving background posture, so the driver returns
+`browser_input_trust_unavailable` before dispatch. Retry with
+`delivery_mode:"foreground"` only when foreground activation is acceptable.
+
+## Assign files without a picker
+
+Take a semantic snapshot and choose a file-input ref whose `actions` contains
+`upload`. Call `browser_set_input_files` with absolute paths to direct regular
+files. Symlinks, directories, missing paths, and more than 32 files are
+refused. Verify the page's uploaded-file state afterward; the tool response
+contains only a count and never echoes the local paths.
+
+## Download into an approved directory
+
+Choose a current ref that activates the download and call `browser_download`
+through an MCP host that presents its destructive-tool approval. Pass an
+existing canonical absolute `destination_root`. The tool temporarily scopes
+Chromium's browser-wide download behavior, correlates the exact frame and
+opaque download id, restores the default on every outcome, and returns only
+the final byte count and opaque id. It does not return a filename, URL, or
+local path. A direct raw call without host approval is refused.
+
## Navigate
```jsonc
diff --git a/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx
index 131525af32..74fa8ced16 100644
--- a/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx
+++ b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx
@@ -97,9 +97,11 @@ Entries in `refs` declare their supported action kinds:
}
```
-`browser_click` and `browser_type` reject a semantic ref when the requested
-action is absent from `actions`, returning `browser_action_unavailable` before
-delivery. Legacy `dom_refs_v1` refs retain their existing behavior.
+The closed action vocabulary is `click`, `type`, and `upload`.
+`browser_click`, `browser_type`, `browser_set_input_files`, and
+`browser_download` reject a semantic ref when the required action is absent,
+returning `browser_action_unavailable` before delivery. A file input declares
+`upload`, not `type`. Legacy `dom_refs_v1` refs retain their existing behavior.
## Content refs
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index cc7ce61231..726273d5ef 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposes
import { Callout } from 'fumadocs-ui/components/callout';
-`cua-driver` exposes 43 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
+`cua-driver` exposes 47 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`.
@@ -425,7 +425,7 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number
- `y` (number, optional): Window-local screenshot Y. See `x`.
```json
-{"direction":"down"}
+{"direction":"up"}
```
### `move_cursor`
@@ -493,7 +493,7 @@ Actions:
- `window_id` (integer, optional): Target window ID from list_windows.
```json
-{"action":"get_text"}
+{"action":"execute_javascript"}
```
## Recording tools
@@ -827,3 +827,77 @@ Type text into an exactly-bound tab via the Input domain. mode="insert_text" (de
```json
{"ref":"example","tab_id":"example","target_id":"example","text":"hello"}
```
+
+### `browser_dialog`
+
+Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.
+
+**Arguments:**
+
+- `action` (string, required)
+- `delivery_mode` (string, optional): Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only.
+- `dialog_id` (string, optional): Opaque current dialog generation returned by action=inspect.
+- `prompt_text` (string, optional): Sensitive response text, valid only when accepting a prompt dialog.
+- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.
+- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).
+- `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).
+
+```json
+{"action":"inspect","tab_id":"example","target_id":"example"}
+```
+
+### `browser_set_input_files`
+
+Assign one or more explicit absolute local files to an exact live &lt;input type=file&gt; ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.
+
+**Arguments:**
+
+- `files` (array of string, required)
+- `ref` (string, required): Page element ref in the p&lt;snapshot&gt;:&lt;index&gt; namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.
+- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.
+- `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped).
+- `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).
+
+```json
+{"files":["example"],"ref":"example","tab_id":"example","target_id":"example"}
+```
+
+### `browser_download`
+
+Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.
+
+**Arguments:**
+
+- `destination_root` (string, required): Absolute, existing, canonical directory approved to receive the download.
+- `ref` (string, required): Live page ref whose activation initiates the download.
+- `session` (string, required): Explicit caller session owning the browser capabilities.
+- `tab_id` (string, required): Opaque exact tab id from get_browser_state.
+- `target_id` (string, required): Opaque exact browser target id from get_browser_state.
+
+```json
+{"destination_root":"example","ref":"example","session":"example","tab_id":"example","target_id":"example"}
+```
+
+### `browser_pointer`
+
+Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.
+
+**Arguments:**
+
+- `action` (string, required)
+- `delta_x` (number, optional): Horizontal scroll delta in CSS pixels.
+- `delta_y` (number, optional): Vertical scroll delta in CSS pixels.
+- `destination_ref` (string, optional): Drag destination page ref in the exact same frame.
+- `input_route` (string, optional)
+- `ref` (string, optional): Origin page ref. Alternative to x/y.
+- `session` (string, required): Explicit caller session owning the browser capabilities.
+- `tab_id` (string, required): Opaque tab id minted by get_browser_state.
+- `target_id` (string, required): Opaque target id minted by get_browser_state.
+- `to_x` (number, optional): Drag destination viewport x in CSS pixels.
+- `to_y` (number, optional): Drag destination viewport y in CSS pixels.
+- `x` (number, optional): Origin viewport x in CSS pixels.
+- `y` (number, optional): Origin viewport y in CSS pixels.
+
+```json
+{"action":"hover","session":"example","tab_id":"example","target_id":"example"}
+```
diff --git a/docs/content/docs/reference/cua-driver/platform-support.mdx b/docs/content/docs/reference/cua-driver/platform-support.mdx
index c1e347f22d..24398dcbd7 100644
--- a/docs/content/docs/reference/cua-driver/platform-support.mdx
+++ b/docs/content/docs/reference/cua-driver/platform-support.mdx
@@ -38,9 +38,9 @@ harnesses; an unlisted trust class or host shape is not implied.
| Surface | Proven page routes | Explicit limit |
| --- | --- | --- |
-| Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse |
-| Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |
-| Standalone Chrome on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |
+| Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse |
+| Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch |
+| Standalone Chrome and Edge on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, foreground JavaScript-dialog resolution, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input and background JavaScript-dialog resolution return `browser_input_trust_unavailable` before dispatch |
| Electron on Windows, macOS, X11, and validated Sway | Typed mutation while one proven native window maps to one CDP page | A second page or native window invalidates the bounded exact route |
| Tauri, WKWebView, WebKitGTK, and common split-process WebView2 | Browser identity plus a side-effect-free structured refusal | No exact native-host-to-engine relationship is currently proven |
| Safari and Firefox | Native discovery and native AX/PX fallbacks | No typed browser mutation engine is currently advertised |
diff --git a/docs/content/docs/reference/cua-driver/telemetry.mdx b/docs/content/docs/reference/cua-driver/telemetry.mdx
index 4344e54bf6..506a0eee80 100644
--- a/docs/content/docs/reference/cua-driver/telemetry.mdx
+++ b/docs/content/docs/reference/cua-driver/telemetry.mdx
@@ -123,7 +123,7 @@ The start event is emitted for a successful `start_session` declaration or the f
The end event is emitted for `end_session` and idle eviction. It contains aggregate counters and booleans. When the platform has a cursor entry for that session, it also includes bounded cursor categories: enabled state, built-in/default/custom icon class, automatic/custom color source, label presence, motion customization, and an active-cursor count bucket. `cursor_outcome_observed=false` distinguishes sessions with no readable cursor entry from a disabled or default cursor. Cua Driver uses the caller session string only as a process-local map key. The string, a hash of it, and a replacement join token are absent from telemetry payloads.
-Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful `browser_navigate`, `browser_click`, and `browser_type` calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` is a read, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect.
+Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful browser navigation, click, type, pointer, file-assignment, download, or dialog-resolution calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` and `browser_dialog` inspection are reads for this classifier, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect.
The end event includes `used_browser` and a bounded `browser_refusal_count_bucket`. It does not include browser targets, pages, profiles, or per-site information.
@@ -141,6 +141,10 @@ Typed browser tools use these reviewed operation values:
| `browser_prepare` | `browser_prepare_isolated`, `browser_prepare_existing_profile`, or `other` |
| `browser_click` | `browser_click_trusted`, `browser_click_dom_event`, or `other` |
| `browser_type` | `browser_type_insert_text`, `browser_type_keystrokes`, or `other` |
+| `browser_dialog` | `browser_dialog_inspect`, `browser_dialog_accept`, `browser_dialog_dismiss`, or `other` |
+| `browser_set_input_files` | `browser_set_input_files` |
+| `browser_download` | `browser_download` |
+| `browser_pointer` | Action-and-route values such as `browser_pointer_scroll_trusted` or `browser_pointer_drag_dom_event`; invalid combinations become `other` |
| `browser_navigate` | `not_applicable`; the tool name already identifies the operation |
### Structured browser refusals
diff --git a/libs/cua-driver/docs/browser-tool-implementation-journal.md b/libs/cua-driver/docs/browser-tool-implementation-journal.md
index c3f68cd7a3..836a118187 100644
--- a/libs/cua-driver/docs/browser-tool-implementation-journal.md
+++ b/libs/cua-driver/docs/browser-tool-implementation-journal.md
@@ -6,9 +6,8 @@ URLs, profile paths, and raw DevTools endpoint identifiers.
## Definition of done
-- Add a capability-aware browser API with five typed tools:
- `get_browser_state`, `browser_prepare`, `browser_navigate`, `browser_click`,
- and `browser_type`.
+- Add a capability-aware browser API whose typed tools share one exact target,
+ tab, frame, and ref model.
- Keep `get_browser_state` read-only and make setup explicit.
- Bind native `(pid, window_id)` targets to browser tabs exactly or refuse all
mutation. Never promote a heuristic match into an action route.
@@ -719,3 +718,41 @@ link checks. A source-installed macOS product-path replay passed the real Chrome
semantic row with the TCC-authorized daemon, foreground sentinel, fixture
journal, exact DOM click, re-snapshot, and typed-value oracle. Windows and Linux
X11 exact-SHA workflow evidence is the remaining release-acceptance gate.
+
+## 2026-07-18: browser completeness and adversarial tab state
+
+The bind response no longer labels a representative DevTools target as the
+active native tab. A unique exact native-title match produces one `true` and
+the remaining tabs `false`; duplicate, empty, substring-only, or otherwise
+ambiguous titles produce `null` for every tab. The standalone harness adds a
+same-title adversarial row so future list-order regressions are externally
+visible.
+
+The typed surface now includes page-owned JavaScript dialog inspection and
+resolution, exact file-input assignment, extended pointer actions, and
+approval-scoped downloads. All mutations retain the existing exact binding,
+tab lock, reconnect generation, ref/frame identity, and fail-closed posture.
+Synthetic pointer actions are explicit `dom_event` requests; trusted pointer
+actions never fall back after a refusal. Downloads serialize Chromium's
+browser-wide setting, subscribe before triggering, correlate exact frame and
+opaque id events, restore defaults on every outcome, and remove only a proven
+direct partial file after failure.
+
+The real-browser harness adds separate rows for dialogs, upload,
+hover/right-click/double-click/scroll/drag, and downloads. A fixed loopback
+attachment keeps download evidence deterministic. Background rows use the
+foreground sentinel and fixture-owned state rather than treating a CDP reply
+as delivery. Linux Chromium cannot resolve a native JavaScript modal without
+changing foreground posture, so its matrix records a no-dispatch background
+refusal and a separate explicit foreground delivery row. Windows and macOS
+retain strict background dialog resolution. Trajectory recording redacts
+prompt text, upload paths, download directories, and internal approval
+markers; product telemetry retains only fixed operation and refusal
+categories.
+
+The dialog design adapts the exact-target, no-replay security model introduced
+in PR #2166. Its contributor remains credited through a parsed
+`Co-authored-by` trailer in the landing history and `Salvaged from #2166` in
+the pull request body even though this implementation supersedes that earlier
+code shape. Issue authors remain reporter-attributed when the landing PR
+closes their linked issues.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
index 73646a4848..21592797a1 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md
@@ -20,7 +20,8 @@ list_windows or launch_app
get_browser_state(pid, window_id, session) # bind
get_browser_state(target_id, tab_id, session,
snapshot_format=semantic_v2) # snapshot
-browser_navigate / browser_click / browser_type
+browser_navigate / browser_click / browser_type / browser_pointer
+browser_dialog / browser_set_input_files / browser_download
get_browser_state(target_id, tab_id, session,
snapshot_format=semantic_v2) # verify and refresh refs
end_session
@@ -119,7 +120,10 @@ Never:
## 3. Snapshot the selected tab
-Choose a returned `tab_id`, then request the page snapshot:
+Choose a returned `tab_id`, then request the page snapshot. `active` is
+tri-state: `true` is a uniquely proven selected tab, `false` is a proven
+unselected tab, and `null` means native evidence cannot distinguish the
+selection. Never guess from list order when all tabs are `null`.
```bash
cua-driver get_browser_state \
@@ -240,14 +244,61 @@ ownership, and reports requested versus delivered characters. Snapshot again
to verify application state rather than treating transport completion as the
task result.
+### Extended pointer actions
+
+Use `browser_pointer` for `hover`, `right_click`, `double_click`, `scroll`, and
+`drag`. It uses the same `trusted` versus explicit `dom_event` distinction as
+`browser_click`. The synthetic route requires a current ref; drag also requires
+`destination_ref` in the same proven frame. Coordinate origins and destinations
+are available only where the trusted route can preserve the requested posture.
+
+```bash
+cua-driver browser_pointer \
+ '{"target_id":"<target>","tab_id":"<tab>","ref":"p5:2",
+ "action":"scroll","input_route":"dom_event","delta_y":240,
+ "session":"browser-run-1"}'
+```
+
+### JavaScript dialogs
+
+`browser_dialog` handles only page-owned `alert`, `confirm`, `prompt`, and
+`beforeunload` dialogs. First inspect the exact tab, then accept or dismiss the
+returned opaque `dialog_id`. A prompt response is allowed only with
+`action:"accept"` on a current prompt. Browser permission UI and native dialogs
+remain outside this tool. Creating Chromium's native modal can activate the
+browser; after the caller restores occlusion, inspecting and resolving the
+exact page-owned dialog do not require another activation on Windows and
+macOS. Resolution defaults to `delivery_mode:"background"`. Linux Chromium
+cannot resolve its native modal while preserving background posture, so the
+driver refuses that mode before dispatch; retry explicitly with
+`delivery_mode:"foreground"` when foreground activation is acceptable.
+
+### File inputs
+
+Use a current semantic ref whose `actions` contains `upload`, then call
+`browser_set_input_files` with one to 32 absolute regular-file paths. The tool
+rejects symlinks and directories, bypasses the native file picker, and returns
+only the assigned file count. Paths are redacted from trajectory arguments.
+
+### Downloads
+
+`browser_download` activates one exact ref under a destructive MCP-host
+approval and saves the result under an existing canonical absolute
+`destination_root`. It correlates browser download events to the exact frame,
+serializes Chromium's browser-wide download setting, restores that setting on
+every outcome, and returns only an opaque download id and byte count. It never
+returns the source URL, filename, or destination path. Direct raw calls without
+the host approval proof are refused.
+
## Browser chrome and native fallbacks
The browser tools operate on page content, not the surrounding native UI. Use
the normal native loop for:
- tabs, address bar, menus, bookmarks, and extension UI;
-- permission prompts and remote-debugging consent UI;
-- downloads, save dialogs, authentication sheets, and file pickers;
+- permission prompts, remote-debugging consent UI, and authentication sheets;
+- native save dialogs and file pickers that are not represented by an exact
+ page ref;
- WebView2, WKWebView, WebKitGTK, Tauri, or Electron surfaces that cannot be
exactly correlated to a page target;
- Safari and Firefox, whose typed mutation engines are not yet supported.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs
index a445ab9c62..252818b903 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs
@@ -86,6 +86,35 @@ fn title_matches(native_title: &str, candidate_title: &str) -> bool {
!candidate_title.is_empty() && native_title.contains(candidate_title)
}
+/// Prove the selected page in one already-correlated browser window without
+/// activating a target. Chromium exposes the native window title but no
+/// browser-level selected-tab bit. A unique title match is therefore proof;
+/// duplicate, empty, absent, or substring-colliding titles remain unknown.
+/// Embedded single-page endpoints are selected by cardinality.
+pub fn selected_tab_target_id<'a>(
+ native_title: &str,
+ candidates: &'a [CdpWindowCandidate],
+ cdp_window_id: Option<i64>,
+) -> Option<&'a str> {
+ let tabs = candidates
+ .iter()
+ .filter(|candidate| match cdp_window_id {
+ Some(window_id) => candidate.cdp_window_id == Some(window_id),
+ None => candidate.cdp_window_id.is_none(),
+ })
+ .collect::<Vec<_>>();
+
+ if cdp_window_id.is_none() {
+ return (tabs.len() == 1).then(|| tabs[0].cdp_target_id.as_str());
+ }
+
+ let title_hits = tabs
+ .into_iter()
+ .filter(|candidate| title_matches(native_title, &candidate.title))
+ .collect::<Vec<_>>();
+ (title_hits.len() == 1).then(|| title_hits[0].cdp_target_id.as_str())
+}
+
/// Reduce page targets to one representative per proven CDP window.
/// The representative prefers a tab title displayed by the native
/// window, but its target id is only a handle for the selected window;
@@ -328,6 +357,45 @@ mod tests {
));
}
+ #[test]
+ fn selected_tab_requires_one_unique_native_title_match() {
+ let tabs = [cand("first", "Checkout", B), cand("second", "Cart", B)];
+ assert_eq!(
+ selected_tab_target_id("Cart - Chrome", &tabs, Some(1)),
+ Some("second")
+ );
+
+ let duplicate_titles = [cand("first", "Checkout", B), cand("second", "Checkout", B)];
+ assert_eq!(
+ selected_tab_target_id("Checkout - Chrome", &duplicate_titles, Some(1)),
+ None
+ );
+
+ let substring_collision = [cand("first", "Mail", B), cand("second", "Mail Inbox", B)];
+ assert_eq!(
+ selected_tab_target_id("Mail Inbox - Chrome", &substring_collision, Some(1)),
+ None
+ );
+
+ let empty_titles = [cand("first", "", B), cand("second", "", B)];
+ assert_eq!(
+ selected_tab_target_id("Chrome", &empty_titles, Some(1)),
+ None
+ );
+ }
+
+ #[test]
+ fn embedded_single_page_is_selected_by_cardinality_only() {
+ let one = [embedded_cand("only", "Fixture")];
+ assert_eq!(selected_tab_target_id("Fixture", &one, None), Some("only"));
+
+ let two = [
+ embedded_cand("first", "Fixture"),
+ embedded_cand("second", "Fixture"),
+ ];
+ assert_eq!(selected_tab_target_id("Fixture", &two, None), None);
+ }
+
#[test]
fn unique_title_disambiguates_distinct_maximized_windows_with_shared_bounds() {
let n = native("Docs - Chrome", B);
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs
index 122438242c..793a173feb 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs
@@ -61,6 +61,15 @@ pub struct CdpEvent {
pub params: Value,
}
+/// Bounded state for one page-owned JavaScript dialog. Message text and URLs
+/// are deliberately not retained.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CdpDialogState {
+ pub generation: u64,
+ pub kind: String,
+ pub session_id: String,
+}
+
/// What the reader routed back for one in-flight call.
enum CallOutcome {
Result(Value),
@@ -71,6 +80,9 @@ enum CallOutcome {
struct Demux {
pending: StdMutex<HashMap<u64, oneshot::Sender<CallOutcome>>>,
subscribers: StdMutex<Vec<mpsc::UnboundedSender<CdpEvent>>>,
+ session_targets: StdMutex<HashMap<String, String>>,
+ dialogs: StdMutex<HashMap<String, CdpDialogState>>,
+ next_dialog_generation: AtomicU64,
closed: AtomicBool,
}
@@ -81,6 +93,8 @@ impl Demux {
// error, which `call` reports as a closed socket.
self.pending.lock().unwrap().clear();
self.subscribers.lock().unwrap().clear();
+ self.session_targets.lock().unwrap().clear();
+ self.dialogs.lock().unwrap().clear();
}
}
@@ -126,6 +140,43 @@ async fn read_loop(mut read: SplitStream<WsStream>, demux: Arc<Demux>) {
.map(str::to_owned),
params: v.get("params").cloned().unwrap_or(Value::Null),
};
+ if let Some(session_id) = event.session_id.as_deref() {
+ let target_id = demux
+ .session_targets
+ .lock()
+ .unwrap()
+ .get(session_id)
+ .cloned();
+ match event.method.as_str() {
+ "Page.javascriptDialogOpening" => {
+ let kind = match event.params.get("type").and_then(Value::as_str) {
+ Some("alert") => "alert",
+ Some("confirm") => "confirm",
+ Some("prompt") => "prompt",
+ Some("beforeunload") => "beforeunload",
+ _ => "other",
+ };
+ let generation =
+ demux.next_dialog_generation.fetch_add(1, Ordering::Relaxed);
+ if let Some(target_id) = target_id {
+ demux.dialogs.lock().unwrap().insert(
+ target_id,
+ CdpDialogState {
+ generation,
+ kind: kind.to_owned(),
+ session_id: session_id.to_owned(),
+ },
+ );
+ }
+ }
+ "Page.javascriptDialogClosed" => {
+ if let Some(target_id) = target_id {
+ demux.dialogs.lock().unwrap().remove(&target_id);
+ }
+ }
+ _ => {}
+ }
+ }
demux
.subscribers
.lock()
@@ -162,6 +213,9 @@ impl CdpConnection {
let demux = Arc::new(Demux {
pending: StdMutex::new(HashMap::new()),
subscribers: StdMutex::new(Vec::new()),
+ session_targets: StdMutex::new(HashMap::new()),
+ dialogs: StdMutex::new(HashMap::new()),
+ next_dialog_generation: AtomicU64::new(1),
closed: AtomicBool::new(false),
});
let reader = tokio::spawn(read_loop(read, demux.clone()));
@@ -190,6 +244,42 @@ impl CdpConnection {
rx
}
+ /// Associate the one Page-enabled dialog session with its exact target.
+ /// Later calls may use fresh attachment sessions, but dialog events keep
+ /// arriving on this bounded, persistent event session.
+ pub fn register_dialog_session(&self, session_id: &str, target_id: &str) {
+ let mut sessions = self.demux.session_targets.lock().unwrap();
+ sessions.retain(|_, mapped_target| mapped_target != target_id);
+ sessions.insert(session_id.to_owned(), target_id.to_owned());
+ }
+
+ pub fn unregister_dialog_session(&self, session_id: &str, target_id: &str) {
+ let mut sessions = self.demux.session_targets.lock().unwrap();
+ if sessions.get(session_id).map(String::as_str) == Some(target_id) {
+ sessions.remove(session_id);
+ }
+ }
+
+ pub fn has_dialog_session(&self, target_id: &str) -> bool {
+ self.demux
+ .session_targets
+ .lock()
+ .unwrap()
+ .values()
+ .any(|mapped_target| mapped_target == target_id)
+ }
+
+ pub fn dialog_state(&self, target_id: &str) -> Option<CdpDialogState> {
+ self.demux.dialogs.lock().unwrap().get(target_id).cloned()
+ }
+
+ pub fn clear_dialog_state(&self, target_id: &str, generation: u64) {
+ let mut dialogs = self.demux.dialogs.lock().unwrap();
+ if dialogs.get(target_id).map(|state| state.generation) == Some(generation) {
+ dialogs.remove(target_id);
+ }
+ }
+
/// Issue one CDP command and await its `id`-matched reply.
/// `session_id` targets an attached (flattened) target session.
/// Returns the `result` object, or an error carrying the CDP
@@ -589,6 +679,41 @@ mod tests {
assert!(events.try_recv().is_err(), "no phantom events");
}
+ #[tokio::test]
+ async fn javascript_dialog_journal_is_bounded_and_generation_checked() {
+ let server = MockCdpServer::start(StdArc::new(|call| match call.method.as_str() {
+ "Dialog.open" => MockReply::ok(json!({})).with_events(vec![MockEvent {
+ method: "Page.javascriptDialogOpening".into(),
+ session_id: call.session_id.clone(),
+ params: json!({
+ "type": "prompt",
+ "message": "private dialog text",
+ "url": "https://private.example"
+ }),
+ }]),
+ "Dialog.close" => MockReply::ok(json!({})).with_events(vec![MockEvent {
+ method: "Page.javascriptDialogClosed".into(),
+ session_id: call.session_id.clone(),
+ params: json!({}),
+ }]),
+ _ => MockReply::ok(json!({})),
+ }))
+ .await;
+ let conn = CdpConnection::connect(&server.ws_url()).await.unwrap();
+ conn.register_dialog_session("tab-session-a", "page-target");
+ conn.call(Some("tab-session-a"), "Dialog.open", json!({}))
+ .await
+ .unwrap();
+ let dialog = conn.dialog_state("page-target").expect("dialog journaled");
+ assert_eq!(dialog.kind, "prompt");
+ assert_eq!(dialog.session_id, "tab-session-a");
+ assert!(!format!("{dialog:?}").contains("private"));
+ conn.call(Some("tab-session-a"), "Dialog.close", json!({}))
+ .await
+ .unwrap();
+ assert!(conn.dialog_state("page-target").is_none());
+ }
+
#[tokio::test]
async fn out_of_order_replies_route_to_the_right_callers() {
// Raw server: read two requests, reply to the SECOND first.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rs
new file mode 100644
index 0000000000..46871f5218
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/download.rs
@@ -0,0 +1,593 @@
+//! Approval-gated downloads for exact browser tab and element capabilities.
+//!
+//! Chromium's download behavior is browser-wide, so this tool enables it only
+//! for the bounded interval around one exact ref mutation and restores the
+//! default before returning. Output deliberately omits URLs, filenames, and
+//! local paths.
+
+use std::path::{Component, Path, PathBuf};
+use std::sync::{Arc, OnceLock};
+use std::time::Duration;
+
+use async_trait::async_trait;
+use serde_json::{json, Value};
+
+use crate::protocol::ToolResult;
+use crate::tool::{Tool, ToolDef};
+use crate::tool_args::ArgsExt;
+
+use super::cdp_ws::CdpConnection;
+use super::engine::BrowserEngine;
+use super::refusal::{BrowserRefusal, BrowserRefusalCode};
+use super::store::BrowserActionKind;
+
+/// Injected by an MCP host only after its destructive-tool approval flow.
+/// It is intentionally absent from the public input schema.
+pub const MCP_HOST_DOWNLOAD_APPROVAL_ARG: &str = "_cua_browser_download_mcp_host_approved";
+
+const DOWNLOAD_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn browser_download_gate() -> &'static tokio::sync::Mutex<()> {
+ static GATE: OnceLock<tokio::sync::Mutex<()>> = OnceLock::new();
+ GATE.get_or_init(|| tokio::sync::Mutex::new(()))
+}
+
+pub struct BrowserDownloadTool {
+ def: ToolDef,
+ engine: Arc<BrowserEngine>,
+}
+
+impl BrowserDownloadTool {
+ pub fn new(engine: Arc<BrowserEngine>) -> Self {
+ Self {
+ def: ToolDef {
+ name: "browser_download".into(),
+ description: "Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "session": {
+ "type": "string",
+ "description": "Explicit caller session owning the browser capabilities."
+ },
+ "target_id": {
+ "type": "string",
+ "description": "Opaque exact browser target id from get_browser_state."
+ },
+ "tab_id": {
+ "type": "string",
+ "description": "Opaque exact tab id from get_browser_state."
+ },
+ "ref": {
+ "type": "string",
+ "description": "Live page ref whose activation initiates the download."
+ },
+ "destination_root": {
+ "type": "string",
+ "description": "Absolute, existing, canonical directory approved to receive the download."
+ }
+ },
+ "required": ["session", "target_id", "tab_id", "ref", "destination_root"],
+ "additionalProperties": true
+ }),
+ read_only: false,
+ destructive: true,
+ idempotent: false,
+ open_world: true,
+ },
+ engine,
+ }
+ }
+}
+
+fn consent_required() -> ToolResult {
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserConsentRequired,
+ "browser_download requires approval through the MCP host's destructive-tool confirmation flow",
+ )
+ .to_tool_result()
+}
+
+fn explicit_session(args: &Value) -> Result<String, ToolResult> {
+ let session = args.require_str("session")?;
+ if session.trim().is_empty() || session == "default" {
+ return Err(ToolResult::error(
+ "browser_download requires an explicit non-default session",
+ ));
+ }
+ Ok(session)
+}
+
+fn validate_destination_root(raw: &str) -> Result<PathBuf, ToolResult> {
+ let path = Path::new(raw);
+ if !path.is_absolute() {
+ return Err(ToolResult::error("destination_root must be absolute"));
+ }
+
+ let metadata = std::fs::symlink_metadata(path)
+ .map_err(|_| ToolResult::error("destination_root must be an existing directory"))?;
+ if metadata.file_type().is_symlink() {
+ return Err(ToolResult::error(
+ "destination_root must name the directory directly, not a symlink",
+ ));
+ }
+ if !metadata.is_dir() {
+ return Err(ToolResult::error(
+ "destination_root must be an existing directory",
+ ));
+ }
+
+ let canonical = std::fs::canonicalize(path)
+ .map_err(|_| ToolResult::error("destination_root could not be canonicalized"))?;
+ if canonical != path {
+ return Err(ToolResult::error(
+ "destination_root must already be in canonical form",
+ ));
+ }
+ Ok(canonical)
+}
+
+fn download_path_for_cdp(path: &Path) -> String {
+ let raw = path.to_string_lossy();
+ #[cfg(target_os = "windows")]
+ {
+ // `canonicalize` deliberately gives containment checks a verbatim
+ // Windows path. Chromium's download API does not accept that transport
+ // spelling, so remove only the namespace prefix after validation.
+ if let Some(unc) = raw.strip_prefix(r"\\?\UNC\") {
+ return format!(r"\\{unc}");
+ }
+ if let Some(local) = raw.strip_prefix(r"\\?\") {
+ return local.to_owned();
+ }
+ }
+ raw.into_owned()
+}
+
+fn main_frame_id(frame_tree: &Value) -> Option<&str> {
+ frame_tree
+ .get("frameTree")?
+ .get("frame")?
+ .get("id")?
+ .as_str()
+}
+
+fn is_safe_guid(guid: &str) -> bool {
+ let mut components = Path::new(guid).components();
+ matches!(components.next(), Some(Component::Normal(_))) && components.next().is_none()
+}
+
+fn completed_download_size(root: &Path, guid: &str) -> Result<u64, BrowserRefusal> {
+ if !is_safe_guid(guid) {
+ return Err(BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the browser returned an invalid opaque download id",
+ ));
+ }
+ let file = root.join(guid);
+ let metadata = std::fs::symlink_metadata(&file).map_err(|_| {
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the completed download could not be proven inside the approved directory",
+ )
+ })?;
+ if metadata.file_type().is_symlink() || !metadata.is_file() {
+ return Err(BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the completed download is not a direct regular file",
+ ));
+ }
+ let canonical = std::fs::canonicalize(&file).map_err(|_| {
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the completed download could not be canonicalized",
+ )
+ })?;
+ if canonical.parent() != Some(root) {
+ return Err(BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the completed download escaped the approved directory",
+ ));
+ }
+ Ok(metadata.len())
+}
+
+async fn reset_download_behavior(conn: &CdpConnection) {
+ let _ = conn
+ .call(
+ None,
+ "Browser.setDownloadBehavior",
+ json!({ "behavior": "default", "eventsEnabled": false }),
+ )
+ .await;
+}
+
+enum DownloadEventOutcome {
+ Completed { guid: String },
+ Canceled { guid: String },
+ EventStreamClosed,
+}
+
+async fn await_download(
+ events: &mut tokio::sync::mpsc::UnboundedReceiver<super::cdp_ws::CdpEvent>,
+ expected_frame_id: &str,
+ observed_guid: &mut Option<String>,
+) -> DownloadEventOutcome {
+ while let Some(event) = events.recv().await {
+ match event.method.as_str() {
+ "Browser.downloadWillBegin" if observed_guid.is_none() => {
+ if event.params.get("frameId").and_then(Value::as_str) != Some(expected_frame_id) {
+ continue;
+ }
+ *observed_guid = event
+ .params
+ .get("guid")
+ .and_then(Value::as_str)
+ .map(str::to_owned);
+ }
+ "Browser.downloadProgress" => {
+ let Some(guid) = observed_guid.as_deref() else {
+ continue;
+ };
+ if event.params.get("guid").and_then(Value::as_str) != Some(guid) {
+ continue;
+ }
+ match event.params.get("state").and_then(Value::as_str) {
+ Some("completed") => {
+ return DownloadEventOutcome::Completed {
+ guid: guid.to_owned(),
+ }
+ }
+ Some("canceled") => {
+ return DownloadEventOutcome::Canceled {
+ guid: guid.to_owned(),
+ }
+ }
+ _ => {}
+ }
+ }
+ _ => {}
+ }
+ }
+ DownloadEventOutcome::EventStreamClosed
+}
+
+fn remove_proven_partial(root: &Path, guid: Option<&str>) {
+ let Some(guid) = guid.filter(|guid| is_safe_guid(guid)) else {
+ return;
+ };
+ let path = root.join(guid);
+ let Ok(metadata) = std::fs::symlink_metadata(&path) else {
+ return;
+ };
+ if metadata.file_type().is_symlink() || !metadata.is_file() {
+ return;
+ }
+ let Ok(canonical) = std::fs::canonicalize(&path) else {
+ return;
+ };
+ if canonical.parent() == Some(root) {
+ let _ = std::fs::remove_file(canonical);
+ }
+}
+
+#[async_trait]
+impl Tool for BrowserDownloadTool {
+ fn def(&self) -> &ToolDef {
+ &self.def
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let (session, target_id, tab_id, ext_ref, destination_raw) = match (
+ explicit_session(&args),
+ args.require_str("target_id"),
+ args.require_str("tab_id"),
+ args.require_str("ref"),
+ args.require_str("destination_root"),
+ ) {
+ (Ok(session), Ok(target), Ok(tab), Ok(ext_ref), Ok(destination)) => {
+ (session, target, tab, ext_ref, destination)
+ }
+ (Err(error), _, _, _, _)
+ | (_, Err(error), _, _, _)
+ | (_, _, Err(error), _, _)
+ | (_, _, _, Err(error), _)
+ | (_, _, _, _, Err(error)) => return error,
+ };
+
+ if args
+ .get(MCP_HOST_DOWNLOAD_APPROVAL_ARG)
+ .and_then(Value::as_bool)
+ != Some(true)
+ {
+ return consent_required();
+ }
+ let destination_root = match validate_destination_root(&destination_raw) {
+ Ok(root) => root,
+ Err(error) => return error,
+ };
+
+ let _mutation = match self
+ .engine
+ .lock_mutation(&session, &target_id, &tab_id)
+ .await
+ {
+ Ok(guard) => guard,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ // Download routing is browser-wide rather than tab-scoped. Serialize
+ // these bounded mutations process-wide so two exact tabs cannot race
+ // to replace each other's approved destination.
+ let _download_mutation = browser_download_gate().lock().await;
+ let validated = match self
+ .engine
+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))
+ .await
+ {
+ Ok(validated) => validated,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let entry = match self
+ .engine
+ .store
+ .resolve_ref(&session, &target_id, &tab_id, &ext_ref)
+ {
+ Ok(entry) => entry,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Click) {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the semantic ref does not declare an activation action",
+ )
+ .to_tool_result();
+ }
+ let ref_session = match self
+ .engine
+ .frame_session_for_mutation(&session, &target_id, &tab_id, &validated, &entry.frame)
+ .await
+ {
+ Ok(session) => session,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+
+ let frame_tree = match validated
+ .conn
+ .call(Some(&validated.cdp_session), "Page.getFrameTree", json!({}))
+ .await
+ {
+ Ok(tree) => tree,
+ Err(_) => {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRouteUnavailable,
+ "the exact tab's main frame could not be proven",
+ )
+ .to_tool_result()
+ }
+ };
+ let Some(main_frame_id) = main_frame_id(&frame_tree).map(str::to_owned) else {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRouteUnavailable,
+ "Page.getFrameTree did not identify the exact tab's main frame",
+ )
+ .to_tool_result();
+ };
+ let expected_frame_id = entry
+ .frame
+ .identity
+ .as_ref()
+ .map(|identity| identity.frame_id.clone())
+ .unwrap_or(main_frame_id);
+
+ let resolved = match validated
+ .conn
+ .call(
+ Some(&ref_session),
+ "DOM.resolveNode",
+ json!({ "backendNodeId": entry.backend_node_id }),
+ )
+ .await
+ {
+ Ok(resolved) => resolved,
+ Err(_) => {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRefStale,
+ "the download ref no longer resolves in the live page",
+ )
+ .to_tool_result()
+ }
+ };
+ let Some(object_id) = resolved
+ .get("object")
+ .and_then(|object| object.get("objectId"))
+ .and_then(Value::as_str)
+ .map(str::to_owned)
+ else {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRefStale,
+ "the download ref has no live page object",
+ )
+ .to_tool_result();
+ };
+
+ // Browser events are endpoint-wide. Subscribe before enabling and
+ // triggering so no matching event can be lost between command replies.
+ let mut events = validated.conn.subscribe();
+ if validated
+ .conn
+ .call(
+ None,
+ "Browser.setDownloadBehavior",
+ json!({
+ "behavior": "allowAndName",
+ "downloadPath": download_path_for_cdp(&destination_root),
+ "eventsEnabled": true
+ }),
+ )
+ .await
+ .is_err()
+ {
+ reset_download_behavior(&validated.conn).await;
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the browser could not configure the approved download",
+ )
+ .to_tool_result();
+ }
+
+ let trigger = validated
+ .conn
+ .call(
+ Some(&ref_session),
+ "Runtime.callFunctionOn",
+ json!({
+ "objectId": object_id,
+ "functionDeclaration": "function() { this.click(); }",
+ "userGesture": true,
+ "awaitPromise": false
+ }),
+ )
+ .await;
+ let trigger_failed = trigger
+ .as_ref()
+ .map_or(true, |result| result.get("exceptionDetails").is_some());
+ if trigger_failed {
+ reset_download_behavior(&validated.conn).await;
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRefStale,
+ "the exact download ref could not be activated",
+ )
+ .to_tool_result();
+ }
+
+ let mut observed_guid = None;
+ let outcome = tokio::time::timeout(
+ DOWNLOAD_TIMEOUT,
+ await_download(&mut events, &expected_frame_id, &mut observed_guid),
+ )
+ .await;
+ reset_download_behavior(&validated.conn).await;
+
+ match outcome {
+ Ok(DownloadEventOutcome::Completed { guid }) => {
+ match completed_download_size(&destination_root, &guid) {
+ Ok(bytes) => ToolResult::text("browser download completed").with_structured(
+ json!({ "status": "completed", "download_id": guid, "bytes": bytes }),
+ ),
+ Err(refusal) => refusal.to_tool_result(),
+ }
+ }
+ Ok(DownloadEventOutcome::Canceled { guid }) => {
+ remove_proven_partial(&destination_root, Some(&guid));
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the approved browser download was canceled",
+ )
+ .to_tool_result()
+ }
+ Ok(DownloadEventOutcome::EventStreamClosed) => {
+ remove_proven_partial(&destination_root, observed_guid.as_deref());
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the browser connection closed before the approved download completed",
+ )
+ .to_tool_result()
+ }
+ Err(_) => {
+ remove_proven_partial(&destination_root, observed_guid.as_deref());
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the approved browser download did not complete within 30 seconds",
+ )
+ .to_tool_result()
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn destination_must_be_absolute_existing_canonical_directory() {
+ assert!(validate_destination_root("relative").is_err());
+ let parent = tempfile::tempdir().unwrap();
+ let missing = parent.path().join("does-not-exist");
+ assert!(validate_destination_root(missing.to_str().unwrap()).is_err());
+
+ let root = tempfile::tempdir().unwrap();
+ let canonical = std::fs::canonicalize(root.path()).unwrap();
+ assert_eq!(
+ validate_destination_root(canonical.to_str().unwrap()).unwrap(),
+ canonical
+ );
+ }
+
+ #[test]
+ fn destination_rejects_regular_files() {
+ let root = tempfile::tempdir().unwrap();
+ let file = root.path().join("not-a-directory");
+ std::fs::write(&file, b"x").unwrap();
+ assert!(validate_destination_root(file.to_str().unwrap()).is_err());
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn destination_rejects_symlink_root() {
+ use std::os::unix::fs::symlink;
+
+ let parent = tempfile::tempdir().unwrap();
+ let real = parent.path().join("real");
+ let link = parent.path().join("link");
+ std::fs::create_dir(&real).unwrap();
+ symlink(&real, &link).unwrap();
+ assert!(validate_destination_root(link.to_str().unwrap()).is_err());
+ }
+
+ #[test]
+ fn completion_proof_accepts_only_direct_regular_guid_file() {
+ let root = tempfile::tempdir().unwrap();
+ let canonical = std::fs::canonicalize(root.path()).unwrap();
+ std::fs::write(canonical.join("opaque-guid"), b"abc").unwrap();
+ assert_eq!(
+ completed_download_size(&canonical, "opaque-guid").unwrap(),
+ 3
+ );
+ assert!(completed_download_size(&canonical, "../escape").is_err());
+ assert!(completed_download_size(&canonical, "missing").is_err());
+ }
+
+ #[test]
+ fn partial_cleanup_is_confined_to_a_direct_guid_file() {
+ let root = tempfile::tempdir().unwrap();
+ let canonical = std::fs::canonicalize(root.path()).unwrap();
+ let partial = canonical.join("opaque-guid");
+ std::fs::write(&partial, b"partial").unwrap();
+
+ remove_proven_partial(&canonical, Some("../escape"));
+ assert!(partial.exists());
+
+ remove_proven_partial(&canonical, Some("opaque-guid"));
+ assert!(!partial.exists());
+ }
+
+ #[test]
+ fn cdp_download_path_preserves_ordinary_paths() {
+ let path = Path::new("/tmp/approved-downloads");
+ assert_eq!(download_path_for_cdp(path), path.to_string_lossy());
+ }
+
+ #[cfg(target_os = "windows")]
+ #[test]
+ fn cdp_download_path_removes_only_windows_verbatim_prefixes() {
+ assert_eq!(
+ download_path_for_cdp(Path::new(r"\\?\C:\approved\downloads")),
+ r"C:\approved\downloads"
+ );
+ assert_eq!(
+ download_path_for_cdp(Path::new(r"\\?\UNC\server\share\downloads")),
+ r"\\server\share\downloads"
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
index bad656150a..4cb7ac2ce5 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
@@ -31,7 +31,10 @@ use uuid::Uuid;
use crate::session::register_session_end_hook;
-use super::binding::{cardinality_exact_candidate, correlate, BindingOutcome, CdpWindowCandidate};
+use super::binding::{
+ cardinality_exact_candidate, correlate, selected_tab_target_id, BindingOutcome,
+ CdpWindowCandidate,
+};
use super::cdp_ws::{CdpConnection, CdpPool};
use super::grant::{ExistingProfileGrant, ExistingProfileGrants, GrantLookup};
use super::mutation::{MutationGates, MutationKey};
@@ -952,7 +955,11 @@ impl BrowserEngine {
}
};
- // Tabs = page targets living in the bound CDP window.
+ // Tabs = page targets living in the bound CDP window. Selection is a
+ // separate proof from native-window correlation: a representative CDP
+ // target is only a window handle and must never be reported as active.
+ let selected_cdp_target_id =
+ selected_tab_target_id(&native.title, &candidates, candidate.cdp_window_id);
let mut tabs = HashMap::new();
for c in candidates.iter().filter(|c| match candidate.cdp_window_id {
Some(window_id) => c.cdp_window_id == Some(window_id),
@@ -966,6 +973,7 @@ impl BrowserEngine {
cdp_target_id: c.cdp_target_id.clone(),
title: c.title.clone(),
url: c.url.clone(),
+ active: selected_cdp_target_id.map(|selected| selected == c.cdp_target_id),
generation: grant.as_ref().map_or(0, |grant| grant.generation),
snapshots: HashMap::new(),
},
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs
index ea899520c4..cbd3ccb64d 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs
@@ -1,8 +1,7 @@
//! Platform-agnostic browser-tool v1 core.
//!
-//! Five typed tools — `get_browser_state` (strictly read-only),
-//! `browser_prepare`, `browser_navigate`, `browser_click`,
-//! `browser_type` — over an exact-or-refused binding model:
+//! Typed browser inspection, preparation, navigation, input, dialog, upload,
+//! and download tools over an exact-or-refused binding model:
//!
//! - The native entrypoint is `pid + window_id`. Browser target ids,
//! tab ids, and page refs (`p<snapshot>:<index>`) are opaque,
@@ -39,12 +38,14 @@
pub mod approval;
pub mod binding;
pub mod cdp_ws;
+pub mod download;
pub mod engine;
mod grant;
#[cfg(test)]
pub(crate) mod mock_cdp;
mod mutation;
pub mod platform;
+pub mod pointer;
mod prepare;
mod reconnect;
pub mod refusal;
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rs
new file mode 100644
index 0000000000..d8e08507ad
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/pointer.rs
@@ -0,0 +1,871 @@
+//! Extended browser pointer actions.
+//!
+//! This module deliberately shares the exact-or-refused posture of the
+//! original browser click tool. Every mutation is serialized by the real CDP
+//! tab, then revalidates the native window, endpoint, tab, and (for refs) frame
+//! identity before dispatch. It never activates a target or brings a page to
+//! the foreground.
+
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use serde_json::{json, Value};
+
+use crate::protocol::ToolResult;
+use crate::tool::{Tool, ToolDef};
+use crate::tool_args::ArgsExt;
+
+use super::cdp_ws::CdpConnection;
+use super::engine::{BrowserEngine, ValidatedTab};
+use super::refusal::{BrowserRefusal, BrowserRefusalCode};
+use super::store::{FrameKind, FrameRef};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum PointerAction {
+ Hover,
+ RightClick,
+ DoubleClick,
+ Scroll,
+ Drag,
+}
+
+impl PointerAction {
+ fn parse(value: &str) -> Result<Self, String> {
+ match value {
+ "hover" => Ok(Self::Hover),
+ "right_click" => Ok(Self::RightClick),
+ "double_click" => Ok(Self::DoubleClick),
+ "scroll" => Ok(Self::Scroll),
+ "drag" => Ok(Self::Drag),
+ _ => Err(format!(
+ "action must be hover, right_click, double_click, scroll, or drag, got {value:?}"
+ )),
+ }
+ }
+
+ fn as_str(self) -> &'static str {
+ match self {
+ Self::Hover => "hover",
+ Self::RightClick => "right_click",
+ Self::DoubleClick => "double_click",
+ Self::Scroll => "scroll",
+ Self::Drag => "drag",
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum InputRoute {
+ Trusted,
+ DomEvent,
+}
+
+impl InputRoute {
+ fn parse(value: &str) -> Result<Self, String> {
+ match value {
+ "trusted" => Ok(Self::Trusted),
+ "dom_event" => Ok(Self::DomEvent),
+ _ => Err(format!(
+ "input_route must be \"trusted\" or \"dom_event\", got {value:?}"
+ )),
+ }
+ }
+
+ fn as_str(self) -> &'static str {
+ match self {
+ Self::Trusted => "trusted",
+ Self::DomEvent => "dom_event",
+ }
+ }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+enum Location {
+ Ref(String),
+ Coordinates(f64, f64),
+}
+
+#[derive(Debug, Clone, PartialEq)]
+struct PointerRequest {
+ action: PointerAction,
+ route: InputRoute,
+ origin: Location,
+ destination: Option<Location>,
+ delta_x: f64,
+ delta_y: f64,
+}
+
+fn explicit_session(args: &Value) -> Result<String, ToolResult> {
+ let session = args
+ .opt_str("session")
+ .or_else(|| args.opt_str("_session_id"))
+ .unwrap_or_else(|| "default".into());
+ if session.is_empty() || session == "default" {
+ return Err(ToolResult::error(
+ "Browser targets and page refs are session-scoped capabilities - declare an explicit session (start_session) and pass its id on this call.",
+ ));
+ }
+ Ok(session)
+}
+
+fn finite_pair(args: &Value, x_name: &str, y_name: &str) -> Result<Option<(f64, f64)>, String> {
+ match (args.opt_f64(x_name), args.opt_f64(y_name)) {
+ (None, None) => Ok(None),
+ (Some(x), Some(y)) if x.is_finite() && y.is_finite() => Ok(Some((x, y))),
+ (Some(_), Some(_)) => Err(format!("{x_name} and {y_name} must be finite numbers")),
+ _ => Err(format!("pass both {x_name} and {y_name}, or neither")),
+ }
+}
+
+fn one_location(
+ args: &Value,
+ ref_name: &str,
+ x_name: &str,
+ y_name: &str,
+) -> Result<Option<Location>, String> {
+ let reference = args.opt_str(ref_name);
+ let coords = finite_pair(args, x_name, y_name)?;
+ match (reference, coords) {
+ (Some(_), Some(_)) => Err(format!(
+ "pass either {ref_name} or {x_name}/{y_name}, not both"
+ )),
+ (Some(reference), None) => Ok(Some(Location::Ref(reference))),
+ (None, Some((x, y))) => Ok(Some(Location::Coordinates(x, y))),
+ (None, None) => Ok(None),
+ }
+}
+
+fn parse_request(args: &Value) -> Result<PointerRequest, String> {
+ let action = PointerAction::parse(
+ args.get("action")
+ .and_then(Value::as_str)
+ .ok_or_else(|| "missing required string field: action".to_owned())?,
+ )?;
+ let route = InputRoute::parse(
+ args.get("input_route")
+ .and_then(Value::as_str)
+ .unwrap_or("trusted"),
+ )?;
+ let origin = one_location(args, "ref", "x", "y")?
+ .ok_or_else(|| "browser_pointer needs a ref or x/y coordinates".to_owned())?;
+ let destination = one_location(args, "destination_ref", "to_x", "to_y")?;
+
+ if route == InputRoute::DomEvent && !matches!(origin, Location::Ref(_)) {
+ return Err("input_route=dom_event requires a ref".into());
+ }
+ if action == PointerAction::Drag && destination.is_none() {
+ return Err("action=drag requires destination_ref or to_x/to_y".into());
+ }
+ if action != PointerAction::Drag && destination.is_some() {
+ return Err("destination_ref and to_x/to_y are valid only for action=drag".into());
+ }
+
+ let delta_x = args.opt_f64("delta_x").unwrap_or(0.0);
+ let delta_y = args.opt_f64("delta_y").unwrap_or(0.0);
+ if !delta_x.is_finite() || !delta_y.is_finite() {
+ return Err("delta_x and delta_y must be finite numbers".into());
+ }
+ if action == PointerAction::Scroll && delta_x == 0.0 && delta_y == 0.0 {
+ return Err("action=scroll requires a non-zero delta_x or delta_y".into());
+ }
+ if action != PointerAction::Scroll
+ && (args.get("delta_x").is_some() || args.get("delta_y").is_some())
+ {
+ return Err("delta_x and delta_y are valid only for action=scroll".into());
+ }
+
+ Ok(PointerRequest {
+ action,
+ route,
+ origin,
+ destination,
+ delta_x,
+ delta_y,
+ })
+}
+
+struct ResolvedRef {
+ external: String,
+ backend_node_id: i64,
+ frame: FrameRef,
+ cdp_session: String,
+}
+
+fn same_exact_frame(left: &FrameRef, right: &FrameRef) -> bool {
+ left.kind == right.kind
+ && left.oopif_target_id == right.oopif_target_id
+ && left.identity == right.identity
+}
+
+fn stale(message: impl Into<String>) -> ToolResult {
+ BrowserRefusal::new(BrowserRefusalCode::BrowserRefStale, message).to_tool_result()
+}
+
+async fn resolve_object(
+ conn: &CdpConnection,
+ cdp_session: &str,
+ backend_node_id: i64,
+) -> Result<String, ToolResult> {
+ let resolved = conn
+ .call(
+ Some(cdp_session),
+ "DOM.resolveNode",
+ json!({ "backendNodeId": backend_node_id }),
+ )
+ .await
+ .map_err(|_| stale("the ref's node no longer resolves in the live page"))?;
+ resolved
+ .get("object")
+ .and_then(|object| object.get("objectId"))
+ .and_then(Value::as_str)
+ .map(str::to_owned)
+ .ok_or_else(|| stale("the ref's node has no live object in the page"))
+}
+
+async fn point_for_ref(
+ conn: &CdpConnection,
+ cdp_session: &str,
+ backend_node_id: i64,
+) -> Result<(f64, f64), ToolResult> {
+ let _ = conn
+ .call(
+ Some(cdp_session),
+ "DOM.scrollIntoViewIfNeeded",
+ json!({ "backendNodeId": backend_node_id }),
+ )
+ .await;
+ let model = conn
+ .call(
+ Some(cdp_session),
+ "DOM.getBoxModel",
+ json!({ "backendNodeId": backend_node_id }),
+ )
+ .await
+ .map_err(|_| stale("the ref's node has no live layout box"))?;
+ quad_center(&model).ok_or_else(|| stale("the ref's node returned an unusable layout box"))
+}
+
+fn quad_center(box_model: &Value) -> Option<(f64, f64)> {
+ let quad = box_model.get("model")?.get("content")?.as_array()?;
+ if quad.len() < 8 {
+ return None;
+ }
+ let values = quad
+ .iter()
+ .take(8)
+ .map(Value::as_f64)
+ .collect::<Option<Vec<_>>>()?;
+ Some((
+ (values[0] + values[2] + values[4] + values[6]) / 4.0,
+ (values[1] + values[3] + values[5] + values[7]) / 4.0,
+ ))
+}
+
+pub struct BrowserPointerTool {
+ def: ToolDef,
+ engine: Arc<BrowserEngine>,
+}
+
+impl BrowserPointerTool {
+ pub fn new(engine: Arc<BrowserEngine>) -> Self {
+ Self {
+ def: ToolDef {
+ name: "browser_pointer".into(),
+ description: "Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "target_id": { "type": "string", "description": "Opaque target id minted by get_browser_state." },
+ "tab_id": { "type": "string", "description": "Opaque tab id minted by get_browser_state." },
+ "session": { "type": "string", "description": "Explicit caller session owning the browser capabilities." },
+ "action": { "type": "string", "enum": ["hover", "right_click", "double_click", "scroll", "drag"] },
+ "input_route": { "type": "string", "enum": ["trusted", "dom_event"], "default": "trusted" },
+ "ref": { "type": "string", "description": "Origin page ref. Alternative to x/y." },
+ "x": { "type": "number", "description": "Origin viewport x in CSS pixels." },
+ "y": { "type": "number", "description": "Origin viewport y in CSS pixels." },
+ "destination_ref": { "type": "string", "description": "Drag destination page ref in the exact same frame." },
+ "to_x": { "type": "number", "description": "Drag destination viewport x in CSS pixels." },
+ "to_y": { "type": "number", "description": "Drag destination viewport y in CSS pixels." },
+ "delta_x": { "type": "number", "description": "Horizontal scroll delta in CSS pixels." },
+ "delta_y": { "type": "number", "description": "Vertical scroll delta in CSS pixels." }
+ },
+ "required": ["target_id", "tab_id", "session", "action"],
+ "additionalProperties": true
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: false,
+ open_world: false,
+ },
+ engine,
+ }
+ }
+
+ async fn resolve_ref(
+ &self,
+ session: &str,
+ target_id: &str,
+ tab_id: &str,
+ validated: &ValidatedTab,
+ external: &str,
+ ) -> Result<ResolvedRef, ToolResult> {
+ let entry = self
+ .engine
+ .store
+ .resolve_ref(session, target_id, tab_id, external)
+ .map_err(|refusal| refusal.to_tool_result())?;
+ let cdp_session = self
+ .engine
+ .frame_session_for_mutation(session, target_id, tab_id, validated, &entry.frame)
+ .await
+ .map_err(|refusal| refusal.to_tool_result())?;
+ Ok(ResolvedRef {
+ external: external.to_owned(),
+ backend_node_id: entry.backend_node_id,
+ frame: entry.frame,
+ cdp_session,
+ })
+ }
+
+ fn trusted_background_refusal(&self, validated: &ValidatedTab) -> Option<ToolResult> {
+ if validated.record.cdp_window_id.is_some() {
+ if let Some(limitation) = self
+ .engine
+ .platform
+ .standalone_trusted_input_background_limitation()
+ {
+ return Some(
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ format!(
+ "{limitation}; use input_route=\"dom_event\" with refs to explicitly request synthetic full-background pointer delivery"
+ ),
+ )
+ .to_tool_result(),
+ );
+ }
+ }
+ None
+ }
+
+ async fn dom_event(
+ &self,
+ request: &PointerRequest,
+ validated: &ValidatedTab,
+ origin: &ResolvedRef,
+ destination: Option<&ResolvedRef>,
+ ) -> ToolResult {
+ let conn = &validated.conn;
+ let object_id =
+ match resolve_object(conn, &origin.cdp_session, origin.backend_node_id).await {
+ Ok(id) => id,
+ Err(result) => return result,
+ };
+
+ let (function, arguments) = match request.action {
+ PointerAction::Hover => (
+ "function() { const o={bubbles:true,composed:true,view:this.ownerDocument.defaultView}; this.dispatchEvent(new PointerEvent('pointerover',o)); this.dispatchEvent(new PointerEvent('pointerenter',{...o,bubbles:false})); this.dispatchEvent(new MouseEvent('mouseover',o)); this.dispatchEvent(new MouseEvent('mouseenter',{...o,bubbles:false})); return true; }",
+ json!([]),
+ ),
+ PointerAction::RightClick => (
+ "function() { const o={bubbles:true,cancelable:true,composed:true,button:2,buttons:2,view:this.ownerDocument.defaultView}; this.dispatchEvent(new PointerEvent('pointerdown',o)); this.dispatchEvent(new MouseEvent('mousedown',o)); this.dispatchEvent(new MouseEvent('mouseup',{...o,buttons:0})); this.dispatchEvent(new PointerEvent('pointerup',{...o,buttons:0})); this.dispatchEvent(new MouseEvent('contextmenu',{...o,buttons:0})); return true; }",
+ json!([]),
+ ),
+ PointerAction::DoubleClick => (
+ "function() { const o={bubbles:true,cancelable:true,composed:true,button:0,view:this.ownerDocument.defaultView}; for (let detail=1; detail<=2; detail++) { this.dispatchEvent(new PointerEvent('pointerdown',{...o,buttons:1,detail})); this.dispatchEvent(new MouseEvent('mousedown',{...o,buttons:1,detail})); this.dispatchEvent(new MouseEvent('mouseup',{...o,buttons:0,detail})); this.dispatchEvent(new PointerEvent('pointerup',{...o,buttons:0,detail})); this.dispatchEvent(new MouseEvent('click',{...o,buttons:0,detail})); } this.dispatchEvent(new MouseEvent('dblclick',{...o,buttons:0,detail:2})); return true; }",
+ json!([]),
+ ),
+ PointerAction::Scroll => (
+ "function(dx,dy) { const o={deltaX:dx,deltaY:dy,bubbles:true,cancelable:true,composed:true,view:this.ownerDocument.defaultView}; this.dispatchEvent(new WheelEvent('wheel',o)); let n=this; while (n && n !== this.ownerDocument.documentElement) { const s=this.ownerDocument.defaultView.getComputedStyle(n); if (/(auto|scroll)/.test(s.overflow+s.overflowX+s.overflowY)) break; n=n.parentElement; } const target=n || this.ownerDocument.scrollingElement || this.ownerDocument.documentElement; const beforeX=target.scrollLeft; const beforeY=target.scrollTop; target.scrollBy(dx,dy); const changed=target.scrollLeft !== beforeX || target.scrollTop !== beforeY; if (changed) target.dispatchEvent(new Event('scroll')); return changed; }",
+ json!([{ "value": request.delta_x }, { "value": request.delta_y }]),
+ ),
+ PointerAction::Drag => {
+ let destination_argument = if let Some(destination) = destination {
+ let object_id = match resolve_object(
+ conn,
+ &destination.cdp_session,
+ destination.backend_node_id,
+ )
+ .await
+ {
+ Ok(id) => id,
+ Err(result) => return result,
+ };
+ json!([{ "objectId": object_id }, { "value": Value::Null }, { "value": Value::Null }])
+ } else if let Some(Location::Coordinates(x, y)) = &request.destination {
+ if origin.frame.kind != FrameKind::Main {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ "coordinate drag destinations are only provably in the same frame for a main-frame origin; use destination_ref for iframe or OOPIF drag",
+ )
+ .to_tool_result();
+ }
+ json!([{ "value": Value::Null }, { "value": x }, { "value": y }])
+ } else {
+ unreachable!("drag destination validated")
+ };
+ (
+ "function(destination,x,y) { const doc=this.ownerDocument; const dest=destination || doc.elementFromPoint(x,y); if (!dest || dest.ownerDocument !== doc) return false; let data=null; try { data=new DataTransfer(); } catch (_) {} const common={bubbles:true,cancelable:true,composed:true}; this.dispatchEvent(new PointerEvent('pointerdown',{...common,button:0,buttons:1})); this.dispatchEvent(new MouseEvent('mousedown',{...common,button:0,buttons:1})); this.dispatchEvent(new DragEvent('dragstart',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('dragenter',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('dragover',{...common,dataTransfer:data})); dest.dispatchEvent(new DragEvent('drop',{...common,dataTransfer:data})); this.dispatchEvent(new DragEvent('dragend',{...common,dataTransfer:data})); dest.dispatchEvent(new MouseEvent('mouseup',{...common,button:0,buttons:0})); dest.dispatchEvent(new PointerEvent('pointerup',{...common,button:0,buttons:0})); return true; }",
+ destination_argument,
+ )
+ }
+ };
+
+ match conn
+ .call(
+ Some(&origin.cdp_session),
+ "Runtime.callFunctionOn",
+ json!({
+ "objectId": object_id,
+ "functionDeclaration": function,
+ "arguments": arguments,
+ "returnByValue": true,
+ }),
+ )
+ .await
+ {
+ Ok(value)
+ if value.get("exceptionDetails").is_none()
+ && (!matches!(request.action, PointerAction::Scroll | PointerAction::Drag)
+ || value.pointer("/result/value").and_then(Value::as_bool)
+ != Some(false)) =>
+ {
+ ToolResult::text(format!(
+ "dispatched synthetic {} in {}",
+ request.action.as_str(),
+ validated.tab.tab_id
+ ))
+ .with_structured(self.success_json(
+ request,
+ validated,
+ origin,
+ destination,
+ None,
+ None,
+ ))
+ }
+ Ok(value) if value.get("exceptionDetails").is_some() => BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ format!(
+ "synthetic {} raised a page-side exception and delivery was not proven",
+ request.action.as_str()
+ ),
+ )
+ .to_tool_result(),
+ Ok(_) if request.action == PointerAction::Drag => BrowserRefusal::new(
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ "the drag destination did not resolve in the origin ref's exact document",
+ )
+ .to_tool_result(),
+ Ok(_) => BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the synthetic scroll target did not move, so delivery was not proven",
+ )
+ .to_tool_result(),
+ Err(error) => ToolResult::error(format!(
+ "synthetic {} failed: {error}",
+ request.action.as_str()
+ )),
+ }
+ }
+
+ fn success_json(
+ &self,
+ request: &PointerRequest,
+ validated: &ValidatedTab,
+ origin: &ResolvedRef,
+ destination: Option<&ResolvedRef>,
+ origin_point: Option<(f64, f64)>,
+ destination_point: Option<(f64, f64)>,
+ ) -> Value {
+ let external_ref = (!origin.external.is_empty()).then_some(origin.external.as_str());
+ json!({
+ "status": "ok",
+ "action": request.action.as_str(),
+ "route": request.route.as_str(),
+ "target_id": validated.record.target_id,
+ "tab_id": validated.tab.tab_id,
+ "ref": external_ref,
+ "frame": origin.frame.kind.as_str(),
+ "destination_ref": destination.map(|value| value.external.as_str()),
+ "x": origin_point.map(|point| point.0),
+ "y": origin_point.map(|point| point.1),
+ "to_x": destination_point.map(|point| point.0),
+ "to_y": destination_point.map(|point| point.1),
+ "delta_x": (request.action == PointerAction::Scroll).then_some(request.delta_x),
+ "delta_y": (request.action == PointerAction::Scroll).then_some(request.delta_y),
+ })
+ }
+
+ async fn trusted(
+ &self,
+ request: &PointerRequest,
+ validated: &ValidatedTab,
+ origin_ref: Option<&ResolvedRef>,
+ destination_ref: Option<&ResolvedRef>,
+ ) -> ToolResult {
+ let conn = &validated.conn;
+ let cdp_session = origin_ref
+ .map(|reference| reference.cdp_session.as_str())
+ .unwrap_or(validated.cdp_session.as_str());
+ let origin = match (&request.origin, origin_ref) {
+ (Location::Coordinates(x, y), None) => (*x, *y),
+ (Location::Ref(_), Some(reference)) => {
+ match point_for_ref(conn, &reference.cdp_session, reference.backend_node_id).await {
+ Ok(point) => point,
+ Err(result) => return result,
+ }
+ }
+ _ => unreachable!("origin resolution matches request"),
+ };
+ let destination = match (&request.destination, destination_ref) {
+ (Some(Location::Coordinates(x, y)), None) => Some((*x, *y)),
+ (Some(Location::Ref(_)), Some(reference)) => {
+ match point_for_ref(conn, &reference.cdp_session, reference.backend_node_id).await {
+ Ok(point) => Some(point),
+ Err(result) => return result,
+ }
+ }
+ (None, None) => None,
+ _ => unreachable!("destination resolution matches request"),
+ };
+
+ if let Err(error) = conn
+ .call(
+ Some(cdp_session),
+ "Emulation.setFocusEmulationEnabled",
+ json!({ "enabled": true }),
+ )
+ .await
+ {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ format!("the target tab could not enter CDP focus emulation: {error}"),
+ )
+ .to_tool_result();
+ }
+ tokio::time::sleep(Duration::from_millis(25)).await;
+
+ let delivery = self
+ .dispatch_trusted(conn, cdp_session, request, origin, destination)
+ .await;
+ let cleanup = conn
+ .call(
+ Some(cdp_session),
+ "Emulation.setFocusEmulationEnabled",
+ json!({ "enabled": false }),
+ )
+ .await;
+ if let Err(error) = delivery {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ format!(
+ "trusted {} failed ({error}); no synthetic fallback was attempted",
+ request.action.as_str()
+ ),
+ )
+ .to_tool_result();
+ }
+ if let Err(error) = cleanup {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ format!(
+ "trusted {} was acknowledged but focus emulation could not be restored ({error}); delivery is unknown and must not be retried automatically",
+ request.action.as_str()
+ ),
+ )
+ .with_detail(json!({ "delivery": "unknown", "retryable": false }))
+ .to_tool_result();
+ }
+
+ let synthetic_origin = ResolvedRef {
+ external: match &request.origin {
+ Location::Ref(reference) => reference.clone(),
+ Location::Coordinates(_, _) => String::new(),
+ },
+ backend_node_id: origin_ref.map_or(0, |reference| reference.backend_node_id),
+ frame: origin_ref
+ .map(|reference| reference.frame.clone())
+ .unwrap_or_else(FrameRef::main_unproven),
+ cdp_session: cdp_session.to_owned(),
+ };
+ ToolResult::text(format!(
+ "dispatched trusted {} in {}",
+ request.action.as_str(),
+ validated.tab.tab_id
+ ))
+ .with_structured(self.success_json(
+ request,
+ validated,
+ &synthetic_origin,
+ destination_ref,
+ Some(origin),
+ destination,
+ ))
+ }
+
+ async fn dispatch_trusted(
+ &self,
+ conn: &CdpConnection,
+ cdp_session: &str,
+ request: &PointerRequest,
+ origin: (f64, f64),
+ destination: Option<(f64, f64)>,
+ ) -> anyhow::Result<()> {
+ let call = |params: Value| conn.call(Some(cdp_session), "Input.dispatchMouseEvent", params);
+ match request.action {
+ PointerAction::Hover => {
+ call(
+ json!({ "type": "mouseMoved", "x": origin.0, "y": origin.1, "button": "none" }),
+ )
+ .await?;
+ }
+ PointerAction::RightClick => {
+ for kind in ["mousePressed", "mouseReleased"] {
+ call(json!({ "type": kind, "x": origin.0, "y": origin.1, "button": "right", "clickCount": 1 })).await?;
+ }
+ }
+ PointerAction::DoubleClick => {
+ for click_count in [1, 2] {
+ for kind in ["mousePressed", "mouseReleased"] {
+ call(json!({ "type": kind, "x": origin.0, "y": origin.1, "button": "left", "clickCount": click_count })).await?;
+ }
+ }
+ }
+ PointerAction::Scroll => {
+ call(json!({ "type": "mouseWheel", "x": origin.0, "y": origin.1, "deltaX": request.delta_x, "deltaY": request.delta_y })).await?;
+ }
+ PointerAction::Drag => {
+ let destination = destination.expect("drag destination validated");
+ call(
+ json!({ "type": "mouseMoved", "x": origin.0, "y": origin.1, "button": "none" }),
+ )
+ .await?;
+ call(json!({ "type": "mousePressed", "x": origin.0, "y": origin.1, "button": "left", "buttons": 1, "clickCount": 1 })).await?;
+ for step in 1..=8 {
+ let progress = f64::from(step) / 8.0;
+ let x = origin.0 + (destination.0 - origin.0) * progress;
+ let y = origin.1 + (destination.1 - origin.1) * progress;
+ call(json!({ "type": "mouseMoved", "x": x, "y": y, "button": "left", "buttons": 1 })).await?;
+ }
+ call(json!({ "type": "mouseReleased", "x": destination.0, "y": destination.1, "button": "left", "buttons": 0, "clickCount": 1 })).await?;
+ }
+ }
+ Ok(())
+ }
+}
+
+#[async_trait]
+impl Tool for BrowserPointerTool {
+ fn def(&self) -> &ToolDef {
+ &self.def
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let target_id = match args.require_str("target_id") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let tab_id = match args.require_str("tab_id") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let session = match explicit_session(&args) {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let request = match parse_request(&args) {
+ Ok(value) => value,
+ Err(error) => return ToolResult::error(error),
+ };
+
+ let _mutation = match self
+ .engine
+ .lock_mutation(&session, &target_id, &tab_id)
+ .await
+ {
+ Ok(guard) => guard,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let validated = match self
+ .engine
+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))
+ .await
+ {
+ Ok(value) => value,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ if request.route == InputRoute::Trusted {
+ if let Some(refusal) = self.trusted_background_refusal(&validated) {
+ return refusal;
+ }
+ }
+
+ let origin_ref = match &request.origin {
+ Location::Ref(external) => match self
+ .resolve_ref(&session, &target_id, &tab_id, &validated, external)
+ .await
+ {
+ Ok(reference) => Some(reference),
+ Err(result) => return result,
+ },
+ Location::Coordinates(_, _) => None,
+ };
+ let destination_ref = match &request.destination {
+ Some(Location::Ref(external)) => match self
+ .resolve_ref(&session, &target_id, &tab_id, &validated, external)
+ .await
+ {
+ Ok(reference) => Some(reference),
+ Err(result) => return result,
+ },
+ _ => None,
+ };
+
+ if let (Some(origin), Some(destination)) = (&origin_ref, &destination_ref) {
+ if !same_exact_frame(&origin.frame, &destination.frame)
+ || origin.cdp_session != destination.cdp_session
+ {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ "drag origin and destination refs must belong to the exact same live frame",
+ )
+ .to_tool_result();
+ }
+ }
+ if request.action == PointerAction::Drag
+ && origin_ref.is_none()
+ && destination_ref.is_some()
+ {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ "a coordinate drag origin cannot prove it shares a frame with destination_ref; use two refs or two coordinate pairs",
+ )
+ .to_tool_result();
+ }
+ if request.action == PointerAction::Drag
+ && matches!(&request.destination, Some(Location::Coordinates(_, _)))
+ && origin_ref
+ .as_ref()
+ .is_some_and(|reference| reference.frame.kind != FrameKind::Main)
+ {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserWrongTargetRefused,
+ "a viewport-coordinate drag destination is only in the same provable frame as a main-frame ref; use destination_ref for iframe or OOPIF drag",
+ )
+ .to_tool_result();
+ }
+
+ match request.route {
+ InputRoute::DomEvent => {
+ self.dom_event(
+ &request,
+ &validated,
+ origin_ref.as_ref().expect("dom route requires ref"),
+ destination_ref.as_ref(),
+ )
+ .await
+ }
+ InputRoute::Trusted => {
+ self.trusted(
+ &request,
+ &validated,
+ origin_ref.as_ref(),
+ destination_ref.as_ref(),
+ )
+ .await
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn parses_every_action_and_coordinate_origin() {
+ for action in ["hover", "right_click", "double_click"] {
+ let parsed = parse_request(&json!({
+ "action": action,
+ "x": 12.5,
+ "y": 8,
+ }))
+ .unwrap();
+ assert_eq!(parsed.origin, Location::Coordinates(12.5, 8.0));
+ assert_eq!(parsed.route, InputRoute::Trusted);
+ }
+ }
+
+ #[test]
+ fn dom_event_requires_a_ref() {
+ let error = parse_request(&json!({
+ "action": "hover",
+ "input_route": "dom_event",
+ "x": 1,
+ "y": 2,
+ }))
+ .unwrap_err();
+ assert!(error.contains("requires a ref"), "{error}");
+ }
+
+ #[test]
+ fn drag_requires_exactly_one_destination_shape() {
+ assert!(parse_request(&json!({ "action": "drag", "ref": "p1:0" }))
+ .unwrap_err()
+ .contains("requires destination"));
+ assert!(parse_request(&json!({
+ "action": "drag",
+ "ref": "p1:0",
+ "destination_ref": "p1:1",
+ "to_x": 3,
+ "to_y": 4,
+ }))
+ .unwrap_err()
+ .contains("not both"));
+ }
+
+ #[test]
+ fn scroll_requires_a_nonzero_finite_delta() {
+ assert!(parse_request(&json!({ "action": "scroll", "ref": "p1:0" }))
+ .unwrap_err()
+ .contains("non-zero"));
+ let parsed = parse_request(&json!({
+ "action": "scroll",
+ "ref": "p1:0",
+ "delta_y": 240,
+ }))
+ .unwrap();
+ assert_eq!(parsed.delta_y, 240.0);
+ }
+
+ #[test]
+ fn quad_center_rejects_malformed_models() {
+ assert_eq!(
+ quad_center(&json!({ "model": { "content": [0, 0, 10, 0, 10, 20, 0, 20] } })),
+ Some((5.0, 10.0))
+ );
+ assert_eq!(
+ quad_center(&json!({ "model": { "content": [0, 0] } })),
+ None
+ );
+ }
+
+ #[test]
+ fn exact_frame_comparison_includes_document_identity() {
+ let first = FrameRef::main_unproven();
+ let second = FrameRef::main_unproven();
+ assert!(same_exact_frame(&first, &second));
+
+ let navigated = FrameRef {
+ kind: FrameKind::Main,
+ oopif_target_id: None,
+ identity: Some(super::super::store::FrameIdentity {
+ frame_id: "main".into(),
+ loader_id: "new-loader".into(),
+ }),
+ };
+ assert!(!same_exact_frame(&first, &navigated));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs
index 956dc47a75..b2348ef831 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs
@@ -773,7 +773,18 @@ fn action_kinds(
.get("contenteditable")
.is_some_and(|value| value.is_empty() || value.eq_ignore_ascii_case("true"))
});
- if matches!(role, "textbox" | "searchbox") || matches!(tag, "input" | "textarea") || editable {
+ let file_input = tag == "input"
+ && dom.is_some_and(|meta| {
+ meta.attrs
+ .get("type")
+ .is_some_and(|value| value.eq_ignore_ascii_case("file"))
+ });
+ if file_input {
+ actions.push(BrowserActionKind::Upload);
+ } else if matches!(role, "textbox" | "searchbox")
+ || matches!(tag, "input" | "textarea")
+ || editable
+ {
actions.push(BrowserActionKind::Type);
}
if actions.is_empty()
@@ -1073,6 +1084,22 @@ mod tests {
use serde_json::json;
use super::*;
+
+ #[test]
+ fn file_inputs_expose_upload_instead_of_text_typing() {
+ let dom = DomMeta {
+ tag: "input".into(),
+ attrs: HashMap::from([("type".into(), "file".into())]),
+ order: 0,
+ css_hidden: false,
+ parent_backend_node_id: None,
+ frame_id: None,
+ };
+ assert_eq!(
+ action_kinds("textbox", Some(&dom), &BTreeMap::new(), None),
+ vec![BrowserActionKind::Upload]
+ );
+ }
use crate::browser::store::FrameRef;
fn frame() -> FrameRef {
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs
index 0b47f26ddf..7bc9740510 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs
@@ -32,6 +32,7 @@ use super::types::{BindingQuality, ProcessFingerprint, Rect};
pub enum BrowserActionKind {
Click,
Type,
+ Upload,
}
impl BrowserActionKind {
@@ -39,6 +40,7 @@ impl BrowserActionKind {
match self {
Self::Click => "click",
Self::Type => "type",
+ Self::Upload => "upload",
}
}
}
@@ -187,6 +189,10 @@ pub struct TabRecord {
pub cdp_target_id: String,
pub title: String,
pub url: String,
+ /// Native-window selection proof captured at bind time. `None` means the
+ /// selected tab could not be proven without activating or foregrounding a
+ /// page, so the public `active` field must be JSON null.
+ pub active: Option<bool>,
pub generation: u64,
pub snapshots: HashMap<u64, SnapshotRecord>,
}
@@ -509,6 +515,7 @@ mod tests {
cdp_target_id: "CDP1".into(),
title: "Example".into(),
url: "https://example.test".into(),
+ active: Some(true),
generation: 0,
snapshots: HashMap::from([(
snap_id,
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
index 64b1eb7741..fcbe2bcad3 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
@@ -1,5 +1,4 @@
-//! The five browser tools: get_browser_state, browser_prepare,
-//! browser_navigate, browser_click, browser_type.
+//! Browser state, preparation, navigation, input, and page-owned dialog tools.
//!
//! Schemas, annotations, and exact-or-refused semantics live here; OS
//! identity comes from the [`BrowserPlatform`] adapter; CDP mechanics
@@ -16,13 +15,15 @@ use crate::tool::{Tool, ToolDef, ToolRegistry};
use crate::tool_args::ArgsExt;
use super::approval::MCP_HOST_APPROVAL_ARG;
+use super::download::BrowserDownloadTool;
use super::engine::BrowserEngine;
use super::platform::{PrepareAuthorization, PrepareProfile, PrepareRequest, PrepareStrategy};
+use super::pointer::BrowserPointerTool;
use super::refusal::{BrowserRefusal, BrowserRefusalCode};
use super::store::BrowserActionKind;
use super::types::BindingQuality;
-/// Register all five browser tools against one shared engine. Platform
+/// Register the complete browser surface against one shared engine. Platform
/// crates call this from their `register_all` after constructing the
/// engine with their adapter.
pub fn register_browser_tools(engine: &Arc<BrowserEngine>, registry: &mut ToolRegistry) {
@@ -31,6 +32,10 @@ pub fn register_browser_tools(engine: &Arc<BrowserEngine>, registry: &mut ToolRe
registry.register(Box::new(BrowserNavigateTool::new(engine.clone())));
registry.register(Box::new(BrowserClickTool::new(engine.clone())));
registry.register(Box::new(BrowserTypeTool::new(engine.clone())));
+ registry.register(Box::new(BrowserDialogTool::new(engine.clone())));
+ registry.register(Box::new(BrowserSetInputFilesTool::new(engine.clone())));
+ registry.register(Box::new(BrowserDownloadTool::new(engine.clone())));
+ registry.register(Box::new(BrowserPointerTool::new(engine.clone())));
}
// ── Shared helpers ───────────────────────────────────────────────────────────
@@ -342,7 +347,7 @@ impl Tool for GetBrowserStateTool {
"tab_id": t.tab_id,
"title": t.title,
"url": t.url,
- "active": t.cdp_target_id == record.cdp_target_id,
+ "active": t.active,
})
})
.collect();
@@ -1405,6 +1410,386 @@ impl Tool for BrowserTypeTool {
}
}
+// ── browser_dialog ──────────────────────────────────────────────────────────
+
+pub struct BrowserDialogTool {
+ def: ToolDef,
+ engine: Arc<BrowserEngine>,
+}
+
+impl BrowserDialogTool {
+ pub fn new(engine: Arc<BrowserEngine>) -> Self {
+ Self {
+ def: ToolDef {
+ name: "browser_dialog".into(),
+ description: "Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "target_id": schema_target_id(),
+ "tab_id": schema_tab_id(),
+ "session": schema_session(),
+ "action": { "type": "string", "enum": ["inspect", "accept", "dismiss"] },
+ "dialog_id": { "type": "string", "description": "Opaque current dialog generation returned by action=inspect." },
+ "prompt_text": { "type": "string", "description": "Sensitive response text, valid only when accepting a prompt dialog." },
+ "delivery_mode": {
+ "type": "string",
+ "enum": ["background", "foreground"],
+ "default": "background",
+ "description": "Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only."
+ }
+ },
+ "required": ["target_id", "tab_id", "action"],
+ "additionalProperties": true
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: false,
+ open_world: false,
+ },
+ engine,
+ }
+ }
+}
+
+#[async_trait]
+impl Tool for BrowserDialogTool {
+ fn def(&self) -> &ToolDef {
+ &self.def
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let (target_id, tab_id, action) = match (
+ args.require_str("target_id"),
+ args.require_str("tab_id"),
+ args.require_str("action"),
+ ) {
+ (Ok(target), Ok(tab), Ok(action)) => (target, tab, action),
+ (Err(error), _, _) | (_, Err(error), _) | (_, _, Err(error)) => return error,
+ };
+ if !matches!(action.as_str(), "inspect" | "accept" | "dismiss") {
+ return ToolResult::error("action must be inspect, accept, or dismiss");
+ }
+ let delivery_mode = args
+ .opt_str("delivery_mode")
+ .unwrap_or_else(|| "background".into());
+ if !matches!(delivery_mode.as_str(), "background" | "foreground") {
+ return ToolResult::error("delivery_mode must be background or foreground");
+ }
+ let session = match require_explicit_session(&args) {
+ Ok(session) => session,
+ Err(error) => return error,
+ };
+ let _mutation = match self
+ .engine
+ .lock_mutation(&session, &target_id, &tab_id)
+ .await
+ {
+ Ok(guard) => guard,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let validated = match self
+ .engine
+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))
+ .await
+ {
+ Ok(validated) => validated,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ if cfg!(target_os = "linux") && action != "inspect" && delivery_mode == "background" {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserInputTrustUnavailable,
+ "Chromium's native JavaScript modal cannot be resolved on Linux without changing foreground posture; retry with delivery_mode=\"foreground\"",
+ )
+ .with_detail(json!({ "supported_delivery_mode": "foreground" }))
+ .to_tool_result();
+ }
+ let conn = &validated.conn;
+ let cdp_session = validated.cdp_session.as_str();
+ let cdp_target_id = validated.tab.cdp_target_id.as_str();
+ if conn.dialog_state(cdp_target_id).is_none() && !conn.has_dialog_session(cdp_target_id) {
+ // Register before Page.enable so an opening event delivered before
+ // the command reply is still attributed to the exact target.
+ conn.register_dialog_session(cdp_session, cdp_target_id);
+ if let Err(error) = conn.call(Some(cdp_session), "Page.enable", json!({})).await {
+ if conn.dialog_state(cdp_target_id).is_none() {
+ conn.unregister_dialog_session(cdp_session, cdp_target_id);
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ format!("the exact tab does not expose JavaScript dialog events: {error}"),
+ )
+ .to_tool_result();
+ }
+ }
+ tokio::task::yield_now().await;
+ }
+
+ let Some(dialog) = conn.dialog_state(cdp_target_id) else {
+ return if action == "inspect" {
+ ToolResult::text(format!(
+ "no page-owned JavaScript dialog is open in {tab_id}"
+ ))
+ .with_structured(json!({
+ "status": "ok", "target_id": target_id, "tab_id": tab_id,
+ "present": false
+ }))
+ } else {
+ BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "no current page-owned JavaScript dialog exists on the exact tab",
+ )
+ .to_tool_result()
+ };
+ };
+ let dialog_id = format!("dialog-{}", dialog.generation);
+ if action == "inspect" {
+ return ToolResult::text(format!("{} dialog is open in {tab_id}", dialog.kind))
+ .with_structured(json!({
+ "status": "ok", "target_id": target_id, "tab_id": tab_id,
+ "present": true, "dialog_id": dialog_id, "kind": dialog.kind
+ }));
+ }
+
+ let supplied_id = match args.require_str("dialog_id") {
+ Ok(id) => id,
+ Err(error) => return error,
+ };
+ if supplied_id != dialog_id {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the dialog capability is stale or belongs to another dialog",
+ )
+ .to_tool_result();
+ }
+ let prompt_text = args.opt_str("prompt_text");
+ if prompt_text.is_some() && (action != "accept" || dialog.kind != "prompt") {
+ return ToolResult::error("prompt_text is valid only when accepting a prompt dialog");
+ }
+ let mut params = json!({ "accept": action == "accept" });
+ if let Some(text) = prompt_text {
+ params["promptText"] = Value::String(text);
+ }
+ match conn
+ .call(
+ Some(&dialog.session_id),
+ "Page.handleJavaScriptDialog",
+ params,
+ )
+ .await
+ {
+ Ok(_) => {
+ conn.clear_dialog_state(cdp_target_id, dialog.generation);
+ ToolResult::text(format!("{action}ed {} dialog in {tab_id}", dialog.kind))
+ .with_structured(json!({
+ "status": "ok", "target_id": target_id, "tab_id": tab_id,
+ "dialog_id": dialog_id, "kind": dialog.kind, "action": action
+ }))
+ }
+ Err(error) => BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ format!("the exact JavaScript dialog could not be resolved: {error}"),
+ )
+ .to_tool_result(),
+ }
+ }
+}
+
+// ── browser_set_input_files ─────────────────────────────────────────────────
+
+pub struct BrowserSetInputFilesTool {
+ def: ToolDef,
+ engine: Arc<BrowserEngine>,
+}
+
+impl BrowserSetInputFilesTool {
+ pub fn new(engine: Arc<BrowserEngine>) -> Self {
+ Self {
+ def: ToolDef {
+ name: "browser_set_input_files".into(),
+ description: "Assign one or more explicit absolute local files to an exact live <input type=file> ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "target_id": schema_target_id(),
+ "tab_id": schema_tab_id(),
+ "session": schema_session(),
+ "ref": schema_ref(),
+ "files": {
+ "type": "array", "minItems": 1, "maxItems": 32,
+ "items": { "type": "string", "description": "Absolute path to one local regular file." }
+ }
+ },
+ "required": ["target_id", "tab_id", "ref", "files"],
+ "additionalProperties": true
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: false,
+ open_world: true,
+ },
+ engine,
+ }
+ }
+}
+
+fn validated_upload_paths(args: &Value) -> Result<Vec<String>, ToolResult> {
+ let Some(files) = args.get("files").and_then(Value::as_array) else {
+ return Err(ToolResult::error(
+ "files must be a non-empty array of absolute paths",
+ ));
+ };
+ if files.is_empty() || files.len() > 32 {
+ return Err(ToolResult::error(
+ "files must contain between 1 and 32 paths",
+ ));
+ }
+ files
+ .iter()
+ .map(|value| {
+ let raw = value
+ .as_str()
+ .ok_or_else(|| ToolResult::error("every files entry must be a string"))?;
+ let path = std::path::Path::new(raw);
+ if !path.is_absolute() {
+ return Err(ToolResult::error("every upload path must be absolute"));
+ }
+ let metadata = std::fs::symlink_metadata(path)
+ .map_err(|_| ToolResult::error("an approved upload path does not exist"))?;
+ if metadata.file_type().is_symlink() || !metadata.is_file() {
+ return Err(ToolResult::error(
+ "upload paths must name regular files directly, not links or directories",
+ ));
+ }
+ std::fs::canonicalize(path)
+ .map(|path| path.to_string_lossy().into_owned())
+ .map_err(|_| ToolResult::error("an upload path could not be canonicalized"))
+ })
+ .collect()
+}
+
+#[async_trait]
+impl Tool for BrowserSetInputFilesTool {
+ fn def(&self) -> &ToolDef {
+ &self.def
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let (target_id, tab_id, ext_ref) = match (
+ args.require_str("target_id"),
+ args.require_str("tab_id"),
+ args.require_str("ref"),
+ ) {
+ (Ok(target), Ok(tab), Ok(ext_ref)) => (target, tab, ext_ref),
+ (Err(error), _, _) | (_, Err(error), _) | (_, _, Err(error)) => return error,
+ };
+ let files = match validated_upload_paths(&args) {
+ Ok(files) => files,
+ Err(error) => return error,
+ };
+ let session = match require_explicit_session(&args) {
+ Ok(session) => session,
+ Err(error) => return error,
+ };
+ let _mutation = match self
+ .engine
+ .lock_mutation(&session, &target_id, &tab_id)
+ .await
+ {
+ Ok(guard) => guard,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let validated = match self
+ .engine
+ .revalidate_for_mutation(&session, &target_id, Some(&tab_id))
+ .await
+ {
+ Ok(validated) => validated,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let entry = match self
+ .engine
+ .store
+ .resolve_ref(&session, &target_id, &tab_id, &ext_ref)
+ {
+ Ok(entry) => entry,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Upload) {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the semantic ref is not a file-upload control",
+ )
+ .to_tool_result();
+ }
+ let cdp_session = match self
+ .engine
+ .frame_session_for_mutation(&session, &target_id, &tab_id, &validated, &entry.frame)
+ .await
+ {
+ Ok(session) => session,
+ Err(refusal) => return refusal.to_tool_result(),
+ };
+ let described = match validated
+ .conn
+ .call(
+ Some(&cdp_session),
+ "DOM.describeNode",
+ json!({ "backendNodeId": entry.backend_node_id }),
+ )
+ .await
+ {
+ Ok(node) => node,
+ Err(_) => {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserRefStale,
+ "the upload ref no longer resolves in the live page",
+ )
+ .to_tool_result()
+ }
+ };
+ let node = &described["node"];
+ let attributes = node["attributes"].as_array().cloned().unwrap_or_default();
+ let is_file_input = node["nodeName"]
+ .as_str()
+ .is_some_and(|name| name.eq_ignore_ascii_case("input"))
+ && attributes.chunks_exact(2).any(|pair| {
+ pair[0]
+ .as_str()
+ .is_some_and(|name| name.eq_ignore_ascii_case("type"))
+ && pair[1]
+ .as_str()
+ .is_some_and(|value| value.eq_ignore_ascii_case("file"))
+ });
+ if !is_file_input {
+ return BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ "the live ref is not an input[type=file] element",
+ )
+ .to_tool_result();
+ }
+ match validated
+ .conn
+ .call(
+ Some(&cdp_session),
+ "DOM.setFileInputFiles",
+ json!({ "backendNodeId": entry.backend_node_id, "files": files }),
+ )
+ .await
+ {
+ Ok(_) => ToolResult::text(format!("assigned {} file(s) in {tab_id}", files.len()))
+ .with_structured(json!({
+ "status": "ok", "target_id": target_id, "tab_id": tab_id,
+ "ref": ext_ref, "frame": entry.frame.kind.as_str(), "file_count": files.len()
+ })),
+ Err(error) => BrowserRefusal::new(
+ BrowserRefusalCode::BrowserActionUnavailable,
+ format!("the browser refused the exact file input assignment: {error}"),
+ )
+ .to_tool_result(),
+ }
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -1541,6 +1926,12 @@ mod tests {
assert!(!prepare_properties.contains_key("allow_restart"));
assert!(!prepare_properties.contains_key(MCP_HOST_APPROVAL_ARG));
+ let dialog = BrowserDialogTool::new(e.clone());
+ assert_eq!(
+ dialog.def().input_schema["properties"]["delivery_mode"]["default"],
+ "background"
+ );
+
for (def, name) in [
(
BrowserPrepareTool::new(e.clone()).def().clone(),
@@ -1558,18 +1949,37 @@ mod tests {
BrowserTypeTool::new(e.clone()).def().clone(),
"browser_type",
),
+ (
+ BrowserDialogTool::new(e.clone()).def().clone(),
+ "browser_dialog",
+ ),
+ (
+ BrowserSetInputFilesTool::new(e.clone()).def().clone(),
+ "browser_set_input_files",
+ ),
+ (
+ BrowserDownloadTool::new(e.clone()).def().clone(),
+ "browser_download",
+ ),
+ (
+ BrowserPointerTool::new(e.clone()).def().clone(),
+ "browser_pointer",
+ ),
] {
assert_eq!(def.name, name);
assert!(!def.read_only, "{name} is a mutation");
assert!(def.input_schema["properties"].is_object(), "{name} schema");
}
- // Navigate reaches the open web; the input tools do not.
+ // Navigate and download reach the open web; ordinary input does not.
assert!(BrowserNavigateTool::new(e.clone()).def().open_world);
- assert!(!BrowserClickTool::new(e).def().open_world);
+ assert!(BrowserDownloadTool::new(e.clone()).def().open_world);
+ assert!(BrowserDownloadTool::new(e.clone()).def().destructive);
+ assert!(!BrowserClickTool::new(e.clone()).def().open_world);
+ assert!(!BrowserPointerTool::new(e).def().open_world);
}
#[test]
- fn registration_registers_all_five() {
+ fn registration_registers_all_browser_tools() {
let e = engine();
let mut registry = ToolRegistry::new();
register_browser_tools(&e, &mut registry);
@@ -1581,11 +1991,27 @@ mod tests {
"browser_prepare",
"browser_navigate",
"browser_click",
- "browser_type"
+ "browser_type",
+ "browser_dialog",
+ "browser_set_input_files",
+ "browser_download",
+ "browser_pointer"
]
);
}
+ #[test]
+ fn upload_paths_are_absolute_regular_files_and_outputs_need_no_path() {
+ let path = std::env::temp_dir().join(format!("cua-upload-{}.txt", uuid::Uuid::new_v4()));
+ std::fs::write(&path, b"fixture").unwrap();
+ let args = json!({ "files": [path.to_string_lossy()] });
+ let validated = validated_upload_paths(&args).unwrap();
+ assert_eq!(validated.len(), 1);
+ assert!(std::path::Path::new(&validated[0]).is_absolute());
+ assert!(validated_upload_paths(&json!({ "files": ["relative.txt"] })).is_err());
+ std::fs::remove_file(path).unwrap();
+ }
+
#[tokio::test]
async fn bind_requires_an_explicit_session() {
let e = engine();
@@ -1781,6 +2207,7 @@ mod tests {
cdp_target_id: "CDPX".into(),
title: "Mock".into(),
url: "https://example.test".into(),
+ active: Some(true),
generation: 0,
snapshots: HashMap::new(),
},
@@ -1837,6 +2264,7 @@ mod tests {
cdp_target_id: "CDPX".into(),
title: "Mock".into(),
url: "https://example.test".into(),
+ active: Some(true),
generation: 0,
snapshots: HashMap::new(),
},
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
index 3f9bf8e818..60c3161ad1 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs
@@ -265,6 +265,21 @@ pub enum ToolOperation {
BrowserTypeKeystrokes,
BrowserPrepareIsolated,
BrowserPrepareExistingProfile,
+ BrowserDialogInspect,
+ BrowserDialogAccept,
+ BrowserDialogDismiss,
+ BrowserSetInputFiles,
+ BrowserDownload,
+ BrowserPointerHoverTrusted,
+ BrowserPointerHoverDomEvent,
+ BrowserPointerRightClickTrusted,
+ BrowserPointerRightClickDomEvent,
+ BrowserPointerDoubleClickTrusted,
+ BrowserPointerDoubleClickDomEvent,
+ BrowserPointerScrollTrusted,
+ BrowserPointerScrollDomEvent,
+ BrowserPointerDragTrusted,
+ BrowserPointerDragDomEvent,
Other,
}
@@ -288,6 +303,21 @@ impl ToolOperation {
Self::BrowserTypeKeystrokes => "browser_type_keystrokes",
Self::BrowserPrepareIsolated => "browser_prepare_isolated",
Self::BrowserPrepareExistingProfile => "browser_prepare_existing_profile",
+ Self::BrowserDialogInspect => "browser_dialog_inspect",
+ Self::BrowserDialogAccept => "browser_dialog_accept",
+ Self::BrowserDialogDismiss => "browser_dialog_dismiss",
+ Self::BrowserSetInputFiles => "browser_set_input_files",
+ Self::BrowserDownload => "browser_download",
+ Self::BrowserPointerHoverTrusted => "browser_pointer_hover_trusted",
+ Self::BrowserPointerHoverDomEvent => "browser_pointer_hover_dom_event",
+ Self::BrowserPointerRightClickTrusted => "browser_pointer_right_click_trusted",
+ Self::BrowserPointerRightClickDomEvent => "browser_pointer_right_click_dom_event",
+ Self::BrowserPointerDoubleClickTrusted => "browser_pointer_double_click_trusted",
+ Self::BrowserPointerDoubleClickDomEvent => "browser_pointer_double_click_dom_event",
+ Self::BrowserPointerScrollTrusted => "browser_pointer_scroll_trusted",
+ Self::BrowserPointerScrollDomEvent => "browser_pointer_scroll_dom_event",
+ Self::BrowserPointerDragTrusted => "browser_pointer_drag_trusted",
+ Self::BrowserPointerDragDomEvent => "browser_pointer_drag_dom_event",
Self::Other => "other",
}
}
@@ -349,6 +379,35 @@ pub fn tool_operation(tool_name: &str, args: Option<&serde_json::Value>) -> Tool
_ => ToolOperation::Other,
}
}
+ "browser_dialog" => match string_arg("action") {
+ Some("inspect") => ToolOperation::BrowserDialogInspect,
+ Some("accept") => ToolOperation::BrowserDialogAccept,
+ Some("dismiss") => ToolOperation::BrowserDialogDismiss,
+ _ => ToolOperation::Other,
+ },
+ "browser_set_input_files" => ToolOperation::BrowserSetInputFiles,
+ "browser_download" => ToolOperation::BrowserDownload,
+ "browser_pointer" => match (string_arg("action"), string_arg("input_route")) {
+ (Some("hover"), None | Some("trusted")) => ToolOperation::BrowserPointerHoverTrusted,
+ (Some("hover"), Some("dom_event")) => ToolOperation::BrowserPointerHoverDomEvent,
+ (Some("right_click"), None | Some("trusted")) => {
+ ToolOperation::BrowserPointerRightClickTrusted
+ }
+ (Some("right_click"), Some("dom_event")) => {
+ ToolOperation::BrowserPointerRightClickDomEvent
+ }
+ (Some("double_click"), None | Some("trusted")) => {
+ ToolOperation::BrowserPointerDoubleClickTrusted
+ }
+ (Some("double_click"), Some("dom_event")) => {
+ ToolOperation::BrowserPointerDoubleClickDomEvent
+ }
+ (Some("scroll"), None | Some("trusted")) => ToolOperation::BrowserPointerScrollTrusted,
+ (Some("scroll"), Some("dom_event")) => ToolOperation::BrowserPointerScrollDomEvent,
+ (Some("drag"), None | Some("trusted")) => ToolOperation::BrowserPointerDragTrusted,
+ (Some("drag"), Some("dom_event")) => ToolOperation::BrowserPointerDragDomEvent,
+ _ => ToolOperation::Other,
+ },
_ => ToolOperation::NotApplicable,
}
}
@@ -368,6 +427,18 @@ pub fn is_computer_action(tool_name: &str, operation: ToolOperation) -> bool {
| ToolOperation::EnableJavascriptAppleEvents
);
}
+ if tool_name == "browser_dialog" {
+ return matches!(
+ operation,
+ ToolOperation::BrowserDialogAccept | ToolOperation::BrowserDialogDismiss
+ );
+ }
+ if matches!(
+ tool_name,
+ "browser_set_input_files" | "browser_download" | "browser_pointer"
+ ) {
+ return true;
+ }
crate::tool::default_capabilities_for(tool_name)
.iter()
.any(|capability| {
@@ -688,6 +759,10 @@ fn structured_refusal_code(tool_name: &str, result: Option<&serde_json::Value>)
| "browser_navigate"
| "browser_click"
| "browser_type"
+ | "browser_dialog"
+ | "browser_set_input_files"
+ | "browser_download"
+ | "browser_pointer"
) {
return ToolRefusalCode::None;
}
@@ -856,6 +931,14 @@ pub async fn handle_request(
);
}
}
+ if call.name == "browser_download" {
+ if let Some(arguments) = call.args.as_object_mut() {
+ arguments.insert(
+ crate::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG.to_owned(),
+ serde_json::Value::Bool(true),
+ );
+ }
+ }
let result = registry.invoke(&call.name, call.args).await;
match serde_json::to_value(result) {
@@ -911,7 +994,11 @@ mod observation_tests {
#[test]
fn page_operation_is_closed_and_retains_no_argument_content() {
for (action, expected, computer_action) in [
- ("execute_javascript", ToolOperation::ExecuteJavascript, false),
+ (
+ "execute_javascript",
+ ToolOperation::ExecuteJavascript,
+ false,
+ ),
("get_text", ToolOperation::GetText, false),
("query_dom", ToolOperation::QueryDom, false),
("click_element", ToolOperation::ClickElement, true),
@@ -1008,6 +1095,26 @@ mod observation_tests {
}),
ToolOperation::BrowserPrepareExistingProfile,
),
+ (
+ "browser_dialog",
+ serde_json::json!({"action": "accept", "prompt_text": "private prompt"}),
+ ToolOperation::BrowserDialogAccept,
+ ),
+ (
+ "browser_set_input_files",
+ serde_json::json!({"files": ["/private/path"]}),
+ ToolOperation::BrowserSetInputFiles,
+ ),
+ (
+ "browser_download",
+ serde_json::json!({"destination_root": "/private/destination"}),
+ ToolOperation::BrowserDownload,
+ ),
+ (
+ "browser_pointer",
+ serde_json::json!({"action": "drag", "input_route": "dom_event", "ref": "private-ref"}),
+ ToolOperation::BrowserPointerDragDomEvent,
+ ),
] {
let operation = tool_operation(tool_name, Some(&args));
assert_eq!(operation, expected, "tool={tool_name}");
@@ -1019,6 +1126,9 @@ mod observation_tests {
"private typed text",
"private-profile",
"private-token",
+ "private prompt",
+ "/private/path",
+ "/private/destination",
] {
assert!(!debug.contains(forbidden), "observer leaked {forbidden}");
}
@@ -1070,7 +1180,11 @@ mod observation_tests {
ToolRefusalCode::BrowserInputTrustUnavailable
);
let debug = format!("{observation:?}");
- for forbidden in ["private refusal prose", "private refusal message", "private tab id"] {
+ for forbidden in [
+ "private refusal prose",
+ "private refusal message",
+ "private tab id",
+ ] {
assert!(!debug.contains(forbidden), "observer leaked {forbidden}");
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
index 67870568dc..f0e6bf29d0 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
@@ -452,7 +452,14 @@ mod tests {
#[test]
fn browser_mutations_are_computer_actions_but_reads_and_prepare_are_not() {
let args = serde_json::json!({"session": "test-session"});
- for tool_name in ["browser_navigate", "browser_click", "browser_type"] {
+ for tool_name in [
+ "browser_navigate",
+ "browser_click",
+ "browser_type",
+ "browser_set_input_files",
+ "browser_download",
+ "browser_pointer",
+ ] {
let operation = crate::server::tool_operation(tool_name, Some(&args));
assert!(
crate::server::is_computer_action(tool_name, operation),
@@ -466,6 +473,20 @@ mod tests {
"tool={tool_name}"
);
}
+ for action in ["accept", "dismiss"] {
+ let args = serde_json::json!({"action": action});
+ let operation = crate::server::tool_operation("browser_dialog", Some(&args));
+ assert!(crate::server::is_computer_action(
+ "browser_dialog",
+ operation
+ ));
+ }
+ let inspect = serde_json::json!({"action": "inspect"});
+ let operation = crate::server::tool_operation("browser_dialog", Some(&inspect));
+ assert!(!crate::server::is_computer_action(
+ "browser_dialog",
+ operation
+ ));
}
#[test]
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index aa593ca324..da0299c091 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -103,7 +103,8 @@ impl ToolDef {
/// `recording.replay`, `recording.install_dependency`
/// - `page.action`
/// - `browser.state`, `browser.prepare`, `browser.navigate`,
-/// `browser.input.click`, `browser.input.type`
+/// `browser.input.click`, `browser.input.type`, `browser.input.files`,
+/// `browser.dialog`
/// - `driver.update_check`, `driver.probe`
///
/// Tools with no entry get `[]` — that's fine, it just means
@@ -265,6 +266,10 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {
"browser_navigate" => &["browser.navigate"],
"browser_click" => &["browser.input.click"],
"browser_type" => &["browser.input.type"],
+ "browser_dialog" => &["browser.dialog"],
+ "browser_set_input_files" => &["browser.input.files"],
+ "browser_download" => &["browser.download"],
+ "browser_pointer" => &["browser.input.pointer"],
// ── driver self-service ──────────────────────────────────────
"check_for_update" => &["driver.update_check"],
@@ -482,19 +487,48 @@ fn is_existing_profile_prepare(tool_name: &str, args: &Value) -> bool {
}
fn recording_args_for(tool_name: &str, args: &Value) -> Value {
- if tool_name != "browser_prepare" {
- return args.clone();
- }
let mut redacted = args.clone();
if let Some(arguments) = redacted.as_object_mut() {
- if arguments.contains_key("approval_token") {
- arguments.insert(
- "approval_token".to_owned(),
- Value::String("[redacted]".to_owned()),
- );
+ match tool_name {
+ "browser_prepare" => {
+ if arguments.contains_key("approval_token") {
+ arguments.insert(
+ "approval_token".to_owned(),
+ Value::String("[redacted]".to_owned()),
+ );
+ }
+ arguments.remove("_cua_browser_prepare_mcp_host_approved");
+ arguments.remove("_transport_session_id");
+ }
+ "browser_dialog" => {
+ if arguments.contains_key("prompt_text") {
+ arguments.insert(
+ "prompt_text".to_owned(),
+ Value::String("[redacted]".to_owned()),
+ );
+ }
+ }
+ "browser_set_input_files" => {
+ if let Some(count) = arguments
+ .get("files")
+ .and_then(Value::as_array)
+ .map(Vec::len)
+ {
+ arguments.insert("files".to_owned(), serde_json::json!({ "count": count }));
+ }
+ }
+ "browser_download" => {
+ arguments.remove("_cua_browser_download_mcp_host_approved");
+ if arguments.contains_key("destination_root") {
+ arguments.insert(
+ "destination_root".to_owned(),
+ Value::String("[redacted]".to_owned()),
+ );
+ }
+ }
+ "browser_pointer" => {}
+ _ => {}
}
- arguments.remove("_cua_browser_prepare_mcp_host_approved");
- arguments.remove("_transport_session_id");
}
redacted
}
@@ -584,6 +618,46 @@ mod capability_tests {
assert_eq!(recorded["strategy"]["kind"], "existing_profile");
}
+ #[test]
+ fn browser_sensitive_recording_args_are_path_and_text_free() {
+ let dialog = recording_args_for(
+ "browser_dialog",
+ &serde_json::json!({"action": "accept", "prompt_text": "private reply"}),
+ );
+ assert_eq!(dialog["prompt_text"], "[redacted]");
+
+ let upload = recording_args_for(
+ "browser_set_input_files",
+ &serde_json::json!({"files": ["/private/one", "/private/two"]}),
+ );
+ assert_eq!(upload["files"], serde_json::json!({"count": 2}));
+
+ let download = recording_args_for(
+ "browser_download",
+ &serde_json::json!({
+ "destination_root": "/private/destination",
+ "_cua_browser_download_mcp_host_approved": true,
+ }),
+ );
+ assert_eq!(download["destination_root"], "[redacted]");
+ assert!(download
+ .get("_cua_browser_download_mcp_host_approved")
+ .is_none());
+
+ let serialized = serde_json::json!([dialog, upload, download]).to_string();
+ for forbidden in [
+ "private reply",
+ "/private/one",
+ "/private/two",
+ "/private/destination",
+ ] {
+ assert!(
+ !serialized.contains(forbidden),
+ "recording leaked {forbidden}"
+ );
+ }
+ }
+
#[test]
fn existing_profile_prepare_is_a_private_consent_turn() {
assert!(is_existing_profile_prepare(
@@ -667,6 +741,10 @@ mod capability_tests {
"browser_navigate",
"browser_click",
"browser_type",
+ "browser_dialog",
+ "browser_set_input_files",
+ "browser_download",
+ "browser_pointer",
];
/// All capability tokens in the canonical vocabulary. Any token
@@ -739,6 +817,10 @@ mod capability_tests {
"browser.navigate",
"browser.input.click",
"browser.input.type",
+ "browser.input.files",
+ "browser.dialog",
+ "browser.download",
+ "browser.input.pointer",
// driver self
"driver.update_check",
"driver.probe",
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs
index e597dc09d3..eb9532d3c2 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/browser_fixture.rs
@@ -13,6 +13,7 @@ use std::time::Duration;
pub struct BrowserFixtureServer {
page_url: String,
journal_url: String,
+ download_url: String,
latest: Arc<Mutex<serde_json::Value>>,
observed: Arc<Mutex<Vec<serde_json::Value>>>,
stop: Arc<AtomicBool>,
@@ -30,6 +31,7 @@ impl BrowserFixtureServer {
.expect("read browser fixture server address");
let page_url = format!("http://{address}/fixture");
let journal_url = format!("http://{address}/state");
+ let download_url = format!("http://{address}/download");
let journal_script = format!(
"<script>window.__CUA_E2E_FIXTURE_JOURNAL_URL={};</script>",
serde_json::to_string(&journal_url).expect("serialize fixture journal URL")
@@ -73,6 +75,7 @@ impl BrowserFixtureServer {
Self {
page_url,
journal_url,
+ download_url,
latest,
observed,
stop,
@@ -100,6 +103,11 @@ impl BrowserFixtureServer {
&self.journal_url
}
+ /// Fixed loopback attachment used by browser download E2E rows.
+ pub fn download_url(&self) -> &str {
+ &self.download_url
+ }
+
pub fn contains(&self, marker: &str) -> bool {
self.latest
.lock()
@@ -214,6 +222,14 @@ fn handle_connection(
);
let _ = stream.write_all(headers.as_bytes());
let _ = stream.write_all(page);
+ } else if request_line.starts_with("GET /download ") {
+ const DOWNLOAD: &[u8] = b"CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1\n";
+ let headers = format!(
+ "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\nContent-Disposition: attachment; filename=fixture.txt\r\nContent-Length: {}\r\nCache-Control: no-store\r\nConnection: close\r\n\r\n",
+ DOWNLOAD.len()
+ );
+ let _ = stream.write_all(headers.as_bytes());
+ let _ = stream.write_all(DOWNLOAD);
} else if request_line.starts_with("POST /state ") {
if let Some((body_start, content_len)) = body_range {
if let Some(body) = request.get(body_start..body_start + content_len) {
@@ -261,10 +277,12 @@ mod tests {
#[test]
fn serves_fixture_and_receives_external_state() {
let server = BrowserFixtureServer::start("<html><head></head><body>marker</body></html>");
- let address = address(server.page_url(), "/fixture");
- let mut stream = TcpStream::connect(&address).expect("connect browser fixture");
+ let fixture_address = address(server.page_url(), "/fixture");
+ let mut stream = TcpStream::connect(&fixture_address).expect("connect browser fixture");
stream
- .write_all(format!("GET /fixture HTTP/1.1\r\nHost: {address}\r\n\r\n").as_bytes())
+ .write_all(
+ format!("GET /fixture HTTP/1.1\r\nHost: {fixture_address}\r\n\r\n").as_bytes(),
+ )
.expect("request browser fixture");
let mut response = String::new();
stream
@@ -274,12 +292,27 @@ mod tests {
assert!(response.contains("marker"));
assert!(response.contains(server.journal_url()));
+ let download_address = address(server.download_url(), "/download");
+ let mut stream =
+ TcpStream::connect(&download_address).expect("connect browser download fixture");
+ stream
+ .write_all(
+ format!("GET /download HTTP/1.1\r\nHost: {download_address}\r\n\r\n").as_bytes(),
+ )
+ .expect("request browser download fixture");
+ let mut response = String::new();
+ stream
+ .read_to_string(&mut response)
+ .expect("read browser download fixture");
+ assert!(response.contains("Content-Disposition: attachment"));
+ assert!(response.contains("CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1"));
+
let body = r#"{"status":{"text":"last_action=left_click"}}"#;
- let mut stream = TcpStream::connect(&address).expect("connect browser journal");
+ let mut stream = TcpStream::connect(&fixture_address).expect("connect browser journal");
stream
.write_all(
format!(
- "POST /state HTTP/1.1\r\nHost: {address}\r\nContent-Length: {}\r\n\r\n{body}",
+ "POST /state HTTP/1.1\r\nHost: {fixture_address}\r\nContent-Length: {}\r\n\r\n{body}",
body.len()
)
.as_bytes(),
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
index a636d86c70..45552e7bfd 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
@@ -31,7 +31,7 @@ fn is_active_proxy_session(session: Option<&str>) -> bool {
session.is_some_and(|session| active_proxy_sessions().lock().unwrap().contains(session))
}
-fn inject_browser_prepare_approval(
+fn inject_browser_approvals(
tool_name: &str,
args: &mut serde_json::Value,
session: Option<&str>,
@@ -44,6 +44,14 @@ fn inject_browser_prepare_approval(
);
}
}
+ if tool_name == "browser_download" && is_active_proxy_session(session) {
+ if let Some(arguments) = args.as_object_mut() {
+ arguments.insert(
+ cua_driver_core::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG.to_owned(),
+ serde_json::Value::Bool(true),
+ );
+ }
+ }
}
// ── Recording idle-TTL backstop (#1764) ─────────────────────────────────────────
@@ -482,7 +490,7 @@ async fn invoke_daemon_tool(
}
}
- inject_browser_prepare_approval(&tool_name, &mut args, req.session_id.as_deref());
+ inject_browser_approvals(&tool_name, &mut args, req.session_id.as_deref());
let session_context = observation_transport.and_then(|transport| {
let transport = match transport {
@@ -1843,8 +1851,9 @@ mod telemetry_routing_tests {
#[cfg(test)]
mod session_boundary_tests {
- use super::{active_proxy_sessions, apply_session_identity, inject_browser_prepare_approval};
+ use super::{active_proxy_sessions, apply_session_identity, inject_browser_approvals};
use cua_driver_core::browser::approval::MCP_HOST_APPROVAL_ARG;
+ use cua_driver_core::browser::download::MCP_HOST_DOWNLOAD_APPROVAL_ARG;
use serde_json::json;
#[test]
@@ -1899,7 +1908,7 @@ mod session_boundary_tests {
fn browser_prepare_approval_requires_a_live_proxy_session() {
let session = "approval-boundary-test";
let mut raw_args = json!({"pid": 42});
- inject_browser_prepare_approval("browser_prepare", &mut raw_args, Some(session));
+ inject_browser_approvals("browser_prepare", &mut raw_args, Some(session));
assert!(raw_args.get(MCP_HOST_APPROVAL_ARG).is_none());
active_proxy_sessions()
@@ -1907,7 +1916,7 @@ mod session_boundary_tests {
.unwrap()
.insert(session.to_owned());
let mut proxy_args = json!({"pid": 42});
- inject_browser_prepare_approval("browser_prepare", &mut proxy_args, Some(session));
+ inject_browser_approvals("browser_prepare", &mut proxy_args, Some(session));
active_proxy_sessions().lock().unwrap().remove(session);
assert_eq!(proxy_args[MCP_HOST_APPROVAL_ARG], true);
@@ -1916,8 +1925,21 @@ mod session_boundary_tests {
.lock()
.unwrap()
.insert(session.to_owned());
- inject_browser_prepare_approval("get_browser_state", &mut other_tool, Some(session));
+ inject_browser_approvals("get_browser_state", &mut other_tool, Some(session));
active_proxy_sessions().lock().unwrap().remove(session);
assert!(other_tool.get(MCP_HOST_APPROVAL_ARG).is_none());
+
+ let mut raw_download = json!({"destination_root": "/private/path"});
+ inject_browser_approvals("browser_download", &mut raw_download, Some(session));
+ assert!(raw_download.get(MCP_HOST_DOWNLOAD_APPROVAL_ARG).is_none());
+
+ active_proxy_sessions()
+ .lock()
+ .unwrap()
+ .insert(session.to_owned());
+ let mut proxy_download = json!({"destination_root": "/private/path"});
+ inject_browser_approvals("browser_download", &mut proxy_download, Some(session));
+ active_proxy_sessions().lock().unwrap().remove(session);
+ assert_eq!(proxy_download[MCP_HOST_DOWNLOAD_APPROVAL_ARG], true);
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
index b7335212f6..2d1e03a378 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/telemetry.rs
@@ -426,6 +426,10 @@ impl AgentSessionState {
| "browser_navigate"
| "browser_click"
| "browser_type"
+ | "browser_dialog"
+ | "browser_set_input_files"
+ | "browser_download"
+ | "browser_pointer"
);
let refused = outcome.refusal_code.is_refusal();
let completed_computer_action = computer_action && !refused;
@@ -2270,6 +2274,7 @@ mod tests {
#[test]
fn v3_payload_allows_server_geoip_without_sending_an_ip_or_client_timestamp() {
+ let _guard = ENV_LOCK.lock().unwrap();
let identity = InstallationIdentity { id: "test-id".into(), persisted: true };
let payload = build_payload(
event::MCP_TOOL_COMPLETED,
@@ -2568,6 +2573,55 @@ mod tests {
assert_eq!(session_properties["used_browser"], true);
}
+ #[test]
+ fn every_browser_tool_marks_the_session_without_retaining_arguments() {
+ use cua_driver_core::server::{
+ DurationBucket, OutputSizeBucket, OutputType, ToolCompletionObservation,
+ ToolErrorClass, ToolOperation, ToolRefusalCode,
+ };
+
+ for (tool_name, operation) in [
+ ("browser_dialog", ToolOperation::BrowserDialogAccept),
+ (
+ "browser_set_input_files",
+ ToolOperation::BrowserSetInputFiles,
+ ),
+ ("browser_download", ToolOperation::BrowserDownload),
+ (
+ "browser_pointer",
+ ToolOperation::BrowserPointerDoubleClickDomEvent,
+ ),
+ ] {
+ let mut state = AgentSessionState::new(Transport::McpStdio);
+ state.observe(
+ Transport::McpStdio,
+ true,
+ &ToolCompletionObservation {
+ tool_name: tool_name.into(),
+ operation,
+ computer_action: true,
+ success: true,
+ error_class: ToolErrorClass::None,
+ refusal_code: ToolRefusalCode::None,
+ duration_bucket: DurationBucket::Under10Ms,
+ output_type: OutputType::Text,
+ output_size_bucket: OutputSizeBucket::Under1KiB,
+ },
+ );
+ let properties =
+ state.ended_properties(cua_driver_core::session::SessionEndReason::Explicit, None);
+ assert_eq!(properties["used_browser"], true, "tool={tool_name}");
+ assert_eq!(
+ properties["had_successful_computer_action"], true,
+ "tool={tool_name}"
+ );
+ let serialized = serde_json::to_string(&properties).unwrap();
+ assert!(!serialized.contains("destination_root"));
+ assert!(!serialized.contains("prompt_text"));
+ assert!(!serialized.contains("file_paths"));
+ }
+ }
+
#[test]
fn agent_session_buckets_have_fixed_boundaries() {
for (count, expected) in [
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
index 3697cd733c..43701e107b 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs
@@ -45,6 +45,55 @@ fn standalone_fixture_html() -> String {
)
}
+fn standalone_browser_completeness_html() -> String {
+ standalone_fixture_html().replace(
+ "</body>",
+ r#"<fieldset>
+ <legend>browser completeness</legend>
+ <button id="standalone-alert" data-cua-id="standalone-alert">Open alert</button>
+ <button id="standalone-confirm" data-cua-id="standalone-confirm">Open confirm</button>
+ <button id="standalone-prompt" data-cua-id="standalone-prompt">Open prompt</button>
+ <span id="standalone-dialog-state" data-cua-id="standalone-dialog-state">dialog=none</span>
+ <input id="standalone-upload" data-cua-id="standalone-upload" aria-label="standalone-upload" type="file" multiple>
+ <span id="standalone-upload-state" data-cua-id="standalone-upload-state">upload=0</span>
+ <a id="standalone-download" data-cua-id="standalone-download" aria-label="standalone-download" href="/download" download>Download fixture</a>
+ <span id="standalone-hover-state" data-cua-id="standalone-hover-state">hover=false</span>
+</fieldset>
+<script>
+ const dialogState = document.getElementById('standalone-dialog-state');
+ document.getElementById('standalone-alert').addEventListener('click', () => {
+ setTimeout(() => { alert('fixture alert'); dialogState.textContent = 'dialog=alert:accepted'; }, 0);
+ });
+ document.getElementById('standalone-confirm').addEventListener('click', () => {
+ setTimeout(() => { dialogState.textContent = `dialog=confirm:${confirm('fixture confirm')}`; }, 0);
+ });
+ document.getElementById('standalone-prompt').addEventListener('click', () => {
+ setTimeout(() => { dialogState.textContent = `dialog=prompt:${prompt('fixture prompt', '')}`; }, 0);
+ });
+ document.getElementById('standalone-upload').addEventListener('change', event => {
+ const names = Array.from(event.target.files).map(file => file.name).join(',');
+ document.getElementById('standalone-upload-state').textContent = `upload=${event.target.files.length}:${names}`;
+ });
+ document.getElementById('scroll-tall').setAttribute('tabindex', '0');
+ document.getElementById('scroll-tall').setAttribute('role', 'region');
+ document.getElementById('drag-source').setAttribute('tabindex', '0');
+ document.getElementById('drag-source').setAttribute('role', 'button');
+ document.getElementById('drag-source').setAttribute('draggable', 'true');
+ document.getElementById('drop-target').setAttribute('tabindex', '0');
+ document.getElementById('drop-target').setAttribute('role', 'button');
+ document.getElementById('click-target').addEventListener('pointerover', () => {
+ document.getElementById('standalone-hover-state').textContent = 'hover=true';
+ });
+ document.getElementById('drop-target').addEventListener('dragover', event => event.preventDefault());
+ document.getElementById('drop-target').addEventListener('drop', event => {
+ event.preventDefault();
+ document.getElementById('drag-status').textContent = 'drag_status=dropped';
+ });
+</script>
+</body>"#,
+ )
+}
+
fn standalone_semantic_fixture_html() -> String {
let hidden = (0..320)
.map(|index| {
@@ -685,13 +734,16 @@ fn wait_for_exact_browser_binding(
"session": session,
}),
);
- if state.structured()["binding_quality"] == "exact" {
+ let has_unique_active_tab = state.structured()["tabs"]
+ .as_array()
+ .is_some_and(|tabs| tabs.iter().filter(|tab| tab["active"] == true).count() == 1);
+ if state.structured()["binding_quality"] == "exact" && has_unique_active_tab {
return Some((window_id, state));
}
}
if Instant::now() >= deadline {
eprintln!(
- "no exact browser window binding for prepared pid {pid}; last windows={}",
+ "no exact browser window binding with one active tab for prepared pid {pid}; last windows={}",
windows.raw
);
return None;
@@ -982,6 +1034,20 @@ fn refusal_case(browser: &str, action: &str, code: RefusalCode) -> CaseSpec {
case(browser, action).expecting_refusal(vec![code])
}
+fn foreground_page_case(browser: &str, action: &str) -> CaseSpec {
+ CaseSpec::delivered(
+ format!("{}-{browser}-standalone-{action}", std::env::consts::OS),
+ browser,
+ "standalone-chromium",
+ action,
+ Targeting::Page,
+ Delivery::Foreground,
+ Scope::Window,
+ DriverRoute::Cdp,
+ vec![OracleKind::FixtureState],
+ )
+}
+
fn run_with_background_oracles(
fixture: &mut BrowserFixture,
action: impl FnOnce(&mut BrowserFixture) -> Observation,
@@ -2134,6 +2200,485 @@ fn run_multi_tab(spec: &BrowserSpec) {
});
}
+fn run_same_title_tabs(spec: &BrowserSpec) {
+ let scenario = format!(
+ "{}-{}-standalone-same-title-tabs",
+ std::env::consts::OS,
+ spec.name
+ );
+ execute_case(
+ case(&spec.name, "same_title_tab_selection_unknown"),
+ |evidence| {
+ let mut fixture = launch_browser(spec, &scenario);
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ let session = format!("standalone-same-title-{}", fixture.pid);
+ let _ = bind(&mut fixture, &session);
+ let second_server = BrowserFixtureServer::start(&standalone_fixture_html());
+ let created = harness_cdp_call(
+ fixture.cdp_port,
+ "Target.createTarget",
+ serde_json::json!({
+ "url": second_server.page_url(),
+ "newWindow": false,
+ "background": true,
+ }),
+ );
+ assert!(created["targetId"].is_string(), "{created}");
+ wait_for_observed(&second_server, "WEB_HARNESS_MARKER_v1");
+ bring_harness_page_to_front(fixture.cdp_port, &fixture.server.page_url());
+
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let rebound = loop {
+ let state = fixture.driver.call(
+ "get_browser_state",
+ serde_json::json!({
+ "pid": fixture.pid as i64,
+ "window_id": fixture.window_id,
+ "session": session,
+ }),
+ );
+ if state.structured()["tabs"]
+ .as_array()
+ .is_some_and(|tabs| tabs.len() >= 2)
+ {
+ break state;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "same-title bind failed: {}",
+ state.raw
+ );
+ thread::sleep(Duration::from_millis(100));
+ };
+ assert_eq!(rebound.structured()["binding_quality"], "exact");
+ let tabs = rebound.structured()["tabs"]
+ .as_array()
+ .expect("same-title tab list");
+ assert!(tabs
+ .iter()
+ .any(|tab| tab["url"] == fixture.server.page_url()));
+ assert!(tabs
+ .iter()
+ .any(|tab| tab["url"] == second_server.page_url()));
+ assert!(
+ tabs.iter().all(|tab| tab["active"].is_null()),
+ "same-title selection must be unknown instead of guessed: {}",
+ rebound.raw
+ );
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ },
+ );
+}
+
+fn run_dialogs(spec: &BrowserSpec) {
+ let scenario = format!("{}-{}-standalone-dialogs", std::env::consts::OS, spec.name);
+ let spec_case = if cfg!(target_os = "linux") {
+ foreground_page_case(&spec.name, "browser_dialogs")
+ } else {
+ case(&spec.name, "browser_dialogs")
+ };
+ execute_case(spec_case, |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ let session = format!("standalone-dialogs-{}", fixture.pid);
+ let (target_id, tab_id, snapshot) = bind(&mut fixture, &session);
+ let primed = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "inspect",
+ "session": session,
+ }),
+ );
+ assert_eq!(primed.structured()["present"], false, "{}", primed.raw);
+
+ let sentinel = if cfg!(target_os = "linux") {
+ None
+ } else {
+ Some(ForegroundSentinel::launch(&mut fixture.driver))
+ };
+ let target = TargetWindow {
+ pid: fixture.pid,
+ native_id: fixture.window_id,
+ };
+ fixture.driver.start_behavior_recording();
+ let mut passed_oracles = vec![OracleKind::FixtureState];
+
+ for (button, kind, action, prompt_text, expected) in [
+ (
+ "id=standalone-alert",
+ "alert",
+ "accept",
+ None,
+ "dialog=alert:accepted",
+ ),
+ (
+ "id=standalone-confirm",
+ "confirm",
+ "dismiss",
+ None,
+ "dialog=confirm:false",
+ ),
+ (
+ "id=standalone-prompt",
+ "prompt",
+ "accept",
+ Some("fixture response"),
+ "dialog=prompt:fixture response",
+ ),
+ ] {
+ // Opening a native browser modal can activate Chromium. Treat that
+ // as fixture setup, then prove the typed inspect/resolve operation
+ // itself preserves a fully occluded background posture.
+ let clicked = fixture.driver.call(
+ "browser_click",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "ref": ref_by_label(&snapshot, button),
+ "input_route": "dom_event",
+ "session": session,
+ }),
+ );
+ assert_eq!(clicked.structured()["status"], "ok", "{}", clicked.raw);
+ thread::sleep(Duration::from_millis(100));
+
+ if cfg!(target_os = "linux") {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let current = loop {
+ let current = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "inspect",
+ "session": session,
+ }),
+ );
+ if current.structured()["present"] == true {
+ break current;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "{kind} dialog was not observed: {}",
+ current.raw
+ );
+ thread::sleep(Duration::from_millis(25));
+ };
+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);
+ let dialog_id = current.structured()["dialog_id"]
+ .as_str()
+ .expect("opaque dialog id")
+ .to_owned();
+ let mut args = serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": action,
+ "dialog_id": dialog_id,
+ "delivery_mode": "foreground",
+ "session": session,
+ });
+ if let Some(prompt_text) = prompt_text {
+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());
+ }
+ let handled = fixture.driver.call("browser_dialog", args);
+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);
+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);
+ continue;
+ }
+
+ sentinel
+ .as_ref()
+ .expect("background dialog sentinel")
+ .prepare_background_observation(&mut fixture.driver, target)
+ .expect("occlude the open JavaScript dialog");
+
+ let (_, passed) = sentinel
+ .as_ref()
+ .expect("background dialog sentinel")
+ .observe_background(target, || {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let current = loop {
+ let current = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "inspect",
+ "session": session,
+ }),
+ );
+ if current.structured()["present"] == true {
+ break current;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "{kind} dialog was not observed: {}",
+ current.raw
+ );
+ thread::sleep(Duration::from_millis(25));
+ };
+ assert_eq!(current.structured()["kind"], kind, "{}", current.raw);
+ let dialog_id = current.structured()["dialog_id"]
+ .as_str()
+ .expect("opaque dialog id")
+ .to_owned();
+ let mut args = serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": action,
+ "dialog_id": dialog_id,
+ "session": session,
+ });
+ if let Some(prompt_text) = prompt_text {
+ args["prompt_text"] = serde_json::Value::String(prompt_text.to_owned());
+ }
+ let handled = fixture.driver.call("browser_dialog", args);
+ assert_eq!(handled.structured()["status"], "ok", "{}", handled.raw);
+ wait_for_text(&fixture.server, "standalone-dialog-state", expected);
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ .expect("observe background JavaScript dialog resolution");
+ passed_oracles.extend(passed);
+ }
+
+ Observation::delivered(passed_oracles, Evidence::default())
+ });
+}
+
+#[cfg(target_os = "linux")]
+fn run_dialog_background_refusal(spec: &BrowserSpec) {
+ let scenario = format!(
+ "{}-{}-standalone-dialog-background-refusal",
+ std::env::consts::OS,
+ spec.name
+ );
+ execute_case(
+ refusal_case(
+ &spec.name,
+ "browser_dialogs_background",
+ RefusalCode::BrowserInputTrustUnavailable,
+ ),
+ |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-dialog-refusal-{}", fixture.pid);
+ let (target_id, tab_id, _) = bind(fixture, &session);
+ let refused = fixture.driver.call(
+ "browser_dialog",
+ serde_json::json!({
+ "target_id": target_id,
+ "tab_id": tab_id,
+ "action": "accept",
+ "dialog_id": "dialog-unavailable",
+ "session": session,
+ }),
+ );
+ assert_eq!(refused.structured()["status"], "refused", "{}", refused.raw);
+ assert_eq!(
+ refused.structured()["refusal"]["code"],
+ "browser_input_trust_unavailable",
+ "{}",
+ refused.raw
+ );
+ wait_for_text(&fixture.server, "standalone-dialog-state", "dialog=none");
+ Observation::refused(
+ RefusalCode::BrowserInputTrustUnavailable,
+ vec![OracleKind::FixtureState],
+ refused.text(),
+ Evidence::default(),
+ )
+ })
+ },
+ );
+}
+
+fn run_upload(spec: &BrowserSpec) {
+ let scenario = format!("{}-{}-standalone-upload", std::env::consts::OS, spec.name);
+ execute_case(case(&spec.name, "browser_file_upload"), |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-upload-{}", fixture.pid);
+ let (target, tab, _) = bind(fixture, &session);
+ let snapshot = fixture.driver.call(
+ "get_browser_state",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "session": session,
+ "snapshot_format": "semantic_v2",
+ "query": "standalone-upload",
+ }),
+ );
+ let upload_ref = semantic_ref_by_name(&snapshot, "standalone-upload", "upload");
+ let directory = tempfile::tempdir().expect("create upload fixture directory");
+ let path = directory.path().join("fixture-upload.txt");
+ std::fs::write(&path, b"fixture upload payload").expect("write upload fixture");
+ let canonical = std::fs::canonicalize(&path).expect("canonical upload fixture");
+ let uploaded = fixture.driver.call(
+ "browser_set_input_files",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": upload_ref,
+ "files": [canonical],
+ "session": session,
+ }),
+ );
+ assert_eq!(uploaded.structured()["status"], "ok", "{}", uploaded.raw);
+ assert_eq!(uploaded.structured()["file_count"], 1, "{}", uploaded.raw);
+ let public = uploaded.raw.to_string();
+ assert!(
+ !public.contains(&path.display().to_string()),
+ "{}",
+ uploaded.raw
+ );
+ assert!(!public.contains("fixture-upload.txt"), "{}", uploaded.raw);
+ wait_for_text(
+ &fixture.server,
+ "standalone-upload-state",
+ "upload=1:fixture-upload.txt",
+ );
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ });
+}
+
+fn run_pointer_actions(spec: &BrowserSpec) {
+ let scenario = format!("{}-{}-standalone-pointer", std::env::consts::OS, spec.name);
+ execute_case(case(&spec.name, "browser_pointer_actions"), |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-pointer-{}", fixture.pid);
+ let (target, tab, snapshot) = bind(fixture, &session);
+ let click_ref = ref_by_label(&snapshot, "id=click-target");
+
+ for action in ["hover", "right_click", "double_click"] {
+ let response = fixture.driver.call(
+ "browser_pointer",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": click_ref,
+ "action": action,
+ "input_route": "dom_event",
+ "session": session,
+ }),
+ );
+ assert_eq!(response.structured()["status"], "ok", "{}", response.raw);
+ }
+ wait_for_text(&fixture.server, "standalone-hover-state", "hover=true");
+ wait_for_text(
+ &fixture.server,
+ "lbl-last-action",
+ "last_action=double_click",
+ );
+
+ let scrolled = fixture.driver.call(
+ "browser_pointer",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": ref_by_label(&snapshot, "id=scroll-tall"),
+ "action": "scroll",
+ "input_route": "dom_event",
+ "delta_y": 240,
+ "session": session,
+ }),
+ );
+ assert_eq!(scrolled.structured()["status"], "ok", "{}", scrolled.raw);
+ let deadline = Instant::now() + Duration::from_secs(5);
+ loop {
+ if fixture
+ .server
+ .text("lbl-scroll-offset")
+ .is_some_and(|value| value != "scroll_offset=0")
+ {
+ break;
+ }
+ assert!(Instant::now() < deadline, "scroll oracle did not change");
+ thread::sleep(Duration::from_millis(25));
+ }
+
+ let dragged = fixture.driver.call(
+ "browser_pointer",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": ref_by_label(&snapshot, "id=drag-source"),
+ "destination_ref": ref_by_label(&snapshot, "id=drop-target"),
+ "action": "drag",
+ "input_route": "dom_event",
+ "session": session,
+ }),
+ );
+ assert_eq!(dragged.structured()["status"], "ok", "{}", dragged.raw);
+ wait_for_text(&fixture.server, "drag-status", "drag_status=dropped");
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ });
+}
+
+fn run_download(spec: &BrowserSpec) {
+ let scenario = format!("{}-{}-standalone-download", std::env::consts::OS, spec.name);
+ execute_case(case(&spec.name, "browser_download"), |evidence| {
+ let mut fixture =
+ launch_browser_with_html(spec, &scenario, standalone_browser_completeness_html());
+ *evidence = recording_evidence(fixture.driver.recording_dir());
+ run_with_background_oracles(&mut fixture, |fixture| {
+ let session = format!("standalone-download-{}", fixture.pid);
+ let (target, tab, snapshot) = bind(fixture, &session);
+ let destination = tempfile::tempdir().expect("create approved download directory");
+ let canonical =
+ std::fs::canonicalize(destination.path()).expect("canonical download directory");
+ let downloaded = fixture.driver.call(
+ "browser_download",
+ serde_json::json!({
+ "target_id": target,
+ "tab_id": tab,
+ "ref": ref_by_label(&snapshot, "id=standalone-download"),
+ "destination_root": canonical,
+ "session": session,
+ }),
+ );
+ assert_eq!(
+ downloaded.structured()["status"],
+ "completed",
+ "{}",
+ downloaded.raw
+ );
+ let entries = std::fs::read_dir(destination.path())
+ .expect("read approved download directory")
+ .collect::<Result<Vec<_>, _>>()
+ .expect("enumerate approved download directory");
+ assert_eq!(entries.len(), 1, "expected one completed download");
+ let bytes = std::fs::read(entries[0].path()).expect("read completed download");
+ assert_eq!(bytes, b"CUA_DRIVER_BROWSER_DOWNLOAD_FIXTURE_V1\n");
+ assert_eq!(
+ downloaded.structured()["bytes"],
+ bytes.len() as u64,
+ "{}",
+ downloaded.raw
+ );
+ let public = downloaded.raw.to_string();
+ assert!(!public.contains(&destination.path().display().to_string()));
+ assert!(!public.contains("fixture.txt"));
+ assert!(!public.contains(fixture.server.download_url()));
+ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())
+ })
+ });
+}
+
fn run_two_window_collision(spec: &BrowserSpec) {
let scenario = format!(
"{}-{}-standalone-two-window",
@@ -2261,6 +2806,16 @@ standalone_browser_test!(
standalone_browser_test!(standalone_browser_stale_ref, run_stale_ref);
standalone_browser_test!(standalone_browser_frames, run_frame_roundtrip);
standalone_browser_test!(standalone_browser_multi_tab, run_multi_tab);
+standalone_browser_test!(standalone_browser_same_title_tabs, run_same_title_tabs);
+standalone_browser_test!(standalone_browser_dialogs, run_dialogs);
+#[cfg(target_os = "linux")]
+standalone_browser_test!(
+ standalone_browser_dialog_background_refusal,
+ run_dialog_background_refusal
+);
+standalone_browser_test!(standalone_browser_upload, run_upload);
+standalone_browser_test!(standalone_browser_pointer_actions, run_pointer_actions);
+standalone_browser_test!(standalone_browser_download, run_download);
standalone_browser_test!(
standalone_browser_window_collision,
run_two_window_collision
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
index 063923db88..1494da8fa2 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
@@ -930,7 +930,19 @@ pub fn x11_activate_window_persistent(xid: u64) -> Result<Option<u64>> {
prior.unwrap_or(0) as x11::xlib::Window,
);
unsafe {
+ // Some WMs honor `_NET_ACTIVE_WINDOW` as raise-only. Persistent
+ // activation must establish input focus too, just like the bounded
+ // foreground rung above, or `bring_to_front` can report success while
+ // keyboard focus remains on the previous window.
+ let previous_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));
+ x11::xlib::XSetInputFocus(
+ display,
+ xid as x11::xlib::Window,
+ x11::xlib::RevertToParent,
+ x11::xlib::CurrentTime,
+ );
x11::xlib::XSync(display, 0);
+ x11::xlib::XSetErrorHandler(previous_handler);
x11::xlib::XCloseDisplay(display);
}
Ok(prior)
diff --git a/scripts/ci/run-rust-standalone-browser-e2e.sh b/scripts/ci/run-rust-standalone-browser-e2e.sh
index 8b784c2af3..3289edcd6e 100755
--- a/scripts/ci/run-rust-standalone-browser-e2e.sh
+++ b/scripts/ci/run-rust-standalone-browser-e2e.sh
@@ -90,15 +90,24 @@ fi
tests=(
standalone_browser_background_type
+ standalone_browser_dialogs
+)
+if [[ "${HOST_OS}" == Linux ]]; then
+ tests+=(standalone_browser_dialog_background_refusal)
+fi
+tests+=(
+ standalone_browser_download
standalone_browser_existing_profile
standalone_browser_existing_profile_setup
standalone_browser_frames
standalone_browser_multi_tab
+ standalone_browser_pointer_actions
standalone_browser_prepare_isolated
standalone_browser_roundtrip
standalone_browser_semantic_state
standalone_browser_stale_ref
standalone_browser_trusted_click
+ standalone_browser_upload
standalone_browser_window_collision
)
failure_count=0
diff --git a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1 b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
index 6cb111ac86..cc3417b43e 100644
--- a/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
+++ b/scripts/ci/windows/run-rust-standalone-browser-e2e.ps1
@@ -117,15 +117,19 @@ if (-not (Test-Path $env:CUA_TEST_DRIVER_BIN)) {
$tests = @(
"standalone_browser_background_type",
+ "standalone_browser_dialogs",
+ "standalone_browser_download",
"standalone_browser_existing_profile",
"standalone_browser_existing_profile_setup",
"standalone_browser_frames",
"standalone_browser_multi_tab",
+ "standalone_browser_pointer_actions",
"standalone_browser_prepare_isolated",
"standalone_browser_roundtrip",
"standalone_browser_semantic_state",
"standalone_browser_stale_ref",
"standalone_browser_trusted_click",
+ "standalone_browser_upload",
"standalone_browser_window_collision"
)
$failureCount = 0
diff --git a/scripts/docs-generators/cua-driver.ts b/scripts/docs-generators/cua-driver.ts
index 098486d0ce..59ae169cc3 100644
--- a/scripts/docs-generators/cua-driver.ts
+++ b/scripts/docs-generators/cua-driver.ts
@@ -76,6 +76,8 @@ export interface MCPPropertyDoc {
type: string;
description: string;
items?: { type: string };
+ enum?: string[];
+ minItems?: number;
}
export interface MCPDocumentation {
@@ -773,6 +775,9 @@ function formatPropertyType(prop: MCPPropertyDoc): string {
}
function syntheticExampleValue(name: string, prop: MCPPropertyDoc): unknown {
+ if (prop.enum?.length) {
+ return prop.enum[0];
+ }
switch (prop.type) {
case 'integer':
if (name === 'pid') return 844;
@@ -794,6 +799,7 @@ function syntheticExampleValue(name: string, prop: MCPPropertyDoc): unknown {
case 'array':
if (name === 'keys') return ['cmd', 'c'];
if (name === 'modifiers') return ['cmd'];
+ if (name === 'files' || (prop.minItems ?? 0) > 0) return ['example'];
return [];
case 'string':
if (name === 'text') return 'hello';