Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): add semantic browser snapshots
trycua-release[bot]1a9fbb480605
diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdxindex e758c38145..24f2d827db 100644--- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx+++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx@@ -166,13 +166,38 @@ Snapshot the selected tab before using an element ref:get_browser_state({"target_id": "<target_id>","tab_id": "<tab_id>",- "session": "research-1"+ "session": "research-1",+ "snapshot_format": "semantic_v2"})```-The response contains interactive elements with refs such as `p1:7`. Refs are-valid only for that session, target, tab, and latest snapshot. Take a fresh-snapshot after navigation or whenever a call returns `browser_ref_stale`.+Read `outline` for visible page content. Select actions only from `refs`, and+check that the chosen entry declares `click` or `type` in `actions`.+`content_refs` are read capabilities for `scope_ref`; they cannot be passed to+an unsupported mutation.++If `snapshot.continuation` is non-null, request the next ranked segment without+changing the page:++```jsonc+get_browser_state({+ "target_id": "<target_id>",+ "tab_id": "<tab_id>",+ "session": "research-1",+ "snapshot_format": "semantic_v2",+ "continuation": "<opaque_continuation>"+})+```++Use `query` to find matching roles, accessible names, or visible text. Use a+current `scope_ref` to inspect one semantic subtree when names repeat in+different regions. A continuation is single-use; a newer snapshot invalidates+it.++Refs are valid only for that session, target, tab, document, frame, and latest+snapshot. Take a fresh snapshot after navigation or whenever a call returns+`browser_ref_stale`. `dom_refs_v1` remains available for compatibility when+`snapshot_format` is omitted, but new workflows should request `semantic_v2`.## Click and typediff --git a/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdxnew file mode 100644index 0000000000..131525af32--- /dev/null+++ b/docs/content/docs/reference/cua-driver/browser-semantic-snapshots.mdx@@ -0,0 +1,154 @@+---+title: Browser semantic snapshots+description: Reference for semantic_v2 browser state, refs, scopes, and continuation capabilities.+---++`get_browser_state` snapshot mode supports two versioned contracts:++| Format | Status | Output |+| --- | --- | --- |+| `semantic_v2` | Recommended for new browser workflows | Semantic outline, typed action refs, content refs, visibility, omission counts, and continuation |+| `dom_refs_v1` | Compatibility default when `snapshot_format` is omitted | Interactive DOM refs and a `truncated` flag |++Bind a native browser window first. Snapshot mode requires the returned opaque+`target_id`, one returned `tab_id`, and the same explicit driver `session`.++## Request++```json+{+ "session": "browser-run-1",+ "target_id": "bt-...",+ "tab_id": "tab-...",+ "snapshot_format": "semantic_v2"+}+```++Semantic snapshot requests may also contain one of these read scopes:++| Field | Meaning |+| --- | --- |+| `query` | Match role, accessible name, or visible text and retain semantic ancestor context |+| `scope_ref` | Read the subtree rooted at a current action ref or content ref |+| `continuation` | Read the next ranked segment from the same stored snapshot |++`continuation` cannot be combined with a new `query` or `scope_ref`.++## Response++```json+{+ "status": "ok",+ "mode": "snapshot",+ "target_id": "bt-...",+ "tab_id": "tab-...",+ "snapshot": {+ "id": "p42",+ "format": "semantic_v2",+ "complete": false,+ "scope": "viewport",+ "selected_nodes": 300,+ "total_nodes": 318,+ "node_budget": 300,+ "omitted": {+ "css_hidden": 410,+ "offscreen": 82,+ "page_occluded": 1,+ "no_layout": 0,+ "unknown": 0,+ "budget": 18,+ "unprovable_frame": 0+ },+ "continuation": "bc-..."+ },+ "page": {+ "url": "https://example.test/inbox",+ "title": "Inbox"+ },+ "outline": "- heading \"Message\"\n- textbox \"Reply body\"",+ "refs": [],+ "content_refs": [],+ "oopif": {+ "status": "attached",+ "frames": 1+ }+}+```++`complete` is true only when collection is complete and no ranked state remains+behind the output budget. A non-null continuation identifies that remaining+state. Known oversized-DOM fallback produces a partial snapshot with+`complete: false`; unrelated transport failures remain errors.++## Action refs++Entries in `refs` declare their supported action kinds:++```json+{+ "ref": "p42:8",+ "role": "button",+ "name": "Reply",+ "value": null,+ "states": { "disabled": false },+ "actions": ["click"],+ "frame": "main",+ "visibility": "in_viewport"+}+```++`browser_click` and `browser_type` reject a semantic ref when the requested+action is absent from `actions`, returning `browser_action_unavailable` before+delivery. Legacy `dom_refs_v1` refs retain their existing behavior.++## Content refs++`content_refs` identify readable semantic nodes that have a live backend node+but no declared mutation. They exist so a caller can pass `scope_ref` without+making static text clickable or editable.++## Visibility++`visibility` has this closed vocabulary:++| Value | Meaning |+| --- | --- |+| `in_viewport` | Layout bounds intersect the current page viewport |+| `near_viewport` | Layout bounds are within the bounded near-viewport margin |+| `offscreen` | Layout exists outside that margin |+| `css_hidden` | DOM or computed style proves the node hidden |+| `no_layout` | No non-empty layout box is available |+| `page_occluded` | A higher-painted fixed or absolute page overlay conservatively covers the node |+| `unknown` | Available evidence cannot classify layout visibility |++Page visibility is independent of native window foreground or desktop+occlusion. It does not indicate whether the browser window is covered by+another application.++## Capability lifetime++Targets, tabs, snapshots, refs, and continuations are opaque capabilities.+They do not expose CDP target IDs, backend node IDs, object IDs, or selectors.++- A newer snapshot of the tab invalidates older refs and continuations.+- Navigation invalidates every snapshot of the tab.+- Browser reconnect or process replacement invalidates the target generation.+- Ending the driver session removes the complete capability namespace.+- Continuations are single-use and resolve only in their owning session,+ target, tab, snapshot, and generation.++Every mutation re-proves the native binding, endpoint ownership, tab target,+frame loader identity, and backend-node liveness.++## Composition++Semantic snapshots join the browser accessibility tree, pierced author DOM,+layout snapshot, viewport metrics, and frame tree. They include the main frame,+author shadow roots, proven same-process frames, and capability-tested+out-of-process frames. User-agent shadow roots and unprovable frame content are+omitted.++CSS-hidden retained state is removed before ranking. Active dialogs and focused+context rank first, followed by visible actions, visible content,+near-viewport state, and offscreen state. Document order is stable within each+tier.diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 8f636b60ec..cc7ce61231 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -746,12 +746,16 @@ Install the ffmpeg binary used by start_recording's video capture (Linux/Windows### `get_browser_state`-Read-only browser inspection. Mode 1 (bind): pass pid + window_id of a native browser window to classify it, correlate it to a CDP target (exact-or-refuse), and mint a session-scoped target id plus tab ids. Mode 2 (snapshot): pass target_id + tab_id to snapshot the tab's composed DOM (main frame, shadow DOM, same-process iframes, and capability-tested out-of-process iframes) and mint p<snapshot>:<index> element refs for browser_click / browser_type; each ref reports its frame kind. Never performs setup — a missing endpoint is a structured browser_requires_setup refusal pointing at browser_prepare.+Read-only browser inspection. Mode 1 (bind): pass pid + window_id of a native browser window to classify it, correlate it to a CDP target (exact-or-refuse), and mint a session-scoped target id plus tab ids. Mode 2 (snapshot): pass target_id + tab_id. The dom_refs_v1 compatibility format returns composed DOM refs. semantic_v2 joins accessibility, DOM, layout, and viewport state; ranks visible content before retained/offscreen state; and returns a semantic outline, typed action refs, content refs, scoped reads, and opaque continuation. Never performs setup — a missing endpoint is a structured browser_requires_setup refusal pointing at browser_prepare.**Arguments:**+- `continuation` (string, optional): Opaque continuation minted by an earlier semantic_v2 response.- `pid` (integer, optional): Native browser process id (bind mode).+- `query` (string, optional): Read-only semantic match over role, accessible name, and visible text.+- `scope_ref` (string, optional): Current semantic/content ref whose subtree should be observed.- `session` (string, optional): Stable caller-declared session id. Browser targets, tabs, and refs are scoped to this session.+- `snapshot_format` (string, optional): Versioned snapshot contract. dom_refs_v1 remains the compatibility default.- `tab_id` (string, optional): Opaque tab id from get_browser_state (session-scoped).- `target_id` (string, optional): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).- `window_id` (integer, optional): Native window id owned by pid (bind mode).diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.jsonindex 7571af6a89..1caf6010cb 100644--- a/docs/content/docs/reference/cua-driver/meta.json+++ b/docs/content/docs/reference/cua-driver/meta.json@@ -8,6 +8,7 @@"mcp-tools","mcp-tool-notes","browser-profile-attachment",+ "browser-semantic-snapshots","action-selection-policy","permission-policies","contracts",diff --git a/libs/cua-driver/docs/browser-semantic-state-plan.md b/libs/cua-driver/docs/browser-semantic-state-plan.mdnew file mode 100644index 0000000000..7b8e39d8cf--- /dev/null+++ b/libs/cua-driver/docs/browser-semantic-state-plan.md@@ -0,0 +1,519 @@+# Browser Semantic State Plan++## Status++- **State:** Implemented; cross-platform acceptance in progress+- **Plan date:** 2026-07-17+- **Tracking issue:** [#2299](https://github.com/trycua/cua/issues/2299)+- **Scope:** Rust browser state collection, typed browser actions, shared web test harnesses, and cross-platform acceptance+- **Documentation type:** Internal engineering plan. User-facing reference and how-to documentation are separate deliverables.++The implementation adds `semantic_v2` without changing the `dom_refs_v1`+compatibility default. Deterministic core coverage, the public reference and+how-to pages, the installed skill, and a real standalone-browser harness row+are in place. The final acceptance gate is the exact-SHA Windows and Linux X11+workflow plus the source-installed macOS replay described below.++## Objective++Make `get_browser_state` return the page content and controls that matter in the+current browser view, even when a large application retains hundreds of hidden,+offscreen, or stale nodes. After exact browser attachment, an agent should read+and operate webpage content through browser tools. Native AX and PX remain for+browser chrome, permission prompts, downloads, file pickers, and explicit+fallbacks.++The implementation must preserve these existing guarantees:++- native `pid` and `window_id` remain the binding anchor;+- browser targets, tabs, snapshots, and refs remain session-scoped capabilities;+- every mutation re-proves browser generation, frame identity, document+ identity, and backend-node liveness;+- `get_browser_state` remains read-only;+- page actions remain background-safe when the browser engine supports them;+- ambiguous state produces a structured refusal.++## Current Failure++The collector in `cua-driver-core/src/browser/engine.rs` calls+`DOM.getDocument`, walks the pierced DOM in document order, marks nodes as+interactive from broad tag and attribute rules, and truncates the result to 300+refs after collection.++This causes four problems:++1. Hidden or retained application subtrees can consume the complete ref budget.+2. Visible content has no priority over offscreen content.+3. Static page text is absent because the response contains actionable refs+ only.+4. Truncation reports a boolean but gives the caller no way to inspect omitted+ state.++Increasing the cap would increase output size while preserving the ordering+fault.++## Design Principles++### Observe before acting++`get_browser_state` produces deterministic state and capabilities. It does not+run a model, choose an action, mutate the page, or repair a failed workflow.+Callers inspect state, select a current ref, invoke a typed action, and inspect+state again.++### Separate content from actions++Readable page content and actionable elements are different data sets:++- the semantic outline contains headings, text, landmarks, form state, dialogs,+ and scroll context;+- the ref map contains only elements that support a declared browser action;+- static text may carry an opaque read identifier for scoping, but it must not+ be accepted by `browser_click` or `browser_type`.++### Rank before limiting++The collector classifies and ranks nodes before applying output budgets. The+default order is:++1. active modal, dialog, focused element, and their semantic context;+2. actionable elements inside the current viewport;+3. readable content inside the current viewport;+4. actionable elements near the viewport;+5. readable content near the viewport;+6. remaining laid-out content when requested through continuation or scope.++Document order is retained within each tier.++### Page visibility is not desktop visibility++Browser layout visibility describes whether page content is rendered in or near+the tab viewport. It does not depend on whether the native browser window is+foreground, covered by another window, or outside the visible desktop. This+distinction preserves full-background browser actions.++### Keep policy above transport details++The driver may cache protocol connections and data used within one snapshot. It+must not cache natural-language action plans or replay old selectors. Workflow+repetition and self-repair belong in the calling agent. Cua Driver continues to+require a current capability for each mutation.++## Snapshot Architecture++### Inputs collected per tab++Collect the following browser protocol state concurrently where dependencies+allow:++| Source | Purpose |+| --- | --- |+| Frame tree | Frame topology, loader identity, and parent relationships |+| Accessibility tree | Roles, names, values, states, readable text, and ignored-node pruning |+| Pierced DOM | Backend node IDs, tags, attributes, author shadow roots, and same-process frame documents |+| Layout snapshot | Computed visibility, bounds, viewport relationship, and paint order |+| Layout metrics | Viewport dimensions, scroll offsets, and device scale |++Build one DOM index for each unique CDP session during a snapshot. Same-process+frames share their session index. Out-of-process frames use the existing+capability-tested child-session route.++### Semantic node model++Add an internal `SemanticNode` model with at least:++```text+frame_ref+backend_node_id?+role+name?+value?+states+parent+children+document_order+visibility+action_kinds+```++`visibility` is a closed enum:++```text+in_viewport+near_viewport+offscreen+css_hidden+no_layout+page_occluded+unknown+```++Treat `page_occluded` conservatively. If paint-order evidence is incomplete,+use `unknown` and keep the node. Never infer CSS or page occlusion from native+window occlusion.++### Accessibility-first semantics++Use the accessibility tree for the semantic outline because it already contains+roles, names, values, state, and readable text. Remove ignored and redundant+structural nodes while preserving useful ancestors. Collapse repeated static+text only when the parent already carries the same accessible name.++Some custom controls do not appear as actionable AX nodes. Add a bounded DOM+supplement for elements with clear interaction evidence:++- native interactive tags;+- supported interactive ARIA roles;+- explicit event listeners or handlers;+- editable content;+- pointer cursor with non-zero layout bounds.++The supplement must reject `aria-hidden`, `hidden`, `display:none`,+`visibility:hidden`, zero opacity, and inherited pointer-cursor noise.++### Frame and shadow composition++Preserve the current frame security model:++- every actionable entry records `FrameRef` and document identity;+- same-process iframe entries require a frame-tree identity;+- out-of-process iframe entries require a contained child target and proven+ identity;+- author shadow roots are composed into their host frame;+- user-agent shadow roots remain excluded;+- unprovable frame content is omitted with a reason and count.++Avoid repeated full DOM reads per frame. Index each unique protocol session once+and slice frame subtrees from that index.++### Bounded protocol fallback++Large pages can fail an unbounded `DOM.getDocument` call. Use a bounded fallback:++1. request the full pierced tree;+2. retry with progressively shallower depth only for known depth or serialization+ failures;+3. hydrate missing branches with bounded `DOM.describeNode` calls;+4. enforce time, node, frame, and hydration-call budgets;+5. report partial coverage explicitly when a complete tree cannot be proven.++Transient transport failures remain errors. They must not be treated as a+capability gap or partial success.++## Public Contract++### Versioning++Add a versioned snapshot request without breaking existing clients:++```json+{+ "session": "research-1",+ "target_id": "bt-...",+ "tab_id": "tab-...",+ "snapshot_format": "semantic_v2"+}+```++Keep the existing response available as `dom_refs_v1` during migration. The+skill and harnesses move to `semantic_v2` before it becomes the default. Remove+the old format only through the normal deprecation process.++### Default response++The new response contains:++```json+{+ "status": "ok",+ "mode": "snapshot",+ "snapshot": {+ "id": "p42",+ "format": "semantic_v2",+ "complete": false,+ "scope": "viewport",+ "omitted": {+ "css_hidden": 410,+ "offscreen": 82,+ "unprovable_frame": 0+ },+ "continuation": "opaque-capability-or-null"+ },+ "page": {+ "url": "https://fixture.invalid/inbox",+ "title": "Inbox",+ "focused_ref": "p42:8"+ },+ "outline": "...compact semantic tree...",+ "refs": []+}+```++The outline is compact text for model consumption. `refs` remains structured+JSON for deterministic action routing. The response must not expose raw CDP+target IDs, backend node IDs, object IDs, or selectors.++### Actionable refs++Each ref reports enough public information to choose a typed action:++```json+{+ "ref": "p42:8",+ "role": "button",+ "name": "Reply",+ "states": { "disabled": false },+ "actions": ["click"],+ "frame": "main",+ "visibility": "in_viewport"+}+```++Internal storage keeps the existing backend node and frame evidence. Mutation+tools reject a ref when the requested action is absent from `actions`.++### Scoped observation++Support three read-only scope mechanisms in this order:++1. `scope_ref`: inspect the subtree rooted at a current semantic or actionable+ ref;+2. `query`: return role, accessible-name, and visible-text matches with ancestor+ context;+3. `continuation`: inspect the next ranked segment from the same live snapshot+ generation.++All scope tokens are opaque, session-bound, tab-bound, and generation-bound.+Do not expose XPath or CSS as the primary agent contract. A later expert-only+selector field may be considered after the capability path is accepted.++### Truncation and budgets++Replace the single `truncated` boolean with:++- whether collection was complete;+- which visibility tiers were omitted;+- counts by omission reason;+- the applied node and output budgets;+- an opaque continuation capability when more proven state is available.++Budgets are configuration constants in the first implementation. Public numeric+overrides wait until performance and abuse bounds are known.++## Code Changes++### `cua-driver-core/src/browser/engine.rs`++- split protocol collection from semantic composition;+- collect frame, AX, DOM, layout, and viewport state;+- rank semantic nodes before output limiting;+- mint refs only for declared action kinds;+- preserve OOPIF containment and cleanup;+- return coverage and omission diagnostics.++### `cua-driver-core/src/browser/store.rs`++- store snapshot format and generation;+- add action kinds to `RefEntry`;+- store opaque continuation and scope capabilities;+- keep static semantic nodes outside the actionable ref map;+- invalidate every derived capability on navigation, reconnect, newer snapshot,+ session end, and target replacement.++### `cua-driver-core/src/browser/tools.rs`++- add `snapshot_format`, `scope_ref`, `query`, and `continuation` to snapshot+ mode;+- return semantic outline, structured refs, and coverage diagnostics;+- keep bind mode unchanged;+- remove the duplicate `target_id` field in the current snapshot response;+- keep MCP read-only annotations accurate.++### Browser transport++- add typed protocol response models for AX and layout snapshots where practical;+- classify known depth and serialization failures separately from transport+ failures;+- make concurrent calls cancellation-safe;+- emit timing and count metrics without page text.++### Skills and documentation++- update `Skills/cua-driver/BROWSER.md` to use `semantic_v2`;+- teach the snapshot, choose-current-ref, act, re-snapshot loop;+- explain scoped reads and continuation;+- state that webpage content should stay on browser tools after attachment;+- retain `get_window_state` for browser chrome and native fallbacks;+- add user reference documentation only after the schema is accepted.++## Test Strategy++### Shared deterministic fixture++Extend the repo-local web harness with a large application fixture containing:++- more than 300 hidden or retained controls before the active view;+- a virtualized list and a selected detail view;+- visible heading, message body, editable reply field, and send button;+- a modal overlay that covers controls beneath it;+- nested author shadow DOM;+- same-process and out-of-process iframes;+- dynamic rerender and navigation controls;+- duplicated names in separate regions;+- offscreen controls reachable through continuation or scope.++Use fixture data only. Do not use customer domains, account names, email text,+profile paths, or browser history in source, logs, screenshots, or artifacts.++### Unit tests++Add tests for:++- AX role, name, state, and static-text normalization;+- CSS and layout visibility classification;+- ranking stability and document-order ties;+- action-kind assignment;+- hidden-node exclusion;+- DOM supplement deduplication;+- frame and shadow composition;+- bounded depth fallback and partial-coverage reporting;+- output budgets and continuation invalidation;+- query and scope ambiguity;+- stale refs after navigation, rerender, reconnect, and newer snapshots.++### Browser E2E rows++Run the same source-built Rust rows on each accepted browser and platform:++| Scenario | Foreground posture | Background posture | Required evidence |+| --- | --- | --- | --- |+| Read visible detail text | yes | yes | semantic outline contains fixture text |+| Click visible action | yes | yes | fixture journal changes once |+| Type into visible editor | yes | yes | exact delivered text and no leaked input |+| Hidden-node pressure | yes | yes | visible controls survive the budget |+| Scoped duplicate-name action | yes | yes | only the scoped region changes |+| Modal page occlusion | yes | yes | covered control omitted or marked occluded |+| Continuation | yes | yes | offscreen control becomes addressable |+| Rerender staleness | yes | yes | old ref refuses, new ref succeeds |+| Frame and shadow action | yes | yes | exact contained frame mutates |++Background rows run with the browser natively occluded and with focus, cursor,+and leaked-input sentinels active. The page viewport stays unchanged so browser+layout visibility remains comparable between postures.++### Platform gates++- **macOS:** current host or accepted VM, local source install, browser consent+ already granted through the documented flow.+- **Windows:** interactive GitHub-hosted runner or Azure RDP desktop, never+ session 0.+- **Linux X11:** interactive desktop with the standard browser harness.+- **Linux Wayland:** accepted compositor with exact browser binding; unsupported+ compositor identity remains a structured refusal.++The semantic collector should produce equivalent page results across operating+systems because it runs above the native platform layer. Platform-specific+differences are limited to attachment, endpoint proof, consent, and native+sentinels.++## Delivery Phases++### Phase 0: Reproduce and measure++- add the hidden-node-pressure fixture;+- prove the current collector omits visible content;+- record protocol time, node counts, response bytes, and output lines;+- add the failing test without changing its expectation to fit current behavior.++**Gate:** the fixture reproduces the defect deterministically on a source build.++### Phase 1: Semantic outline++- add the AX-based semantic model;+- return readable page content separately from actionable refs;+- keep the existing DOM ref collector available as v1;+- add action-kind checks.++**Gate:** visible detail text and editor state are available without native AX.++### Phase 2: Layout and ranking++- join layout snapshot and viewport metrics;+- classify visibility;+- apply visible-first ranking and coverage diagnostics;+- add conservative page-occlusion handling.++**Gate:** hidden-node pressure no longer displaces visible state.++### Phase 3: Scope and continuation++- add `scope_ref`, semantic query, and opaque continuation;+- bind each capability to the current session, target, tab, and generation;+- reject stale and ambiguous scopes.++**Gate:** duplicate names and offscreen content are addressable without raw+selectors or an unbounded response.++### Phase 4: Frame and scale hardening++- share DOM indexes by protocol session;+- add bounded depth fallback and hydration;+- verify same-process frames, out-of-process frames, and shadow roots;+- set accepted time, node, frame, memory, and response budgets.++**Gate:** large multi-frame fixtures finish within the accepted budgets and+report partial coverage truthfully when a budget is exhausted.++### Phase 5: Cross-platform acceptance++- update the skill and protocol schema tests;+- run foreground and background harness rows on macOS, Windows, X11, and the+ accepted Wayland compositor;+- publish matrix summaries, traces, and fixture-only videos;+- verify logs and artifacts contain no page text from non-fixture sessions.++**Gate:** all supported rows pass or produce their documented structured+refusal, and the old snapshot format remains compatible.++### Phase 6: Default migration++- make `semantic_v2` the skill default;+- monitor size, latency, refusal, and stale-ref metrics;+- document the old format deprecation window;+- remove v1 only after accepted clients and harnesses have migrated.++**Gate:** one release cycle completes with no unresolved compatibility or+privacy regression.++## Definition of Done++The project is complete when:++1. A source-built Cua Driver can attach to an exact browser tab and read a large+ application view without native AX inspection.+2. Visible text and actions remain available when more than 300 hidden or+ retained nodes precede them.+3. Static content and actionable refs are separate, and mutation tools enforce+ declared action kinds.+4. Scope and continuation are session-bound capabilities with tested stale+ behavior.+5. Frame, shadow, navigation, rerender, and reconnect tests preserve the exact+ mutation contract.+6. Foreground and background E2E rows pass on every advertised browser route for+ macOS, Windows, X11, and accepted Wayland environments.+7. The skill, protocol schema, support reference, CI matrix, and release notes+ describe the shipped behavior.+8. Repository scans and artifact review find no customer identifiers, account+ content, local profile paths, endpoint tokens, or raw browser IDs.++## Explicit Non-Goals++- model inference inside Cua Driver;+- natural-language `browser_act` or `browser_extract` tools;+- automatic replay or self-healing of old action plans;+- exposing XPath, CSS selectors, backend node IDs, or CDP target IDs as the+ primary public contract;+- treating native window visibility as page-element visibility;+- making arbitrary static text clickable;+- claiming complete state when protocol or budget limits produced a partial+ snapshot.diff --git a/libs/cua-driver/docs/browser-tool-implementation-journal.md b/libs/cua-driver/docs/browser-tool-implementation-journal.mdindex 3c77980dff..c3f68cd7a3 100644--- a/libs/cua-driver/docs/browser-tool-implementation-journal.md+++ b/libs/cua-driver/docs/browser-tool-implementation-journal.md@@ -677,8 +677,45 @@ finished, produced its summary, exited from the process table, recorded all 10passing rows, and generated 10 valid videos. The acceptance verdict is based onthose canonical artifacts rather than the disposable wrapper marker.-Focused local validation at the same source SHA passed all 46 testkit library-tests. Branch CI passed Windows and Linux unit/compile, Nix package and policy,-documentation, link, distribution-compatibility, and publication checks. The-later documentation-only commit records this evidence and does not alter the-accepted executable or harness source.+## 2026-07-17: semantic browser state++Added the opt-in `semantic_v2` snapshot contract while retaining+`dom_refs_v1` as the compatibility default. The new collector joins the+Chromium accessibility tree, pierced author DOM, layout snapshot, viewport+metrics, and proven frame identities. It separates compact readable content+from typed action refs, ranks visible state before offscreen state, excludes+CSS-hidden retained controls before budgeting, and conservatively identifies+controls covered by fixed or absolute page overlays.++Semantic action refs declare `click` and/or `type`. Mutation tools now return+`browser_action_unavailable` before delivery when a semantic ref does not+declare the requested action. Static nodes with live backend identity are+returned separately as `content_refs` for read-only subtree scope; they do not+become clickable. A bounded DOM supplement recovers visible custom controls+with explicit interaction evidence when AX omits them.++Read scoping supports role/name/text `query`, current `scope_ref`, and opaque+single-use continuation. Continuations remain inside the session capability+store and are invalidated by use, a newer snapshot, navigation, reconnect,+target replacement, or session end. Raw CDP target IDs, backend node IDs,+object IDs, selectors, and continuation offsets remain private.++Known full-DOM size or serialization failures use a progressively shallower+depth ladder and hydrate truncated branches with bounded `DOM.describeNode`+calls. Time, call-count, and scan-node budgets terminate hydration and retain+`complete:false`; unrelated transport failures remain hard errors. The+deterministic fixture contains 320 CSS-hidden retained controls, visible+message/editor/actions, and more than 300 offscreen controls. Focused tests+prove visible-first state, continuation, query, subtree scope, action-kind+enforcement, modal occlusion, DOM supplementation, bounded fallback and+hydration, and stale continuations after both newer snapshots and out-of-band+navigation. Hidden-node omission counts are deduplicated, and icon-font glyphs+are removed from semantic text.++Local source validation passed the complete core and session-lifecycle suites,+all 46 testkit library tests, source compilation of the standalone browser+harness, generated documentation drift checks, public-doc hygiene, and local+link checks. A source-installed macOS product-path replay passed the real Chrome+semantic row with the TCC-authorized daemon, foreground sentinel, fixture+journal, exact DOM click, re-snapshot, and typed-value oracle. Windows and Linux+X11 exact-SHA workflow evidence is the remaining release-acceptance gate.diff --git a/libs/cua-driver/docs/browser-tool-implementation-plan.md b/libs/cua-driver/docs/browser-tool-implementation-plan.mdindex be97082f48..3e89fb4694 100644--- a/libs/cua-driver/docs/browser-tool-implementation-plan.md+++ b/libs/cua-driver/docs/browser-tool-implementation-plan.md@@ -78,7 +78,7 @@ This project will not:- provide ambient "current browser" or "current tab" global state;- silently retry, self-heal, or switch routes after an exact-target check fails;- promise a stable tab model for browsers or embedded webviews that do not expose one;-- copy an agent-browser-style daemon or make a browser instance equivalent to a Cua session;+- copy a third-party browser daemon or make a browser instance equivalent to a Cua session;- add universal `browser_wait`, `browser_close`, or tab lifecycle operations in v1;- deprecate `page` before the new browser surface has accepted cross-platform evidence.diff --git a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.mdindex d8654d2b1a..73646a4848 100644--- a/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md+++ b/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md@@ -18,9 +18,11 @@ The canonical loop is:start_sessionlist_windows or launch_appget_browser_state(pid, window_id, session) # bind-get_browser_state(target_id, tab_id, session) # snapshot+get_browser_state(target_id, tab_id, session,+ snapshot_format=semantic_v2) # snapshotbrowser_navigate / browser_click / browser_type-get_browser_state(target_id, tab_id, session) # verify and refresh refs+get_browser_state(target_id, tab_id, session,+ snapshot_format=semantic_v2) # verify and refresh refsend_session```@@ -122,14 +124,43 @@ Choose a returned `tab_id`, then request the page snapshot:```bashcua-driver get_browser_state \'{"target_id":"<target>","tab_id":"<tab>",- "session":"browser-run-1"}'+ "session":"browser-run-1","snapshot_format":"semantic_v2"}'```-The response contains compact `p<snapshot>:<index>` refs. Refs are scoped to-the session, target, tab, document, frame, and latest snapshot. Navigation and-newer snapshots invalidate old refs. A stale-ref refusal means snapshot again;-it is not permission to fall back to a CSS selector or coordinate remembered-from an earlier page.+`semantic_v2` composes the page accessibility tree, pierced DOM, layout, and+viewport state. Read the compact `outline` for page content, use `refs` only+for actions declared in each entry's `actions` array, and use `content_refs`+only to scope later reads. A content ref is not an action capability.++The snapshot ranks active dialogs and visible controls before near-viewport+and offscreen content. It excludes CSS-hidden retained state before applying+the output budget. Inspect `snapshot.complete`, `snapshot.omitted`, and+`snapshot.continuation` rather than assuming the first response is exhaustive.+To continue the same ranked snapshot:++```bash+cua-driver get_browser_state \+ '{"target_id":"<target>","tab_id":"<tab>",+ "session":"browser-run-1","snapshot_format":"semantic_v2",+ "continuation":"<opaque-continuation>"}'+```++Continuations are opaque, single-use, and bound to the current session, tab,+snapshot, and browser generation. A newer snapshot invalidates them. For a+bounded read, pass either `query` or a current `scope_ref` from `refs` or+`content_refs`:++```bash+cua-driver get_browser_state \+ '{"target_id":"<target>","tab_id":"<tab>",+ "session":"browser-run-1","snapshot_format":"semantic_v2",+ "query":"Account settings"}'+```++Refs remain scoped to the session, target, tab, document, frame, and latest+snapshot. Navigation and newer snapshots invalidate old refs. A stale-ref+refusal means snapshot again; it is not permission to fall back to a CSS+selector or coordinate remembered from an earlier page.Snapshots traverse the main document, open shadow roots, same-process frames,and capability-tested out-of-process frames. Each ref reports its frame kind.@@ -255,6 +286,8 @@ result from the current host, process, window, session, and tab.- `browser_binding_ambiguous` or heuristic binding: resolve the native-windowambiguity and bind again; do not mutate.- `browser_ref_stale`: snapshot again and use a new ref.+- `browser_action_unavailable`: choose a ref that declares the requested+ action; never treat a readable `content_ref` as clickable or editable.- `browser_input_trust_unavailable`: either request `dom_event` when itssemantics are acceptable or use the native action ladder. Do not foregroundthe browser while calling the action background.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rsindex 842078bddf..a445ab9c62 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/binding.rs@@ -9,8 +9,9 @@//! 2. Filter candidates whose CDP window bounds match the native bounds//! within `tolerance` device pixels.//! 3. Exactly one bounds match → **Exact** binding.-//! 4. Multiple distinct CDP windows remain ambiguous. Maximized windows-//! can share bounds, so titles are not strong enough to choose one.+//! 4. Multiple distinct CDP windows can share maximized bounds. One unique+//! active-tab title match disambiguates them; duplicate or absent title+//! matches remain ambiguous.//! 5. No bounds match → a unique title match degrades to **Heuristic**//! (read-only); otherwise **None**.@@ -160,7 +161,19 @@ pub fn correlate(_ => BindingOutcome::None,}}- _ => BindingOutcome::Ambiguous(bounds_matches.len()),+ _ => {+ let title_hits = bounds_matches+ .iter()+ .filter(|candidate| title_matches(&native.title, &candidate.title))+ .collect::<Vec<_>>();+ match title_hits.as_slice() {+ [candidate] if candidate.cdp_window_id.is_some() => BindingOutcome::Bound {+ candidate: (**candidate).clone(),+ quality: BindingQuality::Exact,+ },+ _ => BindingOutcome::Ambiguous(bounds_matches.len()),+ }+ }}}@@ -316,12 +329,29 @@ mod tests {}#[test]- fn distinct_maximized_windows_with_shared_bounds_are_ambiguous() {+ fn unique_title_disambiguates_distinct_maximized_windows_with_shared_bounds() {let n = native("Docs - Chrome", B);let mut other_window = cand("t2", "Docs", B);other_window.cdp_window_id = Some(2);let cands = [cand("t1", "Mail", B), other_window];- assert_eq!(correlate(&n, &cands, 4.0), BindingOutcome::Ambiguous(2));+ match correlate(&n, &cands, 4.0) {+ BindingOutcome::Bound { candidate, quality } => {+ assert_eq!(candidate.cdp_target_id, "t2");+ assert_eq!(quality, BindingQuality::Exact);+ }+ other => panic!("expected title-disambiguated exact bound, got {other:?}"),+ }+ }++ #[test]+ fn duplicate_titles_keep_distinct_maximized_windows_ambiguous() {+ let n = native("Docs - Chrome", B);+ let mut other_window = cand("t2", "Docs", B);+ other_window.cdp_window_id = Some(2);+ assert_eq!(+ correlate(&n, &[cand("t1", "Docs", B), other_window], 4.0),+ BindingOutcome::Ambiguous(2)+ );}#[test]diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rsindex 18d2a23bc4..fba6034f68 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs@@ -23,10 +23,11 @@//! - Whenever a frame's identity cannot be proven, its content is//! omitted from the snapshot — never guessed.-use std::collections::HashMap;+use std::collections::{HashMap, HashSet};use std::sync::{Arc, Weak};use serde_json::{json, Value};+use uuid::Uuid;use crate::session::register_session_end_hook;@@ -38,9 +39,14 @@ use super::platform::{BrowserConsentOutcome, BrowserConsentRequest, BrowserPlatfuse super::prepare::ManagedBrowsers;use super::reconnect::ReconnectGates;use super::refusal::{BrowserRefusal, BrowserRefusalCode};+use super::semantic::{+ build_dom_index, build_layout_index, compose_accessibility_tree, parse_viewport,+ OmissionCounts, SemanticDocument, SemanticNode, DEFAULT_SEMANTIC_NODE_BUDGET,+ SEMANTIC_COMPUTED_STYLES,+};use super::store::{- format_ref, BrowserStore, FrameIdentity, FrameKind, FrameRef, RefEntry, SnapshotRecord,- TabRecord, TargetRecord,+ format_ref, BrowserStore, FrameIdentity, FrameKind, FrameRef, RefEntry, SemanticContinuation,+ SnapshotRecord, TabRecord, TargetRecord,};use super::types::{BindingQuality, NativeWindowInfo, OwnedEndpoint, Rect};@@ -92,6 +98,89 @@ fn is_method_unsupported(error: &anyhow::Error) -> bool {error.to_string().contains("(-32601)")}+fn is_semantic_document_size_error(error: &anyhow::Error) -> bool {+ let message = error.to_string().to_ascii_lowercase();+ [+ "maximum depth",+ "object reference chain is too long",+ "message is too large",+ "serialization",+ ]+ .iter()+ .any(|needle| message.contains(needle))+}++const SEMANTIC_DOM_FALLBACK_DEPTHS: &[i64] = &[256, 128, 64, 32, 16, 8, 4, 2, 1];+const SEMANTIC_DOM_HYDRATION_DEPTH: i64 = 8;+const MAX_SEMANTIC_DOM_HYDRATION_CALLS: usize = 64;+const MAX_SEMANTIC_DOM_SCAN_NODES: usize = 50_000;++#[derive(Default)]+struct DomCoverageScan {+ truncated: Vec<i64>,+ visited_nodes: usize,+ budget_exhausted: bool,+}++fn scan_dom_coverage(value: &Value, scan: &mut DomCoverageScan) {+ if scan.budget_exhausted {+ return;+ }+ match value {+ Value::Array(values) => {+ for value in values {+ scan_dom_coverage(value, scan);+ }+ }+ Value::Object(object) => {+ if object.contains_key("nodeType") {+ scan.visited_nodes += 1;+ if scan.visited_nodes > MAX_SEMANTIC_DOM_SCAN_NODES {+ scan.budget_exhausted = true;+ return;+ }+ let expected = object+ .get("childNodeCount")+ .and_then(Value::as_u64)+ .unwrap_or(0) as usize;+ let present = object+ .get("children")+ .and_then(Value::as_array)+ .map_or(0, Vec::len);+ if expected > present {+ if let Some(backend_node_id) =+ object.get("backendNodeId").and_then(Value::as_i64)+ {+ scan.truncated.push(backend_node_id);+ }+ }+ }+ for value in object.values() {+ scan_dom_coverage(value, scan);+ }+ }+ _ => {}+ }+}++fn replace_dom_node(value: &mut Value, backend_node_id: i64, replacement: &Value) -> bool {+ match value {+ Value::Array(values) => values+ .iter_mut()+ .any(|value| replace_dom_node(value, backend_node_id, replacement)),+ Value::Object(object) => {+ if object.get("backendNodeId").and_then(Value::as_i64) == Some(backend_node_id) {+ *value = replacement.clone();+ return true;+ }+ object+ .values_mut()+ .any(|value| replace_dom_node(value, backend_node_id, replacement))+ }+ _ => false,+ }+}+/// Result of one tab snapshot: minted refs plus what was (and was not)/// composable.pub(crate) struct SnapshotOutcome {@@ -102,6 +191,27 @@ pub(crate) struct SnapshotOutcome {pub oopif: OopifStatus,}+pub(crate) struct SemanticListedRef {+ pub external: String,+ pub node: SemanticNode,+}++pub(crate) struct SemanticSnapshotOutcome {+ pub snapshot_id: u64,+ pub url: String,+ pub title: String,+ pub outline: String,+ pub refs: Vec<SemanticListedRef>,+ pub content_refs: Vec<SemanticListedRef>,+ pub complete: bool,+ pub scope: &'static str,+ pub selected_nodes: usize,+ pub total_nodes: usize,+ pub omissions: OmissionCounts,+ pub continuation: Option<String>,+ pub oopif: OopifStatus,+}+/// Whether OOPIF content could be composed into the snapshot.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub(crate) enum OopifStatus {@@ -156,6 +266,19 @@ impl LocalFrameTree {fn proves(&self, identity: &FrameIdentity) -> bool {self.frames.get(&identity.frame_id) == Some(&identity.loader_id)}++ fn identities(&self) -> Vec<FrameIdentity> {+ let mut identities = self+ .frames+ .iter()+ .map(|(frame_id, loader_id)| FrameIdentity {+ frame_id: frame_id.clone(),+ loader_id: loader_id.clone(),+ })+ .collect::<Vec<_>>();+ identities.sort_by_key(|identity| identity.frame_id != self.main_frame_id);+ identities+ }}/// Parse a `Page.getFrameTree` result. Frames missing an id or loader@@ -526,8 +649,8 @@ impl BrowserEngine {/// Re-prove the endpoint exposed by an explicitly approved existing/// profile. This route is intentionally separate from driver-managed/// endpoint discovery: Chrome's per-instance remote-debugging toggle can- /// expose only a PID-owned listener, with no DevToolsActivePort file or- /// discoverable driver-owned profile.+ /// expose a PID-owned listener whose exact WebSocket path is available+ /// only through the browser's default-profile DevToolsActivePort file.async fn existing_profile_endpoint(&self,pid: i64,@@ -828,6 +951,8 @@ impl BrowserEngine {TabRecord {tab_id,cdp_target_id: c.cdp_target_id.clone(),+ title: c.title.clone(),+ url: c.url.clone(),generation: grant.as_ref().map_or(0, |grant| grant.generation),snapshots: HashMap::new(),},@@ -1310,6 +1435,9 @@ impl BrowserEngine {backend_node_id: c.backend_node_id,node_name: c.node_name,label: c.label,+ actions: Vec::new(),+ visibility: None,+ semantic: false,frame,});}@@ -1361,6 +1489,9 @@ impl BrowserEngine {backend_node_id: c.backend_node_id,node_name: c.node_name,label: c.label,+ actions: Vec::new(),+ visibility: None,+ semantic: false,frame: FrameRef {kind: FrameKind::Oopif,oopif_target_id: Some(child.target_id.clone()),@@ -1432,6 +1563,9 @@ impl BrowserEngine {generation: record.generation,url: url.clone(),refs,+ semantic: None,+ semantic_root_identity: None,+ continuations: HashMap::new(),},);}@@ -1444,6 +1578,524 @@ impl BrowserEngine {oopif,})}++ async fn collect_semantic_session(+ &self,+ conn: &Arc<CdpConnection>,+ cdp_session: &str,+ document: &Value,+ tree: Option<&LocalFrameTree>,+ oopif_target_id: Option<&str>,+ ) -> Result<SemanticDocument, BrowserRefusal> {+ let root = document.get("root").cloned().unwrap_or(Value::Null);+ let styles = SEMANTIC_COMPUTED_STYLES+ .iter()+ .map(|value| Value::String((*value).to_owned()))+ .collect::<Vec<_>>();+ let (layout, metrics) = tokio::try_join!(+ conn.call(+ Some(cdp_session),+ "DOMSnapshot.captureSnapshot",+ json!({+ "computedStyles": styles,+ "includePaintOrder": true,+ "includeDOMRects": true+ }),+ ),+ conn.call(Some(cdp_session), "Page.getLayoutMetrics", json!({})),+ )+ .map_err(|error| route_err("semantic layout collection failed", error))?;+ let dom = build_dom_index(&root);+ let layout = build_layout_index(&layout);+ let viewport = parse_viewport(&metrics);++ let identities = tree.map(LocalFrameTree::identities).unwrap_or_default();+ let mut result = SemanticDocument::default();+ if identities.is_empty() {+ let ax = conn+ .call(Some(cdp_session), "Accessibility.getFullAXTree", json!({}))+ .await+ .map_err(|error| route_err("Accessibility.getFullAXTree failed", error))?;+ result = compose_accessibility_tree(+ &ax,+ &dom,+ &layout,+ &viewport,+ FrameRef::main_unproven(),+ );+ } else {+ for (index, identity) in identities.into_iter().enumerate() {+ let ax = conn+ .call(+ Some(cdp_session),+ "Accessibility.getFullAXTree",+ json!({ "frameId": identity.frame_id }),+ )+ .await+ .map_err(|error| route_err("Accessibility.getFullAXTree failed", error))?;+ let kind = if oopif_target_id.is_some() {+ FrameKind::Oopif+ } else if index == 0 {+ FrameKind::Main+ } else {+ FrameKind::Iframe+ };+ let mut frame_document = compose_accessibility_tree(+ &ax,+ &dom,+ &layout,+ &viewport,+ FrameRef {+ kind,+ oopif_target_id: oopif_target_id.map(str::to_owned),+ identity: Some(identity),+ },+ );+ if index > 0 {+ frame_document.css_hidden_dom_count = 0;+ }+ result.extend(frame_document);+ }+ }+ Ok(result)+ }++ async fn semantic_document(+ &self,+ conn: &Arc<CdpConnection>,+ cdp_session: &str,+ ) -> Result<(Value, bool), BrowserRefusal> {+ match conn+ .call(+ Some(cdp_session),+ "DOM.getDocument",+ json!({ "depth": -1, "pierce": true }),+ )+ .await+ {+ Ok(document) => Ok((document, true)),+ Err(error) if is_semantic_document_size_error(&error) => {+ let mut last_size_error = error.to_string();+ for depth in SEMANTIC_DOM_FALLBACK_DEPTHS {+ match conn+ .call(+ Some(cdp_session),+ "DOM.getDocument",+ json!({ "depth": depth, "pierce": true }),+ )+ .await+ {+ Ok(document) => {+ let document = self+ .hydrate_semantic_document(conn, cdp_session, document)+ .await?;+ return Ok((document, false));+ }+ Err(error) if is_semantic_document_size_error(&error) => {+ last_size_error = error.to_string();+ }+ Err(error) => {+ return Err(route_err("bounded DOM.getDocument fallback failed", error))+ }+ }+ }+ Err(route_err(+ "bounded DOM.getDocument fallback exhausted every accepted depth",+ last_size_error,+ ))+ }+ Err(error) => Err(route_err("DOM.getDocument failed", error)),+ }+ }++ async fn hydrate_semantic_document(+ &self,+ conn: &Arc<CdpConnection>,+ cdp_session: &str,+ mut document: Value,+ ) -> Result<Value, BrowserRefusal> {+ let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(2);+ let mut attempted = HashSet::new();+ for _ in 0..MAX_SEMANTIC_DOM_HYDRATION_CALLS {+ let mut scan = DomCoverageScan::default();+ scan_dom_coverage(&document, &mut scan);+ if scan.budget_exhausted {+ break;+ }+ let Some(backend_node_id) = scan+ .truncated+ .into_iter()+ .find(|backend_node_id| attempted.insert(*backend_node_id))+ else {+ break;+ };+ let now = tokio::time::Instant::now();+ if now >= deadline {+ break;+ }+ let response = match tokio::time::timeout(+ deadline - now,+ conn.call(+ Some(cdp_session),+ "DOM.describeNode",+ json!({+ "backendNodeId": backend_node_id,+ "depth": SEMANTIC_DOM_HYDRATION_DEPTH,+ "pierce": true+ }),+ ),+ )+ .await+ {+ Err(_) => break,+ Ok(Ok(response)) => response,+ Ok(Err(error)) if is_semantic_document_size_error(&error) => continue,+ Ok(Err(error)) => {+ return Err(route_err("DOM.describeNode hydration failed", error))+ }+ };+ let Some(node) = response.get("node") else {+ continue;+ };+ let _ = replace_dom_node(&mut document, backend_node_id, node);+ }+ Ok(document)+ }++ fn semantic_outcome(+ &self,+ snapshot_id: u64,+ url: String,+ title: String,+ page: super::semantic::SemanticPage,+ document_complete: bool,+ scope: &'static str,+ oopif: OopifStatus,+ start_index: u32,+ ) -> (SemanticSnapshotOutcome, HashMap<u32, RefEntry>) {+ let mut stored_refs = HashMap::new();+ let mut refs = Vec::new();+ let mut content_refs = Vec::new();+ for (position, node) in page.selected.iter().enumerate() {+ let Some(entry) = node.to_ref_entry() else {+ continue;+ };+ let index = start_index.saturating_add(position as u32);+ let external = format_ref(snapshot_id, index);+ stored_refs.insert(index, entry);+ let listed = SemanticListedRef {+ external,+ node: node.clone(),+ };+ if node.actions.is_empty() {+ content_refs.push(listed);+ } else {+ refs.push(listed);+ }+ }+ let complete = document_complete && page.next_offset.is_none();+ (+ SemanticSnapshotOutcome {+ snapshot_id,+ url,+ title,+ outline: page.outline,+ refs,+ content_refs,+ complete,+ scope,+ selected_nodes: page.selected_nodes,+ total_nodes: page.total_nodes,+ omissions: page.omissions,+ continuation: page.next_offset.map(|_| format!("bc-{}", Uuid::new_v4())),+ oopif,+ },+ stored_refs,+ )+ }++ pub(crate) async fn snapshot_tab_semantic(+ &self,+ session: &str,+ target_id: &str,+ tab_id: &str,+ scope_ref: Option<&str>,+ query: Option<&str>,+ continuation: Option<&str>,+ ) -> Result<SemanticSnapshotOutcome, BrowserRefusal> {+ if let Some(token) = continuation {+ if scope_ref.is_some() || query.is_some() {+ return Err(refuse(+ BrowserRefusalCode::BrowserRefStale,+ "continuation cannot be combined with a new scope_ref or query",+ ));+ }+ let (snapshot, continuation) = self+ .store+ .resolve_semantic_continuation(session, target_id, tab_id, token)?;+ let record = self.store.get_target(session, target_id)?;+ let tab = record.tabs.get(tab_id).cloned().ok_or_else(|| {+ refuse(+ BrowserRefusalCode::BrowserTabNotFound,+ format!("tab {tab_id} is not known for target {target_id}"),+ )+ })?;+ if let Some(identity) = &snapshot.semantic_root_identity {+ let conn = self.connection_for_record(session, &record).await?;+ let cdp_session = self.attach(&conn, &tab.cdp_target_id).await?;+ let tree = self.local_frame_tree(&conn, &cdp_session).await.map_err(|error| {+ match error {+ FrameTreeError::Unsupported => refuse(+ BrowserRefusalCode::BrowserRouteUnavailable,+ "the browser no longer reports its frame tree, so the semantic \n+ continuation's document identity cannot be re-proven",+ ),+ FrameTreeError::Failed(error) => route_err(+ "Page.getFrameTree failed during semantic continuation revalidation",+ error,+ ),+ }+ })?;+ if !tree.proves(identity) {+ self.store+ .invalidate_tab_snapshots(session, target_id, tab_id);+ return Err(refuse(+ BrowserRefusalCode::BrowserRefStale,+ "the page navigated since this semantic continuation was minted; \n+ re-run get_browser_state to start a fresh snapshot",+ ));+ }+ }+ let document = snapshot.semantic.clone().ok_or_else(|| {+ refuse(+ BrowserRefusalCode::BrowserRefStale,+ "the continuation no longer has semantic snapshot state",+ )+ })?;+ let page = document.page(+ continuation.offset,+ DEFAULT_SEMANTIC_NODE_BUDGET,+ continuation.query.as_deref(),+ continuation.scope_backend_node_id,+ );+ let start_index = snapshot+ .refs+ .keys()+ .max()+ .copied()+ .map_or(0, |value| value.saturating_add(1));+ let oopif = if continuation.oopif_supported {+ OopifStatus::Attached(continuation.oopif_frames)+ } else {+ OopifStatus::Unsupported+ };+ let next_offset = page.next_offset;+ let (outcome, new_refs) = self.semantic_outcome(+ snapshot.id,+ snapshot.url.clone(),+ tab.title,+ page,+ document.complete,+ "continuation",+ oopif,+ start_index,+ );+ let next_token = outcome.continuation.clone();+ self.store.update_target(session, target_id, |record| {+ if let Some(stored) = record+ .tabs+ .get_mut(tab_id)+ .and_then(|tab| tab.snapshots.get_mut(&snapshot.id))+ {+ stored.continuations.remove(token);+ stored.refs.extend(new_refs);+ if let (Some(token), Some(offset)) = (next_token, next_offset) {+ stored.continuations.insert(+ token,+ SemanticContinuation {+ offset,+ query: continuation.query.clone(),+ scope_backend_node_id: continuation.scope_backend_node_id,+ oopif_supported: continuation.oopif_supported,+ oopif_frames: continuation.oopif_frames,+ },+ );+ }+ }+ });+ return Ok(outcome);+ }++ let scope_backend_node_id = match scope_ref {+ Some(external) => Some(+ self.store+ .resolve_ref(session, target_id, tab_id, external)?+ .backend_node_id,+ ),+ None => None,+ };+ let record = self.store.get_target(session, target_id)?;+ let tab = record.tabs.get(tab_id).cloned().ok_or_else(|| {+ refuse(+ BrowserRefusalCode::BrowserTabNotFound,+ format!("tab {tab_id} is not known for target {target_id}"),+ )+ })?;+ let conn = self.connection_for_record(session, &record).await?;+ let cdp_session = self.attach(&conn, &tab.cdp_target_id).await?;+ let (document, document_complete) = self.semantic_document(&conn, &cdp_session).await?;+ let root = document.get("root").cloned().unwrap_or(Value::Null);+ let url = root+ .get("documentURL")+ .and_then(Value::as_str)+ .unwrap_or(&tab.url)+ .to_owned();+ let local_tree = match self.local_frame_tree(&conn, &cdp_session).await {+ Ok(tree) => Some(tree),+ Err(FrameTreeError::Unsupported) => None,+ Err(FrameTreeError::Failed(error)) => {+ return Err(route_err("Page.getFrameTree failed", error))+ }+ };+ let semantic_root_identity = local_tree.as_ref().map(LocalFrameTree::main_identity);+ let mut semantic = self+ .collect_semantic_session(&conn, &cdp_session, &document, local_tree.as_ref(), None)+ .await?;+ semantic.complete &= document_complete;++ let oopif = if local_tree.is_some() {+ match self.attached_iframe_children(&conn, &cdp_session).await {+ Ok(children) => {+ let mut attached = 0;+ for child in &children {+ let child_tree = match self.local_frame_tree(&conn, &child.session_id).await+ {+ Ok(tree) => tree,+ Err(_) => {+ semantic.unprovable_frame_count += 1;+ semantic.complete = false;+ continue;+ }+ };+ let (child_document, child_complete) =+ match self.semantic_document(&conn, &child.session_id).await {+ Ok(document) => document,+ Err(_) => {+ semantic.unprovable_frame_count += 1;+ semantic.complete = false;+ continue;+ }+ };+ match self+ .collect_semantic_session(+ &conn,+ &child.session_id,+ &child_document,+ Some(&child_tree),+ Some(&child.target_id),+ )+ .await+ {+ Ok(document) => {+ let mut document = document;+ document.complete &= child_complete;+ semantic.extend(document);+ attached += 1;+ }+ Err(_) => {+ semantic.unprovable_frame_count += 1;+ semantic.complete = false;+ }+ }+ }+ let _ = conn+ .call(+ Some(&cdp_session),+ "Target.setAutoAttach",+ json!({+ "autoAttach": false,+ "waitForDebuggerOnStart": false,+ "flatten": true+ }),+ )+ .await;+ for child in &children {+ let _ = conn+ .call(+ Some(&cdp_session),+ "Target.detachFromTarget",+ json!({ "sessionId": child.session_id }),+ )+ .await;+ }+ OopifStatus::Attached(attached)+ }+ Err(AttachError::Unsupported) => OopifStatus::Unsupported,+ Err(AttachError::Failed(error)) => {+ return Err(route_err("Target.setAutoAttach failed", error))+ }+ }+ } else {+ OopifStatus::Unsupported+ };++ let page = semantic.page(+ 0,+ DEFAULT_SEMANTIC_NODE_BUDGET,+ query,+ scope_backend_node_id,+ );+ let next_offset = page.next_offset;+ let snapshot_id = self.store.mint_snapshot_id();+ let scope = if scope_ref.is_some() {+ "subtree"+ } else if query.is_some() {+ "query"+ } else {+ "viewport"+ };+ let (outcome, refs) = self.semantic_outcome(+ snapshot_id,+ url.clone(),+ tab.title.clone(),+ page,+ semantic.complete,+ scope,+ oopif,+ 0,+ );+ let continuation_token = outcome.continuation.clone();+ self.store+ .update_target(session, target_id, |stored_target| {+ if let Some(stored_tab) = stored_target.tabs.get_mut(tab_id) {+ let mut continuations = HashMap::new();+ if let (Some(token), Some(offset)) = (continuation_token, next_offset) {+ continuations.insert(+ token,+ SemanticContinuation {+ offset,+ query: query.map(str::to_owned),+ scope_backend_node_id,+ oopif_supported: matches!(oopif, OopifStatus::Attached(_)),+ oopif_frames: oopif.frames(),+ },+ );+ }+ stored_tab.snapshots.clear();+ stored_tab.snapshots.insert(+ snapshot_id,+ SnapshotRecord {+ id: snapshot_id,+ generation: record.generation,+ url,+ refs,+ semantic: Some(semantic),+ semantic_root_identity,+ continuations,+ },+ );+ }+ });+ Ok(outcome)+ }}/// Attribute names that make an element interactive-enough to ref.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rsindex 2d521943e8..ea899520c4 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs@@ -48,6 +48,7 @@ pub mod platform;mod prepare;mod reconnect;pub mod refusal;+mod semantic;mod setup_descriptor;pub mod store;pub mod tools;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/refusal.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/refusal.rsindex 5de453de29..fb71950b7c 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/refusal.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/refusal.rs@@ -58,6 +58,9 @@ pub enum BrowserRefusalCode {BrowserReconnectExhausted,/// A typing operation delivered only a proven prefix of the request.BrowserInputIncomplete,+ /// A semantic ref is live, but it does not declare the requested typed+ /// browser action.+ BrowserActionUnavailable,}impl BrowserRefusalCode {@@ -78,6 +81,7 @@ impl BrowserRefusalCode {Self::BrowserConsentRevoked => "browser_consent_revoked",Self::BrowserReconnectExhausted => "browser_reconnect_exhausted",Self::BrowserInputIncomplete => "browser_input_incomplete",+ Self::BrowserActionUnavailable => "browser_action_unavailable",}}}@@ -178,6 +182,10 @@ mod tests {BrowserRefusalCode::BrowserInputIncomplete,"browser_input_incomplete",),+ (+ BrowserRefusalCode::BrowserActionUnavailable,+ "browser_action_unavailable",+ ),];for (code, wire) in all {assert_eq!(code.as_str(), wire);diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rsnew file mode 100644index 0000000000..5ae44da70a--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs@@ -0,0 +1,1210 @@+//! Deterministic semantic browser snapshots.+//!+//! The collector joins Chrome's accessibility tree with pierced DOM metadata+//! and layout snapshot evidence. Accessibility supplies readable semantics,+//! DOM backend ids preserve exact mutation capabilities, and layout evidence+//! keeps hidden retained application state from displacing the active view.++use std::collections::{BTreeMap, HashMap, HashSet};++use serde_json::Value;++use super::store::{BrowserActionKind, BrowserVisibility, FrameRef, RefEntry};++pub(crate) const SEMANTIC_COMPUTED_STYLES: &[&str] = &[+ "display",+ "visibility",+ "opacity",+ "pointer-events",+ "cursor",+ "position",+ "z-index",+];+pub(crate) const DEFAULT_SEMANTIC_NODE_BUDGET: usize = 300;+const NEAR_VIEWPORT_MARGIN: f64 = 1_000.0;+const MAX_SEMANTIC_TEXT_CHARS: usize = 1_000;++#[derive(Debug, Clone, Copy, PartialEq)]+struct Rect {+ x: f64,+ y: f64,+ width: f64,+ height: f64,+}++impl Rect {+ fn from_value(value: &Value) -> Option<Self> {+ let values = value.as_array()?;+ Some(Self {+ x: values.first()?.as_f64()?,+ y: values.get(1)?.as_f64()?,+ width: values.get(2)?.as_f64()?,+ height: values.get(3)?.as_f64()?,+ })+ }++ fn has_area(self) -> bool {+ self.width > 0.0 && self.height > 0.0+ }++ fn intersects(self, other: Self) -> bool {+ self.x < other.x + other.width+ && self.x + self.width > other.x+ && self.y < other.y + other.height+ && self.y + self.height > other.y+ }++ fn expanded(self, margin: f64) -> Self {+ Self {+ x: self.x - margin,+ y: self.y - margin,+ width: self.width + margin * 2.0,+ height: self.height + margin * 2.0,+ }+ }++ fn covers(self, other: Self) -> bool {+ self.x <= other.x+ && self.y <= other.y+ && self.x + self.width >= other.x + other.width+ && self.y + self.height >= other.y + other.height+ }+}++#[derive(Debug, Clone, Default)]+struct DomMeta {+ tag: String,+ attrs: HashMap<String, String>,+ order: usize,+ css_hidden: bool,+ parent_backend_node_id: Option<i64>,+ frame_id: Option<String>,+}++#[derive(Debug, Clone, Default)]+pub(crate) struct DomIndex {+ nodes: HashMap<i64, DomMeta>,+ pub(crate) css_hidden_count: usize,+}++#[derive(Debug, Clone, Default)]+struct LayoutMeta {+ bounds: Option<Rect>,+ styles: HashMap<String, String>,+ paint_order: Option<i64>,+}++#[derive(Debug, Clone, Default)]+pub(crate) struct LayoutIndex {+ nodes: HashMap<i64, LayoutMeta>,+}++#[derive(Debug, Clone, Default)]+pub(crate) struct Viewport {+ rect: Option<Rect>,+}++#[derive(Debug, Clone)]+pub(crate) struct SemanticNode {+ pub(crate) ax_id: String,+ pub(crate) parent_ax_id: Option<String>,+ pub(crate) child_ax_ids: Vec<String>,+ pub(crate) backend_node_id: Option<i64>,+ pub(crate) role: String,+ pub(crate) name: Option<String>,+ pub(crate) value: Option<String>,+ pub(crate) states: BTreeMap<String, Value>,+ pub(crate) frame: FrameRef,+ pub(crate) visibility: BrowserVisibility,+ pub(crate) actions: Vec<BrowserActionKind>,+ pub(crate) document_order: usize,+}++impl SemanticNode {+ pub(crate) fn to_ref_entry(&self) -> Option<RefEntry> {+ let backend_node_id = self.backend_node_id?;+ Some(RefEntry {+ backend_node_id,+ node_name: self.role.clone(),+ label: self.name.clone(),+ actions: self.actions.clone(),+ visibility: Some(self.visibility),+ semantic: true,+ frame: self.frame.clone(),+ })+ }+}++#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub(crate) struct OmissionCounts {+ pub(crate) css_hidden: usize,+ pub(crate) offscreen: usize,+ pub(crate) page_occluded: usize,+ pub(crate) no_layout: usize,+ pub(crate) unknown: usize,+ pub(crate) budget: usize,+ pub(crate) unprovable_frame: usize,+}++#[derive(Debug, Clone)]+pub(crate) struct SemanticPage {+ pub(crate) outline: String,+ pub(crate) selected: Vec<SemanticNode>,+ pub(crate) selected_nodes: usize,+ pub(crate) total_nodes: usize,+ pub(crate) next_offset: Option<usize>,+ pub(crate) omissions: OmissionCounts,+}++#[derive(Debug, Clone, Default)]+pub(crate) struct SemanticDocument {+ pub(crate) nodes: Vec<SemanticNode>,+ pub(crate) css_hidden_dom_count: usize,+ pub(crate) unprovable_frame_count: usize,+ pub(crate) complete: bool,+}++impl SemanticDocument {+ pub(crate) fn extend(&mut self, mut other: Self) {+ let was_empty = self.nodes.is_empty();+ let offset = self.nodes.len();+ for node in &mut other.nodes {+ node.document_order += offset;+ }+ self.nodes.extend(other.nodes);+ self.css_hidden_dom_count += other.css_hidden_dom_count;+ self.unprovable_frame_count += other.unprovable_frame_count;+ self.complete = if was_empty {+ other.complete+ } else {+ self.complete && other.complete+ };+ }++ pub(crate) fn page(+ &self,+ offset: usize,+ budget: usize,+ query: Option<&str>,+ scope_backend_node_id: Option<i64>,+ ) -> SemanticPage {+ let mut candidates = scoped_indices(&self.nodes, query, scope_backend_node_id);+ candidates.retain(|idx| {+ !matches!(+ self.nodes[*idx].visibility,+ BrowserVisibility::CssHidden | BrowserVisibility::PageOccluded+ )+ });+ candidates.sort_by_key(|idx| (rank(&self.nodes[*idx]), self.nodes[*idx].document_order));++ let start = offset.min(candidates.len());+ let end = (start + budget.max(1)).min(candidates.len());+ let page_slice = &candidates[start..end];+ let selected = with_ancestors(&self.nodes, page_slice);+ let outline = render_outline(&self.nodes, &selected);+ let selected_nodes = page_slice+ .iter()+ .map(|idx| self.nodes[*idx].clone())+ .collect::<Vec<_>>();+ let semantic_css_hidden = self+ .nodes+ .iter()+ .filter(|node| node.visibility == BrowserVisibility::CssHidden)+ .count();+ let mut omissions = OmissionCounts {+ css_hidden: self.css_hidden_dom_count.max(semantic_css_hidden),+ unprovable_frame: self.unprovable_frame_count,+ ..Default::default()+ };+ for node in &self.nodes {+ match node.visibility {+ BrowserVisibility::CssHidden => {}+ BrowserVisibility::Offscreen => omissions.offscreen += 1,+ BrowserVisibility::PageOccluded => omissions.page_occluded += 1,+ BrowserVisibility::NoLayout => omissions.no_layout += 1,+ BrowserVisibility::Unknown => omissions.unknown += 1,+ BrowserVisibility::InViewport | BrowserVisibility::NearViewport => {}+ }+ }+ omissions.budget = candidates.len().saturating_sub(end);++ SemanticPage {+ outline,+ selected: selected_nodes,+ selected_nodes: page_slice.len(),+ total_nodes: candidates.len(),+ next_offset: (end < candidates.len()).then_some(end),+ omissions,+ }+ }+}++pub(crate) fn build_dom_index(root: &Value) -> DomIndex {+ fn walk(+ node: &Value,+ inherited_hidden: bool,+ parent_backend_node_id: Option<i64>,+ inherited_frame_id: Option<&str>,+ order: &mut usize,+ index: &mut DomIndex,+ ) {+ let node_type = node.get("nodeType").and_then(Value::as_i64).unwrap_or(0);+ let attrs = attributes(node);+ let hidden = inherited_hidden || statically_hidden(&attrs);+ let backend_node_id = node.get("backendNodeId").and_then(Value::as_i64);+ let frame_id = if node_type == 9 {+ node.get("frameId").and_then(Value::as_str)+ } else {+ inherited_frame_id+ };+ if let Some(backend) = backend_node_id {+ let tag = node+ .get("nodeName")+ .and_then(Value::as_str)+ .unwrap_or("")+ .to_ascii_lowercase();+ if node_type == 1 && hidden && !inherited_hidden {+ index.css_hidden_count += 1;+ }+ index.nodes.insert(+ backend,+ DomMeta {+ tag,+ attrs,+ order: *order,+ css_hidden: hidden,+ parent_backend_node_id,+ frame_id: frame_id.map(str::to_owned),+ },+ );+ *order += 1;+ }+ if let Some(children) = node.get("children").and_then(Value::as_array) {+ for child in children {+ walk(+ child,+ hidden,+ backend_node_id.or(parent_backend_node_id),+ frame_id,+ order,+ index,+ );+ }+ }+ if let Some(shadow_roots) = node.get("shadowRoots").and_then(Value::as_array) {+ for shadow_root in shadow_roots {+ if shadow_root.get("shadowRootType").and_then(Value::as_str) == Some("user-agent") {+ continue;+ }+ walk(+ shadow_root,+ hidden,+ backend_node_id.or(parent_backend_node_id),+ frame_id,+ order,+ index,+ );+ }+ }+ if let Some(content_document) = node.get("contentDocument") {+ walk(+ content_document,+ hidden,+ backend_node_id.or(parent_backend_node_id),+ content_document.get("frameId").and_then(Value::as_str),+ order,+ index,+ );+ }+ }++ let mut index = DomIndex::default();+ let mut order = 0;+ walk(+ root,+ false,+ None,+ root.get("frameId").and_then(Value::as_str),+ &mut order,+ &mut index,+ );+ index+}++pub(crate) fn build_layout_index(snapshot: &Value) -> LayoutIndex {+ let Some(strings) = snapshot.get("strings").and_then(Value::as_array) else {+ return LayoutIndex::default();+ };+ let string_at = |idx: i64| -> Option<String> {+ usize::try_from(idx)+ .ok()+ .and_then(|idx| strings.get(idx))+ .and_then(Value::as_str)+ .map(str::to_owned)+ };++ let mut out = LayoutIndex::default();+ let Some(documents) = snapshot.get("documents").and_then(Value::as_array) else {+ return out;+ };+ for document in documents {+ let Some(backend_ids) = document+ .pointer("/nodes/backendNodeId")+ .and_then(Value::as_array)+ else {+ continue;+ };+ let layout = document.get("layout").unwrap_or(&Value::Null);+ let node_indices = layout+ .get("nodeIndex")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let bounds = layout+ .get("bounds")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let styles = layout+ .get("styles")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let paint_orders = layout+ .get("paintOrders")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();++ for (layout_idx, node_index) in node_indices.iter().enumerate() {+ let Some(node_index) = node_index.as_u64().and_then(|v| usize::try_from(v).ok()) else {+ continue;+ };+ let Some(backend) = backend_ids.get(node_index).and_then(Value::as_i64) else {+ continue;+ };+ let mut computed = HashMap::new();+ if let Some(style_indices) = styles.get(layout_idx).and_then(Value::as_array) {+ for (name, value_idx) in SEMANTIC_COMPUTED_STYLES+ .iter()+ .zip(style_indices.iter().filter_map(Value::as_i64))+ {+ if let Some(value) = string_at(value_idx) {+ computed.insert((*name).to_owned(), value);+ }+ }+ }+ out.nodes.insert(+ backend,+ LayoutMeta {+ bounds: bounds.get(layout_idx).and_then(Rect::from_value),+ styles: computed,+ paint_order: paint_orders.get(layout_idx).and_then(Value::as_i64),+ },+ );+ }+ }+ out+}++pub(crate) fn parse_viewport(metrics: &Value) -> Viewport {+ let viewport = metrics+ .get("cssVisualViewport")+ .or_else(|| metrics.get("visualViewport"));+ let Some(viewport) = viewport else {+ return Viewport::default();+ };+ let x = viewport+ .get("pageX")+ .or_else(|| viewport.get("offsetX"))+ .and_then(Value::as_f64)+ .unwrap_or(0.0);+ let y = viewport+ .get("pageY")+ .or_else(|| viewport.get("offsetY"))+ .and_then(Value::as_f64)+ .unwrap_or(0.0);+ let width = viewport.get("clientWidth").and_then(Value::as_f64);+ let height = viewport.get("clientHeight").and_then(Value::as_f64);+ Viewport {+ rect: width.zip(height).map(|(width, height)| Rect {+ x,+ y,+ width,+ height,+ }),+ }+}++pub(crate) fn compose_accessibility_tree(+ ax_tree: &Value,+ dom: &DomIndex,+ layout: &LayoutIndex,+ viewport: &Viewport,+ frame: FrameRef,+) -> SemanticDocument {+ let Some(ax_nodes) = ax_tree.get("nodes").and_then(Value::as_array) else {+ return SemanticDocument {+ complete: false,+ css_hidden_dom_count: dom.css_hidden_count,+ ..Default::default()+ };+ };++ let mut nodes = Vec::new();+ for (fallback_order, ax) in ax_nodes.iter().enumerate() {+ if ax.get("ignored").and_then(Value::as_bool) == Some(true) {+ continue;+ }+ let ax_id = ax+ .get("nodeId")+ .and_then(Value::as_str)+ .unwrap_or("")+ .to_owned();+ if ax_id.is_empty() {+ continue;+ }+ let role = ax_value_string(ax.get("role"))+ .unwrap_or_else(|| "unknown".to_owned())+ .to_ascii_lowercase();+ if role == "inlinetextbox" {+ continue;+ }+ let backend_node_id = ax.get("backendDOMNodeId").and_then(Value::as_i64);+ let dom_meta = backend_node_id.and_then(|backend| dom.nodes.get(&backend));+ let layout_meta = backend_node_id.and_then(|backend| layout.nodes.get(&backend));+ let states = ax_states(ax);+ let visibility = classify_visibility(dom_meta, layout_meta, viewport);+ let actions = action_kinds(&role, dom_meta, &states, layout_meta);+ let name = ax_value_string(ax.get("name")).and_then(clean_semantic_text);+ let value = ax_value_string(ax.get("value")).and_then(clean_semantic_text);+ let document_order = dom_meta.map_or(fallback_order, |meta| meta.order);+ nodes.push(SemanticNode {+ ax_id,+ parent_ax_id: ax+ .get("parentId")+ .and_then(Value::as_str)+ .map(str::to_owned),+ child_ax_ids: ax+ .get("childIds")+ .and_then(Value::as_array)+ .map(|ids| {+ ids.iter()+ .filter_map(Value::as_str)+ .map(str::to_owned)+ .collect()+ })+ .unwrap_or_default(),+ backend_node_id,+ role,+ name,+ value,+ states,+ frame: frame.clone(),+ visibility,+ actions,+ document_order,+ });+ }++ supplement_dom_actions(&mut nodes, dom, layout, viewport, &frame);+ apply_page_occlusion(&mut nodes, layout);+ remove_redundant_static_text(&mut nodes);+ SemanticDocument {+ nodes,+ css_hidden_dom_count: dom.css_hidden_count,+ unprovable_frame_count: 0,+ complete: true,+ }+}++fn apply_page_occlusion(nodes: &mut [SemanticNode], layout: &LayoutIndex) {+ for node in nodes {+ if node.visibility != BrowserVisibility::InViewport {+ continue;+ }+ let Some(target_backend) = node.backend_node_id else {+ continue;+ };+ let Some(target) = layout.nodes.get(&target_backend) else {+ continue;+ };+ let (Some(target_bounds), Some(target_paint)) = (target.bounds, target.paint_order) else {+ continue;+ };+ let covered = layout.nodes.iter().any(|(&backend, overlay)| {+ if backend == target_backend+ || overlay+ .paint_order+ .is_none_or(|paint| paint <= target_paint)+ || !overlay+ .styles+ .get("position")+ .is_some_and(|position| matches!(position.as_str(), "fixed" | "absolute"))+ || overlay+ .styles+ .get("pointer-events")+ .is_some_and(|value| value == "none")+ || layout_hidden(overlay)+ {+ return false;+ }+ overlay+ .bounds+ .is_some_and(|bounds| bounds.has_area() && bounds.covers(target_bounds))+ });+ if covered {+ node.visibility = BrowserVisibility::PageOccluded;+ }+ }+}++fn supplement_dom_actions(+ nodes: &mut Vec<SemanticNode>,+ dom: &DomIndex,+ layout: &LayoutIndex,+ viewport: &Viewport,+ frame: &FrameRef,+) {+ let existing = nodes+ .iter()+ .filter_map(|node| node.backend_node_id)+ .collect::<HashSet<_>>();+ let expected_frame = frame+ .identity+ .as_ref()+ .map(|identity| identity.frame_id.as_str());+ let by_backend = nodes+ .iter()+ .filter_map(|node| Some((node.backend_node_id?, node.ax_id.clone())))+ .collect::<HashMap<_, _>>();+ let mut candidates = dom.nodes.iter().collect::<Vec<_>>();+ candidates.sort_by_key(|(_, meta)| meta.order);+ for (&backend_node_id, meta) in candidates {+ if existing.contains(&backend_node_id)+ || expected_frame.is_some_and(|expected| meta.frame_id.as_deref() != Some(expected))+ {+ continue;+ }+ let layout_meta = layout.nodes.get(&backend_node_id);+ let visibility = classify_visibility(Some(meta), layout_meta, viewport);+ if matches!(+ visibility,+ BrowserVisibility::CssHidden | BrowserVisibility::PageOccluded+ ) {+ continue;+ }+ let role = meta+ .attrs+ .get("role")+ .map(|value| value.to_ascii_lowercase())+ .unwrap_or_else(|| match meta.tag.as_str() {+ "a" => "link".to_owned(),+ "button" => "button".to_owned(),+ "input" => match meta.attrs.get("type").map(String::as_str) {+ Some("checkbox") => "checkbox".to_owned(),+ Some("radio") => "radio".to_owned(),+ Some("button" | "submit" | "reset" | "image") => "button".to_owned(),+ Some("range") => "slider".to_owned(),+ _ => "textbox".to_owned(),+ },+ "textarea" => "textbox".to_owned(),+ "select" => "combobox".to_owned(),+ "option" => "option".to_owned(),+ "summary" => "summary".to_owned(),+ _ => "generic".to_owned(),+ });+ let mut states = BTreeMap::new();+ if meta.attrs.contains_key("disabled") {+ states.insert("disabled".to_owned(), Value::Bool(true));+ }+ let actions = action_kinds(&role, Some(meta), &states, layout_meta);+ if actions.is_empty() {+ continue;+ }+ let name = ["aria-label", "placeholder", "title", "name", "id"]+ .iter()+ .find_map(|key| meta.attrs.get(*key).cloned())+ .and_then(clean_semantic_text);+ nodes.push(SemanticNode {+ ax_id: format!("dom-{backend_node_id}"),+ parent_ax_id: meta+ .parent_backend_node_id+ .and_then(|parent| by_backend.get(&parent).cloned()),+ child_ax_ids: Vec::new(),+ backend_node_id: Some(backend_node_id),+ role,+ name,+ value: meta+ .attrs+ .get("value")+ .cloned()+ .and_then(clean_semantic_text),+ states,+ frame: frame.clone(),+ visibility,+ actions,+ document_order: meta.order,+ });+ }+}++fn attributes(node: &Value) -> HashMap<String, String> {+ node.get("attributes")+ .and_then(Value::as_array)+ .map(|attrs| {+ attrs+ .chunks_exact(2)+ .filter_map(|pair| {+ Some((+ pair[0].as_str()?.to_ascii_lowercase(),+ pair[1].as_str()?.to_owned(),+ ))+ })+ .collect()+ })+ .unwrap_or_default()+}++fn statically_hidden(attrs: &HashMap<String, String>) -> bool {+ if attrs.contains_key("hidden") || attrs.get("aria-hidden").is_some_and(|v| v == "true") {+ return true;+ }+ let style = attrs+ .get("style")+ .map(|value| value.to_ascii_lowercase())+ .unwrap_or_default();+ style.contains("display:none")+ || style.contains("display: none")+ || style.contains("visibility:hidden")+ || style.contains("visibility: hidden")+ || style.contains("opacity:0")+ || style.contains("opacity: 0")+}++fn layout_hidden(layout: &LayoutMeta) -> bool {+ layout+ .styles+ .get("display")+ .is_some_and(|value| value.eq_ignore_ascii_case("none"))+ || layout+ .styles+ .get("visibility")+ .is_some_and(|value| value.eq_ignore_ascii_case("hidden"))+ || layout+ .styles+ .get("opacity")+ .and_then(|value| value.parse::<f64>().ok())+ .is_some_and(|opacity| opacity <= 0.0)+}++fn classify_visibility(+ dom: Option<&DomMeta>,+ layout: Option<&LayoutMeta>,+ viewport: &Viewport,+) -> BrowserVisibility {+ if dom.is_some_and(|meta| meta.css_hidden) || layout.is_some_and(layout_hidden) {+ return BrowserVisibility::CssHidden;+ }+ let Some(layout) = layout else {+ return BrowserVisibility::Unknown;+ };+ let Some(bounds) = layout.bounds else {+ return BrowserVisibility::NoLayout;+ };+ if !bounds.has_area() {+ return BrowserVisibility::NoLayout;+ }+ let Some(viewport) = viewport.rect else {+ return BrowserVisibility::Unknown;+ };+ if bounds.intersects(viewport) {+ BrowserVisibility::InViewport+ } else if bounds.intersects(viewport.expanded(NEAR_VIEWPORT_MARGIN)) {+ BrowserVisibility::NearViewport+ } else {+ BrowserVisibility::Offscreen+ }+}++fn action_kinds(+ role: &str,+ dom: Option<&DomMeta>,+ states: &BTreeMap<String, Value>,+ layout: Option<&LayoutMeta>,+) -> Vec<BrowserActionKind> {+ if states.get("disabled").and_then(Value::as_bool) == Some(true) {+ return Vec::new();+ }+ let mut actions = Vec::new();+ if matches!(+ role,+ "button"+ | "link"+ | "checkbox"+ | "radio"+ | "switch"+ | "tab"+ | "menuitem"+ | "menuitemcheckbox"+ | "menuitemradio"+ | "option"+ | "treeitem"+ | "slider"+ | "spinbutton"+ | "combobox"+ | "listbox"+ | "summary"+ ) {+ actions.push(BrowserActionKind::Click);+ }+ let tag = dom.map(|meta| meta.tag.as_str()).unwrap_or("");+ let editable = states.get("editable").is_some_and(|value| {+ value.as_bool() == Some(true) || value.as_str().is_some_and(|value| value != "false")+ }) || dom.is_some_and(|meta| {+ meta.attrs+ .get("contenteditable")+ .is_some_and(|value| value.is_empty() || value.eq_ignore_ascii_case("true"))+ });+ if matches!(role, "textbox" | "searchbox") || matches!(tag, "input" | "textarea") || editable {+ actions.push(BrowserActionKind::Type);+ }+ if actions.is_empty()+ && dom.is_some_and(|meta| {+ meta.attrs.contains_key("onclick")+ || meta+ .attrs+ .get("tabindex")+ .is_some_and(|value| value != "-1")+ || (!matches!(meta.tag.as_str(), "html" | "body")+ && layout.is_some_and(|layout| {+ layout+ .styles+ .get("cursor")+ .is_some_and(|cursor| cursor.eq_ignore_ascii_case("pointer"))+ && !layout+ .styles+ .get("pointer-events")+ .is_some_and(|value| value.eq_ignore_ascii_case("none"))+ }))+ })+ && layout.is_some_and(|meta| meta.bounds.is_some_and(Rect::has_area))+ {+ actions.push(BrowserActionKind::Click);+ }+ actions+}++fn ax_states(node: &Value) -> BTreeMap<String, Value> {+ const KEPT: &[&str] = &[+ "checked",+ "disabled",+ "editable",+ "expanded",+ "focused",+ "focusable",+ "pressed",+ "required",+ "selected",+ ];+ node.get("properties")+ .and_then(Value::as_array)+ .map(|properties| {+ properties+ .iter()+ .filter_map(|property| {+ let name = property.get("name").and_then(Value::as_str)?;+ KEPT.contains(&name).then(|| {+ (+ name.to_owned(),+ property+ .pointer("/value/value")+ .cloned()+ .unwrap_or(Value::Null),+ )+ })+ })+ .collect()+ })+ .unwrap_or_default()+}++fn ax_value_string(value: Option<&Value>) -> Option<String> {+ let value = value?.get("value")?;+ match value {+ Value::String(value) => Some(value.clone()),+ Value::Bool(value) => Some(value.to_string()),+ Value::Number(value) => Some(value.to_string()),+ _ => None,+ }+}++fn clean_semantic_text(value: String) -> Option<String> {+ let normalized = value+ .chars()+ .map(|ch| match ch {+ '\u{feff}' | '\u{200b}' | '\u{200c}' | '\u{200d}' | '\u{2060}' | '\u{00a0}'+ | '\u{2007}' | '\u{202f}' => ' ',+ '\u{e000}'..='\u{f8ff}' => ' ',+ _ => ch,+ })+ .collect::<String>()+ .split_whitespace()+ .collect::<Vec<_>>()+ .join(" ");+ if normalized.is_empty() {+ return None;+ }+ Some(normalized.chars().take(MAX_SEMANTIC_TEXT_CHARS).collect())+}++fn remove_redundant_static_text(nodes: &mut Vec<SemanticNode>) {+ let names: HashMap<String, String> = nodes+ .iter()+ .filter_map(|node| node.name.clone().map(|name| (node.ax_id.clone(), name)))+ .collect();+ nodes.retain(|node| {+ if node.role != "statictext" && node.role != "text" {+ return true;+ }+ let Some(name) = node.name.as_deref() else {+ return false;+ };+ !node+ .parent_ax_id+ .as_ref()+ .and_then(|parent| names.get(parent))+ .is_some_and(|parent_name| parent_name == name)+ });+}++fn rank(node: &SemanticNode) -> u8 {+ let priority_context = node.states.get("focused").and_then(Value::as_bool) == Some(true)+ || matches!(node.role.as_str(), "dialog" | "alertdialog");+ if priority_context {+ return 0;+ }+ match (node.visibility, node.actions.is_empty()) {+ (BrowserVisibility::InViewport, false) => 1,+ (BrowserVisibility::InViewport, true) => 2,+ (BrowserVisibility::NearViewport, false) => 3,+ (BrowserVisibility::NearViewport, true) => 4,+ (BrowserVisibility::Unknown | BrowserVisibility::NoLayout, false) => 5,+ (BrowserVisibility::Unknown | BrowserVisibility::NoLayout, true) => 6,+ (BrowserVisibility::Offscreen, false) => 7,+ (BrowserVisibility::Offscreen, true) => 8,+ (BrowserVisibility::CssHidden | BrowserVisibility::PageOccluded, _) => 9,+ }+}++fn scoped_indices(+ nodes: &[SemanticNode],+ query: Option<&str>,+ scope_backend_node_id: Option<i64>,+) -> Vec<usize> {+ let by_ax_id: HashMap<&str, usize> = nodes+ .iter()+ .enumerate()+ .map(|(idx, node)| (node.ax_id.as_str(), idx))+ .collect();+ let mut allowed = HashSet::new();+ if let Some(scope_backend) = scope_backend_node_id {+ if let Some(scope) = nodes+ .iter()+ .find(|node| node.backend_node_id == Some(scope_backend))+ {+ let mut stack = vec![scope.ax_id.as_str()];+ while let Some(ax_id) = stack.pop() {+ if !allowed.insert(ax_id.to_owned()) {+ continue;+ }+ if let Some(idx) = by_ax_id.get(ax_id) {+ stack.extend(nodes[*idx].child_ax_ids.iter().map(String::as_str));+ }+ }+ }+ }+ let query = query.map(|value| value.to_ascii_lowercase());+ nodes+ .iter()+ .enumerate()+ .filter(|(_, node)| {+ (scope_backend_node_id.is_none() || allowed.contains(&node.ax_id))+ && query.as_ref().is_none_or(|query| {+ node.role.to_ascii_lowercase().contains(query)+ || node+ .name+ .as_ref()+ .is_some_and(|name| name.to_ascii_lowercase().contains(query))+ || node+ .value+ .as_ref()+ .is_some_and(|value| value.to_ascii_lowercase().contains(query))+ })+ })+ .map(|(idx, _)| idx)+ .collect()+}++fn with_ancestors(nodes: &[SemanticNode], selected: &[usize]) -> HashSet<usize> {+ let by_ax_id: HashMap<&str, usize> = nodes+ .iter()+ .enumerate()+ .map(|(idx, node)| (node.ax_id.as_str(), idx))+ .collect();+ let mut keep: HashSet<usize> = selected.iter().copied().collect();+ for idx in selected {+ let mut parent = nodes[*idx].parent_ax_id.as_deref();+ while let Some(parent_id) = parent {+ let Some(parent_idx) = by_ax_id.get(parent_id).copied() else {+ break;+ };+ if !keep.insert(parent_idx) {+ break;+ }+ parent = nodes[parent_idx].parent_ax_id.as_deref();+ }+ }+ keep+}++fn render_outline(nodes: &[SemanticNode], selected: &HashSet<usize>) -> String {+ let by_ax_id: HashMap<&str, usize> = nodes+ .iter()+ .enumerate()+ .map(|(idx, node)| (node.ax_id.as_str(), idx))+ .collect();+ let mut ordered: Vec<usize> = selected.iter().copied().collect();+ ordered.sort_by_key(|idx| nodes[*idx].document_order);+ let mut lines = Vec::new();+ for idx in ordered {+ let node = &nodes[idx];+ if matches!(node.role.as_str(), "rootwebarea" | "webarea") {+ continue;+ }+ let mut depth = 0;+ let mut parent = node.parent_ax_id.as_deref();+ while let Some(parent_id) = parent {+ let Some(parent_idx) = by_ax_id.get(parent_id).copied() else {+ break;+ };+ if selected.contains(&parent_idx)+ && !matches!(nodes[parent_idx].role.as_str(), "rootwebarea" | "webarea")+ {+ depth += 1;+ }+ parent = nodes[parent_idx].parent_ax_id.as_deref();+ }+ let mut line = format!("{}- {}", " ".repeat(depth), node.role);+ if let Some(name) = &node.name {+ line.push(' ');+ line.push_str(&serde_json::to_string(name).unwrap_or_else(|_| "\"\"".to_owned()));+ }+ if let Some(value) = &node.value {+ if node.name.as_deref() != Some(value) {+ line.push_str(": ");+ line.push_str(value);+ }+ }+ if !node.states.is_empty() {+ let states = node+ .states+ .iter()+ .map(|(name, value)| format!("{name}={value}"))+ .collect::<Vec<_>>()+ .join(",");+ line.push_str(&format!(" [{states}]"));+ }+ lines.push(line);+ }+ lines.join("\n")+}++#[cfg(test)]+mod tests {+ use serde_json::json;++ use super::*;+ use crate::browser::store::FrameRef;++ fn frame() -> FrameRef {+ FrameRef::main_unproven()+ }++ #[test]+ fn hidden_dom_nodes_do_not_enter_the_visible_working_set() {+ let dom = build_dom_index(&json!({+ "nodeType": 9,+ "children": [+ {"nodeType": 1, "nodeName": "BUTTON", "backendNodeId": 1,+ "attributes": ["aria-hidden", "true"]},+ {"nodeType": 1, "nodeName": "BUTTON", "backendNodeId": 2,+ "attributes": ["aria-label", "Reply"]}+ ]+ }));+ let ax = json!({"nodes": [+ {"nodeId": "root", "ignored": false, "role": {"value": "RootWebArea"},+ "childIds": ["retained", "reply"]},+ {"nodeId": "retained", "parentId": "root", "ignored": false,+ "backendDOMNodeId": 1, "role": {"value": "button"},+ "name": {"value": "Retained"}},+ {"nodeId": "reply", "parentId": "root", "ignored": false,+ "backendDOMNodeId": 2, "role": {"value": "button"},+ "name": {"value": "Reply"}}+ ]});+ let document = compose_accessibility_tree(+ &ax,+ &dom,+ &LayoutIndex::default(),+ &Viewport::default(),+ frame(),+ );+ let page = document.page(0, 300, None, None);+ assert_eq!(page.omissions.css_hidden, 1);+ let actionable = page+ .selected+ .iter()+ .filter(|node| !node.actions.is_empty())+ .collect::<Vec<_>>();+ assert_eq!(actionable.len(), 1);+ assert_eq!(actionable[0].name.as_deref(), Some("Reply"));+ }++ #[test]+ fn dom_supplement_adds_only_explicit_visible_custom_actions() {+ let dom = build_dom_index(&json!({+ "nodeType": 9,+ "frameId": "F_MAIN",+ "children": [+ {"nodeType": 1, "nodeName": "DIV", "backendNodeId": 1,+ "attributes": ["aria-label", "Custom action", "onclick", "run()"]},+ {"nodeType": 1, "nodeName": "DIV", "backendNodeId": 2,+ "attributes": ["aria-label", "Static panel"]}+ ]+ }));+ let layout = build_layout_index(&json!({+ "strings": ["block", "visible", "1", "auto"],+ "documents": [{+ "nodes": {"backendNodeId": [1, 2]},+ "layout": {+ "nodeIndex": [0, 1],+ "bounds": [[10, 10, 100, 30], [10, 50, 100, 30]],+ "styles": [[0, 1, 2, 3, 3, 3, 3], [0, 1, 2, 3, 3, 3, 3]],+ "paintOrders": [1, 2]+ }+ }]+ }));+ let viewport = parse_viewport(&json!({+ "cssVisualViewport": {"pageX": 0.0, "pageY": 0.0,+ "clientWidth": 800.0, "clientHeight": 600.0}+ }));+ let document = compose_accessibility_tree(+ &json!({"nodes": [{"nodeId": "root", "ignored": false,+ "role": {"value": "RootWebArea"}, "childIds": []}]}),+ &dom,+ &layout,+ &viewport,+ frame(),+ );+ let page = document.page(0, 300, None, None);+ assert!(page.selected.iter().any(|node| {+ node.name.as_deref() == Some("Custom action")+ && node.actions == vec![BrowserActionKind::Click]+ }));+ assert!(page+ .selected+ .iter()+ .all(|node| node.name.as_deref() != Some("Static panel")));+ }++ #[test]+ fn layout_ranks_in_viewport_actions_before_offscreen_content() {+ let dom = build_dom_index(&json!({+ "nodeType": 9,+ "children": [+ {"nodeType": 1, "nodeName": "P", "backendNodeId": 1},+ {"nodeType": 1, "nodeName": "BUTTON", "backendNodeId": 2}+ ]+ }));+ let layout = build_layout_index(&json!({+ "strings": ["block", "visible", "1", "auto", "pointer"],+ "documents": [{+ "nodes": {"backendNodeId": [1, 2]},+ "layout": {+ "nodeIndex": [0, 1],+ "bounds": [[0, 5000, 100, 20], [10, 10, 100, 30]],+ "styles": [[0, 1, 2, 3, 4], [0, 1, 2, 3, 4]],+ "paintOrders": [1, 2]+ }+ }]+ }));+ let viewport = parse_viewport(&json!({+ "cssVisualViewport": {"pageX": 0.0, "pageY": 0.0,+ "clientWidth": 800.0, "clientHeight": 600.0}+ }));+ let ax = json!({"nodes": [+ {"nodeId": "root", "ignored": false, "role": {"value": "RootWebArea"},+ "childIds": ["text", "reply"]},+ {"nodeId": "text", "parentId": "root", "ignored": false,+ "backendDOMNodeId": 1, "role": {"value": "StaticText"},+ "name": {"value": "Old content"}},+ {"nodeId": "reply", "parentId": "root", "ignored": false,+ "backendDOMNodeId": 2, "role": {"value": "button"},+ "name": {"value": "Reply"}}+ ]});+ let document = compose_accessibility_tree(&ax, &dom, &layout, &viewport, frame());+ let page = document.page(0, 1, None, None);+ assert_eq!(page.selected[0].name.as_deref(), Some("Reply"));+ assert_eq!(page.next_offset, Some(1));+ }++ #[test]+ fn fixed_painted_overlay_marks_covered_action_as_page_occluded() {+ let dom = build_dom_index(&json!({+ "nodeType": 9,+ "frameId": "F_MAIN",+ "children": [+ {"nodeType": 1, "nodeName": "BUTTON", "backendNodeId": 1},+ {"nodeType": 1, "nodeName": "DIV", "backendNodeId": 2}+ ]+ }));+ let layout = build_layout_index(&json!({+ "strings": ["block", "visible", "1", "auto", "fixed", "100"],+ "documents": [{+ "nodes": {"backendNodeId": [1, 2]},+ "layout": {+ "nodeIndex": [0, 1],+ "bounds": [[10, 10, 100, 30], [0, 0, 800, 600]],+ "styles": [[0, 1, 2, 3, 3, 3, 3], [0, 1, 2, 3, 3, 4, 5]],+ "paintOrders": [1, 2]+ }+ }]+ }));+ let viewport = parse_viewport(&json!({+ "cssVisualViewport": {"pageX": 0.0, "pageY": 0.0,+ "clientWidth": 800.0, "clientHeight": 600.0}+ }));+ let ax = json!({"nodes": [+ {"nodeId": "root", "ignored": false, "role": {"value": "RootWebArea"},+ "childIds": ["button"]},+ {"nodeId": "button", "parentId": "root", "ignored": false,+ "backendDOMNodeId": 1, "role": {"value": "button"},+ "name": {"value": "Covered"}}+ ]});+ let document = compose_accessibility_tree(&ax, &dom, &layout, &viewport, frame());+ let page = document.page(0, 300, None, None);+ assert!(page+ .selected+ .iter()+ .all(|node| node.name.as_deref() != Some("Covered")));+ assert_eq!(page.omissions.page_occluded, 1);+ }++ #[test]+ fn semantic_text_removes_icon_glyphs_and_nonbreaking_spaces() {+ assert_eq!(+ clean_semantic_text("\u{e001} Reply\u{00a0}now \u{f8ff}".to_owned()).as_deref(),+ Some("Reply now")+ );+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rsindex 753b372b36..0b47f26ddf 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/store.rs@@ -23,8 +23,54 @@ use serde::Serialize;use uuid::Uuid;use super::refusal::{BrowserRefusal, BrowserRefusalCode};+use super::semantic::SemanticDocument;use super::types::{BindingQuality, ProcessFingerprint, Rect};+/// Browser action kinds proven for one semantic page ref.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]+#[serde(rename_all = "snake_case")]+pub enum BrowserActionKind {+ Click,+ Type,+}++impl BrowserActionKind {+ pub fn as_str(self) -> &'static str {+ match self {+ Self::Click => "click",+ Self::Type => "type",+ }+ }+}++/// Browser-layout visibility. This is independent from native desktop+/// foreground or occlusion state.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]+#[serde(rename_all = "snake_case")]+pub enum BrowserVisibility {+ InViewport,+ NearViewport,+ Offscreen,+ CssHidden,+ NoLayout,+ PageOccluded,+ Unknown,+}++impl BrowserVisibility {+ pub fn as_str(self) -> &'static str {+ match self {+ Self::InViewport => "in_viewport",+ Self::NearViewport => "near_viewport",+ Self::Offscreen => "offscreen",+ Self::CssHidden => "css_hidden",+ Self::NoLayout => "no_layout",+ Self::PageOccluded => "page_occluded",+ Self::Unknown => "unknown",+ }+ }+}+/// Which frame kind a ref was minted in. Exposed on the wire as a/// stable string via [`FrameKind::as_str`]; everything else about the/// frame stays internal.@@ -99,6 +145,16 @@ pub struct RefEntry {pub node_name: String,#[serde(skip_serializing_if = "Option::is_none")]pub label: Option<String>,+ /// Semantic action kinds. Empty for legacy DOM refs, whose existing+ /// mutation behavior remains compatible during the v2 migration.+ #[serde(default, skip_serializing_if = "Vec::is_empty")]+ pub actions: Vec<BrowserActionKind>,+ #[serde(skip_serializing_if = "Option::is_none")]+ pub visibility: Option<BrowserVisibility>,+ /// Semantic refs enforce their declared action set. Legacy DOM refs keep+ /// their existing permissive behavior during the versioned migration.+ #[serde(skip_serializing)]+ pub semantic: bool,/// Frame identity — internal; only the kind string is surfaced.#[serde(skip_serializing)]pub frame: FrameRef,@@ -111,12 +167,26 @@ pub struct SnapshotRecord {pub url: String,/// index → entry; the external ref is `p<id>:<index>`.pub refs: HashMap<u32, RefEntry>,+ pub(crate) semantic: Option<SemanticDocument>,+ pub(crate) semantic_root_identity: Option<FrameIdentity>,+ pub(crate) continuations: HashMap<String, SemanticContinuation>,+}++#[derive(Debug, Clone)]+pub struct SemanticContinuation {+ pub offset: usize,+ pub query: Option<String>,+ pub scope_backend_node_id: Option<i64>,+ pub oopif_supported: bool,+ pub oopif_frames: usize,}#[derive(Debug, Clone)]pub struct TabRecord {pub tab_id: String,pub cdp_target_id: String,+ pub title: String,+ pub url: String,pub generation: u64,pub snapshots: HashMap<u64, SnapshotRecord>,}@@ -294,6 +364,44 @@ impl BrowserStore {})}+ /// Resolve an opaque continuation within the same session, target, tab,+ /// snapshot generation, and currently-live snapshot namespace.+ pub fn resolve_semantic_continuation(+ &self,+ session: &str,+ target_id: &str,+ tab_id: &str,+ token: &str,+ ) -> Result<(SnapshotRecord, SemanticContinuation), BrowserRefusal> {+ let target = self.get_target(session, target_id)?;+ let tab = target.tabs.get(tab_id).ok_or_else(|| {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserTabNotFound,+ format!("tab {tab_id} is not known for target {target_id}"),+ )+ })?;+ tab.snapshots+ .values()+ .find(|snapshot| {+ snapshot.generation == target.generation+ && snapshot.semantic.is_some()+ && snapshot.continuations.contains_key(token)+ })+ .and_then(|snapshot| {+ snapshot+ .continuations+ .get(token)+ .cloned()+ .map(|continuation| (snapshot.clone(), continuation))+ })+ .ok_or_else(|| {+ BrowserRefusal::new(+ BrowserRefusalCode::BrowserRefStale,+ "the semantic continuation is stale or does not belong to this session and tab",+ )+ })+ }+/// Drop every snapshot of one tab (navigation invalidates refs).pub fn invalidate_tab_snapshots(&self, session: &str, target_id: &str, tab_id: &str) {self.update_target(session, target_id, |rec| {@@ -381,6 +489,9 @@ mod tests {backend_node_id: 555,node_name: "button".into(),label: Some("Submit".into()),+ actions: Vec::new(),+ visibility: None,+ semantic: false,frame: FrameRef {kind: FrameKind::Main,oopif_target_id: None,@@ -396,6 +507,8 @@ mod tests {TabRecord {tab_id: tab_id.clone(),cdp_target_id: "CDP1".into(),+ title: "Example".into(),+ url: "https://example.test".into(),generation: 0,snapshots: HashMap::from([(snap_id,@@ -404,6 +517,9 @@ mod tests {generation: 0,url: "https://example.test".into(),refs,+ semantic: None,+ semantic_root_identity: None,+ continuations: HashMap::new(),},)]),},diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rsindex ef31669caf..5c61bc1c5d 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs@@ -19,6 +19,7 @@ use super::approval::MCP_HOST_APPROVAL_ARG;use super::engine::BrowserEngine;use super::platform::{PrepareAuthorization, PrepareProfile, PrepareRequest, PrepareStrategy};use super::refusal::{BrowserRefusal, BrowserRefusalCode};+use super::store::BrowserActionKind;use super::types::BindingQuality;/// Register all five browser tools against one shared engine. Platform@@ -85,6 +86,19 @@ fn schema_session() -> Value {})}+fn semantic_ref_value(listed: &super::engine::SemanticListedRef) -> Value {+ json!({+ "ref": listed.external,+ "role": listed.node.role,+ "name": listed.node.name,+ "value": listed.node.value,+ "states": listed.node.states,+ "actions": listed.node.actions.iter().map(|action| action.as_str()).collect::<Vec<_>>(),+ "frame": listed.node.frame.kind.as_str(),+ "visibility": listed.node.visibility.as_str(),+ })+}+// ── get_browser_state ────────────────────────────────────────────────────────pub struct GetBrowserStateTool {@@ -99,11 +113,12 @@ impl GetBrowserStateTool {description: "Read-only browser inspection. Mode 1 (bind): pass pid + \window_id of a native browser window to classify it, correlate it to \a CDP target (exact-or-refuse), and mint a session-scoped target id \- plus tab ids. Mode 2 (snapshot): pass target_id + tab_id to snapshot \- the tab's composed DOM (main frame, shadow DOM, same-process iframes, \- and capability-tested out-of-process iframes) and mint \- p<snapshot>:<index> element refs for browser_click / browser_type; \- each ref reports its frame kind. Never performs setup — a missing \+ plus tab ids. Mode 2 (snapshot): pass target_id + tab_id. The \+ dom_refs_v1 compatibility format returns composed DOM refs. \+ semantic_v2 joins accessibility, DOM, layout, and viewport state; \+ ranks visible content before retained/offscreen state; and returns a \+ semantic outline, typed action refs, content refs, scoped reads, and \+ opaque continuation. Never performs setup — a missing \endpoint is a structured browser_requires_setup refusal pointing at \browser_prepare.".into(),@@ -115,6 +130,23 @@ impl GetBrowserStateTool {"target_id": schema_target_id(),"tab_id": schema_tab_id(),"session": schema_session(),+ "snapshot_format": {+ "type": "string",+ "enum": ["dom_refs_v1", "semantic_v2"],+ "description": "Versioned snapshot contract. dom_refs_v1 remains the compatibility default."+ },+ "scope_ref": {+ "type": "string",+ "description": "Current semantic/content ref whose subtree should be observed."+ },+ "query": {+ "type": "string",+ "description": "Read-only semantic match over role, accessible name, and visible text."+ },+ "continuation": {+ "type": "string",+ "description": "Opaque continuation minted by an earlier semantic_v2 response."+ },},"additionalProperties": true}),@@ -150,6 +182,94 @@ impl Tool for GetBrowserStateTool {.to_tool_result()}};+ let snapshot_format = args+ .opt_str("snapshot_format")+ .unwrap_or_else(|| "dom_refs_v1".into());+ if snapshot_format != "dom_refs_v1" && snapshot_format != "semantic_v2" {+ return ToolResult::error(format!(+ "snapshot_format must be \"dom_refs_v1\" or \"semantic_v2\", got {snapshot_format:?}"+ ));+ }+ if snapshot_format == "dom_refs_v1"+ && (args.opt_str("scope_ref").is_some()+ || args.opt_str("query").is_some()+ || args.opt_str("continuation").is_some())+ {+ return ToolResult::error(+ "scope_ref, query, and continuation require snapshot_format=\"semantic_v2\"",+ );+ }+ if snapshot_format == "semantic_v2" {+ return match self+ .engine+ .snapshot_tab_semantic(+ &session,+ &target_id,+ &tab_id,+ args.opt_str("scope_ref").as_deref(),+ args.opt_str("query").as_deref(),+ args.opt_str("continuation").as_deref(),+ )+ .await+ {+ Ok(outcome) => {+ let refs = outcome+ .refs+ .iter()+ .map(semantic_ref_value)+ .collect::<Vec<_>>();+ let content_refs = outcome+ .content_refs+ .iter()+ .map(semantic_ref_value)+ .collect::<Vec<_>>();+ ToolResult::text(format!(+ "semantic snapshot p{} of {}: {} action ref(s), {} content ref(s)",+ outcome.snapshot_id,+ outcome.url,+ refs.len(),+ content_refs.len()+ ))+ .with_structured(json!({+ "status": "ok",+ "mode": "snapshot",+ "target_id": target_id,+ "tab_id": tab_id,+ "snapshot": {+ "id": format!("p{}", outcome.snapshot_id),+ "format": "semantic_v2",+ "complete": outcome.complete,+ "scope": outcome.scope,+ "selected_nodes": outcome.selected_nodes,+ "total_nodes": outcome.total_nodes,+ "node_budget": super::semantic::DEFAULT_SEMANTIC_NODE_BUDGET,+ "omitted": {+ "css_hidden": outcome.omissions.css_hidden,+ "offscreen": outcome.omissions.offscreen,+ "page_occluded": outcome.omissions.page_occluded,+ "no_layout": outcome.omissions.no_layout,+ "unknown": outcome.omissions.unknown,+ "budget": outcome.omissions.budget,+ "unprovable_frame": outcome.omissions.unprovable_frame,+ },+ "continuation": outcome.continuation,+ },+ "page": {+ "url": outcome.url,+ "title": outcome.title,+ },+ "outline": outcome.outline,+ "refs": refs,+ "content_refs": content_refs,+ "oopif": {+ "status": outcome.oopif.as_str(),+ "frames": outcome.oopif.frames(),+ },+ }))+ }+ Err(refusal) => refusal.to_tool_result(),+ };+ }return match self.engine.snapshot_tab(&session, &target_id, &tab_id)@@ -220,6 +340,8 @@ impl Tool for GetBrowserStateTool {.map(|t| {json!({"tab_id": t.tab_id,+ "title": t.title,+ "url": t.url,"active": t.cdp_target_id == record.cdp_target_id,})})@@ -651,6 +773,13 @@ impl Tool for BrowserClickTool {Ok(entry) => entry,Err(refusal) => return refusal.to_tool_result(),};+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Click) {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!("semantic ref {r} does not declare the click action"),+ )+ .to_tool_result();+ }let frame_session = match self.engine.frame_session_for_mutation(@@ -951,6 +1080,13 @@ impl Tool for BrowserTypeTool {Ok(e) => e,Err(refusal) => return refusal.to_tool_result(),};+ if entry.semantic && !entry.actions.contains(&BrowserActionKind::Type) {+ return BrowserRefusal::new(+ BrowserRefusalCode::BrowserActionUnavailable,+ format!("semantic ref {ext_ref} does not declare the type action"),+ )+ .to_tool_result();+ }// Re-prove the ref's frame identity; typing routes to the frame's// own session (tab, or the contained OOPIF child session).let cdp_session = match self@@ -1475,6 +1611,8 @@ mod tests {crate::browser::store::TabRecord {tab_id: tab_id.clone(),cdp_target_id: "CDPX".into(),+ title: "Mock".into(),+ url: "https://example.test".into(),generation: 0,snapshots: HashMap::new(),},@@ -1529,6 +1667,8 @@ mod tests {crate::browser::store::TabRecord {tab_id: tab_id.clone(),cdp_target_id: "CDPX".into(),+ title: "Mock".into(),+ url: "https://example.test".into(),generation: 0,snapshots: HashMap::new(),},diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rsindex d44bf68450..6efcaa7b0c 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/v2_tests.rs@@ -46,6 +46,9 @@ struct FixtureState {oopif_sessions: u64,fail_key_down_after: Option<usize>,completed_key_pairs: usize,+ semantic_large_page: bool,+ semantic_full_dom_fails: bool,+ semantic_truncated_dom: bool,/// Every incoming CDP call: (sessionId, method, params).calls: Vec<(Option<String>, String, Value)>,}@@ -63,6 +66,9 @@ impl Default for FixtureState {oopif_sessions: 0,fail_key_down_after: None,completed_key_pairs: 0,+ semantic_large_page: false,+ semantic_full_dom_fails: false,+ semantic_truncated_dom: false,calls: Vec::new(),}}@@ -155,6 +161,205 @@ fn main_document() -> Value {})}+/// Large application document used to prove that hidden retained controls do+/// not consume the semantic snapshot budget ahead of the active view.+fn large_semantic_document() -> Value {+ let mut children = Vec::new();+ for id in 0..320_i64 {+ children.push(json!({+ "nodeType": 1,+ "nodeName": "BUTTON",+ "backendNodeId": 1_000 + id,+ "attributes": [+ "aria-hidden", "true",+ "style", "display:none",+ "aria-label", format!("Retained control {id}"),+ ],+ }));+ }+ children.extend([+ json!({+ "nodeType": 1,+ "nodeName": "H1",+ "backendNodeId": 2_000,+ "children": [{+ "nodeType": 3,+ "nodeName": "#text",+ "nodeValue": "Visible message",+ "backendNodeId": 2_001,+ }],+ }),+ json!({+ "nodeType": 1,+ "nodeName": "P",+ "backendNodeId": 2_002,+ "children": [{+ "nodeType": 3,+ "nodeName": "#text",+ "nodeValue": "Please review the attached fixture report.",+ "backendNodeId": 2_003,+ }],+ }),+ json!({+ "nodeType": 1,+ "nodeName": "TEXTAREA",+ "backendNodeId": 2_010,+ "attributes": ["aria-label", "Reply body"],+ }),+ json!({+ "nodeType": 1,+ "nodeName": "BUTTON",+ "backendNodeId": 2_011,+ "attributes": ["aria-label", "Reply"],+ }),+ ]);+ for id in 0..305_i64 {+ children.push(json!({+ "nodeType": 1,+ "nodeName": "BUTTON",+ "backendNodeId": 3_000 + id,+ "attributes": ["aria-label", format!("Archive item {id}")],+ }));+ }+ json!({+ "root": {+ "nodeType": 9,+ "nodeName": "#document",+ "documentURL": "https://fixture.test/inbox/item-2",+ "frameId": "F_MAIN",+ "children": [{+ "nodeType": 1,+ "nodeName": "HTML",+ "backendNodeId": 999,+ "children": children,+ }]+ }+ })+}++fn truncated_semantic_document() -> Value {+ json!({+ "root": {+ "nodeType": 9,+ "nodeName": "#document",+ "documentURL": "https://fixture.test/inbox/item-2",+ "frameId": "F_MAIN",+ "children": [{+ "nodeType": 1,+ "nodeName": "HTML",+ "backendNodeId": 999,+ "childNodeCount": 629,+ "children": [],+ }]+ }+ })+}++fn large_semantic_ax_tree(frame_id: &str) -> Value {+ if frame_id != "F_MAIN" {+ return json!({"nodes": [{+ "nodeId": format!("root-{frame_id}"),+ "ignored": false,+ "role": {"value": "RootWebArea"},+ "name": {"value": "Inner fixture"},+ "childIds": []+ }]});+ }+ let mut child_ids = vec![+ "heading".to_owned(),+ "body".to_owned(),+ "editor".to_owned(),+ "reply".to_owned(),+ ];+ child_ids.extend((0..305).map(|id| format!("archive-{id}")));+ let mut nodes = vec![+ json!({+ "nodeId": "root-main",+ "ignored": false,+ "role": {"value": "RootWebArea"},+ "name": {"value": "Fixture inbox"},+ "childIds": child_ids+ }),+ json!({+ "nodeId": "heading",+ "parentId": "root-main",+ "ignored": false,+ "backendDOMNodeId": 2000,+ "role": {"value": "heading"},+ "name": {"value": "Visible message"},+ "childIds": []+ }),+ json!({+ "nodeId": "body",+ "parentId": "root-main",+ "ignored": false,+ "backendDOMNodeId": 2003,+ "role": {"value": "StaticText"},+ "name": {"value": "Please review the attached fixture report."},+ "childIds": []+ }),+ json!({+ "nodeId": "editor",+ "parentId": "root-main",+ "ignored": false,+ "backendDOMNodeId": 2010,+ "role": {"value": "textbox"},+ "name": {"value": "Reply body"},+ "value": {"value": ""},+ "properties": [+ {"name": "editable", "value": {"value": "plaintext"}},+ {"name": "focusable", "value": {"value": true}}+ ],+ "childIds": []+ }),+ json!({+ "nodeId": "reply",+ "parentId": "root-main",+ "ignored": false,+ "backendDOMNodeId": 2011,+ "role": {"value": "button"},+ "name": {"value": "Reply"},+ "properties": [+ {"name": "disabled", "value": {"value": false}},+ {"name": "focusable", "value": {"value": true}}+ ],+ "childIds": []+ }),+ ];+ for id in 0..305_i64 {+ nodes.push(json!({+ "nodeId": format!("archive-{id}"),+ "parentId": "root-main",+ "ignored": false,+ "backendDOMNodeId": 3_000 + id,+ "role": {"value": "button"},+ "name": {"value": format!("Archive item {id}")},+ "childIds": []+ }));+ }+ json!({"nodes": nodes})+}++fn semantic_layout_snapshot(backends: &[i64], bounds: &[[f64; 4]]) -> Value {+ let styles = (0..backends.len())+ .map(|_| json!([0, 1, 2, 3, 3]))+ .collect::<Vec<_>>();+ let node_indices = (0..backends.len()).collect::<Vec<_>>();+ let paint_orders = (1..=backends.len()).collect::<Vec<_>>();+ json!({+ "strings": ["block", "visible", "1", "auto", "pointer"],+ "documents": [{+ "nodes": {"backendNodeId": backends},+ "layout": {+ "nodeIndex": node_indices,+ "bounds": bounds,+ "styles": styles,+ "paintOrders": paint_orders+ }+ }]+ })+}+fn oopif_document() -> Value {json!({"root": {@@ -233,8 +438,74 @@ fn fixture_handler(state: SharedState) -> MockHandler {}}})),- "DOM.getDocument" if is_tab => MockReply::ok(main_document()),+ "DOM.getDocument" if is_tab => {+ let depth = call.params["depth"].as_i64().unwrap_or(-1);+ if st.semantic_full_dom_fails && (depth == -1 || depth > 8) {+ MockReply::err(-32000, "Object reference chain is too long")+ } else if st.semantic_truncated_dom && depth == 8 {+ MockReply::ok(truncated_semantic_document())+ } else {+ MockReply::ok(if st.semantic_large_page {+ large_semantic_document()+ } else {+ main_document()+ })+ }+ }+ "DOM.describeNode" if is_tab && call.params["backendNodeId"] == 999 => {+ MockReply::ok(json!({+ "node": large_semantic_document()["root"]["children"][0].clone()+ }))+ }"DOM.getDocument" if is_oopif => MockReply::ok(oopif_document()),+ "Accessibility.getFullAXTree" if is_tab => {+ let frame_id = call.params["frameId"].as_str().unwrap_or("F_MAIN");+ if st.semantic_large_page {+ MockReply::ok(large_semantic_ax_tree(frame_id))+ } else {+ MockReply::ok(json!({"nodes": []}))+ }+ }+ "Accessibility.getFullAXTree" if is_oopif => MockReply::ok(json!({"nodes": [+ {"nodeId": "oopif-root", "ignored": false,+ "role": {"value": "RootWebArea"}, "childIds": ["oopif-input"]},+ {"nodeId": "oopif-input", "parentId": "oopif-root", "ignored": false,+ "backendDOMNodeId": 100, "role": {"value": "textbox"},+ "name": {"value": "Embedded input"},+ "properties": [{"name": "editable", "value": {"value": "plaintext"}}],+ "childIds": []}+ ]})),+ "DOMSnapshot.captureSnapshot" if is_tab => {+ if st.semantic_large_page {+ let mut backends = vec![999, 2000, 2003, 2010, 2011];+ let mut bounds = vec![+ [0.0, 0.0, 800.0, 600.0],+ [20.0, 20.0, 500.0, 40.0],+ [20.0, 80.0, 600.0, 80.0],+ [20.0, 180.0, 600.0, 120.0],+ [20.0, 320.0, 100.0, 36.0],+ ];+ for id in 0..305_i64 {+ backends.push(3_000 + id);+ bounds.push([20.0, 2_000.0 + id as f64 * 40.0, 160.0, 30.0]);+ }+ MockReply::ok(semantic_layout_snapshot(&backends, &bounds))+ } else {+ MockReply::ok(semantic_layout_snapshot(&[], &[]))+ }+ }+ "DOMSnapshot.captureSnapshot" if is_oopif => MockReply::ok(semantic_layout_snapshot(+ &[90, 100],+ &[[0.0, 0.0, 300.0, 100.0], [10.0, 10.0, 120.0, 30.0]],+ )),+ "Page.getLayoutMetrics" => MockReply::ok(json!({+ "cssVisualViewport": {+ "pageX": 0.0,+ "pageY": 0.0,+ "clientWidth": 800.0,+ "clientHeight": 600.0+ }+ })),"Target.setAutoAttach" if is_tab => {if call.params["autoAttach"].as_bool() == Some(false) {return MockReply::ok(json!({}));@@ -818,6 +1089,35 @@ async fn snapshot(f: &Fixture, target_id: &str, tab_id: &str) -> Value {structured(&result).clone()}+async fn semantic_snapshot(f: &Fixture, target_id: &str, tab_id: &str) -> Value {+ let tool = GetBrowserStateTool::new(f.engine.clone());+ let result = tool+ .invoke(json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "session": SESSION,+ "snapshot_format": "semantic_v2"+ }))+ .await;+ structured(&result).clone()+}++async fn semantic_snapshot_with(f: &Fixture, target_id: &str, tab_id: &str, extra: Value) -> Value {+ let mut args = json!({+ "target_id": target_id,+ "tab_id": tab_id,+ "session": SESSION,+ "snapshot_format": "semantic_v2"+ });+ args.as_object_mut()+ .unwrap()+ .extend(extra.as_object().unwrap().clone());+ let result = GetBrowserStateTool::new(f.engine.clone())+ .invoke(args)+ .await;+ structured(&result).clone()+}+/// The `ref` string of the first snapshot entry in the given frame kind/// with the given backing label fragment (or any, if empty).fn ref_of(snapshot: &Value, frame: &str, label_fragment: &str) -> String {@@ -887,6 +1187,231 @@ async fn snapshot_composes_shadow_iframe_and_oopif_refs() {);}+#[tokio::test]+async fn semantic_snapshot_keeps_visible_content_after_hidden_node_pressure() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let snap = semantic_snapshot(&f, &target, &tab).await;++ assert_eq!(snap["status"], "ok", "{snap}");+ assert_eq!(snap["snapshot"]["format"], "semantic_v2", "{snap}");+ assert!(+ snap["outline"]+ .as_str()+ .is_some_and(|outline| outline.contains("Visible message")+ && outline.contains("Please review the attached fixture report.")),+ "visible semantic content was omitted: {snap}"+ );+ let refs = snap["refs"].as_array().expect("semantic refs");+ assert!(+ refs.iter().any(|entry| entry["name"] == "Reply"+ && entry["actions"]+ .as_array()+ .is_some_and(|actions| actions.iter().any(|action| action == "click"))),+ "visible Reply action was omitted: {snap}"+ );+ assert!(+ refs.iter().any(|entry| entry["name"] == "Reply body"+ && entry["actions"]+ .as_array()+ .is_some_and(|actions| actions.iter().any(|action| action == "type"))),+ "visible editor was omitted: {snap}"+ );+ assert!(+ refs.iter().all(|entry| !entry["name"]+ .as_str()+ .unwrap_or("")+ .starts_with("Retained control")),+ "CSS-hidden retained controls leaked into refs: {snap}"+ );+ assert_eq!(snap["snapshot"]["omitted"]["css_hidden"], 320);+}++#[tokio::test]+async fn semantic_continuation_is_opaque_single_use_and_reaches_offscreen_content() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let first = semantic_snapshot(&f, &target, &tab).await;+ let token = first["snapshot"]["continuation"]+ .as_str()+ .expect("large fixture continuation")+ .to_owned();+ assert!(token.starts_with("bc-"), "{first}");++ let continued = semantic_snapshot_with(&f, &target, &tab, json!({"continuation": token})).await;+ assert_eq!(continued["status"], "ok", "{continued}");+ assert_eq!(continued["snapshot"]["scope"], "continuation");+ assert!(+ continued["refs"]+ .as_array()+ .unwrap()+ .iter()+ .any(|entry| { entry["name"] == "Archive item 304" }),+ "last offscreen action was not reachable: {continued}"+ );++ let reused = semantic_snapshot_with(&f, &target, &tab, json!({"continuation": token})).await;+ assert_eq!(reused["status"], "refused", "{reused}");+ assert_eq!(reused["refusal"]["code"], "browser_ref_stale");+}++#[tokio::test]+async fn newer_semantic_snapshot_invalidates_prior_continuations() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let first = semantic_snapshot(&f, &target, &tab).await;+ let token = first["snapshot"]["continuation"]+ .as_str()+ .expect("large fixture continuation")+ .to_owned();+ let newer = semantic_snapshot(&f, &target, &tab).await;+ assert_eq!(newer["status"], "ok", "{newer}");++ let stale = semantic_snapshot_with(&f, &target, &tab, json!({"continuation": token})).await;+ assert_eq!(stale["status"], "refused", "{stale}");+ assert_eq!(stale["refusal"]["code"], "browser_ref_stale");+}++#[tokio::test]+async fn main_frame_navigation_invalidates_semantic_continuations() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let first = semantic_snapshot(&f, &target, &tab).await;+ let token = first["snapshot"]["continuation"]+ .as_str()+ .expect("large fixture continuation")+ .to_owned();++ f.state.lock().unwrap().main_loader = "L_MAIN_2".into();++ let stale = semantic_snapshot_with(&f, &target, &tab, json!({"continuation": token})).await;+ assert_eq!(stale["status"], "refused", "{stale}");+ assert_eq!(stale["refusal"]["code"], "browser_ref_stale");+}++#[tokio::test]+async fn semantic_query_and_content_scope_are_read_only_and_precise() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let queried =+ semantic_snapshot_with(&f, &target, &tab, json!({"query": "Archive item 304"})).await;+ assert_eq!(queried["snapshot"]["scope"], "query", "{queried}");+ assert_eq!(queried["refs"].as_array().unwrap().len(), 1, "{queried}");+ assert_eq!(queried["refs"][0]["name"], "Archive item 304");++ let fresh = semantic_snapshot(&f, &target, &tab).await;+ let heading_ref = fresh["content_refs"]+ .as_array()+ .unwrap()+ .iter()+ .find(|entry| entry["name"] == "Visible message")+ .and_then(|entry| entry["ref"].as_str())+ .expect("heading content ref")+ .to_owned();+ let scoped = semantic_snapshot_with(&f, &target, &tab, json!({"scope_ref": heading_ref})).await;+ assert_eq!(scoped["snapshot"]["scope"], "subtree", "{scoped}");+ assert_eq!(scoped["refs"].as_array().unwrap().len(), 0, "{scoped}");+ assert_eq!(+ scoped["content_refs"].as_array().unwrap().len(),+ 1,+ "{scoped}"+ );+ assert_eq!(scoped["content_refs"][0]["name"], "Visible message");+}++#[tokio::test]+async fn semantic_refs_enforce_declared_action_kinds_before_delivery() {+ let f = fixture_with(|st| st.semantic_large_page = true).await;+ let (target, tab) = bind(&f).await;+ let snap = semantic_snapshot(&f, &target, &tab).await;+ let content_ref = snap["content_refs"]+ .as_array()+ .unwrap()+ .iter()+ .find(|entry| entry["name"] == "Visible message")+ .and_then(|entry| entry["ref"].as_str())+ .unwrap();+ let click = BrowserClickTool::new(f.engine.clone())+ .invoke(json!({+ "target_id": target,+ "tab_id": tab,+ "session": SESSION,+ "ref": content_ref,+ "input_route": "dom_event"+ }))+ .await;+ assert_eq!(+ structured(&click)["refusal"]["code"],+ "browser_action_unavailable"+ );++ let button_ref = snap["refs"]+ .as_array()+ .unwrap()+ .iter()+ .find(|entry| entry["name"] == "Reply")+ .and_then(|entry| entry["ref"].as_str())+ .unwrap();+ let typed = BrowserTypeTool::new(f.engine.clone())+ .invoke(json!({+ "target_id": target,+ "tab_id": tab,+ "session": SESSION,+ "ref": button_ref,+ "text": "must not deliver"+ }))+ .await;+ assert_eq!(+ structured(&typed)["refusal"]["code"],+ "browser_action_unavailable"+ );+ assert!(recorded_calls(&f, "Input.insertText").is_empty());+ assert!(recorded_calls(&f, "Runtime.callFunctionOn").is_empty());+}++#[tokio::test]+async fn semantic_snapshot_uses_bounded_dom_fallback_only_for_known_size_failures() {+ let f = fixture_with(|st| {+ st.semantic_large_page = true;+ st.semantic_full_dom_fails = true;+ })+ .await;+ let (target, tab) = bind(&f).await;+ let snap = semantic_snapshot_with(&f, &target, &tab, json!({"query": "Reply"})).await;+ assert_eq!(snap["status"], "ok", "{snap}");+ assert_eq!(snap["snapshot"]["complete"], false, "{snap}");+ let document_calls = recorded_calls(&f, "DOM.getDocument");+ assert!(document_calls+ .iter()+ .any(|(_, params)| params["depth"] == -1));+ assert!(document_calls+ .iter()+ .any(|(_, params)| params["depth"] == 256));+ assert!(document_calls+ .iter()+ .any(|(_, params)| params["depth"] == 8));+}++#[tokio::test]+async fn semantic_snapshot_hydrates_truncated_fallback_branches() {+ let f = fixture_with(|st| {+ st.semantic_large_page = true;+ st.semantic_full_dom_fails = true;+ st.semantic_truncated_dom = true;+ })+ .await;+ let (target, tab) = bind(&f).await;+ let snap = semantic_snapshot_with(&f, &target, &tab, json!({"query": "Reply"})).await;+ assert_eq!(snap["status"], "ok", "{snap}");+ assert_eq!(snap["snapshot"]["complete"], false, "{snap}");+ assert!(snap["refs"]+ .as_array()+ .is_some_and(|refs| refs.iter().any(|entry| entry["name"] == "Reply")));+ assert!(recorded_calls(&f, "DOM.describeNode")+ .iter()+ .any(|(_, params)| params["backendNodeId"] == 999));+}+#[tokio::test]async fn unproven_oopif_capability_is_omitted_not_guessed() {let f = fixture_with(|st| st.oopif_supported = false).await;diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rsindex ab8707cd25..efba0fa669 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs@@ -313,6 +313,7 @@ pub enum RefusalCode {BrowserConsentRevoked,BrowserReconnectExhausted,BrowserInputIncomplete,+ BrowserActionUnavailable,}impl RefusalCode {@@ -333,6 +334,7 @@ impl RefusalCode {"browser_consent_revoked" => Some(Self::BrowserConsentRevoked),"browser_reconnect_exhausted" => Some(Self::BrowserReconnectExhausted),"browser_input_incomplete" => Some(Self::BrowserInputIncomplete),+ "browser_action_unavailable" => Some(Self::BrowserActionUnavailable),_ => None,}}diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rsindex 6487468d09..49e79ef992 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/standalone_browser_behavior_test.rs@@ -45,6 +45,29 @@ fn standalone_fixture_html() -> String {)}+fn standalone_semantic_fixture_html() -> String {+ let hidden = (0..320)+ .map(|index| {+ format!(+ r#"<button hidden aria-hidden="true" aria-label="Retained control {index}">hidden</button>"#+ )+ })+ .collect::<String>();+ let offscreen = (0..305)+ .map(|index| {+ format!(r#"<button aria-label="Archive item {index}">Archive item {index}</button>"#)+ })+ .collect::<String>();+ standalone_fixture_html()+ .replace("<body>", &format!("<body>{hidden}"))+ .replace(+ "</body>",+ &format!(+ r#"<section aria-label="Retained archive" style="margin-top:2000px">{offscreen}</section></body>"#+ ),+ )+}+fn standalone_frame_fixture_html(oopif_url: &str) -> String {let oopif_url = serde_json::to_string(oopif_url).expect("serialize OOPIF fixture URL");FIXTURE_HTML.replace(@@ -837,6 +860,22 @@ fn ref_by_frame_label(snapshot: &ToolResponse, frame: &str, fragment: &str) -> S.to_owned()}+fn semantic_ref_by_name(snapshot: &ToolResponse, name: &str, action: &str) -> String {+ snapshot.structured()["refs"]+ .as_array()+ .and_then(|refs| {+ refs.iter().find(|entry| {+ entry["name"] == name+ && entry["actions"]+ .as_array()+ .is_some_and(|actions| actions.iter().any(|value| value == action))+ })+ })+ .and_then(|entry| entry["ref"].as_str())+ .unwrap_or_else(|| panic!("missing semantic {action} ref {name:?}: {}", snapshot.raw))+ .to_owned()+}+fn bind(fixture: &mut BrowserFixture, session: &str) -> (String, String, ToolResponse) {let started = fixture.driver@@ -1001,6 +1040,110 @@ fn run_roundtrip(spec: &BrowserSpec) {});}+fn run_semantic_state(spec: &BrowserSpec) {+ let scenario = format!(+ "{}-{}-standalone-semantic-state",+ std::env::consts::OS,+ spec.name+ );+ execute_case(case(&spec.name, "browser_semantic_state"), |evidence| {+ let mut fixture =+ launch_browser_with_html(spec, &scenario, standalone_semantic_fixture_html());+ *evidence = recording_evidence(fixture.driver.recording_dir());+ run_with_background_oracles(&mut fixture, |fixture| {+ let session = format!("standalone-semantic-state-{}", fixture.pid);+ let (target, tab, _) = bind(fixture, &session);+ let snapshot = fixture.driver.call(+ "get_browser_state",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "session": session,+ "snapshot_format": "semantic_v2",+ }),+ );+ assert_eq!(snapshot.structured()["status"], "ok", "{}", snapshot.raw);+ assert_eq!(+ snapshot.structured()["snapshot"]["format"],+ "semantic_v2",+ "{}",+ snapshot.raw+ );+ assert!(+ snapshot.structured()["outline"]+ .as_str()+ .is_some_and(|outline| outline.contains("WEB_HARNESS_MARKER_v1")),+ "visible fixture marker missing from semantic outline: {}",+ snapshot.raw+ );+ assert!(+ snapshot.structured()["snapshot"]["omitted"]["css_hidden"]+ .as_u64()+ .is_some_and(|count| count >= 320),+ "hidden retained controls were not accounted for: {}",+ snapshot.raw+ );+ assert!(+ snapshot.structured()["snapshot"]["continuation"].is_string(),+ "offscreen fixture state did not produce continuation: {}",+ snapshot.raw+ );+ assert!(+ snapshot.structured()["refs"]+ .as_array()+ .is_some_and(|refs| refs.iter().all(|entry| {+ !entry["name"]+ .as_str()+ .unwrap_or_default()+ .starts_with("Retained control")+ })),+ "hidden retained action leaked into semantic refs: {}",+ snapshot.raw+ );++ let increment_ref = semantic_ref_by_name(&snapshot, "Increment", "click");+ let clicked = fixture.driver.call(+ "browser_click",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": increment_ref,+ "input_route": "dom_event",+ "session": session,+ }),+ );+ assert_eq!(clicked.structured()["status"], "ok", "{}", clicked.raw);+ wait_for_text(&fixture.server, "lbl-counter", "counter=1");++ let refreshed = fixture.driver.call(+ "get_browser_state",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "session": session,+ "snapshot_format": "semantic_v2",+ "query": "txt-input",+ }),+ );+ let input_ref = semantic_ref_by_name(&refreshed, "txt-input", "type");+ let typed = fixture.driver.call(+ "browser_type",+ serde_json::json!({+ "target_id": target,+ "tab_id": tab,+ "ref": input_ref,+ "text": "semantic-browser",+ "session": session,+ }),+ );+ assert_eq!(typed.structured()["status"], "ok", "{}", typed.raw);+ wait_for_value(&fixture.server, "txt-input", "semantic-browser");++ Observation::delivered(vec![OracleKind::FixtureState], Evidence::default())+ })+ });+}+fn run_background_type(spec: &BrowserSpec) {let scenario = format!("{}-{}-standalone-background-type",@@ -1863,6 +2006,7 @@ macro_rules! standalone_browser_test {}standalone_browser_test!(standalone_browser_roundtrip, run_roundtrip);+standalone_browser_test!(standalone_browser_semantic_state, run_semantic_state);standalone_browser_test!(standalone_browser_background_type, run_background_type);standalone_browser_test!(standalone_browser_trusted_click, run_trusted_click);standalone_browser_test!(diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/consent_ui.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/consent_ui.rsindex 1840138770..0fe597da57 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/consent_ui.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/consent_ui.rs@@ -1,6 +1,7 @@//! Exact macOS handling for Chrome's browser-owned remote-debugging consent.use std::time::{Duration, Instant};+use std::{collections::HashSet, iter};use core_foundation::base::{CFRelease, CFTypeRef};use cua_driver_core::browser::{@@ -35,6 +36,29 @@ fn release_actionable_nodes(nodes: &[AXNode]) {}}+fn consent_surface_ids(+ windows: impl IntoIterator<Item = crate::windows::WindowInfo>,+ pid: i32,+ approved_window_id: u32,+) -> Vec<u32> {+ let mut windows = windows+ .into_iter()+ .filter(|window| {+ window.pid == pid+ && window.title != "Allow remote debugging?"+ && !window.title.trim().is_empty()+ && window.bounds.width > 0.0+ && window.bounds.height > 0.0+ })+ .collect::<Vec<_>>();+ windows.sort_by(|left, right| right.z_index.cmp(&left.z_index));+ let mut seen = HashSet::new();+ iter::once(approved_window_id)+ .chain(windows.into_iter().map(|window| window.window_id))+ .filter(|window_id| seen.insert(*window_id))+ .collect()+}+fn remote_debugging_sheet_present(nodes: &[AXNode]) -> bool {nodes.iter().enumerate().any(|(sheet_index, sheet)| {if sheet.role != "AXSheet" {@@ -120,34 +144,64 @@ pub async fn handle(let deadline = Instant::now() + Duration::from_secs(4);let mut saw_prompt = false;loop {- let tree = tokio::task::spawn_blocking(move || walk_tree(pid, Some(window_id), None))- .await- .map_err(|error| {- refusal(- BrowserRefusalCode::BrowserRouteUnavailable,- format!("could not inspect the browser consent UI: {error}"),- )- })?;- let prompt_present = remote_debugging_sheet_present(&tree.nodes);+ let trees = tokio::task::spawn_blocking(move || {+ consent_surface_ids(crate::windows::all_windows(), pid, window_id)+ .into_iter()+ .map(|candidate_window_id| walk_tree(pid, Some(candidate_window_id), None).nodes)+ .collect::<Vec<_>>()+ })+ .await+ .map_err(|error| {+ refusal(+ BrowserRefusalCode::BrowserRouteUnavailable,+ format!("could not inspect the browser consent UI: {error}"),+ )+ })?;+ let prompt_present = trees+ .iter()+ .any(|nodes| remote_debugging_sheet_present(nodes));saw_prompt |= prompt_present;- let candidate = exact_allow_button(&tree.nodes);- match candidate {- Ok(Some(element)) => {- let pressed = unsafe { perform_action(element as AXUIElementRef, "AXPress") };- release_actionable_nodes(&tree.nodes);- if pressed != kAXErrorSuccess {- return Err(refusal(- BrowserRefusalCode::BrowserWrongTargetRefused,- "the exact browser consent action became stale before AXPress",- ));+ let mut candidates = Vec::new();+ let mut matcher_error = None;+ for nodes in &trees {+ match exact_allow_button(nodes) {+ Ok(Some(element)) => candidates.push(element),+ Ok(None) => {}+ Err(error) => {+ matcher_error = Some(error);+ break;}- return Ok(BrowserConsentOutcome::Accepted);}- Ok(None) => release_actionable_nodes(&tree.nodes),- Err(error) => {- release_actionable_nodes(&tree.nodes);- return Err(error);+ }+ candidates.sort_unstable();+ candidates.dedup();+ if let Some(error) = matcher_error {+ for nodes in &trees {+ release_actionable_nodes(nodes);+ }+ return Err(error);+ }+ if let [element] = candidates.as_slice() {+ let pressed = unsafe { perform_action(*element as AXUIElementRef, "AXPress") };+ for nodes in &trees {+ release_actionable_nodes(nodes);}+ if pressed != kAXErrorSuccess {+ return Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "the exact browser consent action became stale before AXPress",+ ));+ }+ return Ok(BrowserConsentOutcome::Accepted);+ }+ for nodes in &trees {+ release_actionable_nodes(nodes);+ }+ if candidates.len() > 1 {+ return Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ "multiple Chrome-owned remote-debugging consent sheets exposed semantic allow actions",+ ));}if saw_prompt && !prompt_present {return Err(refusal(@@ -215,4 +269,37 @@ mod tests {BrowserRefusalCode::BrowserWrongTargetRefused);}++ #[test]+ fn consent_surfaces_keep_approved_window_then_frontmost_normal_windows() {+ let window = |window_id, title: &str, z_index| crate::windows::WindowInfo {+ window_id,+ pid: 42,+ app_name: "Google Chrome".to_owned(),+ title: title.to_owned(),+ bounds: crate::windows::WindowBounds {+ x: 0.0,+ y: 0.0,+ width: 1200.0,+ height: 800.0,+ },+ layer: 0,+ z_index,+ is_on_screen: true,+ on_current_space: Some(true),+ space_ids: None,+ };+ assert_eq!(+ consent_surface_ids(+ [+ window(7, "Approved", 10),+ window(8, "Frontmost", 30),+ window(9, "Allow remote debugging?", 40),+ ],+ 42,+ 7,+ ),+ vec![7, 8]+ );+ }}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/platform.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/platform.rsindex 7f62284f1f..d4e1145c1d 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/platform.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/platform.rs@@ -1,6 +1,7 @@//! macOS identity and endpoint evidence for the first-class browser tools.use std::hash::{Hash, Hasher};+use std::path::PathBuf;use std::process::Stdio;use std::time::Duration;@@ -89,6 +90,54 @@ fn stable_hash(value: &str) -> u64 {hasher.finish()}+fn default_user_data_dir(product: BrowserProduct) -> Option<PathBuf> {+ let home = std::env::var_os("HOME").map(PathBuf::from)?;+ let relative = match product {+ BrowserProduct::GoogleChrome => "Library/Application Support/Google/Chrome",+ BrowserProduct::MicrosoftEdge => "Library/Application Support/Microsoft Edge",+ BrowserProduct::Chromium => "Library/Application Support/Chromium",+ _ => return None,+ };+ Some(home.join(relative))+}++fn parse_devtools_active_port(text: &str) -> Option<(u16, &str)> {+ let mut lines = text.lines().map(str::trim).filter(|line| !line.is_empty());+ let port = lines.next()?.parse::<u16>().ok()?;+ let path = lines.next()?;+ if lines.next().is_some() {+ return None;+ }+ let instance = path.strip_prefix("/devtools/browser/")?;+ (!instance.is_empty()+ && instance+ .bytes()+ .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_'))+ .then_some((port, path))+}++async fn process_uses_custom_user_data_dir(pid: i64) -> Result<bool, BrowserRefusal> {+ let output = tokio::process::Command::new("ps")+ .args(["-p", &pid.to_string(), "-o", "command="])+ .stdin(Stdio::null())+ .stderr(Stdio::null())+ .output()+ .await+ .map_err(|error| {+ refusal(+ BrowserRefusalCode::BrowserRouteUnavailable,+ format!("could not inspect browser arguments for pid {pid}: {error}"),+ )+ })?;+ if !output.status.success() {+ return Err(refusal(+ BrowserRefusalCode::BrowserBindingStale,+ format!("browser process {pid} is no longer available"),+ ));+ }+ Ok(String::from_utf8_lossy(&output.stdout).contains("--user-data-dir"))+}+fn exact_browser_surface_ids(windows: impl IntoIterator<Item = crate::windows::WindowInfo>,pid: i64,@@ -183,6 +232,54 @@ async fn loopback_ports_for_pid(pid: i64) -> Result<Vec<u16>, BrowserRefusal> {)))}+async fn active_port_endpoint(+ pid: i64,+ product: BrowserProduct,+) -> Result<Option<OwnedEndpoint>, BrowserRefusal> {+ // The Chrome 144+ existing-profile bridge deliberately returns 404 for+ // legacy /json discovery. Its exact browser WebSocket path is instead+ // published in the default profile's DevToolsActivePort file. Custom+ // user-data dirs remain on the launch/HTTP discovery paths because this+ // adapter cannot safely recover an arbitrary argv path from `ps` text.+ if process_uses_custom_user_data_dir(pid).await? {+ return Ok(None);+ }+ let Some(user_data_dir) = default_user_data_dir(product) else {+ return Ok(None);+ };+ let text = match tokio::fs::read_to_string(user_data_dir.join("DevToolsActivePort")).await {+ Ok(text) => text,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),+ Err(error) => {+ return Err(refusal(+ BrowserRefusalCode::BrowserRouteUnavailable,+ format!("could not read the browser's DevToolsActivePort file: {error}"),+ ))+ }+ };+ let Some((port, path)) = parse_devtools_active_port(&text) else {+ return Err(refusal(+ BrowserRefusalCode::BrowserEndpointOwnerMismatch,+ "the browser's DevToolsActivePort file did not contain one exact browser endpoint",+ ));+ };+ if !loopback_ports_for_pid(pid).await?.contains(&port) {+ return Ok(None);+ }+ Ok(Some(OwnedEndpoint {+ ws_url: format!("ws://127.0.0.1:{port}{path}"),+ http_port: Some(port),+ ownership: EndpointOwnershipProof {+ method: EndpointOwnershipMethod::DevtoolsActivePortsFile,+ owner_pid: pid,+ detail: Some(+ "exact default-profile DevToolsActivePort path plus lsof loopback listener owner"+ .to_owned(),+ ),+ },+ }))+}+async fn browser_websocket_url(port: u16) -> Option<String> {tokio::time::timeout(Duration::from_secs(2), async move {let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port))@@ -368,6 +465,10 @@ impl BrowserPlatform for MacOsBrowserPlatform {&self,pid: i64,) -> Result<Option<OwnedEndpoint>, BrowserRefusal> {+ let classification = self.classify_browser(pid).await?;+ if let Some(endpoint) = active_port_endpoint(pid, classification.product_kind).await? {+ return Ok(Some(endpoint));+ }let ports = loopback_ports_for_pid(pid).await?;let mut discovered = Vec::new();for port in &ports {@@ -414,6 +515,16 @@ impl BrowserPlatform for MacOsBrowserPlatform {if !loopback_ports_for_pid(pid).await?.contains(&port) {return Ok(None);}+ let classification = self.classify_browser(pid).await?;+ if let Some(endpoint) = active_port_endpoint(pid, classification.product_kind).await? {+ if endpoint.ws_url != expected_ws_url {+ return Err(refusal(+ BrowserRefusalCode::BrowserEndpointOwnerMismatch,+ "the browser's exact DevTools endpoint changed after approval",+ ));+ }+ return Ok(Some(endpoint));+ }Ok(Some(OwnedEndpoint {ws_url: expected_ws_url.to_owned(),http_port: Some(port),@@ -479,6 +590,11 @@ impl BrowserPlatform for MacOsBrowserPlatform {let deadline = std::time::Instant::now() + Duration::from_secs(5);let endpoint_result = loop {+ match active_port_endpoint(request.pid, request.browser).await {+ Ok(Some(endpoint)) => break Ok(endpoint),+ Ok(None) => {}+ Err(error) => break Err(error),+ }let ports = match loopback_ports_for_pid(request.pid).await {Ok(ports) => ports,Err(error) => break Err(error),@@ -721,6 +837,31 @@ mod tests {);}+ #[test]+ fn active_port_parser_requires_one_exact_browser_path() {+ assert_eq!(+ parse_devtools_active_port(+ "9222\n/devtools/browser/f1d991b4-2694-4b28-b63a-1f2a8da3a435\n"+ ),+ Some((+ 9222,+ "/devtools/browser/f1d991b4-2694-4b28-b63a-1f2a8da3a435"+ ))+ );+ assert_eq!(+ parse_devtools_active_port("9222\n/devtools/page/id\n"),+ None+ );+ assert_eq!(+ parse_devtools_active_port("9222\n/devtools/browser/id\nextra\n"),+ None+ );+ assert_eq!(+ parse_devtools_active_port("9222\n/devtools/browser/../page\n"),+ None+ );+ }+#[test]fn embedded_window_proof_ignores_untitled_window_server_helpers() {let mut helper = window(8, 42, "");diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rsindex d758f5ec87..be651347cc 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs@@ -13,8 +13,8 @@ use cua_driver_core::browser::{};use crate::ax::bindings::{- copy_number_attr, copy_string_attr, element_screen_center, focused_element_of_pid,- kAXErrorSuccess, perform_action, set_bool_attr_true, AXUIElementRef,+ copy_number_attr, copy_string_attr, element_screen_center, kAXErrorSuccess, perform_action,+ set_string_attr, AXUIElementRef,};use crate::ax::tree::{walk_tree, AXNode};@@ -112,6 +112,100 @@ fn unique_omnibox(})}+fn is_exact_setup_suggestion(value: &str, setup_url: &str) -> bool {+ let value = value.trim();+ value.eq_ignore_ascii_case(setup_url)+ || value.strip_prefix(setup_url).is_some_and(|suffix| {+ suffix.eq_ignore_ascii_case(", press Tab then Enter to Remove Suggestion.")+ })+}++fn node_is_exact_setup_suggestion(node: &AXNode, setup_url: &str) -> bool {+ [+ node.title.as_deref(),+ node.value.as_deref(),+ node.description.as_deref(),+ node.help.as_deref(),+ ]+ .into_iter()+ .flatten()+ .any(|value| is_exact_setup_suggestion(value, setup_url))+}++fn exact_omnibox_suggestion(+ nodes: &[AXNode],+ descriptor: &BrowserSetupDescriptor,+) -> Result<Option<usize>, BrowserRefusal> {+ let omniboxes = nodes+ .iter()+ .filter(|node| {+ node.role == "AXTextField"+ && field_equals(node, "Address and search bar")+ && node+ .value+ .as_deref()+ .is_some_and(|value| value.trim().eq_ignore_ascii_case(descriptor.setup_url))+ })+ .count();+ match omniboxes {+ 0 => return Ok(None),+ 1 => {}+ _ => {+ return Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ format!(+ "{} exposed multiple exact address fields containing the setup URL",+ descriptor.product_name+ ),+ ))+ }+ }+ let popups = nodes+ .iter()+ .enumerate()+ .filter(|(_, node)| node.role == "AXWebArea" && field_equals(node, "Omnibox Popup"))+ .collect::<Vec<_>>();+ let (popup_index, popup) = match popups.as_slice() {+ [] => return Ok(None),+ [(index, popup)] => (*index, *popup),+ _ => {+ return Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ format!(+ "{} exposed multiple exact omnibox suggestion popups",+ descriptor.product_name+ ),+ ))+ }+ };+ let end = nodes+ .iter()+ .enumerate()+ .skip(popup_index + 1)+ .find(|(_, node)| node.depth <= popup.depth)+ .map_or(nodes.len(), |(index, _)| index);+ let matches = nodes[popup_index + 1..end]+ .iter()+ .filter(|node| {+ node.role == "AXMenuItem"+ && node_is_exact_setup_suggestion(node, descriptor.setup_url)+ && has_action(node, "AXPress")+ })+ .map(|node| node.element_ptr)+ .collect::<Vec<_>>();+ match matches.as_slice() {+ [] => Ok(None),+ [element] => Ok(Some(*element)),+ _ => Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ format!(+ "{} exposed multiple exact setup URL suggestions",+ descriptor.product_name+ ),+ )),+ }+}+fn new_tab_button(nodes: &[AXNode],descriptor: &BrowserSetupDescriptor,@@ -530,7 +624,7 @@ pub fn enable(}};release_actionable_nodes(&initial.nodes);- let mut handle = SetupUiHandle {+ let handle = SetupUiHandle {descriptor,close_button: Some(close_button),enable_attempted: false,@@ -549,43 +643,10 @@ pub fn enable(return Err(handle.abort(pid, window_id, error));}};- let pressed = unsafe { perform_action(omnibox as AXUIElementRef, "AXPress") };- let focused = unsafe { set_bool_attr_true(omnibox as AXUIElementRef, "AXFocused") };- if pressed != kAXErrorSuccess || focused != kAXErrorSuccess {- release_actionable_nodes(&created.nodes);- return Err(handle.abort(- pid,- window_id,- refusal(- BrowserRefusalCode::BrowserWrongTargetRefused,- format!(- "{}'s exact address field rejected the bounded setup navigation",- descriptor.product_name- ),- ),- ));- }- handle.focused_setup_address_field = true;- let exact_focus = unsafe {- focused_element_of_pid(pid).is_some_and(|focused_element| {- let equal = is_same_element(focused_element as usize, omnibox);- CFRelease(focused_element as CFTypeRef);- equal- })+ let wrote_url = unsafe {+ set_string_attr(omnibox as AXUIElementRef, "AXValue", descriptor.setup_url)};- if !exact_focus {- release_actionable_nodes(&created.nodes);- return Err(handle.abort(pid, window_id, refusal(- BrowserRefusalCode::BrowserWrongTargetRefused,- format!(- "{}'s exact address field lost focus before setup navigation could be confirmed",- descriptor.product_name- ),- )));- }- if let Err(error) = crate::input::keyboard::hotkey(pid, "a", &["cmd"]).and_then(|_| {- crate::input::keyboard::type_text_with_delay(pid, descriptor.setup_url, 30)- }) {+ if wrote_url != kAXErrorSuccess {release_actionable_nodes(&created.nodes);return Err(handle.abort(pid,@@ -593,7 +654,7 @@ pub fn enable(refusal(BrowserRefusalCode::BrowserWrongTargetRefused,format!(- "could not type {}'s fixed setup URL into the exact address field: {error}",+ "{}'s exact address field rejected the fixed setup URL",descriptor.product_name),),@@ -601,8 +662,8 @@ pub fn enable(}let exact_value = unsafe { copy_string_attr(omnibox as AXUIElementRef, "AXValue") }.is_some_and(|value| value.trim().eq_ignore_ascii_case(descriptor.setup_url));+ release_actionable_nodes(&created.nodes);if !exact_value {- release_actionable_nodes(&created.nodes);return Err(handle.abort(pid,window_id,@@ -615,41 +676,45 @@ pub fn enable(),));}- let navigation = if descriptor.product == BrowserProduct::MicrosoftEdge {- handle.foregrounded_window = true;- handle.injected_global_input = true;- crate::input::skylight::with_foreground_assist(pid, window_id, || {- std::thread::sleep(Duration::from_millis(60));- if crate::apps::frontmost_pid() != Some(pid) {- anyhow::bail!(- "the approved browser lost foreground before setup navigation"- );+ let deadline = Instant::now() + Duration::from_secs(2);+ let suggestion = loop {+ let popup = walk_tree(pid, Some(window_id), None);+ match exact_omnibox_suggestion(&popup.nodes, descriptor) {+ Ok(Some(element)) => break Ok((popup, element)),+ Ok(None) if Instant::now() < deadline => {+ release_actionable_nodes(&popup.nodes);+ std::thread::sleep(Duration::from_millis(50));}- crate::input::keyboard::press_key_global("return", &[])?;- std::thread::sleep(Duration::from_millis(60));- Ok(())- })- .and_then(|fronted| {- if fronted {- Ok(())- } else {- Err(anyhow::anyhow!(- "the bounded Microsoft Edge foreground assist was unavailable"- ))+ Ok(None) => {+ release_actionable_nodes(&popup.nodes);+ break Err(refusal(+ BrowserRefusalCode::BrowserWrongTargetRefused,+ format!(+ "{} did not expose one exact fixed-URL omnibox suggestion",+ descriptor.product_name+ ),+ ));}- })- } else {- crate::input::keyboard::press_key(pid, "return", &[])+ Err(error) => {+ release_actionable_nodes(&popup.nodes);+ break Err(error);+ }+ }};- release_actionable_nodes(&created.nodes);- if let Err(error) = navigation {+ let (popup, suggestion) = match suggestion {+ Ok(value) => value,+ Err(error) => return Err(handle.abort(pid, window_id, error)),+ };+ let navigation = unsafe { perform_action(suggestion as AXUIElementRef, "AXPress") };+ release_actionable_nodes(&popup.nodes);+ if navigation != kAXErrorSuccess {return Err(handle.abort(pid,window_id,refusal(BrowserRefusalCode::BrowserWrongTargetRefused,format!(- "could not confirm {}'s bounded setup navigation: {error}",+ "{}'s exact fixed-URL suggestion became stale before AXPress",descriptor.product_name),),@@ -910,6 +975,83 @@ mod tests {);}+ #[test]+ fn setup_navigation_selects_only_the_exact_omnibox_suggestion() {+ let omnibox = node(+ "AXTextField",+ Some("Address and search bar"),+ Some(chrome().setup_url),+ &["AXPress"],+ );+ let mut popup = node("AXWebArea", Some("Omnibox Popup"), None, &[]);+ popup.depth = 1;+ let mut menu = node("AXMenu", None, None, &[]);+ menu.depth = 2;+ let mut exact = node(+ "AXMenuItem",+ Some(&format!(+ "{}, press Tab then Enter to Remove Suggestion.",+ chrome().setup_url+ )),+ None,+ &["AXPress"],+ );+ exact.element_ptr = 42;+ exact.depth = 3;+ let mut search = node(+ "AXMenuItem",+ Some("chrome://inspect/#remote-debugging search, Google Search"),+ None,+ &["AXPress"],+ );+ search.depth = 3;+ let mut outside = node(+ "AXMenuItem",+ Some(chrome().setup_url),+ Some(chrome().setup_url),+ &["AXPress"],+ );+ outside.element_ptr = 99;+ outside.depth = 1;++ assert_eq!(+ exact_omnibox_suggestion(&[omnibox, popup, menu, exact, search, outside], chrome())+ .unwrap(),+ Some(42)+ );+ }++ #[test]+ fn setup_navigation_rejects_search_suggestion() {+ assert!(!is_exact_setup_suggestion(+ "chrome://inspect/#remote-debugging search, Google Search",+ chrome().setup_url+ ));+ }++ #[test]+ fn setup_navigation_refuses_multiple_exact_suggestions() {+ let omnibox = node(+ "AXTextField",+ Some("Address and search bar"),+ Some(chrome().setup_url),+ &["AXPress"],+ );+ let mut popup = node("AXWebArea", Some("Omnibox Popup"), None, &[]);+ popup.depth = 1;+ let mut first = node("AXMenuItem", Some(chrome().setup_url), None, &["AXPress"]);+ first.depth = 2;+ let mut second = first.clone();+ second.element_ptr = 8;++ assert_eq!(+ exact_omnibox_suggestion(&[omnibox, popup, first, second], chrome())+ .unwrap_err()+ .code,+ BrowserRefusalCode::BrowserWrongTargetRefused+ );+ }+#[test]fn unknown_checkbox_values_refuse() {assert_eq!(checkbox_state(Some(0.0)).unwrap(), CheckboxState::Off);