Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Send vision mode's Firefox clicks and hovers as trusted input
Emre Sokullu8bbb9b5d4a98
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex d24763c27..696590c2b 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -120,6 +120,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.+- **Firefox with trusted automation**: when **Firefox trusted automation** is on (Settings β Advanced, the local companion in `firefox-companion/`) and its companion reports point input (`point_input`), a left click and a hover go out as trusted input at the point (`bidi/client.js` `visionInput`). The content script runs every check it runs for the synthetic sequence (the viewport, a frame it refuses, a disabled control), then marks the element the top document shows at the point, the one the page monitor prepared there (`_visionBidiPrepare`); the companion checks that the element is still at the point and moves the real pointer there (`hover_at`), then presses and releases its left button (`click_at`), so `:hover`, focus and pointer capture are the browser's own. A click still runs inside the file-chooser guard, settled as the synthetic click's is. The page monitor attributes it as it does the companion's other clicks (a one-time marker on that element). A double, triple, right or middle click, a drag and a wheel stay synthetic (a BiDi wheel scrolls only the window's selected tab, and a run's tab can be in the background), and so does a point over an embedded frame or inside an open shadow root (the monitor binds the shadow host there, but sees the events come from inside it, where it cannot tell them from a person's; the synthetic sequence reaches into both). A companion without point input leaves all vision input synthetic; one that disconnects mid-task stops its trusted input, as for its other actions.In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.@@ -156,7 +157,8 @@ Every hard guard and quality check runs on vision calls; these are the places wh| `vision-runtime.js` | agent methods: keyframes, observation after batches, native tools, pruning, dispatch preflights || `vision-dom-probe.js` | the in-page probe behind the dispatch checks, and the checks themselves || `agent.js` | platform layer (`_visionPlatform*`, `_visionDispatchClick`) and hooks |-| `content.js` | `visionInput` (Firefox input path; also loaded in Chrome) |+| `content.js` | `visionInput` (Firefox input path, and the marking for trusted input; also loaded in Chrome) |+| `bidi/client.js` (Firefox) | `visionInput`: a left click and a hover as trusted input through the companion |## Tests@@ -184,4 +186,4 @@ The older MV3 quality fixtures script an accessibility-tree model (they answer w- HTML5 drag and drop: in Firefox a drag that starts on a `draggable="true"` item runs the drag-and-drop event sequence itself with a constructed `DataTransfer`; in Chrome the CDP pointer drag is left to the browser, as for `drag_drop`, so a page that relies only on `dataTransfer` may not react there.- A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.- The 45-second window that holds back a second click on the same committing caption covers text clicks (`click({text})`) only. Like a ref click on the tree path, a repeated vision click relies on the submit confirmation and the deletion and publication guards instead.-- Firefox input is synthetic (`isTrusted: false`); pages that require trusted events need Chrome. Its emulated `:hover` covers the style sheets the content script can read: a `:hover` rule in another origin's style sheet does not apply, page scripts that test `:hover` themselves see no hover, and while it lasts a script reading a rule's `selectorText` sees the added `:is()`.+- Firefox input is synthetic (`isTrusted: false`) unless the companion sends it (a left click or a hover, above); a page that requires trusted events for anything else needs Chrome. The emulated `:hover` covers the style sheets the content script can read: a `:hover` rule in another origin's style sheet does not apply, page scripts that test `:hover` themselves see no hover, and while it lasts a script reading a rule's `selectorText` sees the added `:is()`.diff --git a/firefox-companion/README.md b/firefox-companion/README.mdindex a69092546..e69ec7c1e 100644--- a/firefox-companion/README.md+++ b/firefox-companion/README.md@@ -28,6 +28,7 @@ The connection test checks the real BiDi session. It does not submit forms or se- Typing dispatches one character at a time, checking run ownership, deadline and focus before each step. Stop prevents the next step; a single already-dispatched character can still arrive. Newlines use a literal editing command, never Enter; single-line fields reject multiline payloads before clearing.- Exact value readback after typing. `set_field` submits with Enter only after verification. An uncertain dispatch is not safe to repeat automatically.- Active-run dialogs: acknowledge alerts; dismiss confirmations and prompts; stay on beforeunload unless an explicit navigate action is in progress. A Leave allowance is single use. This prevents prompts from hanging input without blanket approval of destructive confirmations. Idle tabs are not auto-handled. Pending dialog events are retained and handled when a task takes ownership; manually closed dialogs are discarded. Browser permission dialogs and arbitrary operating-system dialogs are outside this scope.+- Vision mode's point input (capability `point_input`, which `connect` reports): `click_at` (one left click) and `hover_at` at a whole-pixel CSS point of the run's top document, rechecked against the element the extension marked there. A drag, a wheel, a right, middle or repeated click and a point over an embedded frame or inside an open shadow root stay synthetic; a BiDi wheel scrolls only the window's selected tab. With an older companion, which reports no capability, all vision input stays synthetic.- Uploads keep the existing WebBrain attachment/download/picker authorization. Bytes are written to a private temporary file, passed to `input.setFiles`, and the file input's name and size are checked. The input must be unique, enabled, and in the current document's light DOM. Attachment is not proof the site has uploaded or submitted the file. Arbitrary model-supplied filesystem paths are not accepted.Target markers bind the extension-selected page element to a BiDi shared node. Duplicate or stale bindings fail. Existing permission, recipient, toolbar and dispatch guards run before target preparation. Native input rechecks connection, run ownership, and target visibility before dispatch. Page DOM is not a security boundary against a hostile page rearranging itself between validation and input; existing prompt-injection and action-authorization checks remain necessary.@@ -38,6 +39,7 @@ The host stores temporary upload files only for its connection lifetime and dele- `npm run test:firefox-bidi`: deterministic screenshot transfer, document binding, privacy gating, cancellation, disconnection and dialog regressions (also in `npm test`).- `npm run test:firefox-bidi:e2e`: starts an isolated headless Firefox profile and local fixture, exercises the production BiDi session, verifies full-page pixels, large image transfers, below-fold privacy geometry, PNG saving, `isTrusted`, dialog dismissal, exact typing, and uploaded bytes, then removes the profile. Set `FIREFOX_BINARY` if Firefox is not in the macOS default location. It also temporarily installs the packaged extension and exercises its content-script target preparation. The isolated test enables system access solely to inspect its extension Settings page; normal companion use does not need that flag. It does not install the native host in your normal browser profile.+- `npm run test:firefox-vision:e2e`: vision mode's content-script input in an isolated headless Firefox, then its trusted `click_at` and `hover_at` through this session on the element the content script marked. Set `FIREFOX_BINARY` as for the test above.## Uninstalldiff --git a/firefox-companion/session.mjs b/firefox-companion/session.mjsindex 8db8c8700..cbeb0da31 100644--- a/firefox-companion/session.mjs+++ b/firefox-companion/session.mjs@@ -181,6 +181,13 @@ function unpackRemoteObjectValue(value) {}));}+// What this companion can do beyond the first protocol, reported by+// `connect`. point_input: vision mode's click_at and hover_at, trusted input+// at a viewport point of the run's top document.+export const COMPANION_CAPABILITIES = Object.freeze(['point_input']);+// Vision mode's point actions (point_input).+const POINT_ACTIONS = new Set(['click_at', 'hover_at']);+export class BidiSession {constructor(WebSocketImpl = WebSocket) {this.WebSocketImpl = WebSocketImpl;@@ -194,7 +201,7 @@ export class BidiSession {}async connectSession(port) {if (!Number.isInteger(port) || port < 1024 || port > 65535) throw new Error('Invalid local Firefox port');- if (this.socket?.readyState === 1) return { connected: true };+ if (this.socket?.readyState === 1) return { connected: true, capabilities: [...COMPANION_CAPABILITIES] };const socket = this.socket = new this.WebSocketImpl(`ws://127.0.0.1:${port}/session`);socket.addEventListener('message', event => this.receive(JSON.parse(event.data)));socket.addEventListener('close', () => {@@ -208,7 +215,7 @@ export class BidiSession {});await this.send('session.new', { capabilities: { alwaysMatch: { unhandledPromptBehavior: { default: 'ignore' } } } });await this.send('session.subscribe', { events: ['browsingContext.userPromptOpened', 'browsingContext.userPromptClosed', 'browsingContext.contextDestroyed'] });- return { connected: true };+ return { connected: true, capabilities: [...COMPANION_CAPABILITIES] };}send(method, params) {if (this.socket?.readyState !== 1) return Promise.reject(new Error('Firefox BiDi is disconnected'));@@ -388,7 +395,7 @@ export class BidiSession {}}async performAction(id, action, payload, dispatch, validatePageDispatch = null) {- if (!['navigate', 'click', 'hover', 'type', 'field', 'key', 'upload'].includes(action)) throw new Error('Unsupported BiDi action');+ if (!['navigate', 'click', 'hover', 'type', 'field', 'key', 'upload', ...POINT_ACTIONS].includes(action)) throw new Error('Unsupported BiDi action');if (payload.modifiers) throw new Error('Key modifiers are not supported by this tool');const run = this.runs.get(id); if (!run) throw new Error('Run stopped or disconnected');const assertLive = () => {@@ -410,8 +417,15 @@ export class BidiSession {finally { run.navigation = false; }return { success: true, dispatched: true };}+ const pointAction = POINT_ACTIONS.has(action);+ if (pointAction) {+ // A vision point is in the top document's viewport, so the element the+ // extension marked there is one of that document (content.js).+ if (!Number.isInteger(payload.point?.x) || !Number.isInteger(payload.point?.y)) throw new Error('Invalid point coordinates');+ }const match = await this.locate(payload.token, payload.url, run.context);assertLive();+ if (pointAction && match.context !== run.context) throw new Error('Point input targets the top document; no input sent');let nativeSequence = 0;const markNativeDispatch = async (kind, { rebindFocus = false, domDispatch = false } = {}) => {assertLive();@@ -471,7 +485,7 @@ export class BidiSession {}`, [{ type: 'string', value: marker }, { type: 'string', value: action }]);} catch { /* The document may have navigated or the monitor already consumed the marker. */ }};- const point = action === 'click' && payload.point != null ? payload.point : null;+ const point = (action === 'click' && payload.point != null) || pointAction ? payload.point : null;if (point && (!Number.isInteger(point.x) || !Number.isInteger(point.y))) throw new Error('Invalid click coordinates');const focusGuard = payload.pageFeedbackGuard;if (['type', 'field', 'key'].includes(action) && focusGuard@@ -482,10 +496,12 @@ export class BidiSession {}assertLive();const check = await this.call(match, `(el, token, action, x, y) => {- if (!el.isConnected || el.getAttribute('data-webbrain-bidi') !== token || el.disabled) return false;+ // A pointer may rest on a disabled control.+ if (!el.isConnected || el.getAttribute('data-webbrain-bidi') !== token+ || (el.disabled && action !== 'hover_at')) return false;el.removeAttribute('data-webbrain-bidi');if (action === 'upload') return el.tagName === 'INPUT' && el.type === 'file';- if (action === 'click' || action === 'hover') {+ if (action === 'click' || action === 'hover' || action === 'click_at' || action === 'hover_at') {const r = el.getBoundingClientRect();const px = x === null ? r.x+r.width/2 : x;const py = y === null ? r.y+r.height/2 : y;@@ -588,11 +604,12 @@ export class BidiSession {finally { await this.send('input.releaseActions', { context: match.context }).catch(() => {}); }assertLive();};- if (action === 'click' || action === 'hover') {+ if (action === 'click' || action === 'hover' || action === 'click_at' || action === 'hover_at') {+ const pressing = action === 'click' || action === 'click_at';const source = { type: 'pointer', id: 'webbrain-pointer', parameters: { pointerType: 'mouse' }, actions: [point ? { type: 'pointerMove', x: point.x, y: point.y, origin: 'viewport' }: { type: 'pointerMove', x: 0, y: 0, origin: { type: 'element', element: { sharedId: match.node.sharedId } } },- ...(action === 'click' ? [{ type: 'pointerDown', button: 0 }, { type: 'pointerUp', button: 0 }] : []),+ ...(pressing ? [{ type: 'pointerDown', button: 0 }, { type: 'pointerUp', button: 0 }] : []),] };assertLive();const marker = await markNativeDispatch('click');diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex f0eb39ba0..9d836237a 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -2260,6 +2260,31 @@};}+ // Trusted input (the Firefox companion, bidi/client.js visionInput) takes+ // the element the top document shows at a viewport point, the one the page+ // monitor prepared there, and marks it for the companion, which moves (and+ // for a click presses) the real pointer there itself; the monitor knows+ // that input by the mark. A point over an embedded frame, or inside an open+ // shadow root (the monitor binds its host, but sees the events come from+ // inside, where it cannot tell them from a person's), stays with the+ // synthetic sequence, which reaches into both.+ function _visionBidiPrepare(x, y, token) {+ if (!/^[a-f0-9-]{36}$/.test(String(token || ''))) {+ return { success: false, dispatched: false, noDispatch: true, error: 'Invalid trusted-input target.' };+ }+ const el = document.elementFromPoint(x, y);+ if (!el) return { success: false, dispatched: false, noDispatch: true, error: 'Nothing is rendered at that point.' };+ const inner = el.shadowRoot?.elementFromPoint?.(x, y);+ if (/^(IFRAME|FRAME|OBJECT|EMBED)$/i.test(String(el.tagName || '')) || (inner && inner !== el)) {+ return { bidiUnavailable: true, success: false, dispatched: false, noDispatch: true };+ }+ el.setAttribute('data-webbrain-bidi', token);+ setTimeout(() => { if (el.getAttribute('data-webbrain-bidi') === token) el.removeAttribute('data-webbrain-bidi'); }, 10000);+ // The companion takes whole CSS pixels.+ return { bidiPrepared: true, success: false, dispatched: false, noDispatch: true, url: location.href,+ point: { x: Math.min(Math.round(x), Math.max(0, innerWidth - 1)), y: Math.min(Math.round(y), Math.max(0, innerHeight - 1)) } };+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -2347,6 +2372,20 @@error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,};}+ if (params._bidiPrepare) {+ if (type !== 'click' && type !== 'hover') {+ return { success: false, dispatched: false, noDispatch: true, error: `Trusted vision input does not send a ${type}.` };+ }+ const prepared = _visionBidiPrepare(x, y, params._bidiPrepare);+ if (!prepared.bidiPrepared) return prepared;+ // The real pointer is there after this; a later synthetic sequence+ // starts as one arriving from elsewhere.+ _visionPointerOver = null;+ // A click runs inside the native file-chooser guard, as the synthetic+ // click does (settled by the agent like that one).+ return { ...prepared, target: describe,+ ...(type === 'click' ? { _filePickerGuardId: clickWithoutNativeFilePicker(() => {}).guardId } : {}) };+ }// A click or drag presses the pointer, which the page may capture// (_visionPointerCapture); its capture methods are the page's own again// once the sequence ends.diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 3f8c0b7e3..613f192dc 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -17153,7 +17153,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// vision-runtime.js owns the logic. Firefox has no CDP: captures use// tabs.captureTab with a render scale, and pointer input is dispatched by// the content script at the exact point (content.js visionInput), behind- // the same page-agent fence as every other content action.+ // the same page-agent fence as every other content action. With the local+ // companion (Settings β Advanced), a left click and a hover go out as+ // trusted input at the point instead (bidi/client.js visionInput).async _visionPlatformProbe(tabId) {const results = await browser.tabs.executeScript(tabId, { code: `(() => ({@@ -17213,6 +17215,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}async _visionContentInput(tabId, params, kind = 'click') {+ if (firefoxBidi.runs.has(tabId)) {+ let trusted;+ try {+ trusted = await firefoxBidi.visionInput(tabId, params);+ } catch (error) {+ if (error?.name === 'AbortError') throw error;+ // Binding the run's document and preparing the point send nothing.+ const message = String(error?.message || error).slice(0, 160);+ return { success: false, dispatched: false, noDispatch: true, error: `The page could not take trusted input yet (${message}). Take a screenshot and try again.` };+ }+ if (trusted) return trusted;+ }await beforePageAgentDispatch(browser, tabId, {kind, frameId: 0, navigationCandidate: kind === 'click' && params.type === 'click', fenceOnly: true,});diff --git a/src/firefox/src/bidi/client.js b/src/firefox/src/bidi/client.jsindex 12cab921d..a8ffa3639 100644--- a/src/firefox/src/bidi/client.js+++ b/src/firefox/src/bidi/client.js@@ -30,6 +30,17 @@ function trustedTypingReadback(readback, fieldMeta) {return { unverified: { reason: readback.reason === 'element_replaced' ? 'element_replaced' : 'readback_unavailable', reader: 'none' },readbackExact: false };}+// Vision mode's input that goes out as trusted input at a point+// (visionInput): a left click and a hover. A double, triple, right or middle+// click, a drag and a wheel stay synthetic (content.js visionInput). A BiDi+// wheel scrolls only the window's selected tab, and a run's tab may be in+// the background.+function trustedVisionAction(params = {}) {+ if (params.type === 'hover') return 'hover_at';+ if (params.type === 'click' && (params.button == null || params.button === 'left')+ && (params.clickCount == null || Number(params.clickCount) === 1)) return 'click_at';+ return null;+}export class FirefoxBidiClient {constructor(api) { this.apiOverride = api; this.pending = new Map(); this.runs = new Map(); this.captures = new Map(); this.sequence = 0; this.connectionEpoch = 0; }get api() { return this.apiOverride || globalThis.browser; }@@ -189,16 +200,20 @@ export class FirefoxBidiClient {if (!owner || owner.signal?.aborted || owner.disconnected) throw new Error('No connected Firefox BiDi run; restart the task after reconnecting');await this.bindRun(tabId, owner);if (this.runs.get(tabId) !== owner || owner.disconnected || owner.signal?.aborted) throw new Error('Run stopped');- pageFeedbackGuard = await beforePageAgentDispatch(this.api, tabId, {- kind: action === 'navigate' ? 'navigate' : action === 'scroll' ? 'scroll'- : ['type', 'field', 'key', 'upload'].includes(action) ? 'input' : 'click',- ...(payload.point ? { x: payload.point.x, y: payload.point.y } : {}),- selector: payload.selector, ref_id: payload.ref_id, frameId: payload.frameId || 0,- tool: action, prepareMonitor: true,- navigationCandidate: ['click', 'checked'].includes(action),- fenceOnly: true,- url: action === 'navigate' ? payload.url : undefined,- });+ pageFeedbackGuard = await beforePageAgentDispatch(this.api, tabId, action === 'click_at' || action === 'hover_at'+ // The run already prepared the monitor at this vision point+ // (page-feedback.js); like the synthetic sequence, only fence it.+ ? { kind: 'click', frameId: 0, navigationCandidate: action === 'click_at', fenceOnly: true }+ : {+ kind: action === 'navigate' ? 'navigate' : action === 'scroll' ? 'scroll'+ : ['type', 'field', 'key', 'upload'].includes(action) ? 'input' : 'click',+ ...(payload.point ? { x: payload.point.x, y: payload.point.y } : {}),+ selector: payload.selector, ref_id: payload.ref_id, frameId: payload.frameId || 0,+ tool: action, prepareMonitor: true,+ navigationCandidate: ['click', 'checked'].includes(action),+ fenceOnly: true,+ url: action === 'navigate' ? payload.url : undefined,+ });if (!owner.bound && (action !== 'navigate' || !/^https?:\/\//.test(payload.url || ''))) throw new Error('Navigate to a web page before trusted input');onDispatch?.();} catch (error) { throw notSent(error); }@@ -208,6 +223,42 @@ export class FirefoxBidiClient {}return this.request('perform', { runId: owner.runId, action, payload: { ...payload, pageFeedbackGuard } });}+ // Vision mode's point input as trusted input. The content script runs every+ // check of its synthetic sequence (content.js visionInput) and marks the+ // element the top document shows at the point; the companion then moves+ // the real pointer there and, for a click, presses it. null: this run has+ // no trusted input, its companion has no point input, the input is one+ // that stays synthetic (trustedVisionAction) or the point is over an+ // embedded frame. Nothing was sent, and the caller sends it synthetically.+ async visionInput(tabId, params = {}) {+ const action = trustedVisionAction(params);+ const owner = this.runs.get(tabId);+ if (!owner || !action) return null;+ if (owner.disconnected || owner.signal?.aborted) return { success: false, dispatched: false, noDispatch: true, retryable: false, error: 'Firefox connection stopped; restart the task.' };+ const connection = await this.connect();+ if (!Array.isArray(connection?.capabilities) || !connection.capabilities.includes('point_input')) return null;+ await this.bindRun(tabId, owner);+ if (!owner.bound) return { success: false, dispatched: false, noDispatch: true, error: 'Navigate to a web page before trusted input.' };+ const token = crypto.randomUUID();+ const prepared = await this.api.tabs.sendMessage(tabId, { target: 'content', action: 'vision_input',+ params: { ...params, _bidiPrepare: token } }, { frameId: 0 });+ if (prepared?.bidiUnavailable === true) return null;+ if (!prepared?.bidiPrepared) return prepared || { success: false, dispatched: false, noDispatch: true, error: 'The page did not answer the vision input.' };+ if (this.runs.get(tabId) !== owner || owner.disconnected || owner.signal?.aborted) return { success: false, dispatched: false, noDispatch: true, retryable: false, error: 'Run stopped before trusted input.' };+ const evidence = { ...(prepared.target ? { target: prepared.target } : {}),+ ...(prepared._filePickerGuardId ? { _filePickerGuardId: prepared._filePickerGuardId } : {}) };+ try {+ const performed = await this.perform(tabId, action, { token, url: prepared.url, point: prepared.point });+ return { ...performed, ...evidence, method: 'firefox-bidi' };+ } catch (error) {+ if (error?.name === 'AbortError' && isBidiRequestNotSentError(error)) throw error;+ if (error?.code === 'page_feedback_pending') return { ...evidence, success: false, dispatched: false,+ noDispatch: true, pageFeedbackPending: true, error: error.message };+ if (isBidiRequestNotSentError(error)) return { ...evidence, success: false, dispatched: false, noDispatch: true,+ outcomeUnknown: false, retryable: false, error: error.message };+ return { ...evidence, success: false, dispatched: true, outcomeUnknown: true, retryable: false, ...(error.dispatchState || {}), error: error.message };+ }+ }// `onDispatch` marks the transport boundary of whichever route is taken: the// content message, or the native request after its preparation and fence.async sendContent(tabId, message, options, { onDispatch = null } = {}) {diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 641ce6286..9ea3186d9 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2554,6 +2554,31 @@};}+ // Trusted input (the Firefox companion, bidi/client.js visionInput) takes+ // the element the top document shows at a viewport point, the one the page+ // monitor prepared there, and marks it for the companion, which moves (and+ // for a click presses) the real pointer there itself; the monitor knows+ // that input by the mark. A point over an embedded frame, or inside an open+ // shadow root (the monitor binds its host, but sees the events come from+ // inside, where it cannot tell them from a person's), stays with the+ // synthetic sequence, which reaches into both.+ function _visionBidiPrepare(x, y, token) {+ if (!/^[a-f0-9-]{36}$/.test(String(token || ''))) {+ return { success: false, dispatched: false, noDispatch: true, error: 'Invalid trusted-input target.' };+ }+ const el = document.elementFromPoint(x, y);+ if (!el) return { success: false, dispatched: false, noDispatch: true, error: 'Nothing is rendered at that point.' };+ const inner = el.shadowRoot?.elementFromPoint?.(x, y);+ if (/^(IFRAME|FRAME|OBJECT|EMBED)$/i.test(String(el.tagName || '')) || (inner && inner !== el)) {+ return { bidiUnavailable: true, success: false, dispatched: false, noDispatch: true };+ }+ el.setAttribute('data-webbrain-bidi', token);+ setTimeout(() => { if (el.getAttribute('data-webbrain-bidi') === token) el.removeAttribute('data-webbrain-bidi'); }, 10000);+ // The companion takes whole CSS pixels.+ return { bidiPrepared: true, success: false, dispatched: false, noDispatch: true, url: location.href,+ point: { x: Math.min(Math.round(x), Math.max(0, innerWidth - 1)), y: Math.min(Math.round(y), Math.max(0, innerHeight - 1)) } };+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -2641,6 +2666,20 @@error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,};}+ if (params._bidiPrepare) {+ if (type !== 'click' && type !== 'hover') {+ return { success: false, dispatched: false, noDispatch: true, error: `Trusted vision input does not send a ${type}.` };+ }+ const prepared = _visionBidiPrepare(x, y, params._bidiPrepare);+ if (!prepared.bidiPrepared) return prepared;+ // The real pointer is there after this; a later synthetic sequence+ // starts as one arriving from elsewhere.+ _visionPointerOver = null;+ // A click runs inside the native file-chooser guard, as the synthetic+ // click does (settled by the agent like that one).+ return { ...prepared, target: describe,+ ...(type === 'click' ? { _filePickerGuardId: clickWithoutNativeFilePicker(() => {}).guardId } : {}) };+ }// A click or drag presses the pointer, which the page may capture// (_visionPointerCapture); its capture methods are the page's own again// once the sequence ends.diff --git a/test/firefox-bidi.mjs b/test/firefox-bidi.mjsindex 85095724b..76594777e 100644--- a/test/firefox-bidi.mjs+++ b/test/firefox-bidi.mjs@@ -708,3 +708,207 @@ test('release workflows run the full suite on the companion runtime', async () =assert.ok(main.includes(` - ${shard.replace('test:shard:','')}`),`main.yml matrix should run ${shard}`);}});++// ββ Vision mode's point input (point_input) ββββββββββββββββββββββββββββββββ++test('the companion reports point input when it connects', async () => {+ const { COMPANION_CAPABILITIES } = await import('../firefox-companion/session.mjs');+ assert.deepEqual([...COMPANION_CAPABILITIES], ['point_input']);+ const session = new BidiSession();+ session.socket = { readyState: 1 };+ assert.deepEqual(await session.connectSession(9222), { connected: true, capabilities: ['point_input'] });+});++function pointSession({ context = 'tab', disabled = false, validate = async () => true } = {}) {+ const session = new BidiSession(), runId = id(), token = id(), sent = [], checks = [], markers = [];+ const el = {+ isConnected: true, disabled, tagName: 'DIV', attributes: new Map([['data-webbrain-bidi', token]]),+ getAttribute(name) { return this.attributes.get(name) ?? null; },+ setAttribute(name, value) { this.attributes.set(name, value); if (name === 'data-webbrain-native-action') markers.push(JSON.parse(value).kind); },+ removeAttribute(name) { this.attributes.delete(name); },+ getBoundingClientRect: () => ({ x: 0, y: 0, width: 200, height: 100 }),+ getRootNode: () => ({ activeElement: null }),+ };+ session.runs.set(runId, { context: 'tab' });+ session.locate = async () => ({ context, node: { sharedId: 'marked' } });+ session.call = async (_match, declaration, args = []) => {+ checks.push(declaration);+ const document = { elementFromPoint: () => el, documentElement: { getAttribute: () => 'point-document:0' } };+ return { result: { value: vm.runInNewContext(`(${declaration})`, { document, innerWidth: 1000, innerHeight: 700 })(el, ...args.map(arg => arg.value)) } };+ };+ session.send = async (method, params) => { if (method === 'input.performActions') sent.push(params.actions[0]); return {}; };+ return { session, runId, token, sent, checks, markers, el, url: 'https://example.com/', guard: { documentToken: 'point-document', revision: 0, operationId: 'vision', nativeSecret: 'secret' }, validate };+}++test('click_at presses the left button once at the point, under a click marker', async () => {+ const run = pointSession();+ const result = await run.session.perform(run.runId, 'click_at', { token: run.token, url: run.url, point: { x: 120, y: 40 }, pageFeedbackGuard: run.guard }, async () => true);+ assert.equal(result.success, true);+ assert.equal(result.method, 'firefox-bidi');+ assert.deepEqual(run.sent, [{ type: 'pointer', id: 'webbrain-pointer', parameters: { pointerType: 'mouse' }, actions: [+ { type: 'pointerMove', x: 120, y: 40, origin: 'viewport' }, { type: 'pointerDown', button: 0 }, { type: 'pointerUp', button: 0 }] }]);+ assert.deepEqual(run.markers, ['click']);+ assert.equal(run.el.getAttribute('data-webbrain-bidi'), null, 'the binding is used once');+});++test('hover_at moves the pointer to the point and may rest on a disabled control', async () => {+ const run = pointSession({ disabled: true });+ assert.equal((await run.session.perform(run.runId, 'hover_at', { token: run.token, url: run.url, point: { x: 30, y: 60 } })).success, true);+ assert.deepEqual(run.sent[0].actions, [{ type: 'pointerMove', x: 30, y: 60, origin: 'viewport' }]);+ const pressed = pointSession({ disabled: true });+ await assert.rejects(pressed.session.perform(pressed.runId, 'click_at', { token: pressed.token, url: pressed.url, point: { x: 30, y: 60 } }), /covered/);+ assert.deepEqual(pressed.sent, []);+});++test('point input refuses a fractional or missing point, a wheel and a frame target before any input', async () => {+ for (const [action, payload, pattern] of [+ ['click_at', { point: { x: 10.5, y: 4 } }, /Invalid point/],+ ['hover_at', {}, /Invalid point/],+ ['scroll_at', { point: { x: 1, y: 1 }, deltaY: 100 }, /Unsupported BiDi action/],+ ]) {+ const run = pointSession();+ await assert.rejects(run.session.perform(run.runId, action, { token: run.token, url: run.url, ...payload }),+ error => pattern.test(error.message) && error.dispatchState.noDispatch === true, `${action} ${JSON.stringify(payload)}`);+ assert.deepEqual(run.sent, []);+ }+ const framed = pointSession({ context: 'child-frame' });+ await assert.rejects(framed.session.perform(framed.runId, 'click_at', { token: framed.token, url: framed.url, point: { x: 5, y: 5 } }), /top document/);+ assert.deepEqual(framed.sent, []);+});++function visionClient({ prepared = null, capabilities = ['point_input'], perform = null, sendMessage = null } = {}) {+ const messages = [];+ const client = new FirefoxBidiClient({ tabs: { sendMessage: sendMessage || (async (_tab, message, options) => {+ messages.push({ message, options });+ return prepared ? prepared(message) : { bidiPrepared: true, success: false, dispatched: false, noDispatch: true,+ url: 'https://example.com/', point: { x: 120, y: 40 }, target: { tag: 'button' }, _filePickerGuardId: 'fpg_1' };+ }) } });+ client.connect = async () => (capabilities ? { connected: true, capabilities } : { connected: true });+ client.runs.set(1, { runId: id(), bound: true });+ const performed = [];+ client.perform = perform || (async (_tab, action, payload) => { performed.push({ action, payload }); return { success: true, dispatched: true, method: 'firefox-bidi' }; });+ return { client, messages, performed };+}++test('trusted vision input: a left click is prepared by the content script and pressed by the companion', async () => {+ const { client, messages, performed } = visionClient();+ const result = await client.visionInput(1, { type: 'click', x: 120.4, y: 39.6, button: 'left', clickCount: 1 });+ assert.equal(messages.length, 1);+ assert.equal(messages[0].message.action, 'vision_input');+ assert.equal(messages[0].options.frameId, 0);+ assert.match(messages[0].message.params._bidiPrepare, /^[a-f0-9-]{36}$/);+ assert.equal(performed.length, 1);+ assert.equal(performed[0].action, 'click_at');+ assert.deepEqual(performed[0].payload, { token: messages[0].message.params._bidiPrepare, url: 'https://example.com/', point: { x: 120, y: 40 } });+ assert.deepEqual(result, { success: true, dispatched: true, method: 'firefox-bidi', target: { tag: 'button' }, _filePickerGuardId: 'fpg_1' });+});++test('trusted vision input: a hover goes to the companion at its prepared point', async () => {+ const { client, messages, performed } = visionClient();+ await client.visionInput(1, { type: 'hover', x: 10, y: 10 });+ assert.deepEqual(performed.map(entry => entry.action), ['hover_at']);+ assert.deepEqual(performed[0].payload, { token: messages[0].message.params._bidiPrepare, url: 'https://example.com/', point: { x: 120, y: 40 } });+});++test('trusted vision input leaves synthetic what it does not send, and sends nothing for it', async () => {+ // A BiDi wheel scrolls only the window's selected tab; a run's tab can be in the background.+ for (const params of [{ type: 'drag', x: 1, y: 1, toX: 50, toY: 1 }, { type: 'click', x: 1, y: 1, clickCount: 2 },+ { type: 'click', x: 1, y: 1, button: 'right' }, { type: 'click', x: 1, y: 1, button: 'middle' },+ { type: 'wheel', x: 1, y: 1, deltaY: 300 }]) {+ const { client, messages, performed } = visionClient();+ assert.equal(await client.visionInput(1, params), null, JSON.stringify(params));+ assert.equal(messages.length + performed.length, 0);+ }+ const older = visionClient({ capabilities: null });+ assert.equal(await older.client.visionInput(1, { type: 'click', x: 1, y: 1 }), null, 'a companion without point input');+ assert.equal(older.messages.length, 0);+ const noRun = visionClient();+ noRun.client.runs.clear();+ assert.equal(await noRun.client.visionInput(1, { type: 'click', x: 1, y: 1 }), null, 'a run without trusted input');+ const framed = visionClient({ prepared: () => ({ bidiUnavailable: true, success: false, dispatched: false, noDispatch: true }) });+ assert.equal(await framed.client.visionInput(1, { type: 'click', x: 1, y: 1 }), null, 'a point over an embedded frame');+ assert.equal(framed.performed.length, 0);+});++test('trusted vision input returns the content refusal and never falls back after a disconnect', async () => {+ const refusal = { success: false, dispatched: false, noDispatch: true, visionInputRefused: 'disabled_control', error: 'disabled' };+ const refused = visionClient({ prepared: () => refusal });+ assert.deepEqual(await refused.client.visionInput(1, { type: 'click', x: 1, y: 1 }), refusal);+ assert.equal(refused.performed.length, 0);+ const lost = visionClient();+ lost.client.runs.get(1).disconnected = true;+ const result = await lost.client.visionInput(1, { type: 'click', x: 1, y: 1 });+ assert.equal(result.noDispatch, true);+ assert.equal(result.retryable, false);+ assert.equal(lost.messages.length, 0);+});++test('trusted vision input keeps no-dispatch and uncertain outcomes apart', async () => {+ const vetoed = visionClient({ perform: async () => { const error = new Error('Page changed'); error.code = 'page_feedback_pending'; throw error; } });+ const pending = await vetoed.client.visionInput(1, { type: 'click', x: 1, y: 1 });+ assert.equal(pending.noDispatch, true);+ assert.equal(pending.pageFeedbackPending, true);+ assert.equal(pending._filePickerGuardId, 'fpg_1', 'the armed file-chooser guard is still settled');+ const uncertain = visionClient({ perform: async () => { throw new Error('Firefox companion timed out'); } });+ const unknown = await uncertain.client.visionInput(1, { type: 'click', x: 1, y: 1 });+ assert.equal(unknown.dispatched, true);+ assert.equal(unknown.outcomeUnknown, true);+ assert.equal(unknown.retryable, false);+});++test('trusted vision input fences its click or hover like the synthetic sequence', async () => {+ for (const [action, expected] of [+ ['click_at', { kind: 'click', fenceOnly: true, navigationCandidate: true }],+ ['hover_at', { kind: 'click', fenceOnly: true, navigationCandidate: false }],+ ]) {+ const tab = 957, owner = { ...pageFeedbackMethods, isRunning: () => true, _checkAbort: () => false };+ const registrations = [];+ const client = new FirefoxBidiClient({ tabs: { sendMessage: async (_tab, message) => {+ registrations.push(message);+ return { ready: true, guard: { documentToken: 'd', revision: 0, operationId: 'o' } };+ } } });+ client.runs.set(tab, { runId: id(), bound: true });+ client.request = async (_command, args) => { assert.equal(args.action, action); return { success: true }; };+ await owner._beginPageFeedbackRun(tab, 'interactive');+ try {+ await client.perform(tab, action, { token: id(), url: 'https://example.com/', point: { x: 5, y: 6 } });+ assert.deepEqual(registrations.map(message => message.action), ['page_monitor_dispatch'], `${action}: no second preparation of the vision point`);+ const params = registrations[0].params;+ for (const [key, value] of Object.entries(expected)) assert.deepEqual(params[key], value, `${action}: ${key}`);+ assert.equal(params.x, undefined, `${action}: the prepared point stands`);+ assert.equal(params.prepareMonitor, undefined);+ } finally { owner._finishPageFeedbackRun(tab); }+ }+});++test('Firefox vision input goes through the companion when the run has one, and stays synthetic otherwise', async () => {+ const { Agent } = await import('../src/firefox/src/agent/agent.js');+ const { firefoxBidi } = await import('../src/firefox/src/bidi/client.js');+ const originalBrowser = globalThis.browser, originalVisionInput = firefoxBidi.visionInput;+ const contentMessages = [];+ globalThis.browser = { tabs: { sendMessage: async (_tab, message) => { contentMessages.push(message); return { success: true, dispatched: true, method: 'synthetic-vision-pointer' }; } } };+ const agent = Object.create(Agent.prototype);+ try {+ firefoxBidi.runs.set(31, { runId: id(), bound: true });+ firefoxBidi.visionInput = async () => ({ success: true, dispatched: true, method: 'firefox-bidi' });+ assert.equal((await agent._visionContentInput(31, { type: 'click', x: 5, y: 5 })).method, 'firefox-bidi');+ assert.equal(contentMessages.length, 0);+ firefoxBidi.visionInput = async () => null;+ assert.equal((await agent._visionContentInput(31, { type: 'drag', x: 5, y: 5, toX: 50, toY: 5 }, 'click')).method, 'synthetic-vision-pointer');+ assert.equal(contentMessages.at(-1).action, 'vision_input');+ firefoxBidi.visionInput = async () => { throw new Error('Tab or element binding is stale'); };+ const failed = await agent._visionContentInput(31, { type: 'hover', x: 5, y: 5 });+ assert.equal(failed.noDispatch, true);+ assert.equal(contentMessages.length, 1, 'a failed preparation is not retried synthetically');+ firefoxBidi.visionInput = async () => { const error = new Error('stopped'); error.name = 'AbortError'; throw error; };+ await assert.rejects(agent._visionContentInput(31, { type: 'hover', x: 5, y: 5 }), /stopped/);+ firefoxBidi.runs.delete(31);+ firefoxBidi.visionInput = async () => { throw new Error('a run without trusted input never asks'); };+ assert.equal((await agent._visionContentInput(31, { type: 'click', x: 5, y: 5 })).method, 'synthetic-vision-pointer');+ } finally {+ firefoxBidi.runs.delete(31);+ firefoxBidi.visionInput = originalVisionInput;+ if (originalBrowser === undefined) delete globalThis.browser;+ else globalThis.browser = originalBrowser;+ }+});diff --git a/test/firefox-vision-e2e.mjs b/test/firefox-vision-e2e.mjsindex 7078da666..ae8efd177 100644--- a/test/firefox-vision-e2e.mjs+++ b/test/firefox-vision-e2e.mjs@@ -2,6 +2,8 @@// isolated headless Firefox over WebDriver BiDi, opens a local fixture and// sends `vision_input` to the real content script (its Xray world,// wrappedJSObject and exportFunction included), then reads what the page saw.+// Trusted input (point_input: a click and a hover) runs through the+// production companion session on the element the real content script marked.// Set FIREFOX_BINARY if Firefox is not on PATH or in the macOS default place.//// npm run test:firefox-vision:e2e@@ -30,6 +32,7 @@ await writeFile(join(profile, 'user.js'), 'user_pref("ui.prefersReducedMotion",const delay = ms => new Promise(resolve => setTimeout(resolve, ms));const results = [];+let locatedByScript = false;const check = (name, ok, detail = '') => {results.push({ name, ok: !!ok, ...(ok ? {} : { detail: String(detail).slice(0, 800) }) });};@@ -48,13 +51,14 @@ const page = port => `<!doctype html><meta charset="utf-8"><title>vision in Fire#volume{position:absolute;left:20px;top:280px;width:300px;height:20px;margin:0}#words{position:absolute;left:20px;top:330px;margin:0;font:20px monospace;white-space:nowrap}#scroller{position:absolute;left:20px;top:380px;width:200px;height:100px;overflow:auto;background:#f4f4f4}+#trust{position:absolute;left:700px;top:250px;width:120px;height:40px}#scaled{position:absolute;left:400px;top:300px;width:400px;height:300px;border:3px solid #000;transform:scale(0.5);transform-origin:0 0}#cross{position:absolute;left:700px;top:100px;width:200px;height:100px;border:0}</style><div id="menu">Menu<div class="submenu">Item</div></div><div class="save" id="save">Save</div><div id="track"></div><div id="handle"></div><button id="press">Press</button><input id="volume" type="range" min="0" max="100" step="5" value="10"><p id="words">alpha bravo charlie</p>-<div id="scroller"><div style="height:1000px"></div></div>+<div id="scroller"><div style="height:1000px"></div></div><button id="trust">Trust</button><iframe id="scaled" srcdoc="${scaledFrame.replace(/"/g, '"')}"></iframe><iframe id="cross" src="http://localhost:${port}/cross"></iframe><script>@@ -68,6 +72,7 @@ const press=document.getElementById('press');press.addEventListener('pointerdown',e=>{e.target.releasePointerCapture(e.pointerId);document.addEventListener('pointerup',()=>log.push(['pressed']),{once:true});});for(const t of ['input','change'])document.getElementById('volume').addEventListener(t,e=>log.push([t,e.target.value]));window.nativeCapture=Element.prototype.setPointerCapture;+window.trusted=[];for(const t of ['pointerdown','click'])document.addEventListener(t,e=>trusted.push([t,e.isTrusted,Math.round(e.clientX),Math.round(e.clientY),e.target.id||e.target.tagName]),true);// The centre of a word of #words, from the layout this Firefox drew.window.wordPoint=(word,edge)=>{const text=document.getElementById('words').firstChild;const start=text.nodeValue.indexOf(word);const range=document.createRange();range.setStart(text,start);range.setEnd(text,start+word.length);const r=range.getBoundingClientRect();@@ -216,6 +221,67 @@ try {result = await vision({ type: 'click', x: 800, y: 150 });check('a click on a cross-origin frame is refused',result?.visionInputRefused === 'opaque_frame' && result?.noDispatch === true, JSON.stringify(result));++ // 9. Trusted input: the content script marks the element at the point and+ // the companion session moves (and for a click presses) the real pointer there.+ // Playwright's Firefox build answers browsingContext.locateNodes with no+ // node at all. There the marked node is found by script instead, exactly+ // one in the frames at the binding's URL as BidiSession.locate requires;+ // every other step is the production session's.+ const locator = await session.send('browsingContext.locateNodes', { context, locator: { type: 'css', value: 'html' } });+ if (!locator.nodes?.length) {+ locatedByScript = true;+ session.locate = async (token, targetUrl, root = null) => {+ if (!/^[a-f0-9-]{36}$/.test(token) || !/^https?:\/\//.test(targetUrl)) throw new Error('Invalid tab binding');+ const tree = await session.send('browsingContext.getTree', root ? { root } : {});+ const contexts = [];+ const visit = nodes => { for (const node of nodes || []) { contexts.push(node); visit(node.children); } };+ visit(tree.contexts);+ const matches = [];+ for (const frame of contexts) {+ if (frame.url !== targetUrl) continue;+ const found = await session.send('script.evaluate', { target: { context: frame.context }, awaitPromise: false,+ expression: `(() => { const nodes = document.querySelectorAll('[data-webbrain-bidi="${token}"]'); return nodes.length === 1 ? nodes[0] : nodes.length; })()` });+ if (found.result?.sharedId) matches.push({ context: frame.context, node: { sharedId: found.result.sharedId } });+ else if (found.result?.value > 1) matches.push(null, null);+ }+ if (matches.length !== 1) throw new Error('Tab or element binding is stale or ambiguous; re-read the page');+ return matches[0];+ };+ }+ const bindToken = crypto.randomUUID();+ await evaluate(`document.documentElement.setAttribute('data-webbrain-bidi', ${JSON.stringify(bindToken)})`);+ const runId = crypto.randomUUID();+ await session.openRun(runId, bindToken, url);+ const trustedLog = async () => evaluateJson('window.trusted.splice(0)');+ const perform = (action, payload) => session.perform(runId, action, payload)+ .catch(error => ({ success: false, error: error.message, ...error.dispatchState }));+ await trustedLog();+ const markedToken = () => evaluate(`document.querySelector('[data-webbrain-bidi]')?.getAttribute('data-webbrain-bidi') || null`);+ let prepared = await vision({ type: 'click', x: 760.4, y: 269.6, _bidiPrepare: crypto.randomUUID() });+ const clickToken = await evaluate(`document.getElementById('trust').getAttribute('data-webbrain-bidi')`);+ check('the content script marks the element at a trusted click point', prepared?.bidiPrepared === true && prepared?.noDispatch === true+ && JSON.stringify(prepared.point) === JSON.stringify({ x: 760, y: 270 }) && typeof clickToken === 'string' && /^fpg_/.test(prepared._filePickerGuardId || ''), JSON.stringify({ prepared, clickToken }));+ check('preparing sends nothing to the page', (await trustedLog()).length === 0);+ result = await perform('click_at', { token: clickToken, url: prepared.url, point: prepared.point });+ const clicked = await trustedLog();+ check('click_at is one trusted press and click at the point', result?.success === true+ && JSON.stringify(clicked) === JSON.stringify([['pointerdown', true, 760, 270, 'trust'], ['click', true, 760, 270, 'trust']]), JSON.stringify({ result, clicked }));+ // The companion checks the marked element is still the one at the point.+ prepared = await vision({ type: 'click', x: 760, y: 270, _bidiPrepare: crypto.randomUUID() });+ const staleToken = await markedToken();+ const refusal = await perform('click_at', { token: staleToken, url: prepared.url, point: { x: 120, y: 40 } });+ check('a point that no longer hits the marked element sends nothing', refusal?.success === false && refusal.noDispatch === true+ && (await trustedLog()).length === 0, JSON.stringify(refusal));+ // A trusted hover is the browser's own :hover, with no emulation.+ prepared = await vision({ type: 'hover', x: 120, y: 40, _bidiPrepare: crypto.randomUUID() });+ const hoverToken = await markedToken();+ result = await perform('hover_at', { token: hoverToken, url: prepared.url, point: prepared.point });+ const realHover = await evaluateJson(`({ submenu: getComputedStyle(document.querySelector('#menu .submenu')).display,+ emulated: document.querySelectorAll('[data-webbrain-hover]').length, matches: document.getElementById('menu').matches(':hover') })`);+ check('hover_at opens a CSS :hover menu through the real :hover', result?.success === true && realHover.submenu === 'block'+ && realHover.emulated === 0 && realHover.matches === true, JSON.stringify({ result, realHover }));+ await session.closeRun(runId);} catch (error) {check('harness', false, error?.stack || error);} finally {@@ -238,5 +304,5 @@ if (failed.length) {console.error(`FAIL: ${failed.length} of ${results.length} Firefox vision input checks`);process.exitCode = 1;} else {- console.log(`PASS: ${results.length} Firefox vision input checks (CSS :hover and its restore, pointer capture, react-aria press, range track click, word and drag text selection, wheel, scaled frame click, cross-origin frame refusal)`);+ console.log(`PASS: ${results.length} Firefox vision input checks (CSS :hover and its restore, pointer capture, react-aria press, range track click, word and drag text selection, wheel, scaled frame click, cross-origin frame refusal; trusted click_at and hover_at through the companion session${locatedByScript ? ', its marked node found by script' : ''})`);}diff --git a/test/monitor-intervention-contract.mjs b/test/monitor-intervention-contract.mjsindex ba16f2c3b..eb4be11e7 100644--- a/test/monitor-intervention-contract.mjs+++ b/test/monitor-intervention-contract.mjs@@ -305,6 +305,75 @@ for (const [build, engine] of [['chrome', chromium], ['firefox', firefox]]) {} finally { await browser.close(); }});+ test(`${build}: a vision point's trusted click and hover through the companion belong to the agent`, { timeout: 60000 }, async t => {+ const browser = await engine.launch({ headless: true });+ // As the run prepares a vision point (page-feedback.js) and the companion+ // validates and marks before trusted input (firefox-companion/session.mjs),+ // with the element content.js _visionBidiPrepare marks: the one+ // document.elementFromPoint gives for the point.+ const trusted = async (page, { x, y, kind = 'click', mark = null }) => {+ const prepared = await message(page, 'page_monitor_prepare', { operationId: 'vision', tool: 'click', allowPassiveRebase: true, visionPoint: true, x, y });+ assert.equal(prepared.ready, true);+ const fence = await message(page, 'page_monitor_dispatch', { operationId: 'vision', kind: 'click', frameId: 0, navigationCandidate: kind === 'click', fenceOnly: true });+ assert.equal(fence.ready, true, JSON.stringify(fence));+ const validation = await message(page, 'page_monitor_validate', { ...fence.guard, kind: 'click', rebindFocus: false });+ assert.equal(validation.ready, true, JSON.stringify(validation));+ const marker = createNativeActionMarker({ ...fence.guard, revision: validation.revision }, 'click', 1);+ await page.evaluate(({ x, y, marker, mark }) => {+ const el = mark ? eval(mark) : document.elementFromPoint(x, y);+ el.setAttribute('data-webbrain-native-action', marker);+ }, { x, y, marker, mark });+ if (kind === 'click') await page.mouse.click(x, y);+ else await page.mouse.move(x, y, { steps: 2 });+ await message(page, 'page_monitor_finish', { operationId: 'vision' });+ await settle(page);+ return sources(page);+ };+ const shadowHost = page => page.evaluate(() => {+ const host = document.createElement('div');+ host.id = 'shadow-host';+ host.style.cssText = 'position:absolute;left:600px;top:300px;width:200px;height:60px';+ document.body.append(host);+ const root = host.attachShadow({ mode: 'open' });+ root.innerHTML = '<button id="inner" style="width:200px;height:60px">Inner</button>';+ window.innerClicks = 0;+ root.getElementById('inner').addEventListener('click', () => { window.innerClicks++; });+ }).then(() => settle(page)).then(() => page.evaluate(() => { feedback = []; delivered = []; }));+ try {+ await t.test('a trusted click on the marked element is no user input', async () => {+ const { context, page } = await fixture(browser, build);+ try {+ const box = await page.locator('#toggle').boundingBox();+ const events = await trusted(page, { x: Math.round(box.x + box.width / 2), y: Math.round(box.y + box.height / 2) });+ assert.equal(await page.evaluate(() => document.getElementById('hidden-switch').checked), true, 'the click landed');+ assert.equal(events.some(event => event.endsWith('/user')), false, `the trusted click is the agent's: ${events}`);+ } finally { await context.close(); }+ });+ await t.test('a trusted hover on the marked element is no user input', async () => {+ const { context, page } = await fixture(browser, build);+ try {+ const box = await page.locator('#send').boundingBox();+ const events = await trusted(page, { x: Math.round(box.x + box.width / 2), y: Math.round(box.y + box.height / 2), kind: 'hover' });+ assert.equal(events.some(event => event.endsWith('/user')), false, `the trusted hover is the agent's: ${events}`);+ } finally { await context.close(); }+ });+ // Why content.js leaves a point inside an open shadow root synthetic: the+ // monitor binds the host there, and neither a mark on the host nor one on+ // the element inside covers the trusted events, which come from inside.+ for (const [name, mark] of [['the host', null], ['the element inside', "document.getElementById('shadow-host').shadowRoot.getElementById('inner')"]]) {+ await t.test(`inside an open shadow root a mark on ${name} does not cover a trusted click`, async () => {+ const { context, page } = await fixture(browser, build);+ try {+ await shadowHost(page);+ const events = await trusted(page, { x: 700, y: 330, mark });+ assert.equal(await page.evaluate(() => window.innerClicks), 1);+ assert.ok(events.some(event => event.endsWith('/user')), `the click is not shown as the agent's: ${events}`);+ } finally { await context.close(); }+ });+ }+ } finally { await browser.close(); }+ });+test(`${build}: finishing an operation delivers its own pending reactions first`, { timeout: 60000 }, async t => {const browser = await engine.launch({ headless: true });try {diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 4225b9d71..55d036f96 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -226,6 +226,79 @@ test('content-script vision input (Firefox path) clicks, double-clicks, focuses,}});+test('content-script vision input (Firefox path) marks the element at the point for trusted input, after its checks', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ await page.setContent(`<!doctype html><style>body{margin:0}button,iframe,#host{position:absolute;margin:0;border:0}+ #go{left:100px;top:100px;width:200px;height:60px}#off{left:400px;top:100px;width:200px;height:60px}+ #frame{left:100px;top:300px;width:300px;height:150px}#host{left:500px;top:300px;width:200px;height:80px}</style>+ <button id="go">Go</button><button id="off" disabled>Off</button>+ <iframe id="frame" srcdoc="<button style='width:100%;height:100%'>Inside</button>"></iframe><div id="host"></div>+ <script>window.log=[];for(const t of ['pointerover','pointerdown','mousedown','mouseup','click','wheel'])document.addEventListener(t,e=>log.push(t),true);+ const root=document.getElementById('host').attachShadow({mode:'open'});root.innerHTML='<span id="inner" style="display:block;width:200px;height:80px">In</span>';</script>`);+ await page.waitForFunction(() => document.getElementById('frame').contentDocument?.querySelector('button'));+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ await page.evaluate(({ code, guardBlock }) => {+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)();+ }, { code: block, guardBlock });+ const token = '0b8d6f43-6c3a-4c43-9d0e-5b5b0f7f4c21';+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const marked = () => page.evaluate(t => [...document.querySelectorAll(`[data-webbrain-bidi="${t}"]`)].map(el => el.id), token);+ const clear = () => page.evaluate(() => { for (const el of document.querySelectorAll('[data-webbrain-bidi]')) el.removeAttribute('data-webbrain-bidi'); window.log.length = 0; });++ // A click: the element at the point is marked, the guard armed, and nothing synthetic is sent.+ let result = await run({ type: 'click', x: 150.6, y: 120.2, _bidiPrepare: token });+ assert.equal(result.bidiPrepared, true);+ assert.equal(result.noDispatch, true);+ assert.deepEqual(result.point, { x: 151, y: 120 });+ assert.deepEqual(result.target, { tag: 'button' });+ assert.match(result._filePickerGuardId, /^fpg_/);+ assert.equal(result.url, await page.evaluate(() => location.href));+ assert.deepEqual(await marked(), ['go']);+ assert.deepEqual(await page.evaluate(() => window.log), [], 'no synthetic event');+ await clear();+ // A hover is marked too, and arms no file-chooser guard.+ result = await run({ type: 'hover', x: 150, y: 120, _bidiPrepare: token });+ assert.equal(result.bidiPrepared, true);+ assert.equal(result._filePickerGuardId, undefined);+ await clear();+ // A point inside an open shadow root stays with the synthetic sequence:+ // the monitor binds the host, but the events come from inside it.+ result = await run({ type: 'click', x: 600, y: 340, _bidiPrepare: token });+ assert.equal(result.bidiUnavailable, true);+ assert.deepEqual(await page.evaluate(() => document.querySelectorAll('[data-webbrain-bidi]').length+ + document.getElementById('host').shadowRoot.querySelectorAll('[data-webbrain-bidi]').length), 0);+ await clear();+ // The synthetic sequence's checks come first: a disabled control is refused.+ result = await run({ type: 'click', x: 500, y: 120, _bidiPrepare: token });+ assert.equal(result.visionInputRefused, 'disabled_control');+ assert.deepEqual(await marked(), []);+ // A point over an embedded frame stays with the synthetic sequence.+ result = await run({ type: 'click', x: 250, y: 375, _bidiPrepare: token });+ assert.equal(result.bidiUnavailable, true);+ assert.deepEqual(await marked(), []);+ // A drag or a wheel is never prepared, and a malformed binding marks nothing.+ for (const params of [{ type: 'drag', x: 150, y: 120, toX: 250, toY: 120 }, { type: 'wheel', x: 150, y: 120, deltaY: 100 }]) {+ result = await run({ ...params, _bidiPrepare: token });+ assert.equal(result.noDispatch, true, params.type);+ assert.equal(result.bidiPrepared, undefined, params.type);+ }+ result = await run({ type: 'click', x: 150, y: 120, _bidiPrepare: 'not-a-token' });+ assert.equal(result.noDispatch, true);+ assert.deepEqual(await page.evaluate(() => document.querySelectorAll('[data-webbrain-bidi]').length), 0);+ assert.deepEqual(await page.evaluate(() => window.log), [], 'still no synthetic event');+ } finally {+ await browser.close();+ }+});+test('content-script vision drag (Firefox path) moves and releases in a same-origin frame\'s own coordinates', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {