Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode (Firefox): keep the pointer capture a page takes on a vision press
Emre Sokullua7220563c41b
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 12b5d0b26..60cf7470b 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.@@ -169,7 +169,7 @@ npm run test:vision-mode- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 7c7d7e6c8..b7bfbf864 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -2022,6 +2022,92 @@return chain;}+ // The pointerId of the vision pointer's events (a mouse's in Chrome).+ const _VISION_POINTER_ID = 1;++ // A slider, canvas or press-and-hold control often calls setPointerCapture+ // on the pointerdown it gets, then reads the moves and the release from+ // the capturing element wherever the pointer goes. A synthetic pointer is+ // no active browser pointer: Firefox throws for its id (a handler that+ // captures or releases first stops there), and no engine routes its+ // events. So while a vision click or drag runs, the capture methods of+ // the press's window answer for the vision pointer here (other ids still+ // reach the page's own methods, which are put back by end()), and the+ // sequence sends each later pointer event, with its compatibility mouse+ // event, to target(): the capturing element, if any. As in the browser,+ // gotpointercapture comes before the first of those events and+ // lostpointercapture right after the release; `fire(target, type, init)`+ // dispatches them.+ function _visionPointerCapture(view, fire) {+ let active = true;+ let pressed = false;+ let pending = null;+ let captured = null;+ const restore = [];+ let proto = null;+ let exported = fn => fn;+ try {+ // A Firefox content script reaches the page's own prototype through+ // wrappedJSObject, and hands it functions with exportFunction.+ const page = view.wrappedJSObject;+ if (page && typeof exportFunction === 'function') {+ proto = page.Element?.prototype || null;+ exported = fn => exportFunction(fn, view);+ } else {+ proto = view.Element?.prototype || null;+ }+ } catch { proto = null; }+ const answers = {+ // As for a mouse: a pressed pointer can be captured, a hovering one not.+ setPointerCapture: element => { if (pressed) pending = element; },+ releasePointerCapture: element => { if (pending === element) pending = null; },+ hasPointerCapture: element => pending === element,+ };+ for (const name of Object.keys(answers)) {+ try {+ const original = proto?.[name];+ if (typeof original !== 'function') continue;+ proto[name] = exported(function (pointerId, ...rest) {+ // An element out of the document cannot take capture: the page's+ // method throws for it, as the browser's does.+ if (active && this?.nodeType === 1 && Math.trunc(Number(pointerId)) === _VISION_POINTER_ID+ && (name !== 'setPointerCapture' || this.isConnected)) return answers[name](this);+ return original.call(this, pointerId, ...rest);+ });+ restore.push(() => { proto[name] = original; });+ } catch { /* the page keeps its own method */ }+ }+ const settle = init => {+ // A capturing element taken out of the document loses capture, which+ // its document is told of.+ if (captured && !captured.isConnected) {+ const doc = captured.ownerDocument;+ if (pending === captured) pending = null;+ captured = null;+ if (doc) fire(doc, 'lostpointercapture', init);+ }+ if (pending && !pending.isConnected) pending = null;+ const next = pending;+ if (captured && captured !== next) fire(captured, 'lostpointercapture', init);+ if (next && next !== captured) fire(next, 'gotpointercapture', init);+ captured = next;+ return captured;+ };+ return {+ press() { pressed = true; },+ // Before each pointer event after the press: the element it goes to,+ // or null for the element under the pointer.+ target: settle,+ // Right after the pointerup or pointercancel: capture ends.+ release(init) { pressed = false; pending = null; settle(init); },+ end() {+ active = false;+ pressed = false;+ for (const undo of restore.splice(0)) { try { undo(); } catch { /* left to the page */ } }+ },+ };+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -2042,7 +2128,7 @@...extra,});const PointerCtor = view.PointerEvent || view.MouseEvent;- const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...extra })));+ const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: _VISION_POINTER_ID, pointerType: 'mouse', isPrimary: true, ...extra })));const mouse = (target, name, extra = {}) => target.dispatchEvent(new view.MouseEvent(name, init(extra)));// The pointer moving onto `target` (with `extra` init, e.g. a drag's// point and buttons): out and leave on what it left, over and enter on@@ -2089,7 +2175,13 @@error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,};}- return withLocalPageDispatch(() => {+ // A click or drag presses the pointer, which the page may capture+ // (_visionPointerCapture); its capture methods are the page's own again+ // once the sequence ends.+ const capture = type === 'click' || type === 'drag'+ ? _visionPointerCapture(view, (target, name, extra) => pointer(target, name, { ...extra, cancelable: false, composed: false }))+ : null;+ const dispatchSequence = () => {if (type === 'hover') {hover(el);return { success: true, dispatched: true, method: 'synthetic-vision-pointer', target: describe };@@ -2119,6 +2211,7 @@const toY = Number(params.toY);const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));hover(el);+ capture.press();pointer(el, 'pointerdown', { button: 0, buttons: 1 });const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });// The press focuses what it lands on, as a click's does, and starts@@ -2152,6 +2245,10 @@if (at) { offsetX = px - at.cx; offsetY = py - at.cy; }return { clientX: px - offsetX, clientY: py - offsetY, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };};+ // An element holding the pointer's capture gets the events wherever+ // the pointer is, in its own document's coordinates: the press's,+ // whose window alone answered the capture call.+ const heldAt = (point, px, py) => ({ ...point, clientX: px - pressOffsetX, clientY: py - pressOffsetY });let last = el;for (let i = 1; i <= steps; i++) {const px = x + (toX - x) * (i / steps);@@ -2161,16 +2258,24 @@const point = pointAt(at, px, py);if (html && i === 1) {htmlStarted = html.start(point);- if (htmlStarted) pointer(el, 'pointercancel', { ...point, buttons: 0, cancelable: false });+ if (htmlStarted) {+ const holder = capture.target({ ...heldAt(point, px, py), buttons: 1 });+ const cancelInit = { ...(holder ? heldAt(point, px, py) : point), buttons: 0, cancelable: false };+ pointer(holder || el, 'pointercancel', cancelInit);+ capture.release(cancelInit);+ }}if (htmlStarted) { html.move(last, point); continue; }- moveOnto(last, { button: 0, buttons: 1, ...point });+ const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });+ moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });range?.to(px - pressOffsetX, py - pressOffsetY);selection?.to(px - pressOffsetX, py - pressOffsetY);}const end = _visionTargetAt(toX, toY);- const releaseTarget = end?.el || last;- const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };+ const endPoint = pointAt(end, toX, toY);+ const holder = htmlStarted ? null : capture.target({ button: 0, buttons: 1, ...heldAt(endPoint, toX, toY) });+ const releaseTarget = holder || end?.el || last;+ const endInit = { button: 0, buttons: 0, ...(holder ? heldAt(endPoint, toX, toY) : endPoint) };if (htmlStarted) {const html5Drag = html.end(endInit);_visionPointerOver = releaseTarget;@@ -2178,6 +2283,7 @@}if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });pointer(releaseTarget, 'pointerup', endInit);+ capture.release(endInit);mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });const rangeValue = range?.release();return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,@@ -2189,6 +2295,7 @@// the guard blocks it, and the agent settles the guard afterwards.const filePickerGuard = clickWithoutNativeFilePicker(() => {for (let count = 1; count <= clicks; count++) {+ capture.press();pointer(el, 'pointerdown', { button, buttons: buttonsMask });const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });if (count === 1 && button === 0 && pressAllowed) {@@ -2197,8 +2304,11 @@else if (!focusable) el.ownerDocument.activeElement?.blur?.();if (focusable) _visionPlaceCaret(el, cx, cy);}- pointer(el, 'pointerup', { button, buttons: 0 });- mouse(el, 'mouseup', { button, buttons: 0, detail: count });+ // A press the page captured is released on the capturing element.+ const holder = capture.target({ button, buttons: buttonsMask }) || el;+ pointer(holder, 'pointerup', { button, buttons: 0 });+ capture.release({ button, buttons: 0 });+ mouse(holder, 'mouseup', { button, buttons: 0, detail: count });if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });}@@ -2231,7 +2341,12 @@if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };- });+ };+ try {+ return withLocalPageDispatch(dispatchSequence);+ } finally {+ capture?.end();+ }}let _lastClickIdent = null;diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 479da1386..17f5e3bce 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2316,6 +2316,92 @@return chain;}+ // The pointerId of the vision pointer's events (a mouse's in Chrome).+ const _VISION_POINTER_ID = 1;++ // A slider, canvas or press-and-hold control often calls setPointerCapture+ // on the pointerdown it gets, then reads the moves and the release from+ // the capturing element wherever the pointer goes. A synthetic pointer is+ // no active browser pointer: Firefox throws for its id (a handler that+ // captures or releases first stops there), and no engine routes its+ // events. So while a vision click or drag runs, the capture methods of+ // the press's window answer for the vision pointer here (other ids still+ // reach the page's own methods, which are put back by end()), and the+ // sequence sends each later pointer event, with its compatibility mouse+ // event, to target(): the capturing element, if any. As in the browser,+ // gotpointercapture comes before the first of those events and+ // lostpointercapture right after the release; `fire(target, type, init)`+ // dispatches them.+ function _visionPointerCapture(view, fire) {+ let active = true;+ let pressed = false;+ let pending = null;+ let captured = null;+ const restore = [];+ let proto = null;+ let exported = fn => fn;+ try {+ // A Firefox content script reaches the page's own prototype through+ // wrappedJSObject, and hands it functions with exportFunction.+ const page = view.wrappedJSObject;+ if (page && typeof exportFunction === 'function') {+ proto = page.Element?.prototype || null;+ exported = fn => exportFunction(fn, view);+ } else {+ proto = view.Element?.prototype || null;+ }+ } catch { proto = null; }+ const answers = {+ // As for a mouse: a pressed pointer can be captured, a hovering one not.+ setPointerCapture: element => { if (pressed) pending = element; },+ releasePointerCapture: element => { if (pending === element) pending = null; },+ hasPointerCapture: element => pending === element,+ };+ for (const name of Object.keys(answers)) {+ try {+ const original = proto?.[name];+ if (typeof original !== 'function') continue;+ proto[name] = exported(function (pointerId, ...rest) {+ // An element out of the document cannot take capture: the page's+ // method throws for it, as the browser's does.+ if (active && this?.nodeType === 1 && Math.trunc(Number(pointerId)) === _VISION_POINTER_ID+ && (name !== 'setPointerCapture' || this.isConnected)) return answers[name](this);+ return original.call(this, pointerId, ...rest);+ });+ restore.push(() => { proto[name] = original; });+ } catch { /* the page keeps its own method */ }+ }+ const settle = init => {+ // A capturing element taken out of the document loses capture, which+ // its document is told of.+ if (captured && !captured.isConnected) {+ const doc = captured.ownerDocument;+ if (pending === captured) pending = null;+ captured = null;+ if (doc) fire(doc, 'lostpointercapture', init);+ }+ if (pending && !pending.isConnected) pending = null;+ const next = pending;+ if (captured && captured !== next) fire(captured, 'lostpointercapture', init);+ if (next && next !== captured) fire(next, 'gotpointercapture', init);+ captured = next;+ return captured;+ };+ return {+ press() { pressed = true; },+ // Before each pointer event after the press: the element it goes to,+ // or null for the element under the pointer.+ target: settle,+ // Right after the pointerup or pointercancel: capture ends.+ release(init) { pressed = false; pending = null; settle(init); },+ end() {+ active = false;+ pressed = false;+ for (const undo of restore.splice(0)) { try { undo(); } catch { /* left to the page */ } }+ },+ };+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -2336,7 +2422,7 @@...extra,});const PointerCtor = view.PointerEvent || view.MouseEvent;- const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...extra })));+ const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: _VISION_POINTER_ID, pointerType: 'mouse', isPrimary: true, ...extra })));const mouse = (target, name, extra = {}) => target.dispatchEvent(new view.MouseEvent(name, init(extra)));// The pointer moving onto `target` (with `extra` init, e.g. a drag's// point and buttons): out and leave on what it left, over and enter on@@ -2383,7 +2469,13 @@error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,};}- return withLocalPageDispatch(() => {+ // A click or drag presses the pointer, which the page may capture+ // (_visionPointerCapture); its capture methods are the page's own again+ // once the sequence ends.+ const capture = type === 'click' || type === 'drag'+ ? _visionPointerCapture(view, (target, name, extra) => pointer(target, name, { ...extra, cancelable: false, composed: false }))+ : null;+ const dispatchSequence = () => {if (type === 'hover') {hover(el);return { success: true, dispatched: true, method: 'synthetic-vision-pointer', target: describe };@@ -2413,6 +2505,7 @@const toY = Number(params.toY);const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));hover(el);+ capture.press();pointer(el, 'pointerdown', { button: 0, buttons: 1 });const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });// The press focuses what it lands on, as a click's does, and starts@@ -2446,6 +2539,10 @@if (at) { offsetX = px - at.cx; offsetY = py - at.cy; }return { clientX: px - offsetX, clientY: py - offsetY, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };};+ // An element holding the pointer's capture gets the events wherever+ // the pointer is, in its own document's coordinates: the press's,+ // whose window alone answered the capture call.+ const heldAt = (point, px, py) => ({ ...point, clientX: px - pressOffsetX, clientY: py - pressOffsetY });let last = el;for (let i = 1; i <= steps; i++) {const px = x + (toX - x) * (i / steps);@@ -2455,16 +2552,24 @@const point = pointAt(at, px, py);if (html && i === 1) {htmlStarted = html.start(point);- if (htmlStarted) pointer(el, 'pointercancel', { ...point, buttons: 0, cancelable: false });+ if (htmlStarted) {+ const holder = capture.target({ ...heldAt(point, px, py), buttons: 1 });+ const cancelInit = { ...(holder ? heldAt(point, px, py) : point), buttons: 0, cancelable: false };+ pointer(holder || el, 'pointercancel', cancelInit);+ capture.release(cancelInit);+ }}if (htmlStarted) { html.move(last, point); continue; }- moveOnto(last, { button: 0, buttons: 1, ...point });+ const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });+ moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });range?.to(px - pressOffsetX, py - pressOffsetY);selection?.to(px - pressOffsetX, py - pressOffsetY);}const end = _visionTargetAt(toX, toY);- const releaseTarget = end?.el || last;- const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };+ const endPoint = pointAt(end, toX, toY);+ const holder = htmlStarted ? null : capture.target({ button: 0, buttons: 1, ...heldAt(endPoint, toX, toY) });+ const releaseTarget = holder || end?.el || last;+ const endInit = { button: 0, buttons: 0, ...(holder ? heldAt(endPoint, toX, toY) : endPoint) };if (htmlStarted) {const html5Drag = html.end(endInit);_visionPointerOver = releaseTarget;@@ -2472,6 +2577,7 @@}if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });pointer(releaseTarget, 'pointerup', endInit);+ capture.release(endInit);mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });const rangeValue = range?.release();return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,@@ -2483,6 +2589,7 @@// the guard blocks it, and the agent settles the guard afterwards.const filePickerGuard = clickWithoutNativeFilePicker(() => {for (let count = 1; count <= clicks; count++) {+ capture.press();pointer(el, 'pointerdown', { button, buttons: buttonsMask });const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });if (count === 1 && button === 0 && pressAllowed) {@@ -2491,8 +2598,11 @@else if (!focusable) el.ownerDocument.activeElement?.blur?.();if (focusable) _visionPlaceCaret(el, cx, cy);}- pointer(el, 'pointerup', { button, buttons: 0 });- mouse(el, 'mouseup', { button, buttons: 0, detail: count });+ // A press the page captured is released on the capturing element.+ const holder = capture.target({ button, buttons: buttonsMask }) || el;+ pointer(holder, 'pointerup', { button, buttons: 0 });+ capture.release({ button, buttons: 0 });+ mouse(holder, 'mouseup', { button, buttons: 0, detail: count });if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });}@@ -2525,7 +2635,12 @@if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };- });+ };+ try {+ return withLocalPageDispatch(dispatchSequence);+ } finally {+ capture?.end();+ }}let _lastClickIdent = null;diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 07fc371b0..2cb8f9c48 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -500,6 +500,89 @@ test('content-script vision input (Firefox path) moves the pointer off one eleme}});+test('content-script vision input (Firefox path) honours the pointer capture a page takes on the press', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ // Run as here (the page's own world), and as from a Firefox content+ // script, which reaches the page's prototype through wrappedJSObject and+ // hands it functions with exportFunction (stood in for here).+ for (const firefoxContentScript of [false, true]) {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ // #handle is a slider thumb whose pointer handlers are on itself; #track+ // captures the thumb when pressed; #press works like a react-aria button.+ await page.setContent(`<!doctype html><style>body{margin:0}#track{position:absolute;left:100px;top:100px;width:600px;height:20px;background:#ddd}+ #handle{position:absolute;left:100px;top:90px;width:40px;height:40px;background:#08f}#press{position:absolute;left:100px;top:300px;width:120px;height:40px}</style>+ <div id="track"></div><div id="handle"></div><button id="press">Press</button>+ <script>+ // As in Firefox, the browser's capture methods know no pointer with the vision pointer's id.+ for (const name of ['setPointerCapture', 'releasePointerCapture']) {+ const native = Element.prototype[name];+ Element.prototype[name] = function (id) { if (id === 1) throw new DOMException('Invalid pointer id', 'NotFoundError'); return native.call(this, id); };+ }+ window.own = { set: Element.prototype.setPointerCapture, release: Element.prototype.releasePointerCapture, has: Element.prototype.hasPointerCapture };+ window.log = [];+ const handle = document.getElementById('handle');+ let grab = false;+ handle.addEventListener('pointerdown', e => { handle.setPointerCapture(e.pointerId); grab = true; log.push(['down', e.clientX]); });+ handle.addEventListener('pointermove', e => { if (grab) log.push(['move', e.clientX, handle.hasPointerCapture(e.pointerId)]); });+ handle.addEventListener('pointerup', e => { log.push(['up', e.clientX, e.clientY]); grab = false; handle.releasePointerCapture(e.pointerId); });+ handle.addEventListener('mouseup', e => log.push(['mouseup', e.clientX]));+ for (const type of ['gotpointercapture', 'lostpointercapture']) handle.addEventListener(type, e => log.push([type.replace('pointercapture', ''), e.target.id]));+ document.getElementById('track').addEventListener('pointerdown', e => handle.setPointerCapture(e.pointerId));+ const press = document.getElementById('press');+ press.addEventListener('pointerdown', e => {+ e.target.releasePointerCapture(e.pointerId);+ try { press.setPointerCapture(7); } catch (error) { log.push(['other id', error.name]); }+ document.addEventListener('pointerup', () => log.push(['pressed']), { once: true });+ });+ </script>`);+ await page.evaluate(({ code, guardBlock, firefoxContentScript }) => {+ window.__exported = [];+ if (firefoxContentScript) Object.defineProperty(window, 'wrappedJSObject', { value: window, configurable: true });+ const exportFunction = firefoxContentScript ? (fn, scope) => { window.__exported.push(scope === window); return fn; } : undefined;+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function('exportFunction', `function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)(exportFunction);+ }, { code: block, guardBlock, firefoxContentScript });+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const take = () => page.evaluate(() => window.log.splice(0));+ const label = firefoxContentScript ? 'Firefox content script' : 'page world';+ // The thumb takes capture on the press: every move and the release reach+ // it, in its document's coordinates, far past its own box.+ let result = await run({ type: 'drag', x: 120, y: 110, toX: 620, toY: 160, steps: 5 });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.deepEqual(await take(), [+ ['down', 120], ['got', 'handle'], ['move', 220, true], ['move', 320, true], ['move', 420, true], ['move', 520, true], ['move', 620, true],+ ['up', 620, 160], ['lost', 'handle'], ['mouseup', 620],+ ], label);+ // A press on the track that captures the thumb is released on the thumb.+ result = await run({ type: 'click', x: 400, y: 110 });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.deepEqual(await take(), [['got', 'handle'], ['up', 400, 110], ['lost', 'handle'], ['mouseup', 400]], label);+ // A control that drops the implicit capture first, then waits for the+ // release, completes its press; other pointer ids stay the page's own.+ result = await run({ type: 'click', x: 160, y: 320 });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.deepEqual(await take(), [['other id', 'NotFoundError'], ['pressed']], label);+ // The page's own capture methods are back once each sequence ends.+ assert.equal(await page.evaluate(() => Element.prototype.setPointerCapture === own.set+ && Element.prototype.releasePointerCapture === own.release && Element.prototype.hasPointerCapture === own.has), true, label);+ assert.deepEqual(await page.evaluate(() => {+ try { document.body.setPointerCapture(1); return 'no error'; } catch (error) { return error.name; }+ }), 'NotFoundError', label);+ if (firefoxContentScript) assert.deepEqual(await page.evaluate(() => window.__exported), Array(9).fill(true), 'each method of each sequence is exported into the page window');+ await page.close();+ }+ } finally {+ await browser.close();+ }+});+test('content-script vision wheel (Firefox path) scrolls what the browser would', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {