Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: place points inside scaled same-origin frames
Emre Sokullu856d2f0d06af
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 60cf7470b..de7521b2d 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -118,11 +118,11 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.-Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:+Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls. It maps a point into a same-origin frame as the Firefox input does; a frame no scale maps counts as one it cannot inspect. (The click submit probe's Chrome frame mapping likewise takes the frame's border, padding and drawn scale into account.)- `click_at` with `clicks: 2|3` is refused on a button, link or other control (including an element drawn with a pointer cursor, which is how a framework button made of a bare `<div>` shows itself), on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.- `drag_at` is refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a cross-origin frame, when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging page `drag_at` is blocked outright, like `iframe_click`.@@ -169,7 +169,7 @@ npm run test:vision-mode- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex bb464ccc5..e8d0213e8 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -33932,11 +33932,22 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dhost = parsed.hostname;}} catch {}+ // Where the frame's own viewport starts inside its box, and+ // the scale a CSS transform or zoom draws it at (its layout+ // box against the drawn one; 1 up to rounding), so the probe+ // in the frame maps the point onto what is drawn there.+ const style = getComputedStyle(frame);+ const length = value => Number.parseFloat(value) || 0;+ const scale = (drawn, laid) => (laid > 0 && Math.abs(drawn - laid) > 1 ? drawn / laid : 1);return {left: rect.left,top: rect.top,width: rect.width,height: rect.height,+ insetLeft: length(style.borderLeftWidth) + length(style.paddingLeft),+ insetTop: length(style.borderTopWidth) + length(style.paddingTop),+ scaleX: scale(rect.width, frame.offsetWidth),+ scaleY: scale(rect.height, frame.offsetHeight),url,host,index,@@ -35343,10 +35354,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst candidateRects = matchingRects.length ? matchingRects : (frameRects.length === 1 ? frameRects : []);let fallbackTarget = null;for (const frame of candidateRects) {- const localX = Number(args.x) - Number(frame?.left);- const localY = Number(args.y) - Number(frame?.top);- const width = Number(frame?.width);- const height = Number(frame?.height);+ // The frame's drawn box, scaled back to its own viewport.+ const scaleX = Number(frame?.scaleX) > 0 ? Number(frame.scaleX) : 1;+ const scaleY = Number(frame?.scaleY) > 0 ? Number(frame.scaleY) : 1;+ const localX = (Number(args.x) - Number(frame?.left)) / scaleX - (Number(frame?.insetLeft) || 0);+ const localY = (Number(args.y) - Number(frame?.top)) / scaleY - (Number(frame?.insetTop) || 0);+ const width = Number(frame?.width) / scaleX;+ const height = Number(frame?.height) / scaleY;if (!Number.isFinite(localX) || !Number.isFinite(localY)) continue;if (Number.isFinite(width) && (localX < 0 || localX > width)) continue;if (Number.isFinite(height) && (localY < 0 || localY > height)) continue;diff --git a/src/chrome/src/agent/vision-dom-probe.js b/src/chrome/src/agent/vision-dom-probe.jsindex 75b85538a..61828c085 100644--- a/src/chrome/src/agent/vision-dom-probe.js+++ b/src/chrome/src/agent/vision-dom-probe.js@@ -52,6 +52,49 @@ export function visionDomProbe(request) {if (parent && parent.nodeType === 11) return parent.host || null;return parent || null;};+ // How a point of a document maps into a same-origin frame drawn in it+ // (content.js _visionFrameMap does the same for Firefox input): the+ // frame's position, borders and padding are taken off, and whatever scale+ // a CSS transform or zoom draws it at is undone, by comparing its drawn box+ // with the size of its own viewport. Null when the frame or an ancestor is+ // rotated, skewed, flipped, set on a motion path or drawn in 3D, which no+ // scale maps.+ const frameMap = (frame, innerView) => {+ const view = frame.ownerDocument && frame.ownerDocument.defaultView;+ if (!view) return null;+ const length = value => Number.parseFloat(value) || 0;+ for (let node = frame, depth = 0; node && node.nodeType === 1 && depth < 256; depth++) {+ const style = view.getComputedStyle(node);+ const transform = String(style.transform || 'none');+ if (transform !== 'none') {+ const matrix = /^matrix\(([^)]*)\)$/.exec(transform);+ const parts = matrix ? matrix[1].split(',').map(Number) : [];+ if (!matrix || Math.abs(parts[1]) > 1e-6 || Math.abs(parts[2]) > 1e-6 || !(parts[0] > 0) || !(parts[3] > 0)) return null;+ }+ const rotate = String(style.rotate || 'none');+ if (rotate !== 'none' && !/^0(deg|rad|grad|turn)?$/.test(rotate)) return null;+ const scale = String(style.scale || 'none');+ if (scale !== 'none' && scale.split(/\s+/).some(part => !(length(part) > 0))) return null;+ // A motion path turns the element along it.+ if (String(style.offsetPath || 'none') !== 'none') return null;+ const root = node.getRootNode ? node.getRootNode() : null;+ node = node.parentElement || (root && root.nodeType === 11 ? root.host : null);+ }+ const rect = frame.getBoundingClientRect();+ const style = view.getComputedStyle(frame);+ const left = length(style.borderLeftWidth) + length(style.paddingLeft);+ const top = length(style.borderTopWidth) + length(style.paddingTop);+ const right = length(style.borderRightWidth) + length(style.paddingRight);+ const bottom = length(style.borderBottomWidth) + length(style.paddingBottom);+ const innerWidth = Number(innerView && innerView.innerWidth) || Math.max(0, frame.clientWidth - length(style.paddingLeft) - length(style.paddingRight));+ const innerHeight = Number(innerView && innerView.innerHeight) || Math.max(0, frame.clientHeight - length(style.paddingTop) - length(style.paddingBottom));+ const width = innerWidth + left + right;+ const height = innerHeight + top + bottom;+ // An unscaled frame's drawn box differs from that size by rounding only.+ const sx = width > 0 && Math.abs(rect.width - width) > 1 ? rect.width / width : 1;+ const sy = height > 0 && Math.abs(rect.height - height) > 1 ? rect.height / height : 1;+ return (px, py) => ({ x: (px - rect.left) / sx - left, y: (py - rect.top) / sy - top });+ };const hit = (x, y) => {let doc = document;let cx = x;@@ -68,14 +111,16 @@ export function visionDomProbe(request) {if (tag === 'IFRAME' || tag === 'FRAME' || tag === 'OBJECT' || tag === 'EMBED') {let inner = null;try { inner = el.contentDocument; } catch (e) { inner = null; }- if (inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME')) {- const rect = el.getBoundingClientRect();- cx -= rect.left + (el.clientLeft || 0);- cy -= rect.top + (el.clientTop || 0);+ const map = inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME') ? frameMap(el, inner.defaultView) : null;+ if (map) {+ const point = map(cx, cy);+ cx = point.x;+ cy = point.y;doc = inner;continue;}- // A cross-origin frame (or plugin): its content is not inspectable.+ // A cross-origin frame (or plugin), or one drawn at an angle no+ // scale maps: what is under the point inside it cannot be read.return { el, opaque: true };}return { el, opaque: false };diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex b7bfbf864..8aa563781 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -1700,11 +1700,18 @@// coordinates, so canvases, sliders and maps see where the model pointed;// the default action of a press (focus) and of a click (activation) is// reproduced the way a browser does it. Same-origin iframes are entered.+ //+ // What is under a top-level viewport point: the element, its document and+ // window, the point in that document's client coordinates (cx, cy), and+ // toDoc(x, y), which maps any top-level point into that document the same+ // way. A point on a same-origin frame that no scale maps (_visionFrameMap)+ // gives { el: the frame, unmappable: true }.function _visionTargetAt(x, y) {let doc = document;let view = window;let cx = x;let cy = y;+ const maps = [];for (let depth = 0; depth < 6; depth++) {let el = doc.elementFromPoint(cx, cy);const seen = new Set();@@ -1720,19 +1727,58 @@let inner = null;try { inner = el.contentDocument; } catch { inner = null; }if (inner?.documentElement) {- const rect = el.getBoundingClientRect();- cx -= rect.left + (el.clientLeft || 0);- cy -= rect.top + (el.clientTop || 0);+ const map = _visionFrameMap(el, view, inner.defaultView);+ if (!map) return { el, doc, view, cx, cy, unmappable: true };+ ({ x: cx, y: cy } = map(cx, cy));+ maps.push(map);doc = inner;view = inner.defaultView || view;continue;}}- return { el, doc, view, cx, cy };+ const toDoc = (px, py) => maps.reduce((point, map) => map(point.x, point.y), { x: px, y: py });+ return { el, doc, view, cx, cy, toDoc };}return null;}+ // How a point of a document maps into a same-origin frame drawn in it:+ // the frame's position, borders and padding are taken off, and whatever+ // scale a CSS transform or zoom draws it at (on itself or an ancestor) is+ // undone, by comparing its drawn box with the size of its own viewport.+ // Null when the frame or an ancestor is rotated, skewed, flipped, set on+ // a motion path or drawn in 3D, which no scale maps.+ function _visionFrameMap(frame, view, innerView) {+ const length = value => Number.parseFloat(value) || 0;+ for (let node = frame, depth = 0; node?.nodeType === 1 && depth < 256; node = node.parentElement || node.getRootNode?.()?.host, depth++) {+ const style = view.getComputedStyle(node);+ const transform = String(style.transform || 'none');+ if (transform !== 'none') {+ const matrix = /^matrix\(([^)]*)\)$/.exec(transform);+ const [a, b, c, d] = matrix ? matrix[1].split(',').map(Number) : [];+ if (!matrix || Math.abs(b) > 1e-6 || Math.abs(c) > 1e-6 || !(a > 0) || !(d > 0)) return null;+ }+ const rotate = String(style.rotate || 'none');+ if (rotate !== 'none' && !/^0(deg|rad|grad|turn)?$/.test(rotate)) return null;+ const scale = String(style.scale || 'none');+ if (scale !== 'none' && scale.split(/\s+/).some(part => !(length(part) > 0))) return null;+ // A motion path turns the element along it.+ if (String(style.offsetPath || 'none') !== 'none') return null;+ }+ const rect = frame.getBoundingClientRect();+ const style = view.getComputedStyle(frame);+ const left = length(style.borderLeftWidth) + length(style.paddingLeft);+ const top = length(style.borderTopWidth) + length(style.paddingTop);+ const right = length(style.borderRightWidth) + length(style.paddingRight);+ const bottom = length(style.borderBottomWidth) + length(style.paddingBottom);+ const width = (Number(innerView?.innerWidth) || Math.max(0, frame.clientWidth - length(style.paddingLeft) - length(style.paddingRight))) + left + right;+ const height = (Number(innerView?.innerHeight) || Math.max(0, frame.clientHeight - length(style.paddingTop) - length(style.paddingBottom))) + top + bottom;+ // An unscaled frame's drawn box differs from that size by rounding only.+ const sx = width > 0 && Math.abs(rect.width - width) > 1 ? rect.width / width : 1;+ const sy = height > 0 && Math.abs(rect.height - height) > 1 ? rect.height / height : 1;+ return (x, y) => ({ x: (x - rect.left) / sx - left, y: (y - rect.top) / sy - top });+ }+function _visionFocusable(el) {for (let node = el; node && node.nodeType === 1; node = node.parentElement || node.getRootNode?.()?.host) {if (node.matches?.('input:not([disabled]),textarea:not([disabled]),select:not([disabled]),button:not([disabled]),a[href],[tabindex],[contenteditable=""],[contenteditable="true"],summary')) return node;@@ -2117,6 +2163,12 @@}const hit = _visionTargetAt(x, y);if (!hit) return { success: false, dispatched: false, noDispatch: true, error: 'Nothing is rendered at that point.' };+ if (hit.unmappable) {+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'frame_unmappable',+ error: 'Not dispatched: the point is inside an embedded frame drawn rotated, skewed, flipped or in 3D, so it cannot be placed on the frame\'s content. Use a control outside that frame.',+ };+ }if (actionDeadlineExpired()) return { success: false, dispatched: false, noDispatch: true, deadlineExpired: true, error: 'The action deadline expired before dispatch.' };const { el, view, cx, cy } = hit;const button = params.button === 'right' ? 2 : params.button === 'middle' ? 1 : 0;@@ -2231,29 +2283,35 @@// gets drag events instead of pointer and mouse events.const html = pressAllowed && !range ? _visionHtmlDrag(el) : null;let htmlStarted = false;- const pressOffsetX = x - cx;- const pressOffsetY = y - cy;range?.to(cx, cy);if (selection && !selection.begin(cx, cy)) selection = null;// Every point, the release included, is given in the coordinates of- // the document under it (a same-origin frame has its own), so the- // events a slider or canvas reads all agree. A point with nothing- // under it keeps the offset of the last frame that had something.- let offsetX = x - cx;- let offsetY = y - cy;+ // the document under it (a same-origin frame has its own, scaled as+ // the frame is drawn), so the events a slider or canvas reads all+ // agree. A point with nothing under it, or on a frame no scale maps,+ // is mapped into the last document that had something.+ let lastToDoc = hit.toDoc;const pointAt = (at, px, py) => {- if (at) { offsetX = px - at.cx; offsetY = py - at.cy; }- return { clientX: px - offsetX, clientY: py - offsetY, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };+ if (at) lastToDoc = at.toDoc;+ const local = at ? { x: at.cx, y: at.cy } : lastToDoc(px, py);+ return { clientX: local.x, clientY: local.y, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };+ };+ const mappedAt = (px, py) => {+ const at = _visionTargetAt(px, py);+ return at && !at.unmappable ? at : null;};// An element holding the pointer's capture gets the events wherever// the pointer is, in its own document's coordinates: the press's,// whose window alone answered the capture call.- const heldAt = (point, px, py) => ({ ...point, clientX: px - pressOffsetX, clientY: py - pressOffsetY });+ const heldAt = (point, px, py) => {+ const local = hit.toDoc(px, py);+ return { ...point, clientX: local.x, clientY: local.y };+ };let last = el;for (let i = 1; i <= steps; i++) {const px = x + (toX - x) * (i / steps);const py = y + (toY - y) * (i / steps);- const at = _visionTargetAt(px, py);+ const at = mappedAt(px, py);last = at?.el || last;const point = pointAt(at, px, py);if (html && i === 1) {@@ -2268,10 +2326,11 @@if (htmlStarted) { html.move(last, point); continue; }const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });- range?.to(px - pressOffsetX, py - pressOffsetY);- selection?.to(px - pressOffsetX, py - pressOffsetY);+ const local = hit.toDoc(px, py);+ range?.to(local.x, local.y);+ selection?.to(local.x, local.y);}- const end = _visionTargetAt(toX, toY);+ const end = mappedAt(toX, toY);const endPoint = pointAt(end, toX, toY);const holder = htmlStarted ? null : capture.target({ button: 0, buttons: 1, ...heldAt(endPoint, toX, toY) });const releaseTarget = holder || end?.el || last;diff --git a/src/firefox/src/agent/vision-dom-probe.js b/src/firefox/src/agent/vision-dom-probe.jsindex 75b85538a..61828c085 100644--- a/src/firefox/src/agent/vision-dom-probe.js+++ b/src/firefox/src/agent/vision-dom-probe.js@@ -52,6 +52,49 @@ export function visionDomProbe(request) {if (parent && parent.nodeType === 11) return parent.host || null;return parent || null;};+ // How a point of a document maps into a same-origin frame drawn in it+ // (content.js _visionFrameMap does the same for Firefox input): the+ // frame's position, borders and padding are taken off, and whatever scale+ // a CSS transform or zoom draws it at is undone, by comparing its drawn box+ // with the size of its own viewport. Null when the frame or an ancestor is+ // rotated, skewed, flipped, set on a motion path or drawn in 3D, which no+ // scale maps.+ const frameMap = (frame, innerView) => {+ const view = frame.ownerDocument && frame.ownerDocument.defaultView;+ if (!view) return null;+ const length = value => Number.parseFloat(value) || 0;+ for (let node = frame, depth = 0; node && node.nodeType === 1 && depth < 256; depth++) {+ const style = view.getComputedStyle(node);+ const transform = String(style.transform || 'none');+ if (transform !== 'none') {+ const matrix = /^matrix\(([^)]*)\)$/.exec(transform);+ const parts = matrix ? matrix[1].split(',').map(Number) : [];+ if (!matrix || Math.abs(parts[1]) > 1e-6 || Math.abs(parts[2]) > 1e-6 || !(parts[0] > 0) || !(parts[3] > 0)) return null;+ }+ const rotate = String(style.rotate || 'none');+ if (rotate !== 'none' && !/^0(deg|rad|grad|turn)?$/.test(rotate)) return null;+ const scale = String(style.scale || 'none');+ if (scale !== 'none' && scale.split(/\s+/).some(part => !(length(part) > 0))) return null;+ // A motion path turns the element along it.+ if (String(style.offsetPath || 'none') !== 'none') return null;+ const root = node.getRootNode ? node.getRootNode() : null;+ node = node.parentElement || (root && root.nodeType === 11 ? root.host : null);+ }+ const rect = frame.getBoundingClientRect();+ const style = view.getComputedStyle(frame);+ const left = length(style.borderLeftWidth) + length(style.paddingLeft);+ const top = length(style.borderTopWidth) + length(style.paddingTop);+ const right = length(style.borderRightWidth) + length(style.paddingRight);+ const bottom = length(style.borderBottomWidth) + length(style.paddingBottom);+ const innerWidth = Number(innerView && innerView.innerWidth) || Math.max(0, frame.clientWidth - length(style.paddingLeft) - length(style.paddingRight));+ const innerHeight = Number(innerView && innerView.innerHeight) || Math.max(0, frame.clientHeight - length(style.paddingTop) - length(style.paddingBottom));+ const width = innerWidth + left + right;+ const height = innerHeight + top + bottom;+ // An unscaled frame's drawn box differs from that size by rounding only.+ const sx = width > 0 && Math.abs(rect.width - width) > 1 ? rect.width / width : 1;+ const sy = height > 0 && Math.abs(rect.height - height) > 1 ? rect.height / height : 1;+ return (px, py) => ({ x: (px - rect.left) / sx - left, y: (py - rect.top) / sy - top });+ };const hit = (x, y) => {let doc = document;let cx = x;@@ -68,14 +111,16 @@ export function visionDomProbe(request) {if (tag === 'IFRAME' || tag === 'FRAME' || tag === 'OBJECT' || tag === 'EMBED') {let inner = null;try { inner = el.contentDocument; } catch (e) { inner = null; }- if (inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME')) {- const rect = el.getBoundingClientRect();- cx -= rect.left + (el.clientLeft || 0);- cy -= rect.top + (el.clientTop || 0);+ const map = inner && inner.documentElement && (tag === 'IFRAME' || tag === 'FRAME') ? frameMap(el, inner.defaultView) : null;+ if (map) {+ const point = map(cx, cy);+ cx = point.x;+ cy = point.y;doc = inner;continue;}- // A cross-origin frame (or plugin): its content is not inspectable.+ // A cross-origin frame (or plugin), or one drawn at an angle no+ // scale maps: what is under the point inside it cannot be read.return { el, opaque: true };}return { el, opaque: false };diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 17f5e3bce..aa1e56453 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -1994,11 +1994,18 @@// coordinates, so canvases, sliders and maps see where the model pointed;// the default action of a press (focus) and of a click (activation) is// reproduced the way a browser does it. Same-origin iframes are entered.+ //+ // What is under a top-level viewport point: the element, its document and+ // window, the point in that document's client coordinates (cx, cy), and+ // toDoc(x, y), which maps any top-level point into that document the same+ // way. A point on a same-origin frame that no scale maps (_visionFrameMap)+ // gives { el: the frame, unmappable: true }.function _visionTargetAt(x, y) {let doc = document;let view = window;let cx = x;let cy = y;+ const maps = [];for (let depth = 0; depth < 6; depth++) {let el = doc.elementFromPoint(cx, cy);const seen = new Set();@@ -2014,19 +2021,58 @@let inner = null;try { inner = el.contentDocument; } catch { inner = null; }if (inner?.documentElement) {- const rect = el.getBoundingClientRect();- cx -= rect.left + (el.clientLeft || 0);- cy -= rect.top + (el.clientTop || 0);+ const map = _visionFrameMap(el, view, inner.defaultView);+ if (!map) return { el, doc, view, cx, cy, unmappable: true };+ ({ x: cx, y: cy } = map(cx, cy));+ maps.push(map);doc = inner;view = inner.defaultView || view;continue;}}- return { el, doc, view, cx, cy };+ const toDoc = (px, py) => maps.reduce((point, map) => map(point.x, point.y), { x: px, y: py });+ return { el, doc, view, cx, cy, toDoc };}return null;}+ // How a point of a document maps into a same-origin frame drawn in it:+ // the frame's position, borders and padding are taken off, and whatever+ // scale a CSS transform or zoom draws it at (on itself or an ancestor) is+ // undone, by comparing its drawn box with the size of its own viewport.+ // Null when the frame or an ancestor is rotated, skewed, flipped, set on+ // a motion path or drawn in 3D, which no scale maps.+ function _visionFrameMap(frame, view, innerView) {+ const length = value => Number.parseFloat(value) || 0;+ for (let node = frame, depth = 0; node?.nodeType === 1 && depth < 256; node = node.parentElement || node.getRootNode?.()?.host, depth++) {+ const style = view.getComputedStyle(node);+ const transform = String(style.transform || 'none');+ if (transform !== 'none') {+ const matrix = /^matrix\(([^)]*)\)$/.exec(transform);+ const [a, b, c, d] = matrix ? matrix[1].split(',').map(Number) : [];+ if (!matrix || Math.abs(b) > 1e-6 || Math.abs(c) > 1e-6 || !(a > 0) || !(d > 0)) return null;+ }+ const rotate = String(style.rotate || 'none');+ if (rotate !== 'none' && !/^0(deg|rad|grad|turn)?$/.test(rotate)) return null;+ const scale = String(style.scale || 'none');+ if (scale !== 'none' && scale.split(/\s+/).some(part => !(length(part) > 0))) return null;+ // A motion path turns the element along it.+ if (String(style.offsetPath || 'none') !== 'none') return null;+ }+ const rect = frame.getBoundingClientRect();+ const style = view.getComputedStyle(frame);+ const left = length(style.borderLeftWidth) + length(style.paddingLeft);+ const top = length(style.borderTopWidth) + length(style.paddingTop);+ const right = length(style.borderRightWidth) + length(style.paddingRight);+ const bottom = length(style.borderBottomWidth) + length(style.paddingBottom);+ const width = (Number(innerView?.innerWidth) || Math.max(0, frame.clientWidth - length(style.paddingLeft) - length(style.paddingRight))) + left + right;+ const height = (Number(innerView?.innerHeight) || Math.max(0, frame.clientHeight - length(style.paddingTop) - length(style.paddingBottom))) + top + bottom;+ // An unscaled frame's drawn box differs from that size by rounding only.+ const sx = width > 0 && Math.abs(rect.width - width) > 1 ? rect.width / width : 1;+ const sy = height > 0 && Math.abs(rect.height - height) > 1 ? rect.height / height : 1;+ return (x, y) => ({ x: (x - rect.left) / sx - left, y: (y - rect.top) / sy - top });+ }+function _visionFocusable(el) {for (let node = el; node && node.nodeType === 1; node = node.parentElement || node.getRootNode?.()?.host) {if (node.matches?.('input:not([disabled]),textarea:not([disabled]),select:not([disabled]),button:not([disabled]),a[href],[tabindex],[contenteditable=""],[contenteditable="true"],summary')) return node;@@ -2411,6 +2457,12 @@}const hit = _visionTargetAt(x, y);if (!hit) return { success: false, dispatched: false, noDispatch: true, error: 'Nothing is rendered at that point.' };+ if (hit.unmappable) {+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'frame_unmappable',+ error: 'Not dispatched: the point is inside an embedded frame drawn rotated, skewed, flipped or in 3D, so it cannot be placed on the frame\'s content. Use a control outside that frame.',+ };+ }if (actionDeadlineExpired()) return { success: false, dispatched: false, noDispatch: true, deadlineExpired: true, error: 'The action deadline expired before dispatch.' };const { el, view, cx, cy } = hit;const button = params.button === 'right' ? 2 : params.button === 'middle' ? 1 : 0;@@ -2525,29 +2577,35 @@// gets drag events instead of pointer and mouse events.const html = pressAllowed && !range ? _visionHtmlDrag(el) : null;let htmlStarted = false;- const pressOffsetX = x - cx;- const pressOffsetY = y - cy;range?.to(cx, cy);if (selection && !selection.begin(cx, cy)) selection = null;// Every point, the release included, is given in the coordinates of- // the document under it (a same-origin frame has its own), so the- // events a slider or canvas reads all agree. A point with nothing- // under it keeps the offset of the last frame that had something.- let offsetX = x - cx;- let offsetY = y - cy;+ // the document under it (a same-origin frame has its own, scaled as+ // the frame is drawn), so the events a slider or canvas reads all+ // agree. A point with nothing under it, or on a frame no scale maps,+ // is mapped into the last document that had something.+ let lastToDoc = hit.toDoc;const pointAt = (at, px, py) => {- if (at) { offsetX = px - at.cx; offsetY = py - at.cy; }- return { clientX: px - offsetX, clientY: py - offsetY, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };+ if (at) lastToDoc = at.toDoc;+ const local = at ? { x: at.cx, y: at.cy } : lastToDoc(px, py);+ return { clientX: local.x, clientY: local.y, screenX: px + (window.screenX || 0), screenY: py + (window.screenY || 0) };+ };+ const mappedAt = (px, py) => {+ const at = _visionTargetAt(px, py);+ return at && !at.unmappable ? at : null;};// An element holding the pointer's capture gets the events wherever// the pointer is, in its own document's coordinates: the press's,// whose window alone answered the capture call.- const heldAt = (point, px, py) => ({ ...point, clientX: px - pressOffsetX, clientY: py - pressOffsetY });+ const heldAt = (point, px, py) => {+ const local = hit.toDoc(px, py);+ return { ...point, clientX: local.x, clientY: local.y };+ };let last = el;for (let i = 1; i <= steps; i++) {const px = x + (toX - x) * (i / steps);const py = y + (toY - y) * (i / steps);- const at = _visionTargetAt(px, py);+ const at = mappedAt(px, py);last = at?.el || last;const point = pointAt(at, px, py);if (html && i === 1) {@@ -2562,10 +2620,11 @@if (htmlStarted) { html.move(last, point); continue; }const holder = capture.target({ button: 0, buttons: 1, ...heldAt(point, px, py) });moveOnto(holder || last, { button: 0, buttons: 1, ...(holder ? heldAt(point, px, py) : point) });- range?.to(px - pressOffsetX, py - pressOffsetY);- selection?.to(px - pressOffsetX, py - pressOffsetY);+ const local = hit.toDoc(px, py);+ range?.to(local.x, local.y);+ selection?.to(local.x, local.y);}- const end = _visionTargetAt(toX, toY);+ const end = mappedAt(toX, toY);const endPoint = pointAt(end, toX, toY);const holder = htmlStarted ? null : capture.target({ button: 0, buttons: 1, ...heldAt(endPoint, toX, toY) });const releaseTarget = holder || end?.el || last;diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 2cb8f9c48..616ab619b 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -864,6 +864,105 @@ test('the DOM probe behind the dispatch checks reads controls, text fields, focu}});+test('vision input (Firefox path) and the DOM probe place points inside scaled same-origin frames', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ // Each frame holds a button at its own (200,100)-(300,140) and a track at+ // (0,180)-(400,220) whose document logs the press, the moves and the release.+ const inner = (left = 200, top = 100) => `<style>body{margin:0}#b{position:absolute;left:${left}px;top:${top}px;width:100px;height:40px}+ #t{position:absolute;left:0;top:180px;width:400px;height:40px;background:#ddd}</style><button id="b">Go</button><div id="t"></div>+ <script>window.log=[];let down=false;document.getElementById('b').addEventListener('click',e=>log.push(['click',e.clientX,e.clientY]));+ document.getElementById('t').addEventListener('mousedown',e=>{down=true;log.push(['down',e.clientX,e.clientY])});+ document.addEventListener('mousemove',e=>{if(down)log.push(['move',e.clientX,e.clientY])});+ document.addEventListener('mouseup',e=>{if(down)log.push(['up',e.clientX,e.clientY]);down=false});<\/script>`;+ const frame = (id, style, html = inner()) => `<iframe id="${id}" style="${style}" srcdoc="${html.replace(/"/g, '"')}"></iframe>`;+ await page.setContent(`<!doctype html><style>body{margin:0}iframe{position:absolute;width:400px;height:300px;border:3px solid #000}</style>+ ${frame('scaled', 'left:100px;top:50px;transform:scale(0.5);transform-origin:0 0')}+ ${frame('tilted', 'left:600px;top:50px;transform:rotate(10deg)')}+ <div style="position:absolute;left:100px;top:400px;transform:scale(1.5);transform-origin:0 0">${frame('nested', 'position:static;width:200px;height:100px;border:1px solid #000', inner(20, 10))}</div>`);+ await page.waitForFunction(() => [...document.querySelectorAll('iframe')].every(item => item.contentWindow?.log));+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ await page.evaluate(({ code, guardBlock }) => {+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)();+ }, { code: block, guardBlock });+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const take = id => page.evaluate(id => document.getElementById(id).contentWindow.log.splice(0), id);+ // The scaled frame's (x, y) is drawn at its box (100, 50) plus its 3 px border, at half size.+ const scaled = (x, y) => ({ x: 100 + (3 + x) * 0.5, y: 50 + (3 + y) * 0.5 });+ let result = await run({ type: 'click', ...scaled(250, 120) });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.deepEqual(await take('scaled'), [['click', 250, 120]]);+ // An ancestor's scale counts too: the nested frame is drawn at 1.5x inside its 1 px border.+ result = await run({ type: 'click', x: 100 + (1 + 70) * 1.5, y: 400 + (1 + 30) * 1.5 });+ assert.equal(result.success, true, JSON.stringify(result));+ assert.deepEqual(await take('nested'), [['click', 70, 30]]);+ // A drag gives every point in the scaled frame's own coordinates.+ const from = scaled(50, 200);+ const to = scaled(350, 200);+ result = await run({ type: 'drag', x: from.x, y: from.y, toX: to.x, toY: to.y, steps: 6 });+ assert.equal(result.success, true, JSON.stringify(result));+ const drag = await take('scaled');+ assert.deepEqual(drag[0], ['down', 50, 200], JSON.stringify(drag));+ assert.deepEqual(drag.filter(entry => entry[0] === 'move').map(entry => entry[1]), [100, 150, 200, 250, 300, 350], JSON.stringify(drag));+ assert.deepEqual(drag.at(-1), ['up', 350, 200], JSON.stringify(drag));+ // No scale maps a rotated frame: the click is refused, nothing is sent.+ const rect = await page.evaluate(() => { const box = document.getElementById('tilted').getBoundingClientRect(); return { x: box.left + box.width / 2, y: box.top + box.height / 2 }; });+ result = await run({ type: 'click', ...rect });+ assert.equal(result.visionInputRefused, 'frame_unmappable', JSON.stringify(result));+ assert.equal(result.noDispatch, true);+ assert.deepEqual(await take('tilted'), []);+ // The DOM probe behind the dispatch checks reads the same elements, in both builds.+ for (const build of ['chrome', 'firefox']) {+ const { visionDomProbeExpression } = await import(`../src/${build}/src/agent/vision-dom-probe.js`);+ const probe = async point => (await page.evaluate(visionDomProbeExpression({ points: [point] }))).points[0];+ assert.deepEqual(await probe(scaled(250, 120)), { found: true, tag: 'button', clickable: true, opaqueFrame: false, control: 'button' }, build);+ assert.equal((await probe(scaled(250, 200))).tag, 'div', build);+ assert.equal((await probe(rect)).opaqueFrame, true, `${build}: a rotated frame cannot be read`);+ }+ } finally {+ await browser.close();+ }+});++test('the click submit probe (Chrome) reads a point inside a scaled, bordered child frame', async () => {+ const { Agent } = await import('../src/chrome/src/agent/agent.js');+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ const inner = `<style>body{margin:0}button{position:absolute;left:200px;top:100px;width:100px;height:40px}</style>+ <form action="/pay" method="post"><input name="amount" value="5"><button type="submit">Pay now</button></form>`;+ await page.setContent(`<!doctype html><style>body{margin:0}</style>+ <iframe style="position:absolute;left:100px;top:50px;width:400px;height:300px;border:3px solid #000;transform:scale(0.5);transform-origin:0 0"+ srcdoc="${inner.replace(/"/g, '"')}"></iframe>`);+ await page.waitForFunction(() => document.querySelector('iframe').contentDocument?.querySelector('button'));+ // The frame rectangles come from the agent's own top-frame script.+ const agent = new Agent({ getVisionProvider: async () => null });+ const realChrome = globalThis.chrome;+ globalThis.chrome = { scripting: { executeScript: async ({ func, args }) => [{ result: await page.evaluate(({ source, args }) => new Function(`return (${source})`)()(...args), { source: func.toString(), args }) }] } };+ let rects;+ // The button's centre, (250, 120) in the frame, is drawn at (100 + (3 + 250) / 2, 50 + (3 + 120) / 2).+ const point = { x: 226.5, y: 111.5 };+ try { rects = await agent._iframeRectsForCoordinate(1, point.x, point.y); } finally { globalThis.chrome = realChrome; }+ assert.equal(rects.length, 1, JSON.stringify(rects));+ assert.deepEqual({ scaleX: rects[0].scaleX, scaleY: rects[0].scaleY, insetLeft: rects[0].insetLeft, insetTop: rects[0].insetTop }, { scaleX: 0.5, scaleY: 0.5, insetLeft: 3, insetTop: 3 });+ const child = page.frames().find(item => item !== page.mainFrame());+ const result = await child.evaluate(({ source, args }) => new Function(`return (${source})`)()('click', args), {+ source: Agent._submitActionProbe.toString(),+ args: { ...point, coordinate_space: 'css', __wbTopLevelCoordinateFrames: rects, __wbCaptionPass: 'final', __wbCaptionReadings: {} },+ });+ assert.equal(result?.isSubmit, true, JSON.stringify(result)?.slice(0, 400));+ } finally {+ await browser.close();+ }+});+test('the click submit probe reads a vision point inside open shadow roots and on plain content (both builds)', async () => {const page_ = `<!doctype html><meta charset="utf-8"><title>Probe fixture</title><style>body{margin:0;font:16px sans-serif}#post{position:absolute;left:20px;top:20px;width:400px;height:120px;cursor:pointer}