Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: a Firefox wheel goes where the browser would send it

Emre Sokullu2b415ee34283
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 91787d115..bddb332de 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:
@@ -152,7 +152,7 @@ npm run test:vision-mode
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 3a98d658e..fb3d13789 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -1740,12 +1740,46 @@
return null;
}
- function _visionScrollable(el, axis) {
- for (let node = el; node && node.nodeType === 1; node = node.parentElement || node.getRootNode?.()?.host) {
- const style = node.ownerDocument.defaultView.getComputedStyle(node);
- const overflow = axis === 'x' ? style.overflowX : style.overflowY;
- const room = axis === 'x' ? node.scrollWidth - node.clientWidth : node.scrollHeight - node.clientHeight;
- if (room > 1 && /(auto|scroll|overlay)/.test(overflow)) return node;
+ // Where a wheel delta along `axis` goes, routed as the browser routes it:
+ // the nearest scroll container under the point that can still move that
+ // way (an exhausted one passes the wheel on unless its overscroll-behavior
+ // keeps it), then the document unless its overflow locks it (a page
+ // behind a modal), then the page around a same-origin frame. Null when
+ // nothing would scroll. A programmatic scroll ignores all of this (it
+ // scrolls a locked page too), so it is checked here.
+ function _visionWheelTarget(el, axis, delta) {
+ const x = axis === 'x';
+ const canMove = node => {
+ const max = x ? node.scrollWidth - node.clientWidth : node.scrollHeight - node.clientHeight;
+ const at = Math.abs(x ? node.scrollLeft : node.scrollTop);
+ return max > 1 && (delta < 0 ? at > 0.5 : at < max - 0.5);
+ };
+ const contains = style => /contain|none/.test(String((x ? style.overscrollBehaviorX : style.overscrollBehaviorY) || ''));
+ let node = el;
+ for (let depth = 0; node && depth < 8; depth++) {
+ const doc = node.ownerDocument;
+ const view = doc?.defaultView;
+ if (!view) return null;
+ const root = doc.scrollingElement || doc.documentElement;
+ // The viewport's overflow is the root element's, or the body's when
+ // the root's is visible (the body is then no scroller of its own).
+ const htmlStyle = view.getComputedStyle(doc.documentElement);
+ const htmlOverflow = x ? htmlStyle.overflowX : htmlStyle.overflowY;
+ const bodyIsViewport = !!doc.body && htmlOverflow === 'visible';
+ for (let n = node; n && n.nodeType === 1 && n !== doc.documentElement; n = n.parentElement || n.getRootNode?.()?.host) {
+ if (n === doc.body && bodyIsViewport) break;
+ const style = view.getComputedStyle(n);
+ if (!/(auto|scroll|overlay)/.test(x ? style.overflowX : style.overflowY)) continue;
+ if (canMove(n)) return n;
+ if (contains(style)) return null;
+ }
+ const bodyStyle = bodyIsViewport ? view.getComputedStyle(doc.body) : null;
+ const viewportOverflow = bodyStyle ? (x ? bodyStyle.overflowX : bodyStyle.overflowY) : htmlOverflow;
+ if (!/hidden|clip/.test(viewportOverflow) && root && canMove(root)) return root;
+ if (contains(htmlStyle)) return null;
+ let frame = null;
+ try { frame = view.frameElement; } catch { frame = null; }
+ node = frame;
}
return null;
}
@@ -2013,9 +2047,8 @@
if (notCancelled) {
for (const [axis, delta] of [['x', deltaX], ['y', deltaY]]) {
if (!delta) continue;
- const scroller = _visionScrollable(el, axis);
- const scrollingElement = el.ownerDocument.scrollingElement || el.ownerDocument.documentElement;
- const target = scroller || scrollingElement;
+ const target = _visionWheelTarget(el, axis, delta);
+ if (!target) continue;
const before = axis === 'x' ? target.scrollLeft : target.scrollTop;
target.scrollBy({ left: axis === 'x' ? delta : 0, top: axis === 'y' ? delta : 0, behavior: 'instant' });
const after = axis === 'x' ? target.scrollLeft : target.scrollTop;
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 70ade3b17..b19390ee0 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2034,12 +2034,46 @@
return null;
}
- function _visionScrollable(el, axis) {
- for (let node = el; node && node.nodeType === 1; node = node.parentElement || node.getRootNode?.()?.host) {
- const style = node.ownerDocument.defaultView.getComputedStyle(node);
- const overflow = axis === 'x' ? style.overflowX : style.overflowY;
- const room = axis === 'x' ? node.scrollWidth - node.clientWidth : node.scrollHeight - node.clientHeight;
- if (room > 1 && /(auto|scroll|overlay)/.test(overflow)) return node;
+ // Where a wheel delta along `axis` goes, routed as the browser routes it:
+ // the nearest scroll container under the point that can still move that
+ // way (an exhausted one passes the wheel on unless its overscroll-behavior
+ // keeps it), then the document unless its overflow locks it (a page
+ // behind a modal), then the page around a same-origin frame. Null when
+ // nothing would scroll. A programmatic scroll ignores all of this (it
+ // scrolls a locked page too), so it is checked here.
+ function _visionWheelTarget(el, axis, delta) {
+ const x = axis === 'x';
+ const canMove = node => {
+ const max = x ? node.scrollWidth - node.clientWidth : node.scrollHeight - node.clientHeight;
+ const at = Math.abs(x ? node.scrollLeft : node.scrollTop);
+ return max > 1 && (delta < 0 ? at > 0.5 : at < max - 0.5);
+ };
+ const contains = style => /contain|none/.test(String((x ? style.overscrollBehaviorX : style.overscrollBehaviorY) || ''));
+ let node = el;
+ for (let depth = 0; node && depth < 8; depth++) {
+ const doc = node.ownerDocument;
+ const view = doc?.defaultView;
+ if (!view) return null;
+ const root = doc.scrollingElement || doc.documentElement;
+ // The viewport's overflow is the root element's, or the body's when
+ // the root's is visible (the body is then no scroller of its own).
+ const htmlStyle = view.getComputedStyle(doc.documentElement);
+ const htmlOverflow = x ? htmlStyle.overflowX : htmlStyle.overflowY;
+ const bodyIsViewport = !!doc.body && htmlOverflow === 'visible';
+ for (let n = node; n && n.nodeType === 1 && n !== doc.documentElement; n = n.parentElement || n.getRootNode?.()?.host) {
+ if (n === doc.body && bodyIsViewport) break;
+ const style = view.getComputedStyle(n);
+ if (!/(auto|scroll|overlay)/.test(x ? style.overflowX : style.overflowY)) continue;
+ if (canMove(n)) return n;
+ if (contains(style)) return null;
+ }
+ const bodyStyle = bodyIsViewport ? view.getComputedStyle(doc.body) : null;
+ const viewportOverflow = bodyStyle ? (x ? bodyStyle.overflowX : bodyStyle.overflowY) : htmlOverflow;
+ if (!/hidden|clip/.test(viewportOverflow) && root && canMove(root)) return root;
+ if (contains(htmlStyle)) return null;
+ let frame = null;
+ try { frame = view.frameElement; } catch { frame = null; }
+ node = frame;
}
return null;
}
@@ -2307,9 +2341,8 @@
if (notCancelled) {
for (const [axis, delta] of [['x', deltaX], ['y', deltaY]]) {
if (!delta) continue;
- const scroller = _visionScrollable(el, axis);
- const scrollingElement = el.ownerDocument.scrollingElement || el.ownerDocument.documentElement;
- const target = scroller || scrollingElement;
+ const target = _visionWheelTarget(el, axis, delta);
+ if (!target) continue;
const before = axis === 'x' ? target.scrollLeft : target.scrollTop;
target.scrollBy({ left: axis === 'x' ? delta : 0, top: axis === 'y' ? delta : 0, behavior: 'instant' });
const after = axis === 'x' ? target.scrollLeft : target.scrollTop;
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index bcda0775e..46054ee8a 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -500,6 +500,54 @@ test('content-script vision input (Firefox path) moves the pointer off one eleme
}
});
+test('content-script vision wheel (Firefox path) scrolls what the browser would', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ const pane = (id, left, extra = '') => `<div id="${id}" style="position:fixed;left:${left}px;top:100px;width:300px;height:200px;overflow:auto;${extra}"><div style="height:1000px"></div></div>`;
+ await page.setContent(`<!doctype html><style>body{margin:0}iframe{position:fixed;left:100px;top:400px;width:300px;height:150px;border:0}</style>
+ ${pane('pane', 100)}${pane('kept', 500, 'overscroll-behavior:contain')}<iframe srcdoc="<p>short</p>"></iframe><div style="height:3000px"></div>`);
+ await page.waitForFunction(() => document.querySelector('iframe').contentDocument?.body);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const wheel = (x, y) => page.evaluate(p => window.__visionInput(p), { type: 'wheel', x, y, deltaY: 300 });
+ const state = () => page.evaluate(() => [document.getElementById('pane').scrollTop, document.getElementById('kept').scrollTop, window.scrollY]);
+ const reset = (pane = 0, kept = 0) => page.evaluate(([pane, kept]) => {
+ document.getElementById('pane').scrollTop = pane; document.getElementById('kept').scrollTop = kept; window.scrollTo(0, 0);
+ }, [pane, kept]);
+ // A pane that can still move takes the wheel.
+ assert.equal((await wheel(250, 200)).moved, true);
+ assert.deepEqual(await state(), [300, 0, 0]);
+ // An exhausted pane passes it to the page; one with overscroll-behavior: contain keeps it.
+ await reset(800, 800);
+ assert.equal((await wheel(250, 200)).moved, true);
+ assert.deepEqual(await state(), [800, 800, 300]);
+ await reset(800, 800);
+ assert.equal((await wheel(650, 200)).moved, false);
+ assert.deepEqual(await state(), [800, 800, 0]);
+ // A same-origin frame that cannot scroll passes the wheel to the page around it.
+ await reset();
+ assert.equal((await wheel(250, 450)).moved, true);
+ assert.equal((await state())[2], 300);
+ // A page locked behind a modal (overflow: hidden) does not scroll; a pane on it still does.
+ await reset();
+ await page.evaluate(() => { document.documentElement.style.overflow = 'hidden'; });
+ assert.equal((await wheel(450, 650)).moved, false);
+ assert.equal((await wheel(250, 200)).moved, true);
+ assert.deepEqual(await state(), [300, 0, 0]);
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision clicks and drags (Firefox path) leave disabled controls alone', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {