Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: the Firefox vision pointer moves off one element onto the next

Emre Sokulluee81b3ab1d87
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 9d2e584c4..91787d115 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; a wheel moves the pointer to its point first), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:
@@ -152,7 +152,7 @@ npm run test:vision-mode
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 373165825..3a98d658e 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -1919,6 +1919,18 @@
return [index > 0 ? value.lastIndexOf('\n', index - 1) + 1 : 0, end < 0 ? value.length : end];
}
+ // What the vision pointer was last over, so the next input moves off it
+ // the way a mouse does (out and leave events) instead of appearing on the
+ // new target: a hover menu or tooltip that closes on mouseleave closes.
+ let _visionPointerOver = null;
+
+ // An element and its ancestors, through shadow roots to their hosts.
+ function _visionPointerChain(node) {
+ const chain = [];
+ for (let n = node, depth = 0; n?.nodeType === 1 && depth < 256; n = n.parentElement || n.getRootNode?.()?.host, depth++) chain.push(n);
+ return chain;
+ }
+
function visionInput(params = {}, actionDeadlineExpired = () => false) {
const type = String(params.type || 'click');
const x = Number(params.x);
@@ -1941,11 +1953,36 @@
const PointerCtor = view.PointerEvent || view.MouseEvent;
const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...extra })));
const mouse = (target, name, extra = {}) => target.dispatchEvent(new view.MouseEvent(name, init(extra)));
- const hover = target => {
- pointer(target, 'pointerover'); pointer(target, 'pointerenter', { bubbles: false });
- mouse(target, 'mouseover'); mouse(target, 'mouseenter', { bubbles: false });
- pointer(target, 'pointermove'); mouse(target, 'mousemove');
+ // The pointer moving onto `target` (with `extra` init, e.g. a drag's
+ // point and buttons): out and leave on what it left, over and enter on
+ // what it entered (shared ancestors get neither), pointer events before
+ // mouse events, then the move. Arriving with no known previous target,
+ // or again on the same one (`arrive`), over and enter go to the target.
+ const moveOnto = (target, extra = {}, { arrive = false } = {}) => {
+ const last = _visionPointerOver;
+ if (last !== target || arrive) {
+ const previous = last && last !== target && last.isConnected ? last : null;
+ const related = (a, b) => (a && b && a.ownerDocument === b.ownerDocument ? a : null);
+ const targetChain = _visionPointerChain(target);
+ const previousChain = previous ? _visionPointerChain(previous) : [];
+ const kept = new Set(targetChain);
+ const leaving = [];
+ for (const node of previousChain) { if (kept.has(node)) break; leaving.push(node); }
+ const left = new Set(previousChain);
+ const entering = previous ? targetChain.filter(node => !left.has(node)).reverse() : [target];
+ for (const [send, prefix] of [[pointer, 'pointer'], [mouse, 'mouse']]) {
+ if (previous) {
+ send(previous, `${prefix}out`, { ...extra, relatedTarget: related(target, previous) });
+ for (const node of leaving) send(node, `${prefix}leave`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(target, node) });
+ }
+ send(target, `${prefix}over`, { ...extra, relatedTarget: related(previous, target) });
+ for (const node of entering) send(node, `${prefix}enter`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(previous, node) });
+ }
+ }
+ pointer(target, 'pointermove', extra); mouse(target, 'mousemove', extra);
+ _visionPointerOver = target;
};
+ const hover = target => moveOnto(target, {}, { arrive: true });
const tag = String(el.tagName || '').toLowerCase();
const role = el.getAttribute?.('role') || '';
const describe = { tag, ...(role ? { role: role.slice(0, 32) } : {}) };
@@ -1969,6 +2006,8 @@
if (type === 'wheel') {
const deltaX = Number(params.deltaX) || 0;
const deltaY = Number(params.deltaY) || 0;
+ // A wheel turns where the mouse is: it moves there first.
+ hover(el);
const notCancelled = el.dispatchEvent(new view.WheelEvent('wheel', init({ deltaX, deltaY, deltaMode: 0 })));
let moved = false;
if (notCancelled) {
@@ -2024,17 +2063,16 @@
const py = y + (toY - y) * (i / steps);
const at = _visionTargetAt(px, py);
last = at?.el || last;
- const moveInit = { button: 0, buttons: 1, ...pointAt(at, px, py) };
- last.dispatchEvent(new PointerCtor('pointermove', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...moveInit })));
- last.dispatchEvent(new view.MouseEvent('mousemove', init(moveInit)));
+ moveOnto(last, { button: 0, buttons: 1, ...pointAt(at, px, py) });
range?.to(px - pressOffsetX, py - pressOffsetY);
selection?.to(px - pressOffsetX, py - pressOffsetY);
}
const end = _visionTargetAt(toX, toY);
const releaseTarget = end?.el || last;
const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };
- releaseTarget.dispatchEvent(new PointerCtor('pointerup', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...endInit })));
- releaseTarget.dispatchEvent(new view.MouseEvent('mouseup', init({ ...endInit, detail: 1 })));
+ if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });
+ pointer(releaseTarget, 'pointerup', endInit);
+ mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
const rangeValue = range?.release();
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 8a6a3eb68..70ade3b17 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2213,6 +2213,18 @@
return [index > 0 ? value.lastIndexOf('\n', index - 1) + 1 : 0, end < 0 ? value.length : end];
}
+ // What the vision pointer was last over, so the next input moves off it
+ // the way a mouse does (out and leave events) instead of appearing on the
+ // new target: a hover menu or tooltip that closes on mouseleave closes.
+ let _visionPointerOver = null;
+
+ // An element and its ancestors, through shadow roots to their hosts.
+ function _visionPointerChain(node) {
+ const chain = [];
+ for (let n = node, depth = 0; n?.nodeType === 1 && depth < 256; n = n.parentElement || n.getRootNode?.()?.host, depth++) chain.push(n);
+ return chain;
+ }
+
function visionInput(params = {}, actionDeadlineExpired = () => false) {
const type = String(params.type || 'click');
const x = Number(params.x);
@@ -2235,11 +2247,36 @@
const PointerCtor = view.PointerEvent || view.MouseEvent;
const pointer = (target, name, extra = {}) => target.dispatchEvent(new PointerCtor(name, init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...extra })));
const mouse = (target, name, extra = {}) => target.dispatchEvent(new view.MouseEvent(name, init(extra)));
- const hover = target => {
- pointer(target, 'pointerover'); pointer(target, 'pointerenter', { bubbles: false });
- mouse(target, 'mouseover'); mouse(target, 'mouseenter', { bubbles: false });
- pointer(target, 'pointermove'); mouse(target, 'mousemove');
+ // The pointer moving onto `target` (with `extra` init, e.g. a drag's
+ // point and buttons): out and leave on what it left, over and enter on
+ // what it entered (shared ancestors get neither), pointer events before
+ // mouse events, then the move. Arriving with no known previous target,
+ // or again on the same one (`arrive`), over and enter go to the target.
+ const moveOnto = (target, extra = {}, { arrive = false } = {}) => {
+ const last = _visionPointerOver;
+ if (last !== target || arrive) {
+ const previous = last && last !== target && last.isConnected ? last : null;
+ const related = (a, b) => (a && b && a.ownerDocument === b.ownerDocument ? a : null);
+ const targetChain = _visionPointerChain(target);
+ const previousChain = previous ? _visionPointerChain(previous) : [];
+ const kept = new Set(targetChain);
+ const leaving = [];
+ for (const node of previousChain) { if (kept.has(node)) break; leaving.push(node); }
+ const left = new Set(previousChain);
+ const entering = previous ? targetChain.filter(node => !left.has(node)).reverse() : [target];
+ for (const [send, prefix] of [[pointer, 'pointer'], [mouse, 'mouse']]) {
+ if (previous) {
+ send(previous, `${prefix}out`, { ...extra, relatedTarget: related(target, previous) });
+ for (const node of leaving) send(node, `${prefix}leave`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(target, node) });
+ }
+ send(target, `${prefix}over`, { ...extra, relatedTarget: related(previous, target) });
+ for (const node of entering) send(node, `${prefix}enter`, { ...extra, bubbles: false, cancelable: false, relatedTarget: related(previous, node) });
+ }
+ }
+ pointer(target, 'pointermove', extra); mouse(target, 'mousemove', extra);
+ _visionPointerOver = target;
};
+ const hover = target => moveOnto(target, {}, { arrive: true });
const tag = String(el.tagName || '').toLowerCase();
const role = el.getAttribute?.('role') || '';
const describe = { tag, ...(role ? { role: role.slice(0, 32) } : {}) };
@@ -2263,6 +2300,8 @@
if (type === 'wheel') {
const deltaX = Number(params.deltaX) || 0;
const deltaY = Number(params.deltaY) || 0;
+ // A wheel turns where the mouse is: it moves there first.
+ hover(el);
const notCancelled = el.dispatchEvent(new view.WheelEvent('wheel', init({ deltaX, deltaY, deltaMode: 0 })));
let moved = false;
if (notCancelled) {
@@ -2318,17 +2357,16 @@
const py = y + (toY - y) * (i / steps);
const at = _visionTargetAt(px, py);
last = at?.el || last;
- const moveInit = { button: 0, buttons: 1, ...pointAt(at, px, py) };
- last.dispatchEvent(new PointerCtor('pointermove', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...moveInit })));
- last.dispatchEvent(new view.MouseEvent('mousemove', init(moveInit)));
+ moveOnto(last, { button: 0, buttons: 1, ...pointAt(at, px, py) });
range?.to(px - pressOffsetX, py - pressOffsetY);
selection?.to(px - pressOffsetX, py - pressOffsetY);
}
const end = _visionTargetAt(toX, toY);
const releaseTarget = end?.el || last;
const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };
- releaseTarget.dispatchEvent(new PointerCtor('pointerup', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...endInit })));
- releaseTarget.dispatchEvent(new view.MouseEvent('mouseup', init({ ...endInit, detail: 1 })));
+ if (releaseTarget !== _visionPointerOver) moveOnto(releaseTarget, { ...endInit, buttons: 1 });
+ pointer(releaseTarget, 'pointerup', endInit);
+ mouse(releaseTarget, 'mouseup', { ...endInit, detail: 1 });
const rangeValue = range?.release();
return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 16d163c0e..bcda0775e 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -455,6 +455,51 @@ third</textarea>
}
});
+test('content-script vision input (Firefox path) moves the pointer off one element onto the next like a mouse', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ const box = (id, left, top, inner = '') => `<div id="${id}" style="position:absolute;left:${left}px;top:${top}px;width:200px;height:100px;background:#eee">${inner}</div>`;
+ // #menu holds #item (at its top left); #drop is a drop zone right of #source.
+ await page.setContent(`<!doctype html><style>body{margin:0}#item{position:absolute;left:10px;top:10px;width:80px;height:30px;background:#ccc}</style>
+ ${box('menu', 100, 100, '<div id="item"></div>')}${box('other', 400, 100)}${box('drop', 300, 300)}${box('source', 100, 300)}
+ <script>window.log=[];for(const id of ['menu','item','other','drop','source'])for(const t of ['mouseover','mouseout','mouseenter','mouseleave','pointerleave'])
+ document.getElementById(id).addEventListener(t,e=>{if(e.target===e.currentTarget)log.push([id,t,e.relatedTarget?.id||null,e.buttons])});</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ // Mouse events only (pointer events come first, in the same shape).
+ const take = () => page.evaluate(() => window.log.splice(0).filter(entry => entry[1].startsWith('mouse')).map(entry => entry.slice(0, 3)));
+ await run({ type: 'hover', x: 130, y: 120 });
+ assert.deepEqual(await take(), [['item', 'mouseover', null], ['item', 'mouseenter', null]], 'the first hover arrives on the item');
+ // Item -> its menu: out of the item, over the menu; no leave or enter on the menu.
+ await run({ type: 'hover', x: 250, y: 180 });
+ assert.deepEqual(await take(), [['item', 'mouseout', 'menu'], ['item', 'mouseleave', 'menu'], ['menu', 'mouseover', 'item']]);
+ // Menu -> another element: the menu is left (a hover menu that closes on mouseleave closes).
+ await run({ type: 'hover', x: 500, y: 150 });
+ const away = await take();
+ assert.deepEqual(away.filter(entry => entry[0] === 'menu'), [['menu', 'mouseout', 'other'], ['menu', 'mouseleave', 'other']]);
+ assert.deepEqual(away.filter(entry => entry[0] === 'other'), [['other', 'mouseover', 'menu'], ['other', 'mouseenter', 'menu']]);
+ // A click elsewhere leaves it too, and a drag crosses into the drop zone pressed.
+ await run({ type: 'click', x: 200, y: 350 });
+ assert.ok((await take()).some(entry => entry[0] === 'other' && entry[1] === 'mouseleave'));
+ await run({ type: 'drag', x: 200, y: 350, toX: 450, toY: 350, steps: 6 });
+ const drag = await page.evaluate(() => window.log.splice(0));
+ assert.ok(drag.some(entry => entry[0] === 'source' && entry[1] === 'mouseleave' && entry[2] === 'drop' && entry[3] === 1), JSON.stringify(drag));
+ assert.ok(drag.some(entry => entry[0] === 'drop' && entry[1] === 'mouseover' && entry[2] === 'source' && entry[3] === 1), JSON.stringify(drag));
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision clicks and drags (Firefox path) leave disabled controls alone', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {