Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Merge branch 'vision-mode' into vision-bidi-input

Emre Sokullu14ff20886fe2
diff --cc docs/vision-mode.md
index 696590c2b,3352fa3e0..c234627d7
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@@ -119,8 -119,7 +119,8 @@@ With an instant model the MV3 fixture'
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.
- - **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
+ - **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A press whose `pointerdown` the page cancels gets what a browser gives it: no compatibility `mousedown`, `mousemove` or `mouseup` until its release (the boundary events still go out) and none of the press's defaults below (focus, the caret, a range thumb, a selection, drag and drop, a double click's word), while its `click` still fires. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
+- **Firefox with trusted automation**: when **Firefox trusted automation** is on (Settings → Advanced, the local companion in `firefox-companion/`) and its companion reports point input (`point_input`), a left click and a hover go out as trusted input at the point (`bidi/client.js` `visionInput`). The content script runs every check it runs for the synthetic sequence (the viewport, a frame it refuses, a disabled control), then marks the element the top document shows at the point, the one the page monitor prepared there (`_visionBidiPrepare`); the companion checks that the element is still at the point and moves the real pointer there (`hover_at`), then presses and releases its left button (`click_at`), so `:hover`, focus and pointer capture are the browser's own. A click still runs inside the file-chooser guard, settled as the synthetic click's is. The page monitor attributes it as it does the companion's other clicks (a one-time marker on that element). A double, triple, right or middle click, a drag and a wheel stay synthetic (a BiDi wheel scrolls only the window's selected tab, and a run's tab can be in the background), and so does a point over an embedded frame or inside an open shadow root (the monitor binds the shadow host there, but sees the events come from inside it, where it cannot tell them from a person's; the synthetic sequence reaches into both). A companion without point input leaves all vision input synthetic; one that disconnects mid-task stops its trusted input, as for its other actions.
In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.
diff --cc test/vision-pipeline-browser.mjs
index 55d036f96,0d5ce6124..ddf44ddc9
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@@ -226,79 -226,80 +226,153 @@@ test('content-script vision input (Fire
}
});
+ test('content-script vision input (Firefox path) honours a pointerdown the page cancels, as a browser does', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ await page.setContent(`<!doctype html><style>body{margin:0;font:20px monospace}
+ #range{position:absolute;left:20px;top:20px;width:300px;height:20px;margin:0}
+ #field{position:absolute;left:20px;top:80px;width:300px;height:30px}
+ #words{position:absolute;left:20px;top:150px;margin:0;white-space:nowrap}
+ #pad{position:absolute;left:20px;top:220px;width:400px;height:60px;background:#ddd}</style>
+ <input id="range" type="range" min="0" max="100" step="5" value="10"><input id="field" value="hello">
+ <p id="words">alpha bravo charlie</p><div id="pad"></div>
+ <script>window.log=[];window.cancel=new Set(['range','field','words','pad']);
+ document.addEventListener('pointerdown',e=>{if(cancel.has(e.target.id))e.preventDefault();},true);
+ for(const t of ['pointerdown','mousedown','pointermove','mousemove','mouseover','pointerup','mouseup','click','dblclick','input','change'])
+ document.addEventListener(t,e=>{if(e.target.id)log.push(t+':'+e.target.id)},true);</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ const take = () => page.evaluate(() => window.log.splice(0));
+ const bravo = await page.evaluate(() => { const text = document.getElementById('words').firstChild; const r = document.createRange();
+ r.setStart(text, 6); r.setEnd(text, 11); const b = r.getBoundingClientRect(); return { x: (b.left + b.right) / 2, y: (b.top + b.bottom) / 2 }; });
+
+ // A click on a range track: no thumb moves and no compatibility mouse
+ // events, but the click still fires.
+ let result = await run({ type: 'click', x: 170, y: 30 });
+ assert.equal(result.success, true);
+ assert.equal(result.rangeValue, undefined);
+ assert.equal(await page.evaluate(() => document.getElementById('range').value), '10');
+ let log = await take();
+ assert.ok(log.includes('pointerdown:range') && log.includes('pointerup:range') && log.includes('click:range'), log.join());
+ assert.ok(!log.some(entry => /^(mousedown|mouseup|input|change):/.test(entry)), log.join());
+ // A click in a text field: no focus.
+ await run({ type: 'click', x: 100, y: 95 });
+ assert.notEqual(await page.evaluate(() => document.activeElement.id), 'field');
+ await take();
+ // A double click on a word: dblclick fires, but no word is selected.
+ await page.evaluate(() => getSelection().removeAllRanges());
+ result = await run({ type: 'click', x: bravo.x, y: bravo.y, clickCount: 2 });
+ log = await take();
+ assert.ok(log.includes('dblclick:words'), log.join());
+ assert.equal(await page.evaluate(() => String(getSelection())), '');
+ // A drag: pointer moves and the release go out, no mousemove or mouseup,
+ // the boundary mouseover still does, and no text is selected.
+ result = await run({ type: 'drag', x: 30, y: 250, toX: 380, toY: 250, steps: 4 });
+ log = await take();
+ assert.equal(result.success, true);
+ const pressed = log.slice(log.indexOf('pointerdown:pad'));
+ assert.ok(pressed.filter(entry => entry === 'pointermove:pad').length >= 4, log.join());
+ assert.ok(pressed.includes('pointerup:pad'), log.join());
+ assert.ok(!pressed.some(entry => /^(mousedown|mousemove|mouseup):pad$/.test(entry)), log.join());
+ // The same sequences on presses the page lets through are unchanged.
+ await page.evaluate(() => window.cancel.clear());
+ result = await run({ type: 'click', x: 170, y: 30 });
+ assert.equal(result.rangeValue, '50');
+ log = await take();
+ assert.ok(log.includes('mousedown:range') && log.includes('mouseup:range'), log.join());
+ result = await run({ type: 'drag', x: 30, y: 250, toX: 380, toY: 250, steps: 4 });
+ log = await take();
+ assert.ok(log.includes('mousemove:pad') && log.includes('mouseup:pad'), log.join());
+ await run({ type: 'click', x: bravo.x, y: bravo.y, clickCount: 2 });
+ assert.equal(await page.evaluate(() => String(getSelection())), 'bravo');
+ } finally {
+ await browser.close();
+ }
+ });
+
+test('content-script vision input (Firefox path) marks the element at the point for trusted input, after its checks', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ await page.setContent(`<!doctype html><style>body{margin:0}button,iframe,#host{position:absolute;margin:0;border:0}
+ #go{left:100px;top:100px;width:200px;height:60px}#off{left:400px;top:100px;width:200px;height:60px}
+ #frame{left:100px;top:300px;width:300px;height:150px}#host{left:500px;top:300px;width:200px;height:80px}</style>
+ <button id="go">Go</button><button id="off" disabled>Off</button>
+ <iframe id="frame" srcdoc="<button style='width:100%;height:100%'>Inside</button>"></iframe><div id="host"></div>
+ <script>window.log=[];for(const t of ['pointerover','pointerdown','mousedown','mouseup','click','wheel'])document.addEventListener(t,e=>log.push(t),true);
+ const root=document.getElementById('host').attachShadow({mode:'open'});root.innerHTML='<span id="inner" style="display:block;width:200px;height:80px">In</span>';</script>`);
+ await page.waitForFunction(() => document.getElementById('frame').contentDocument?.querySelector('button'));
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const token = '0b8d6f43-6c3a-4c43-9d0e-5b5b0f7f4c21';
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ const marked = () => page.evaluate(t => [...document.querySelectorAll(`[data-webbrain-bidi="${t}"]`)].map(el => el.id), token);
+ const clear = () => page.evaluate(() => { for (const el of document.querySelectorAll('[data-webbrain-bidi]')) el.removeAttribute('data-webbrain-bidi'); window.log.length = 0; });
+
+ // A click: the element at the point is marked, the guard armed, and nothing synthetic is sent.
+ let result = await run({ type: 'click', x: 150.6, y: 120.2, _bidiPrepare: token });
+ assert.equal(result.bidiPrepared, true);
+ assert.equal(result.noDispatch, true);
+ assert.deepEqual(result.point, { x: 151, y: 120 });
+ assert.deepEqual(result.target, { tag: 'button' });
+ assert.match(result._filePickerGuardId, /^fpg_/);
+ assert.equal(result.url, await page.evaluate(() => location.href));
+ assert.deepEqual(await marked(), ['go']);
+ assert.deepEqual(await page.evaluate(() => window.log), [], 'no synthetic event');
+ await clear();
+ // A hover is marked too, and arms no file-chooser guard.
+ result = await run({ type: 'hover', x: 150, y: 120, _bidiPrepare: token });
+ assert.equal(result.bidiPrepared, true);
+ assert.equal(result._filePickerGuardId, undefined);
+ await clear();
+ // A point inside an open shadow root stays with the synthetic sequence:
+ // the monitor binds the host, but the events come from inside it.
+ result = await run({ type: 'click', x: 600, y: 340, _bidiPrepare: token });
+ assert.equal(result.bidiUnavailable, true);
+ assert.deepEqual(await page.evaluate(() => document.querySelectorAll('[data-webbrain-bidi]').length
+ + document.getElementById('host').shadowRoot.querySelectorAll('[data-webbrain-bidi]').length), 0);
+ await clear();
+ // The synthetic sequence's checks come first: a disabled control is refused.
+ result = await run({ type: 'click', x: 500, y: 120, _bidiPrepare: token });
+ assert.equal(result.visionInputRefused, 'disabled_control');
+ assert.deepEqual(await marked(), []);
+ // A point over an embedded frame stays with the synthetic sequence.
+ result = await run({ type: 'click', x: 250, y: 375, _bidiPrepare: token });
+ assert.equal(result.bidiUnavailable, true);
+ assert.deepEqual(await marked(), []);
+ // A drag or a wheel is never prepared, and a malformed binding marks nothing.
+ for (const params of [{ type: 'drag', x: 150, y: 120, toX: 250, toY: 120 }, { type: 'wheel', x: 150, y: 120, deltaY: 100 }]) {
+ result = await run({ ...params, _bidiPrepare: token });
+ assert.equal(result.noDispatch, true, params.type);
+ assert.equal(result.bidiPrepared, undefined, params.type);
+ }
+ result = await run({ type: 'click', x: 150, y: 120, _bidiPrepare: 'not-a-token' });
+ assert.equal(result.noDispatch, true);
+ assert.deepEqual(await page.evaluate(() => document.querySelectorAll('[data-webbrain-bidi]').length), 0);
+ assert.deepEqual(await page.evaluate(() => window.log), [], 'still no synthetic event');
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision drag (Firefox path) moves and releases in a same-origin frame\'s own coordinates', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {