Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode (Firefox): a click on a native range track moves its thumb

Emre Sokullu8752d2cd0685
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 40790d5bd..8891c2874 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -119,7 +119,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (`_resourceMutationActionKey`), with the same record, stage, sibling and confirmation checks and the same card, and `_visionDispatchClick` reads the point again right before input goes out (`_visionMutationBindingBlock`): when the authorized control is no longer there, nothing is sent (`resource_changed_before_dispatch`) and the model chooses again on the updated screen. A guard's advice names what the run offers (`click_at`), never a ref or selector. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing `:hover`, so while the pointer rests on an element it and its ancestors carry an attribute, and every `:hover` in the selectors of the style rules the content script can read also accepts it, in place (`:hover` reads `:is(:hover, [data-webbrain-hover])`, keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless its `overscroll-behavior` keeps it, never a page locked with `overflow: hidden`, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (`frame_unmappable`). The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`), and on a `draggable="true"` item HTML5 drag and drop runs as the browser runs it (`dragstart` with a `DataTransfer` the page fills, `drag`/`dragenter`/`dragleave`/`dragover` per step, `drop` where the last `dragover` was accepted, `dragend`; no pointer or mouse events meanwhile). A press the page captures (`setPointerCapture` on its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates, `gotpointercapture` comes before them and `lostpointercapture` right after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (`opaque_frame`): its events would stop at the frame element and reach nothing inside it.
In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (`recipient_bound_click`) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.
@@ -170,7 +170,7 @@ npm run test:vision-mode
- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.
- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a selector listed beside a `:hover` keeping its own cascade, a later rule of equal specificity still winning, a rule inside `@media`, a shadow root's `:host(:hover)`, a rule added later) and ended by a person's own pointer with the page's selectors put back, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.
+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs (dragged, and clicked on the track), text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a selector listed beside a `:hover` keeping its own cascade, a later rule of equal specificity still winning, a rule inside `@media`, a shadow root's `:host(:hover)`, a rule added later) and ended by a person's own pointer with the page's selectors put back, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 3166a6895..f0eb39ba0 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -2472,22 +2472,30 @@
// Like any agent click, a vision click must not open a native file
// chooser (a file input, its label, an upload button that clicks one):
// the guard blocks it, and the agent settles the guard afterwards.
+ let rangeValue;
const filePickerGuard = clickWithoutNativeFilePicker(() => {
for (let count = 1; count <= clicks; count++) {
capture.press();
pointer(el, 'pointerdown', { button, buttons: buttonsMask });
const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ // A primary press on a native range input moves its thumb to the
+ // point, as a drag's does (_visionRangeDrag): `input` now, `change`
+ // on release.
+ let range = null;
if (count === 1 && button === 0 && pressAllowed) {
const focusable = _visionFocusable(el);
if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
else if (!focusable) el.ownerDocument.activeElement?.blur?.();
if (focusable) _visionPlaceCaret(el, cx, cy);
+ range = _visionRangeDrag(el);
+ range?.to(cx, cy);
}
// A press the page captured is released on the capturing element.
const holder = capture.target({ button, buttons: buttonsMask }) || el;
pointer(holder, 'pointerup', { button, buttons: 0 });
capture.release({ button, buttons: 0 });
mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
+ if (range) rangeValue = range.release();
if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
}
@@ -2519,6 +2527,7 @@
});
if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };
return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,
+ ...(rangeValue !== undefined ? { rangeValue } : {}),
...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };
};
try {
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index a98dc3db5..641ce6286 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2766,22 +2766,30 @@
// Like any agent click, a vision click must not open a native file
// chooser (a file input, its label, an upload button that clicks one):
// the guard blocks it, and the agent settles the guard afterwards.
+ let rangeValue;
const filePickerGuard = clickWithoutNativeFilePicker(() => {
for (let count = 1; count <= clicks; count++) {
capture.press();
pointer(el, 'pointerdown', { button, buttons: buttonsMask });
const pressAllowed = mouse(el, 'mousedown', { button, buttons: buttonsMask, detail: count });
+ // A primary press on a native range input moves its thumb to the
+ // point, as a drag's does (_visionRangeDrag): `input` now, `change`
+ // on release.
+ let range = null;
if (count === 1 && button === 0 && pressAllowed) {
const focusable = _visionFocusable(el);
if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
else if (!focusable) el.ownerDocument.activeElement?.blur?.();
if (focusable) _visionPlaceCaret(el, cx, cy);
+ range = _visionRangeDrag(el);
+ range?.to(cx, cy);
}
// A press the page captured is released on the capturing element.
const holder = capture.target({ button, buttons: buttonsMask }) || el;
pointer(holder, 'pointerup', { button, buttons: 0 });
capture.release({ button, buttons: 0 });
mouse(holder, 'mouseup', { button, buttons: 0, detail: count });
+ if (range) rangeValue = range.release();
if (button === 0) mouse(el, 'click', { button: 0, buttons: 0, detail: count });
else if (button === 1) mouse(el, 'auxclick', { button: 1, buttons: 0, detail: count });
}
@@ -2813,6 +2821,7 @@
});
if (filePickerGuard.blocked) return { ...filePickerBlockedResponse(filePickerGuard.blocked), target: describe };
return { success: true, dispatched: true, method: 'synthetic-vision-pointer', clicks, target: describe,
+ ...(rangeValue !== undefined ? { rangeValue } : {}),
...(filePickerGuard.guardId ? { _filePickerGuardId: filePickerGuard.guardId } : {}) };
};
try {
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 25045a230..4225b9d71 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -329,6 +329,42 @@ test('content-script vision drag (Firefox path) moves a native range input like
}
});
+test('content-script vision click (Firefox path) moves a native range input\'s thumb to the point', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ const range = (id, top, extra = '') => `<input id="${id}" type="range" ${extra} style="position:absolute;left:100px;top:${top}px;width:300px;height:20px;margin:0">`;
+ await page.setContent(`<!doctype html><style>body{margin:0}</style>
+ ${range('volume', 100, 'min="0" max="100" step="5" value="10"')}${range('held', 160, 'min="0" max="100" value="20"')}
+ <script>window.events=[];for(const id of ['volume','held'])for(const t of ['input','change'])document.getElementById(id).addEventListener(t,e=>events.push([id,t,e.target.value]));
+ document.getElementById('held').addEventListener('mousedown',e=>e.preventDefault());</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ // The middle of the track (the thumb's centre spans x 108-392): 50, an `input` then a `change`.
+ let result = await run({ type: 'click', x: 250, y: 110 });
+ assert.equal(result.rangeValue, '50', JSON.stringify(result));
+ assert.deepEqual(await page.evaluate(() => window.events.splice(0)), [['volume', 'input', '50'], ['volume', 'change', '50']]);
+ assert.equal(await page.evaluate(() => document.activeElement.id), 'volume');
+ // A press the page cancels moves nothing, and neither does a right click.
+ result = await run({ type: 'click', x: 300, y: 170 });
+ assert.equal(result.rangeValue, undefined, JSON.stringify(result));
+ result = await run({ type: 'click', x: 350, y: 110, button: 'right' });
+ assert.equal(result.rangeValue, undefined, JSON.stringify(result));
+ assert.deepEqual(await page.evaluate(() => [window.events.splice(0), document.getElementById('volume').value, document.getElementById('held').value]), [[], '50', '20']);
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision drag (Firefox path) selects text like a person\'s drag', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {