Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: Firefox drags move native sliders and select text

Emre Sokullud856f7e999a5
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 1f00cf31a..70e035fde 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:
- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.
-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path. A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.
Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:
@@ -152,7 +152,7 @@ npm run test:vision-mode
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (drags in a same-origin frame, native range inputs, text selection, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 071f6acce..c293c4bc2 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -1750,6 +1750,122 @@
return null;
}
+ // A native range input moves its thumb only for trusted input, so under a
+ // synthetic drag the value the pointer gives is set the way the browser
+ // sets it: from the point's place along the track (the thumb's centre
+ // spans the track less half a thumb at each end), snapped to the step by
+ // the input's own sanitization, with an `input` event per change and a
+ // `change` event on release. Points are in the input's document. Null
+ // when `el` is not an enabled range input.
+ function _visionRangeDrag(el) {
+ const view = el?.ownerDocument?.defaultView;
+ if (!view || !(el instanceof view.HTMLInputElement) || String(el.type).toLowerCase() !== 'range' || el.disabled) return null;
+ const setter = Object.getOwnPropertyDescriptor(view.HTMLInputElement.prototype, 'value')?.set;
+ const style = view.getComputedStyle(el);
+ // Vertical by writing mode runs from the inline start (top, or bottom
+ // when right-to-left); the legacy vertical forms put the minimum at the bottom.
+ const legacyVertical = el.getAttribute('orient') === 'vertical' || /slider-vertical/.test(style.appearance || '');
+ const vertical = legacyVertical || /^vertical/.test(style.writingMode || '');
+ const reversed = legacyVertical || style.direction === 'rtl';
+ const number = (value, fallback) => (Number.isFinite(Number.parseFloat(value)) ? Number.parseFloat(value) : fallback);
+ const startValue = el.value;
+ const fire = type => el.dispatchEvent(new view.Event(type, { bubbles: true, composed: type === 'input' }));
+ return {
+ to(px, py) {
+ const rect = el.getBoundingClientRect();
+ const min = number(el.min, 0);
+ const max = Math.max(min, number(el.max, 100));
+ const length = vertical ? rect.height : rect.width;
+ const thumb = Math.min(16, vertical ? rect.width : rect.height, length / 2);
+ const along = vertical ? py - rect.top : px - rect.left;
+ let fraction = Math.min(1, Math.max(0, (along - thumb / 2) / Math.max(1, length - thumb)));
+ if (reversed) fraction = 1 - fraction;
+ const before = el.value;
+ const value = String(min + fraction * (max - min));
+ if (setter) setter.call(el, value); else el.value = value;
+ if (el.value !== before) fire('input');
+ },
+ release() {
+ if (el.value !== startValue) fire('change');
+ return el.value;
+ },
+ };
+ }
+
+ // Where a caret would go at a point of `doc` (its client coordinates, kept
+ // inside its viewport): { node, offset }, the node being the text field
+ // itself when the point is in one.
+ function _visionCaretAt(doc, x, y) {
+ const view = doc.defaultView;
+ const px = Math.min(Math.max(0, x), Math.max(0, (view?.innerWidth || 1) - 1));
+ const py = Math.min(Math.max(0, y), Math.max(0, (view?.innerHeight || 1) - 1));
+ try {
+ const position = doc.caretPositionFromPoint?.(px, py);
+ if (position?.offsetNode) return { node: position.offsetNode, offset: position.offset };
+ } catch { /* fall back to the WebKit form */ }
+ try {
+ const range = doc.caretRangeFromPoint?.(px, py);
+ if (range?.startContainer) return { node: range.startContainer, offset: range.startOffset };
+ } catch { /* no caret here */ }
+ return null;
+ }
+
+ // A synthetic press selects no text either. A drag a person's would turn
+ // into a selection (a press on selectable text or in a text field) selects
+ // from the press to the pointer, after a cancelable `selectstart` like the
+ // browser's. Null where a press would not select: a link, image, draggable
+ // item or other control, `user-select: none`, or no text at the point.
+ // Points are in the element's document.
+ function _visionTextSelection(el) {
+ const doc = el?.ownerDocument;
+ const view = doc?.defaultView;
+ if (!view) return null;
+ const tag = String(el.tagName || '').toUpperCase();
+ const field = !el.disabled && (tag === 'TEXTAREA'
+ || (tag === 'INPUT' && /^(text|search|url|tel|password)?$/.test(String(el.getAttribute('type') || '').toLowerCase()))) ? el : null;
+ if (!field) {
+ if (el.closest?.('a[href],img,[draggable="true"],button,select,option,summary,input,textarea')) return null;
+ // `user-select: auto` takes the parent's; the nearest other value decides.
+ for (let node = el, depth = 0; node?.nodeType === 1 && depth < 64; node = node.parentElement, depth++) {
+ const style = view.getComputedStyle(node);
+ const value = style.userSelect || style.webkitUserSelect || 'auto';
+ if (value === 'none') return null;
+ if (value !== 'auto') break;
+ }
+ }
+ let start = null;
+ const caret = (px, py) => {
+ if (!field) return _visionCaretAt(doc, px, py);
+ // Past the field's edge the selection runs to the edge, as it does.
+ const rect = field.getBoundingClientRect();
+ const at = _visionCaretAt(doc, Math.min(Math.max(px, rect.left + 1), rect.right - 1), Math.min(Math.max(py, rect.top + 1), rect.bottom - 1));
+ return at?.node === field ? at : null;
+ };
+ const to = (px, py) => {
+ const at = start && caret(px, py);
+ if (!at) return;
+ try {
+ if (field) field.setSelectionRange(Math.min(start.offset, at.offset), Math.max(start.offset, at.offset), at.offset < start.offset ? 'backward' : 'forward');
+ else doc.getSelection()?.setBaseAndExtent(start.node, start.offset, at.node, at.offset);
+ } catch { /* a node the selection cannot hold */ }
+ };
+ return {
+ begin(px, py) {
+ const at = caret(px, py);
+ if (!at || (!field && at.node.nodeType !== 3)) return false;
+ if (!el.dispatchEvent(new view.Event('selectstart', { bubbles: true, cancelable: true }))) return false;
+ start = at;
+ to(px, py);
+ return true;
+ },
+ to,
+ selectedChars() {
+ if (field) return Math.abs((field.selectionEnd || 0) - (field.selectionStart || 0));
+ return String(doc.getSelection() || '').length;
+ },
+ };
+ }
+
function visionInput(params = {}, actionDeadlineExpired = () => false) {
const type = String(params.type || 'click');
const x = Number(params.x);
@@ -1809,7 +1925,24 @@
const toY = Number(params.toY);
const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));
hover(el);
- pointer(el, 'pointerdown', { button: 0, buttons: 1 }); mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ pointer(el, 'pointerdown', { button: 0, buttons: 1 });
+ const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ // The press focuses what it lands on, as a click's does, and starts
+ // what a person's press would: on a native range input it takes the
+ // thumb (_visionRangeDrag), on selectable text or in a text field a
+ // selection (_visionTextSelection). Either then follows the pointer
+ // in that element's own document wherever it goes.
+ if (pressAllowed) {
+ const focusable = _visionFocusable(el);
+ if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
+ else if (!focusable) el.ownerDocument.activeElement?.blur?.();
+ }
+ const range = pressAllowed ? _visionRangeDrag(el) : null;
+ let selection = pressAllowed && !range ? _visionTextSelection(el) : null;
+ const pressOffsetX = x - cx;
+ const pressOffsetY = y - cy;
+ range?.to(cx, cy);
+ if (selection && !selection.begin(cx, cy)) selection = null;
// Every point, the release included, is given in the coordinates of
// the document under it (a same-origin frame has its own), so the
// events a slider or canvas reads all agree. A point with nothing
@@ -1829,13 +1962,17 @@
const moveInit = { button: 0, buttons: 1, ...pointAt(at, px, py) };
last.dispatchEvent(new PointerCtor('pointermove', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...moveInit })));
last.dispatchEvent(new view.MouseEvent('mousemove', init(moveInit)));
+ range?.to(px - pressOffsetX, py - pressOffsetY);
+ selection?.to(px - pressOffsetX, py - pressOffsetY);
}
const end = _visionTargetAt(toX, toY);
const releaseTarget = end?.el || last;
const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };
releaseTarget.dispatchEvent(new PointerCtor('pointerup', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...endInit })));
releaseTarget.dispatchEvent(new view.MouseEvent('mouseup', init({ ...endInit, detail: 1 })));
- return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe };
+ const rangeValue = range?.release();
+ return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
+ ...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
}
hover(el);
// Like any agent click, a vision click must not open a native file
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 1671c2f34..343efc43b 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2044,6 +2044,122 @@
return null;
}
+ // A native range input moves its thumb only for trusted input, so under a
+ // synthetic drag the value the pointer gives is set the way the browser
+ // sets it: from the point's place along the track (the thumb's centre
+ // spans the track less half a thumb at each end), snapped to the step by
+ // the input's own sanitization, with an `input` event per change and a
+ // `change` event on release. Points are in the input's document. Null
+ // when `el` is not an enabled range input.
+ function _visionRangeDrag(el) {
+ const view = el?.ownerDocument?.defaultView;
+ if (!view || !(el instanceof view.HTMLInputElement) || String(el.type).toLowerCase() !== 'range' || el.disabled) return null;
+ const setter = Object.getOwnPropertyDescriptor(view.HTMLInputElement.prototype, 'value')?.set;
+ const style = view.getComputedStyle(el);
+ // Vertical by writing mode runs from the inline start (top, or bottom
+ // when right-to-left); the legacy vertical forms put the minimum at the bottom.
+ const legacyVertical = el.getAttribute('orient') === 'vertical' || /slider-vertical/.test(style.appearance || '');
+ const vertical = legacyVertical || /^vertical/.test(style.writingMode || '');
+ const reversed = legacyVertical || style.direction === 'rtl';
+ const number = (value, fallback) => (Number.isFinite(Number.parseFloat(value)) ? Number.parseFloat(value) : fallback);
+ const startValue = el.value;
+ const fire = type => el.dispatchEvent(new view.Event(type, { bubbles: true, composed: type === 'input' }));
+ return {
+ to(px, py) {
+ const rect = el.getBoundingClientRect();
+ const min = number(el.min, 0);
+ const max = Math.max(min, number(el.max, 100));
+ const length = vertical ? rect.height : rect.width;
+ const thumb = Math.min(16, vertical ? rect.width : rect.height, length / 2);
+ const along = vertical ? py - rect.top : px - rect.left;
+ let fraction = Math.min(1, Math.max(0, (along - thumb / 2) / Math.max(1, length - thumb)));
+ if (reversed) fraction = 1 - fraction;
+ const before = el.value;
+ const value = String(min + fraction * (max - min));
+ if (setter) setter.call(el, value); else el.value = value;
+ if (el.value !== before) fire('input');
+ },
+ release() {
+ if (el.value !== startValue) fire('change');
+ return el.value;
+ },
+ };
+ }
+
+ // Where a caret would go at a point of `doc` (its client coordinates, kept
+ // inside its viewport): { node, offset }, the node being the text field
+ // itself when the point is in one.
+ function _visionCaretAt(doc, x, y) {
+ const view = doc.defaultView;
+ const px = Math.min(Math.max(0, x), Math.max(0, (view?.innerWidth || 1) - 1));
+ const py = Math.min(Math.max(0, y), Math.max(0, (view?.innerHeight || 1) - 1));
+ try {
+ const position = doc.caretPositionFromPoint?.(px, py);
+ if (position?.offsetNode) return { node: position.offsetNode, offset: position.offset };
+ } catch { /* fall back to the WebKit form */ }
+ try {
+ const range = doc.caretRangeFromPoint?.(px, py);
+ if (range?.startContainer) return { node: range.startContainer, offset: range.startOffset };
+ } catch { /* no caret here */ }
+ return null;
+ }
+
+ // A synthetic press selects no text either. A drag a person's would turn
+ // into a selection (a press on selectable text or in a text field) selects
+ // from the press to the pointer, after a cancelable `selectstart` like the
+ // browser's. Null where a press would not select: a link, image, draggable
+ // item or other control, `user-select: none`, or no text at the point.
+ // Points are in the element's document.
+ function _visionTextSelection(el) {
+ const doc = el?.ownerDocument;
+ const view = doc?.defaultView;
+ if (!view) return null;
+ const tag = String(el.tagName || '').toUpperCase();
+ const field = !el.disabled && (tag === 'TEXTAREA'
+ || (tag === 'INPUT' && /^(text|search|url|tel|password)?$/.test(String(el.getAttribute('type') || '').toLowerCase()))) ? el : null;
+ if (!field) {
+ if (el.closest?.('a[href],img,[draggable="true"],button,select,option,summary,input,textarea')) return null;
+ // `user-select: auto` takes the parent's; the nearest other value decides.
+ for (let node = el, depth = 0; node?.nodeType === 1 && depth < 64; node = node.parentElement, depth++) {
+ const style = view.getComputedStyle(node);
+ const value = style.userSelect || style.webkitUserSelect || 'auto';
+ if (value === 'none') return null;
+ if (value !== 'auto') break;
+ }
+ }
+ let start = null;
+ const caret = (px, py) => {
+ if (!field) return _visionCaretAt(doc, px, py);
+ // Past the field's edge the selection runs to the edge, as it does.
+ const rect = field.getBoundingClientRect();
+ const at = _visionCaretAt(doc, Math.min(Math.max(px, rect.left + 1), rect.right - 1), Math.min(Math.max(py, rect.top + 1), rect.bottom - 1));
+ return at?.node === field ? at : null;
+ };
+ const to = (px, py) => {
+ const at = start && caret(px, py);
+ if (!at) return;
+ try {
+ if (field) field.setSelectionRange(Math.min(start.offset, at.offset), Math.max(start.offset, at.offset), at.offset < start.offset ? 'backward' : 'forward');
+ else doc.getSelection()?.setBaseAndExtent(start.node, start.offset, at.node, at.offset);
+ } catch { /* a node the selection cannot hold */ }
+ };
+ return {
+ begin(px, py) {
+ const at = caret(px, py);
+ if (!at || (!field && at.node.nodeType !== 3)) return false;
+ if (!el.dispatchEvent(new view.Event('selectstart', { bubbles: true, cancelable: true }))) return false;
+ start = at;
+ to(px, py);
+ return true;
+ },
+ to,
+ selectedChars() {
+ if (field) return Math.abs((field.selectionEnd || 0) - (field.selectionStart || 0));
+ return String(doc.getSelection() || '').length;
+ },
+ };
+ }
+
function visionInput(params = {}, actionDeadlineExpired = () => false) {
const type = String(params.type || 'click');
const x = Number(params.x);
@@ -2103,7 +2219,24 @@
const toY = Number(params.toY);
const steps = Math.min(40, Math.max(2, Math.floor(Number(params.steps) || 12)));
hover(el);
- pointer(el, 'pointerdown', { button: 0, buttons: 1 }); mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ pointer(el, 'pointerdown', { button: 0, buttons: 1 });
+ const pressAllowed = mouse(el, 'mousedown', { button: 0, buttons: 1, detail: 1 });
+ // The press focuses what it lands on, as a click's does, and starts
+ // what a person's press would: on a native range input it takes the
+ // thumb (_visionRangeDrag), on selectable text or in a text field a
+ // selection (_visionTextSelection). Either then follows the pointer
+ // in that element's own document wherever it goes.
+ if (pressAllowed) {
+ const focusable = _visionFocusable(el);
+ if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });
+ else if (!focusable) el.ownerDocument.activeElement?.blur?.();
+ }
+ const range = pressAllowed ? _visionRangeDrag(el) : null;
+ let selection = pressAllowed && !range ? _visionTextSelection(el) : null;
+ const pressOffsetX = x - cx;
+ const pressOffsetY = y - cy;
+ range?.to(cx, cy);
+ if (selection && !selection.begin(cx, cy)) selection = null;
// Every point, the release included, is given in the coordinates of
// the document under it (a same-origin frame has its own), so the
// events a slider or canvas reads all agree. A point with nothing
@@ -2123,13 +2256,17 @@
const moveInit = { button: 0, buttons: 1, ...pointAt(at, px, py) };
last.dispatchEvent(new PointerCtor('pointermove', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...moveInit })));
last.dispatchEvent(new view.MouseEvent('mousemove', init(moveInit)));
+ range?.to(px - pressOffsetX, py - pressOffsetY);
+ selection?.to(px - pressOffsetX, py - pressOffsetY);
}
const end = _visionTargetAt(toX, toY);
const releaseTarget = end?.el || last;
const endInit = { button: 0, buttons: 0, ...pointAt(end, toX, toY) };
releaseTarget.dispatchEvent(new PointerCtor('pointerup', init({ pointerId: 1, pointerType: 'mouse', isPrimary: true, ...endInit })));
releaseTarget.dispatchEvent(new view.MouseEvent('mouseup', init({ ...endInit, detail: 1 })));
- return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe };
+ const rangeValue = range?.release();
+ return { success: true, dispatched: true, method: 'synthetic-vision-drag', target: describe,
+ ...(range ? { rangeValue } : {}), ...(selection ? { selectedChars: selection.selectedChars() } : {}) };
}
hover(el);
// Like any agent click, a vision click must not open a native file
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index 04622bbc6..0614c2f92 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -266,6 +266,129 @@ test('content-script vision drag (Firefox path) moves and releases in a same-ori
}
});
+test('content-script vision drag (Firefox path) moves a native range input like a person\'s drag', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ // Synthetic pointer events never move a native slider by themselves.
+ const range = (id, top, extra = '') => `<input id="${id}" type="range" ${extra} style="position:absolute;left:100px;top:${top}px;width:300px;height:20px;margin:0">`;
+ const frame = `<style>body{margin:0}</style><input id="inner" type="range" min="0" max="100" value="0" style="position:absolute;left:20px;top:20px;width:200px;height:20px;margin:0">
+ <script>window.events=[];for(const t of ['input','change'])document.getElementById('inner').addEventListener(t,e=>events.push([t,e.target.value]));<\/script>`;
+ await page.setContent(`<!doctype html><style>body{margin:0}iframe{position:absolute;left:500px;top:300px;width:250px;height:80px;border:2px solid #000}</style>
+ ${range('steps', 100, 'min="0" max="100" step="5" value="10"')}${range('rtl', 160, 'min="0" max="10" value="0" dir="rtl"')}
+ ${range('off', 220, 'min="0" max="100" value="20" disabled')}${range('held', 280, 'min="0" max="100" value="20"')}
+ <iframe srcdoc="${frame.replace(/"/g, '&quot;')}"></iframe>
+ <script>window.events=[];for(const id of ['steps','rtl','off','held'])for(const t of ['input','change'])document.getElementById(id).addEventListener(t,e=>events.push([id,t,e.target.value]));
+ document.getElementById('held').addEventListener('mousedown',e=>e.preventDefault());</script>`);
+ await page.waitForFunction(() => document.querySelector('iframe').contentWindow?.events);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ const value = id => page.evaluate(id => document.getElementById(id).value, id);
+ const eventsOf = id => page.evaluate(id => window.events.filter(event => event[0] === id).map(event => event.slice(1)), id);
+ // The thumb's centre spans x 108–392 (300 px less half a 16 px thumb at
+ // each end). From the thumb at 10 to the middle: 50, snapped to steps of 5.
+ let result = await run({ type: 'drag', x: 136, y: 110, toX: 250, toY: 110, steps: 6 });
+ assert.equal(result.success, true, JSON.stringify(result));
+ assert.equal(result.rangeValue, '50');
+ assert.equal(await value('steps'), '50');
+ const steps = await eventsOf('steps');
+ assert.deepEqual(steps.at(-1), ['change', '50'], JSON.stringify(steps));
+ assert.ok(steps.filter(event => event[0] === 'input').length >= 3, `an input event per change: ${JSON.stringify(steps)}`);
+ assert.ok(steps.every(event => Number(event[1]) % 5 === 0), JSON.stringify(steps));
+ assert.equal(steps.filter(event => event[0] === 'change').length, 1);
+ assert.equal(await page.evaluate(() => document.activeElement.id), 'steps', 'the press focuses the slider');
+ // Right to left: the minimum is at the right end.
+ result = await run({ type: 'drag', x: 392, y: 170, toX: 250, toY: 170, steps: 4 });
+ assert.equal(await value('rtl'), '5', JSON.stringify(result));
+ // A disabled slider, or one whose press the page cancels, does not move.
+ result = await run({ type: 'drag', x: 160, y: 230, toX: 350, toY: 230, steps: 4 });
+ assert.equal(result.rangeValue, undefined, JSON.stringify(result));
+ result = await run({ type: 'drag', x: 160, y: 290, toX: 350, toY: 290, steps: 4 });
+ assert.equal(result.rangeValue, undefined, JSON.stringify(result));
+ assert.deepEqual([await value('off'), await value('held')], ['20', '20']);
+ assert.deepEqual([await eventsOf('off'), await eventsOf('held')], [[], []]);
+ // In a same-origin frame the thumb follows the pointer past the frame's
+ // edge: frame point (28, 30) is page point (530, 332).
+ result = await run({ type: 'drag', x: 530, y: 332, toX: 800, toY: 332, steps: 6 });
+ assert.equal(result.rangeValue, '100', JSON.stringify(result));
+ const inner = await page.evaluate(() => document.querySelector('iframe').contentWindow.events);
+ assert.deepEqual(inner.at(-1), ['change', '100'], JSON.stringify(inner));
+ } finally {
+ await browser.close();
+ }
+});
+
+test('content-script vision drag (Firefox path) selects text like a person\'s drag', async () => {
+ const browser = await chromium.launch({ channel: 'chromium' });
+ try {
+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });
+ const text = (id, top, body, style = '') => `<p id="${id}" style="position:absolute;left:100px;top:${top}px;margin:0;font:20px monospace;white-space:nowrap;${style}">${body}</p>`;
+ await page.setContent(`<!doctype html><style>body{margin:0}</style>
+ ${text('para', 100, 'alpha beta gamma delta')}
+ <textarea id="area" style="position:absolute;left:100px;top:160px;width:500px;height:40px;font:20px monospace;padding:0;border:0;resize:none;outline:0">one two three four</textarea>
+ ${text('noselect', 230, '<span>cannot select this</span>', 'user-select:none')}${text('guarded', 280, 'guarded text here')}
+ <a id="link" href="#x" style="position:absolute;left:100px;top:330px;font:20px monospace">link text here</a>
+ <script>window.starts=[];document.addEventListener('selectstart',e=>starts.push(e.target.id||e.target.parentNode?.id));
+ document.getElementById('guarded').addEventListener('selectstart',e=>e.preventDefault());</script>`);
+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');
+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));
+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));
+ await page.evaluate(({ code, guardBlock }) => {
+ // eslint-disable-next-line no-new-func
+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }
+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }
+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }
+ ${guardBlock}\n${code}\nreturn visionInput;`)();
+ }, { code: block, guardBlock });
+ const run = params => page.evaluate(p => window.__visionInput(p), params);
+ // Character boxes of a paragraph's text, from the layout itself.
+ const charBox = (id, index) => page.evaluate(({ id, index }) => {
+ const node = document.getElementById(id).firstChild;
+ const range = document.createRange();
+ range.setStart(node, index); range.setEnd(node, index + 1);
+ const r = range.getBoundingClientRect();
+ return { left: r.left, right: r.right, y: r.top + r.height / 2 };
+ }, { id, index });
+ // "alpha beta gamma delta": from the left half of "b" (6) to the right half of the last "a" of "gamma" (15).
+ const from = await charBox('para', 6);
+ const to = await charBox('para', 15);
+ let result = await run({ type: 'drag', x: from.left + 2, y: from.y, toX: to.right - 2, toY: to.y, steps: 5 });
+ assert.equal(result.success, true, JSON.stringify(result));
+ assert.equal(await page.evaluate(() => String(document.getSelection())), 'beta gamma');
+ assert.equal(result.selectedChars, 10);
+ // In a text field the selection is the field's own, past its edge to its end.
+ const width = await page.evaluate(() => {
+ const probe = document.createElement('span');
+ probe.style.font = '20px monospace'; probe.textContent = 'x'.repeat(20);
+ document.body.append(probe); const w = probe.getBoundingClientRect().width / 20; probe.remove(); return w;
+ });
+ result = await run({ type: 'drag', x: 100 + 4.2 * width, y: 172, toX: 100 + 12.7 * width, toY: 172, steps: 4 });
+ assert.deepEqual(await page.evaluate(() => { const a = document.getElementById('area'); return [document.activeElement.id, a.value.slice(a.selectionStart, a.selectionEnd)]; }),
+ ['area', 'two three'], JSON.stringify(result));
+ result = await run({ type: 'drag', x: 100 + 8.2 * width, y: 172, toX: 900, toY: 172, steps: 4 });
+ assert.equal(await page.evaluate(() => { const a = document.getElementById('area'); return a.value.slice(a.selectionStart, a.selectionEnd); }), 'three four');
+ // No selection where a person's drag would make none.
+ for (const [x, y, label] of [[110, 240, 'user-select: none'], [110, 290, 'selectstart cancelled'], [110, 342, 'a link']]) {
+ await page.evaluate(() => { document.activeElement?.blur?.(); document.getSelection().removeAllRanges(); });
+ result = await run({ type: 'drag', x, y, toX: x + 150, toY: y, steps: 4 });
+ assert.equal(result.selectedChars, undefined, `${label}: ${JSON.stringify(result)}`);
+ assert.equal(await page.evaluate(() => String(document.getSelection())), '', label);
+ }
+ assert.ok((await page.evaluate(() => window.starts)).includes('guarded'), 'the page saw the selectstart it cancelled');
+ } finally {
+ await browser.close();
+ }
+});
+
test('content-script vision clicks (Firefox path) never open a native file chooser', async () => {
const browser = await chromium.launch({ channel: 'chromium' });
try {