Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: a Firefox double click selects its word, a click places the caret
Emre Sokullu00ab28f3599e
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 6ff301fdf..9d2e584c4 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:@@ -152,7 +152,7 @@ npm run test:vision-mode- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 9c0c7df24..373165825 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -1867,6 +1867,58 @@};}+ // A person's press in a text field or in editable text puts the caret+ // under the pointer; focus() alone leaves it wherever it was, and typing+ // would land there instead of where the model clicked.+ function _visionPlaceCaret(el, cx, cy) {+ const tag = String(el?.tagName || '').toUpperCase();+ const caret = el ? _visionCaretAt(el.ownerDocument, cx, cy) : null;+ if (!caret) return;+ try {+ if (tag === 'INPUT' || tag === 'TEXTAREA') {+ if (caret.node === el) el.setSelectionRange(caret.offset, caret.offset);+ } else if (caret.node.nodeType === 3 && caret.node.parentElement?.isContentEditable) {+ el.ownerDocument.getSelection()?.collapse(caret.node, caret.offset);+ }+ } catch { /* a field type without a caret (email, number) */ }+ }++ // What a double click selects around a caret `offset` in `text`: the word+ // (or the run of spaces, or the punctuation mark) under the pointer, which+ // is the character after the caret, or the one before it when that one is+ // part of a word and the next is not (the right half of a word's last+ // letter). Returns [start, end].+ function _visionWordAt(text, offset) {+ const value = String(text || '');+ if (!value) return [0, 0];+ const wordChar = ch => /[\p{L}\p{N}\p{M}_']/u.test(ch || '');+ let index = Math.min(Math.max(0, Number(offset) || 0), value.length);+ if (index >= value.length || (!wordChar(value[index]) && wordChar(value[index - 1]))) index = Math.max(0, index - 1);+ if (typeof Intl?.Segmenter === 'function') {+ try {+ for (const part of new Intl.Segmenter(undefined, { granularity: 'word' }).segment(value)) {+ if (index >= part.index && index < part.index + part.segment.length) return [part.index, part.index + part.segment.length];+ }+ } catch { /* character classes below */ }+ }+ const kind = ch => (wordChar(ch) ? 'word' : /\s/.test(ch) ? 'space' : 'mark');+ const at = kind(value[index]);+ if (at === 'mark') return [index, index + 1];+ let start = index;+ let end = index + 1;+ while (start > 0 && kind(value[start - 1]) === at) start--;+ while (end < value.length && kind(value[end]) === at) end++;+ return [start, end];+ }++ // What a triple click in a textarea selects: the line around the caret.+ function _visionLineAt(text, offset) {+ const value = String(text || '');+ const index = Math.min(Math.max(0, Number(offset) || 0), value.length);+ const end = value.indexOf('\n', index);+ return [index > 0 ? value.lastIndexOf('\n', index - 1) + 1 : 0, end < 0 ? value.length : end];+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -1999,6 +2051,7 @@const focusable = _visionFocusable(el);if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });else if (!focusable) el.ownerDocument.activeElement?.blur?.();+ if (focusable) _visionPlaceCaret(el, cx, cy);}pointer(el, 'pointerup', { button, buttons: 0 });mouse(el, 'mouseup', { button, buttons: 0, detail: count });@@ -2007,21 +2060,24 @@}if (button === 0 && clicks >= 2) {mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });- // Browsers select a word (two clicks) or the whole line/field- // (three) as the default action; synthetic events need it done.+ // Browsers select the word under the pointer (two clicks), or its+ // line or the whole field (three), as the default action; synthetic+ // events need it done. A double click never takes a whole field:+ // the typing that usually follows would replace all of it.try {- if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {- el.select();+ const caret = _visionCaretAt(el.ownerDocument, cx, cy);+ if (typeof el.setSelectionRange === 'function' && /^(input|textarea)$/.test(tag)) {+ if (clicks >= 3 && tag === 'input') el.select();+ else if (caret?.node === el) {+ const [start, end] = (clicks >= 3 ? _visionLineAt : _visionWordAt)(el.value, caret.offset);+ el.setSelectionRange(start, end);+ }} else {const selection = el.ownerDocument.getSelection();if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);- else if (typeof selection.modify === 'function') {- const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);- if (caret?.offsetNode) {- selection.collapse(caret.offsetNode, caret.offset);- selection.modify('move', 'backward', 'word');- selection.modify('extend', 'forward', 'word');- }+ else if (caret?.node?.nodeType === 3) {+ const [start, end] = _visionWordAt(caret.node.nodeValue, caret.offset);+ selection.setBaseAndExtent(caret.node, start, caret.node, end);}}} catch { /* selection is best effort */ }diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex fd1ab7ede..8a6a3eb68 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2161,6 +2161,58 @@};}+ // A person's press in a text field or in editable text puts the caret+ // under the pointer; focus() alone leaves it wherever it was, and typing+ // would land there instead of where the model clicked.+ function _visionPlaceCaret(el, cx, cy) {+ const tag = String(el?.tagName || '').toUpperCase();+ const caret = el ? _visionCaretAt(el.ownerDocument, cx, cy) : null;+ if (!caret) return;+ try {+ if (tag === 'INPUT' || tag === 'TEXTAREA') {+ if (caret.node === el) el.setSelectionRange(caret.offset, caret.offset);+ } else if (caret.node.nodeType === 3 && caret.node.parentElement?.isContentEditable) {+ el.ownerDocument.getSelection()?.collapse(caret.node, caret.offset);+ }+ } catch { /* a field type without a caret (email, number) */ }+ }++ // What a double click selects around a caret `offset` in `text`: the word+ // (or the run of spaces, or the punctuation mark) under the pointer, which+ // is the character after the caret, or the one before it when that one is+ // part of a word and the next is not (the right half of a word's last+ // letter). Returns [start, end].+ function _visionWordAt(text, offset) {+ const value = String(text || '');+ if (!value) return [0, 0];+ const wordChar = ch => /[\p{L}\p{N}\p{M}_']/u.test(ch || '');+ let index = Math.min(Math.max(0, Number(offset) || 0), value.length);+ if (index >= value.length || (!wordChar(value[index]) && wordChar(value[index - 1]))) index = Math.max(0, index - 1);+ if (typeof Intl?.Segmenter === 'function') {+ try {+ for (const part of new Intl.Segmenter(undefined, { granularity: 'word' }).segment(value)) {+ if (index >= part.index && index < part.index + part.segment.length) return [part.index, part.index + part.segment.length];+ }+ } catch { /* character classes below */ }+ }+ const kind = ch => (wordChar(ch) ? 'word' : /\s/.test(ch) ? 'space' : 'mark');+ const at = kind(value[index]);+ if (at === 'mark') return [index, index + 1];+ let start = index;+ let end = index + 1;+ while (start > 0 && kind(value[start - 1]) === at) start--;+ while (end < value.length && kind(value[end]) === at) end++;+ return [start, end];+ }++ // What a triple click in a textarea selects: the line around the caret.+ function _visionLineAt(text, offset) {+ const value = String(text || '');+ const index = Math.min(Math.max(0, Number(offset) || 0), value.length);+ const end = value.indexOf('\n', index);+ return [index > 0 ? value.lastIndexOf('\n', index - 1) + 1 : 0, end < 0 ? value.length : end];+ }+function visionInput(params = {}, actionDeadlineExpired = () => false) {const type = String(params.type || 'click');const x = Number(params.x);@@ -2293,6 +2345,7 @@const focusable = _visionFocusable(el);if (focusable && focusable !== el.ownerDocument.activeElement) focusable.focus({ preventScroll: true });else if (!focusable) el.ownerDocument.activeElement?.blur?.();+ if (focusable) _visionPlaceCaret(el, cx, cy);}pointer(el, 'pointerup', { button, buttons: 0 });mouse(el, 'mouseup', { button, buttons: 0, detail: count });@@ -2301,21 +2354,24 @@}if (button === 0 && clicks >= 2) {mouse(el, 'dblclick', { button: 0, buttons: 0, detail: 2 });- // Browsers select a word (two clicks) or the whole line/field- // (three) as the default action; synthetic events need it done.+ // Browsers select the word under the pointer (two clicks), or its+ // line or the whole field (three), as the default action; synthetic+ // events need it done. A double click never takes a whole field:+ // the typing that usually follows would replace all of it.try {- if (typeof el.select === 'function' && /^(input|textarea)$/.test(tag)) {- el.select();+ const caret = _visionCaretAt(el.ownerDocument, cx, cy);+ if (typeof el.setSelectionRange === 'function' && /^(input|textarea)$/.test(tag)) {+ if (clicks >= 3 && tag === 'input') el.select();+ else if (caret?.node === el) {+ const [start, end] = (clicks >= 3 ? _visionLineAt : _visionWordAt)(el.value, caret.offset);+ el.setSelectionRange(start, end);+ }} else {const selection = el.ownerDocument.getSelection();if (clicks >= 3) selection.selectAllChildren(el.closest('p,li,td,th,div,section,article,h1,h2,h3,h4,h5,h6') || el);- else if (typeof selection.modify === 'function') {- const caret = el.ownerDocument.caretPositionFromPoint?.(cx, cy);- if (caret?.offsetNode) {- selection.collapse(caret.offsetNode, caret.offset);- selection.modify('move', 'backward', 'word');- selection.modify('extend', 'forward', 'word');- }+ else if (caret?.node?.nodeType === 3) {+ const [start, end] = _visionWordAt(caret.node.nodeValue, caret.offset);+ selection.setBaseAndExtent(caret.node, start, caret.node, end);}}} catch { /* selection is best effort */ }diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex 445523de7..16d163c0e 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -391,6 +391,70 @@ test('content-script vision drag (Firefox path) selects text like a person\'s dr}});+test('content-script vision clicks (Firefox path) put the caret at the point, double clicks select its word, triple clicks its line', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ const font = 'font:20px monospace;padding:0;border:0;margin:0;outline:0';+ await page.setContent(`<!doctype html><style>body{margin:0}</style>+ <input id="one" value="alpha beta gamma" style="position:absolute;left:100px;top:100px;width:400px;height:30px;${font}">+ <textarea id="two" style="position:absolute;left:100px;top:160px;width:500px;height:90px;line-height:30px;resize:none;${font}">first line here+second line there+third</textarea>+ <p id="three" style="position:absolute;left:100px;top:280px;white-space:nowrap;${font}">one two three</p>+ <div id="editor" contenteditable="true" style="position:absolute;left:100px;top:340px;width:400px;white-space:nowrap;${font}">edit this text</div>`);+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ await page.evaluate(({ code, guardBlock }) => {+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)();+ }, { code: block, guardBlock });+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const w = await page.evaluate(() => {+ const probe = document.createElement('span');+ probe.style.font = '20px monospace'; probe.textContent = 'x'.repeat(20);+ document.body.append(probe); const width = probe.getBoundingClientRect().width / 20; probe.remove(); return width;+ });+ const fieldSelection = id => page.evaluate(id => { const el = document.getElementById(id); return el.value.slice(el.selectionStart, el.selectionEnd); }, id);+ // One click: the caret goes where the pointer is (between "alpha" and its space), in a field and in editable text.+ await run({ type: 'click', x: 100 + 5.2 * w, y: 115 });+ assert.deepEqual(await page.evaluate(() => { const el = document.getElementById('one'); return [document.activeElement.id, el.selectionStart, el.selectionEnd]; }), ['one', 5, 5]);+ await run({ type: 'click', x: 100 + 9.2 * w, y: 350 });+ assert.deepEqual(await page.evaluate(() => { const s = document.getSelection(); return [document.activeElement.id, s.isCollapsed, s.anchorOffset, s.anchorNode?.parentElement?.id]; }),+ ['editor', true, 9, 'editor'], 'the caret before "text"');+ // "alpha beta gamma": inside "beta", on the right half of its last letter, on the left half of "gamma"'s first.+ for (const [at, word] of [[7.5, 'beta'], [9.8, 'beta'], [11.2, 'gamma']]) {+ await run({ type: 'click', x: 100 + at * w, y: 115, clickCount: 2 });+ assert.equal(await fieldSelection('one'), word, `double click at character ${at}`);+ }+ await run({ type: 'click', x: 100 + 7.5 * w, y: 115, clickCount: 3 });+ assert.equal(await fieldSelection('one'), 'alpha beta gamma', 'a triple click takes the whole input');+ // The textarea's second line (y 190–220): the word, then the line.+ await run({ type: 'click', x: 100 + 8.5 * w, y: 205, clickCount: 2 });+ assert.equal(await fieldSelection('two'), 'line');+ await run({ type: 'click', x: 100 + 8.5 * w, y: 205, clickCount: 3 });+ assert.equal(await fieldSelection('two'), 'second line there');+ // Page text: the left half of "two"'s first letter and the right half of its last.+ const charBox = index => page.evaluate(index => {+ const range = document.createRange();+ const node = document.getElementById('three').firstChild;+ range.setStart(node, index); range.setEnd(node, index + 1);+ const r = range.getBoundingClientRect();+ return { left: r.left, right: r.right, y: r.top + r.height / 2 };+ }, index);+ for (const x of [(await charBox(4)).left + 2, (await charBox(6)).right - 2]) {+ await run({ type: 'click', x, y: (await charBox(4)).y, clickCount: 2 });+ assert.equal(await page.evaluate(() => String(document.getSelection())), 'two', `double click at x=${x}`);+ }+ } finally {+ await browser.close();+ }+});+test('content-script vision clicks and drags (Firefox path) leave disabled controls alone', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {