Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: Firefox vision input never presses a disabled control
Emre Sokullud4d34c7ba039
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 70e035fde..6ff301fdf 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -118,7 +118,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll`click_at` is rewritten into an ordinary coordinate `click` (`translateVisionClick`) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. Only the final dispatch differs:- **Chrome**: trusted CDP pointer events at exactly the point (`_visionDispatchClick`), with the coordinate path's dispatch-time protections kept: a native `<select>` is focused for `type_text` instead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding, `target=_blank` redirection into the run tab (also for a tab opened after the click returned, see Latency) and click-progress annotation. The legacy coordinate path's semantic re-targeting and nearby-input redirect are skipped: a vision click lands where the model pointed. Iframes, including cross-origin ones, receive the events natively.-- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.+- **Firefox**: no CDP, so `content.js` `visionInput` dispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus and a click's activation, selecting text for double/triple clicks, and entering same-origin iframes. The click runs inside the same native file-chooser guard as every agent click (`clickWithoutNativeFilePicker`), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (or `el.click()`), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (`disabled_control`). A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone neither move a native slider nor select text: on an `<input type=range>` the value under the pointer is set as the browser sets it (snapped to the step, an `input` event per change, `change` on release), and on selectable text or in a text field the text from the press to the pointer is selected (after a cancelable `selectstart`; not on links, images, draggable items, other controls or `user-select: none`). A click that must pass the recipient binding keeps the content click path. Cross-origin iframes receive the click on the frame element.Right before input goes out, both builds read what is under the pointer with `visionDomProbe` (`vision-dom-probe.js`), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls:@@ -152,7 +152,7 @@ npm run test:vision-mode- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.-- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (drags in a same-origin frame, native range inputs, text selection, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.+- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.The older MV3 quality fixtures script an accessibility-tree model (they answer with `ref_id`s read from `get_accessibility_tree`) under model names that WebBrain recognizes as vision-capable, so they seed `visionMode: 'off'` with their other settings. A new fixture that scripts tree tools needs the same line.diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex fef5fc0e4..9c0c7df24 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -1897,6 +1897,18 @@const tag = String(el.tagName || '').toLowerCase();const role = el.getAttribute?.('role') || '';const describe = { tag, ...(role ? { role: role.slice(0, 32) } : {}) };+ // A disabled native control (itself, or by its fieldset) takes no press+ // and no click from a person, and el.click() honours that, but+ // dispatched events reach its listeners all the same: a click or a drag+ // that would press one is refused before anything is sent.+ const disabled = type === 'click' || type === 'drag' ? el.closest?.(':disabled:not(fieldset)') : null;+ if (disabled) {+ const what = String(disabled.tagName || 'control').toLowerCase();+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'disabled_control', target: { ...describe, disabled: what },+ error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,+ };+ }return withLocalPageDispatch(() => {if (type === 'hover') {hover(el);diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 2ce27df28..fd1ab7ede 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2191,6 +2191,18 @@const tag = String(el.tagName || '').toLowerCase();const role = el.getAttribute?.('role') || '';const describe = { tag, ...(role ? { role: role.slice(0, 32) } : {}) };+ // A disabled native control (itself, or by its fieldset) takes no press+ // and no click from a person, and el.click() honours that, but+ // dispatched events reach its listeners all the same: a click or a drag+ // that would press one is refused before anything is sent.+ const disabled = type === 'click' || type === 'drag' ? el.closest?.(':disabled:not(fieldset)') : null;+ if (disabled) {+ const what = String(disabled.tagName || 'control').toLowerCase();+ return {+ success: false, dispatched: false, noDispatch: true, visionInputRefused: 'disabled_control', target: { ...describe, disabled: what },+ error: `Not dispatched: the ${what} at this point is disabled, so a person's ${type} there would do nothing. Wait until it is enabled, or use another control.`,+ };+ }return withLocalPageDispatch(() => {if (type === 'hover') {hover(el);diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex ec81f5ed6..445523de7 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -391,6 +391,55 @@ test('content-script vision drag (Firefox path) selects text like a person\'s dr}});+test('content-script vision clicks and drags (Firefox path) leave disabled controls alone', async () => {+ const browser = await chromium.launch({ channel: 'chromium' });+ try {+ const page = await browser.newPage({ viewport: { width: 1000, height: 700 } });+ const at = top => `position:absolute;left:20px;top:${top}px;width:200px;height:30px;margin:0`;+ await page.setContent(`<!doctype html><style>body{margin:0}</style>+ <button id="off" disabled style="${at(20)}">Off</button>+ <button id="inner" disabled style="${at(70)}"><span id="innerSpan">Inner</span></button>+ <fieldset disabled style="position:absolute;left:20px;top:120px;width:300px;height:90px;margin:0;padding:0;border:0">+ <button id="fenced" style="${at(0).replace('left:20px', 'left:0')}">Fenced</button>+ <a id="link" href="#ok" style="${at(50).replace('left:20px', 'left:0')};display:block">Link</a>+ </fieldset>+ <button id="on" style="${at(240)}">On</button>+ <script>window.hits=[];for(const id of ['off','inner','innerSpan','fenced','link','on'])for(const t of ['pointerdown','mousedown','mouseup','click'])+ document.getElementById(id).addEventListener(t,e=>{hits.push(id+':'+t);if(id==='link'&&t==='click')e.preventDefault()});</script>`);+ const source = fs.readFileSync(path.join(ROOT, 'src/firefox/src/content/content.js'), 'utf8');+ const block = source.slice(source.indexOf(' function _visionTargetAt('), source.indexOf(' let _lastClickIdent = null;'));+ const guardBlock = source.slice(source.indexOf(' function uniqueFileInputSelector(input) {'), source.indexOf(' /**\n * Click an element by selector or coordinates.'));+ await page.evaluate(({ code, guardBlock }) => {+ // eslint-disable-next-line no-new-func+ window.__visionInput = new Function(`function withLocalPageDispatch(callback) { return callback(); }+ function _uniqueFileInputSelector(input) { return input.id ? '#' + input.id : ''; }+ function _deepSelectorMatches(selector) { return [...document.querySelectorAll(selector)]; }+ ${guardBlock}\n${code}\nreturn visionInput;`)();+ }, { code: block, guardBlock });+ const run = params => page.evaluate(p => window.__visionInput(p), params);+ const hits = () => page.evaluate(() => window.hits.splice(0));+ // A disabled button, a child of one, and a button disabled by its fieldset.+ for (const [y, what] of [[35, 'button'], [85, 'button'], [135, 'button']]) {+ const result = await run({ type: 'click', x: 100, y });+ assert.equal(result.visionInputRefused, 'disabled_control', JSON.stringify(result));+ assert.equal(result.noDispatch, true);+ assert.equal(result.target.disabled, what);+ assert.match(result.error, /is disabled/);+ assert.deepEqual(await hits(), [], `nothing reached the page at y=${y}`);+ }+ const drag = await run({ type: 'drag', x: 100, y: 35, toX: 400, toY: 35, steps: 4 });+ assert.equal(drag.visionInputRefused, 'disabled_control', JSON.stringify(drag));+ assert.deepEqual(await hits(), []);+ // A link inside a disabled fieldset is not a disabled control; an enabled button clicks.+ assert.equal((await run({ type: 'click', x: 100, y: 185 })).success, true);+ assert.ok((await hits()).includes('link:click'));+ assert.equal((await run({ type: 'click', x: 100, y: 255 })).success, true);+ assert.deepEqual(await hits(), ['on:pointerdown', 'on:mousedown', 'on:mouseup', 'on:click']);+ } finally {+ await browser.close();+ }+});+test('content-script vision clicks (Firefox path) never open a native file chooser', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {