Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: drags owe verification; a requested frame verifies nothing until captured
Emre Sokullu594d80cb95ae
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex cdb7d0bb0..0b2bd9c9e 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -140,7 +140,7 @@ Every hard guard and quality check runs on vision calls; these are the places wh- **Loop detection** keys a vision action on the screen it acted on (`VisionSession.screenVersion`, bumped by a patch or a frame with other pixels, not by "no visible change" or a `take_screenshot` of the same screen; `_loopScreenIdentity`). Accepting ten invitations under the same point, pressing Enter after each typed item or scrolling to the next stretch of a list is progress; the same point, key or wheel on a screen that does not change still nudges and stops (the coordinate stop after eight clicks, with the model's own point in the message). Whether a click or an arrow key changed anything is for the screen update to say: the DOM text snapshot behind `noProgress` is not taken in a vision run, since it cannot see a canvas, a map or a value it does not read. Choosing from a native dropdown (`nativeSelect`) is never a failed click. Loop, stop and recovery messages name `click_at`, `zoom` and the screen instead of selectors, refs and the tree.- **The delivery checkpoint** counts `take_screenshot`, `scroll_at` and `wait` like the tree's reads, scrolls and waits, and a screen update that shows a change ends the streak, so only looking at a screen that stays the same adds up. Zooms are left out: reading a dense screen takes several.- **Page feedback** keeps a response of `take_screenshot`, `zoom` or `wait` when a page script moves focus: none of them acts on focus.-- **Completion**: every batch, including one that ends early (a refused `done`, a completion hand-back), is followed by one screen update (`_executeToolBatch`), so the model never acts again on the frame its last actions were chosen on; `done`'s own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence), a zoom or a `take_screenshot` counts as an observation after an action, and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update, `take_screenshot`, `read_page` and `wait` instead of `get_accessibility_tree`, `verify_form` or `wait_for_element`. `gmail_count_results`, which takes no target, stays in the vision tool list for Gmail's count job.+- **Completion**: every batch, including one that ends early (a refused `done`, a completion hand-back), is followed by one screen update (`_executeToolBatch`), so the model never acts again on the frame its last actions were chosen on; `done`'s own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence); after an action a zoom counts as an observation, as does a screen update once it was captured (`auto_screenshot`), but not `take_screenshot` itself, which only asks for the frame the update after its batch captures; a `drag_at` is a consequential action like the `drag_drop` it replaces; and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update, `take_screenshot`, `read_page` and `wait` instead of `get_accessibility_tree`, `verify_form` or `wait_for_element`. `gmail_count_results`, which takes no target, stays in the vision tool list for Gmail's count job.- **CAPTCHA**: a solved or manually completed challenge is seen gone only through an authoritative root tree read; while a gate is pending, WebBrain makes that read itself before the next page action (`_visionCaptchaRootRead`, never shown to the model).## Files@@ -165,8 +165,8 @@ npm run test:vision-mode```- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.-- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action, on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.+- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.diff --git a/src/chrome/src/agent/completion-invariant.js b/src/chrome/src/agent/completion-invariant.jsindex db962e07d..bb6e0bb6b 100644--- a/src/chrome/src/agent/completion-invariant.js+++ b/src/chrome/src/agent/completion-invariant.js@@ -6,6 +6,9 @@ const DIRECT_ACTION_TOOLS = new Set(['set_checked','iframe_click','drag_drop',+ // Vision mode's drag (a slider, a reordered item, a canvas stroke); its+ // click_at is recorded as the coordinate click it runs as.+ 'drag_at','type_text','type_ax','iframe_type',@@ -70,10 +73,9 @@ const OBSERVATION_TOOLS = new Set(['inspect_viewport','screenshot','full_page_screenshot',- // Vision mode: a fresh frame, and a zoom, which captures its region of the- // page as it is now (the screen update after a batch records itself as- // auto_screenshot).- 'take_screenshot',+ // Vision mode: a zoom, which captures its region of the page as it is now.+ // take_screenshot only asks for the frame the screen update after its+ // batch captures, which records itself as auto_screenshot once it exists.'zoom','list_downloads','read_downloaded_file',diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex 987e3c35f..447d71c0b 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -2118,6 +2118,11 @@}// Adopt (or refresh) the copy of `root`'s :hover rules.+ // A sheet's copies are read again only when its rule count changes (a+ // page adding or removing rules), so a large site is read once.+ const _visionHoverSheetCopies = new WeakMap();+ const _visionHoverSheetTexts = new WeakMap();+function _visionHoverAdopt(root) {const view = (root.ownerDocument || root).defaultView;if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;@@ -2127,11 +2132,20 @@if (sheet === ours) continue;let rules = null;try { rules = sheet.cssRules; } catch { rules = null; }- copies.push(..._visionHoverCopies(rules));+ if (!rules) continue;+ const cached = _visionHoverSheetCopies.get(sheet);+ if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }+ const own = _visionHoverCopies(rules);+ _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });+ copies.push(...own);}if (!copies.length) { _visionHoverDrop(root); return; }const sheet = ours || new view.CSSStyleSheet();- try { sheet.replaceSync(copies.join('\n')); } catch { return; }+ const text = copies.join('\n');+ if (_visionHoverSheetTexts.get(sheet) !== text) {+ try { sheet.replaceSync(text); } catch { return; }+ _visionHoverSheetTexts.set(sheet, text);+ }if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];_visionHoverSheets.set(root, sheet);}diff --git a/src/firefox/src/agent/completion-invariant.js b/src/firefox/src/agent/completion-invariant.jsindex db962e07d..bb6e0bb6b 100644--- a/src/firefox/src/agent/completion-invariant.js+++ b/src/firefox/src/agent/completion-invariant.js@@ -6,6 +6,9 @@ const DIRECT_ACTION_TOOLS = new Set(['set_checked','iframe_click','drag_drop',+ // Vision mode's drag (a slider, a reordered item, a canvas stroke); its+ // click_at is recorded as the coordinate click it runs as.+ 'drag_at','type_text','type_ax','iframe_type',@@ -70,10 +73,9 @@ const OBSERVATION_TOOLS = new Set(['inspect_viewport','screenshot','full_page_screenshot',- // Vision mode: a fresh frame, and a zoom, which captures its region of the- // page as it is now (the screen update after a batch records itself as- // auto_screenshot).- 'take_screenshot',+ // Vision mode: a zoom, which captures its region of the page as it is now.+ // take_screenshot only asks for the frame the screen update after its+ // batch captures, which records itself as auto_screenshot once it exists.'zoom','list_downloads','read_downloaded_file',diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 7ba817553..c62700e8d 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -2412,6 +2412,11 @@}// Adopt (or refresh) the copy of `root`'s :hover rules.+ // A sheet's copies are read again only when its rule count changes (a+ // page adding or removing rules), so a large site is read once.+ const _visionHoverSheetCopies = new WeakMap();+ const _visionHoverSheetTexts = new WeakMap();+function _visionHoverAdopt(root) {const view = (root.ownerDocument || root).defaultView;if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;@@ -2421,11 +2426,20 @@if (sheet === ours) continue;let rules = null;try { rules = sheet.cssRules; } catch { rules = null; }- copies.push(..._visionHoverCopies(rules));+ if (!rules) continue;+ const cached = _visionHoverSheetCopies.get(sheet);+ if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }+ const own = _visionHoverCopies(rules);+ _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });+ copies.push(...own);}if (!copies.length) { _visionHoverDrop(root); return; }const sheet = ours || new view.CSSStyleSheet();- try { sheet.replaceSync(copies.join('\n')); } catch { return; }+ const text = copies.join('\n');+ if (_visionHoverSheetTexts.get(sheet) !== text) {+ try { sheet.replaceSync(text); } catch { return; }+ _visionHoverSheetTexts.set(sheet, text);+ }if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];_visionHoverSheets.set(root, sheet);}diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjsindex 4c94118dd..35def1ea3 100644--- a/test/vision-mode-agent.mjs+++ b/test/vision-mode-agent.mjs@@ -795,6 +795,44 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))}});+ test(`${build}: a requested frame verifies an action only once it was captured`, async () => {+ let failing = false;+ const ctx = runAgent(step => {+ if (step === 1) { failing = true; return { toolCalls: [call('click_at', { x: 300, y: 200 })] }; }+ // The screen update after the click could not be captured. Asking for a+ // frame and finishing in the same batch is not a verified finish.+ if (step === 2) { failing = false; return { toolCalls: [call('take_screenshot', {}), call('done', { summary: 'Saved.', outcome: 'success' })] }; }+ // The frame asked for was captured after that batch: now it is.+ if (step === 3) return { toolCalls: [call('done', { summary: 'Saved.', outcome: 'success' })] };+ return { content: 'done' };+ }, { maxSteps: 8 });+ // The completion check advises here (lean mode skips it).+ ctx.agent.leanMode = 'off';+ const capture = ctx.agent._visionPlatformCapture;+ ctx.agent._visionPlatformCapture = async (...args) => {+ if (failing) throw new Error('The browser returned no screenshot.');+ return capture(...args);+ };+ const result = await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.requests.length, 3, JSON.stringify(result).slice(0, 300));+ assert.deepEqual(ctx.dispatched.map(entry => entry.name), ['click', 'take_screenshot', 'done']);+ assert.ok(toolTexts(ctx.requests[2]).some(text => /not been verified by a successful page observation/.test(text)), toolTexts(ctx.requests[2]).join('\n').slice(0, 2000));+ assert.doesNotMatch(String(result?.content ?? result), /could not verify/i, 'the finish after the captured frame carries no caveat');+ });++ test(`${build}: a drag finished in the same batch is not a verified finish`, async () => {+ const ctx = runAgent(step => {+ if (step === 1) return { toolCalls: [call('drag_at', { start_x: 300, start_y: 300, end_x: 600, end_y: 300 }), call('done', { summary: 'Moved the slider.', outcome: 'success' })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'Moved the slider.', outcome: 'success' })] };+ return { content: 'done' };+ }, { maxSteps: 8 });+ ctx.agent.leanMode = 'off';+ const result = await runLoop(ctx.agent, nextTab++, false);+ assert.equal(ctx.requests.length, 2, JSON.stringify(result).slice(0, 300));+ assert.deepEqual(ctx.dispatched.map(entry => entry.name), ['drag_at', 'done']);+ assert.ok(toolTexts(ctx.requests[1]).some(text => /verification|verified|result was seen|same step/i.test(text) && /blockedDone/.test(text)), toolTexts(ctx.requests[1]).join('\n').slice(0, 2000));+ });+test(`${build}: a window resized while the model decides gets a fresh full frame before any point is used`, async () => {let tab;const ctx = runAgent(step => {diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjsindex 8f7cc97d4..56b31d16b 100644--- a/test/vision-mode-guards.mjs+++ b/test/vision-mode-guards.mjs@@ -351,12 +351,21 @@ for (const build of ['chrome', 'firefox']) {assert.ok(names.includes('click_at') && names.includes('gmail_count_results'), names.join(','));});- test(`${build}: after a vision action a zoom or a fresh frame is the observation, and the refusal names what the run has`, () => {+ test(`${build}: after a vision action a zoom or a captured frame is the observation, and the refusal names what the run has`, () => {const acted = recordCompletionToolResult(createCompletionInvariantState('run-1'), 'click',{ x: 40, y: 60, coordinate_space: 'css' }, { success: true, dispatched: true, method: 'cdp-vision-pointer' });assert.equal(acted.verificationDebt, true);assert.equal(recordCompletionToolResult(acted, 'zoom', { x: 0, y: 0, width: 50, height: 40 }, { success: true, image: 'z1' }).verificationDebt, false);- assert.equal(recordCompletionToolResult(acted, 'take_screenshot', {}, { success: true }).verificationDebt, false);+ // take_screenshot only asks for a frame: the capture after the batch,+ // recorded as auto_screenshot when it succeeds, is the observation.+ const asked = recordCompletionToolResult(acted, 'take_screenshot', {}, { success: true, note: 'A fresh full frame follows this result.' });+ assert.equal(asked.verificationDebt, true, 'a requested frame that was never captured verifies nothing');+ assert.equal(recordCompletionToolResult(asked, 'auto_screenshot', {}, { success: true, method: 'image_attach', _attachImage: true }).verificationDebt, false);+ // A drag is a consequential action like the drag_drop it replaces.+ const dragged = recordCompletionToolResult(createCompletionInvariantState('run-2'), 'drag_at',+ { start_x: 10, start_y: 10, end_x: 200, end_y: 10 }, { success: true, dispatched: true, method: 'synthetic-vision-drag' });+ assert.equal(dragged.verificationDebt, true);+ assert.equal(dragged.lastAction?.name, 'drag_at');const agent = agentFor();const tab = 33;agent._effectiveRunMode = () => 'act';diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex fde794fac..2acb10e17 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -553,6 +553,10 @@ test('content-script vision hover (Firefox path) reveals what CSS :hover reveals// A shadow root's :host(:hover) rule applies inside it.await run({ type: 'hover', x: 750, y: 120 });assert.equal((await state()).tip, 'block');+ // A rule the page adds later is read on the next input.+ await page.evaluate(() => document.styleSheets[0].insertRule('#away:hover { color: rgb(0, 128, 0) }', document.styleSheets[0].cssRules.length));+ await run({ type: 'hover', x: 150, y: 420 });+ assert.equal(await page.evaluate(() => getComputedStyle(document.getElementById('away')).color), 'rgb(0, 128, 0)');// A click leaves the pointer, and :hover, where it clicked.await run({ type: 'click', x: 150, y: 120 });assert.equal((await state()).submenu, 'block');