Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: drags owe verification; a requested frame verifies nothing until captured

Emre Sokullu594d80cb95ae
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index cdb7d0bb0..0b2bd9c9e 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -140,7 +140,7 @@ Every hard guard and quality check runs on vision calls; these are the places wh
- **Loop detection** keys a vision action on the screen it acted on (`VisionSession.screenVersion`, bumped by a patch or a frame with other pixels, not by "no visible change" or a `take_screenshot` of the same screen; `_loopScreenIdentity`). Accepting ten invitations under the same point, pressing Enter after each typed item or scrolling to the next stretch of a list is progress; the same point, key or wheel on a screen that does not change still nudges and stops (the coordinate stop after eight clicks, with the model's own point in the message). Whether a click or an arrow key changed anything is for the screen update to say: the DOM text snapshot behind `noProgress` is not taken in a vision run, since it cannot see a canvas, a map or a value it does not read. Choosing from a native dropdown (`nativeSelect`) is never a failed click. Loop, stop and recovery messages name `click_at`, `zoom` and the screen instead of selectors, refs and the tree.
- **The delivery checkpoint** counts `take_screenshot`, `scroll_at` and `wait` like the tree's reads, scrolls and waits, and a screen update that shows a change ends the streak, so only looking at a screen that stays the same adds up. Zooms are left out: reading a dense screen takes several.
- **Page feedback** keeps a response of `take_screenshot`, `zoom` or `wait` when a page script moves focus: none of them acts on focus.
-- **Completion**: every batch, including one that ends early (a refused `done`, a completion hand-back), is followed by one screen update (`_executeToolBatch`), so the model never acts again on the frame its last actions were chosen on; `done`'s own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence), a zoom or a `take_screenshot` counts as an observation after an action, and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update, `take_screenshot`, `read_page` and `wait` instead of `get_accessibility_tree`, `verify_form` or `wait_for_element`. `gmail_count_results`, which takes no target, stays in the vision tool list for Gmail's count job.
+- **Completion**: every batch, including one that ends early (a refused `done`, a completion hand-back), is followed by one screen update (`_executeToolBatch`), so the model never acts again on the frame its last actions were chosen on; `done`'s own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence); after an action a zoom counts as an observation, as does a screen update once it was captured (`auto_screenshot`), but not `take_screenshot` itself, which only asks for the frame the update after its batch captures; a `drag_at` is a consequential action like the `drag_drop` it replaces; and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update, `take_screenshot`, `read_page` and `wait` instead of `get_accessibility_tree`, `verify_form` or `wait_for_element`. `gmail_count_results`, which takes no target, stays in the vision tool list for Gmail's count job.
- **CAPTCHA**: a solved or manually completed challenge is seen gone only through an authoritative root tree read; while a gate is pending, WebBrain makes that read itself before the next page action (`_visionCaptchaRootRead`, never shown to the model).
## Files
@@ -165,8 +165,8 @@ npm run test:vision-mode
```
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
-- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action, on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.
+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
+- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.
- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.
- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
diff --git a/src/chrome/src/agent/completion-invariant.js b/src/chrome/src/agent/completion-invariant.js
index db962e07d..bb6e0bb6b 100644
--- a/src/chrome/src/agent/completion-invariant.js
+++ b/src/chrome/src/agent/completion-invariant.js
@@ -6,6 +6,9 @@ const DIRECT_ACTION_TOOLS = new Set([
'set_checked',
'iframe_click',
'drag_drop',
+ // Vision mode's drag (a slider, a reordered item, a canvas stroke); its
+ // click_at is recorded as the coordinate click it runs as.
+ 'drag_at',
'type_text',
'type_ax',
'iframe_type',
@@ -70,10 +73,9 @@ const OBSERVATION_TOOLS = new Set([
'inspect_viewport',
'screenshot',
'full_page_screenshot',
- // Vision mode: a fresh frame, and a zoom, which captures its region of the
- // page as it is now (the screen update after a batch records itself as
- // auto_screenshot).
- 'take_screenshot',
+ // Vision mode: a zoom, which captures its region of the page as it is now.
+ // take_screenshot only asks for the frame the screen update after its
+ // batch captures, which records itself as auto_screenshot once it exists.
'zoom',
'list_downloads',
'read_downloaded_file',
diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.js
index 987e3c35f..447d71c0b 100644
--- a/src/chrome/src/content/content.js
+++ b/src/chrome/src/content/content.js
@@ -2118,6 +2118,11 @@
}
// Adopt (or refresh) the copy of `root`'s :hover rules.
+ // A sheet's copies are read again only when its rule count changes (a
+ // page adding or removing rules), so a large site is read once.
+ const _visionHoverSheetCopies = new WeakMap();
+ const _visionHoverSheetTexts = new WeakMap();
+
function _visionHoverAdopt(root) {
const view = (root.ownerDocument || root).defaultView;
if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;
@@ -2127,11 +2132,20 @@
if (sheet === ours) continue;
let rules = null;
try { rules = sheet.cssRules; } catch { rules = null; }
- copies.push(..._visionHoverCopies(rules));
+ if (!rules) continue;
+ const cached = _visionHoverSheetCopies.get(sheet);
+ if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }
+ const own = _visionHoverCopies(rules);
+ _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });
+ copies.push(...own);
}
if (!copies.length) { _visionHoverDrop(root); return; }
const sheet = ours || new view.CSSStyleSheet();
- try { sheet.replaceSync(copies.join('\n')); } catch { return; }
+ const text = copies.join('\n');
+ if (_visionHoverSheetTexts.get(sheet) !== text) {
+ try { sheet.replaceSync(text); } catch { return; }
+ _visionHoverSheetTexts.set(sheet, text);
+ }
if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];
_visionHoverSheets.set(root, sheet);
}
diff --git a/src/firefox/src/agent/completion-invariant.js b/src/firefox/src/agent/completion-invariant.js
index db962e07d..bb6e0bb6b 100644
--- a/src/firefox/src/agent/completion-invariant.js
+++ b/src/firefox/src/agent/completion-invariant.js
@@ -6,6 +6,9 @@ const DIRECT_ACTION_TOOLS = new Set([
'set_checked',
'iframe_click',
'drag_drop',
+ // Vision mode's drag (a slider, a reordered item, a canvas stroke); its
+ // click_at is recorded as the coordinate click it runs as.
+ 'drag_at',
'type_text',
'type_ax',
'iframe_type',
@@ -70,10 +73,9 @@ const OBSERVATION_TOOLS = new Set([
'inspect_viewport',
'screenshot',
'full_page_screenshot',
- // Vision mode: a fresh frame, and a zoom, which captures its region of the
- // page as it is now (the screen update after a batch records itself as
- // auto_screenshot).
- 'take_screenshot',
+ // Vision mode: a zoom, which captures its region of the page as it is now.
+ // take_screenshot only asks for the frame the screen update after its
+ // batch captures, which records itself as auto_screenshot once it exists.
'zoom',
'list_downloads',
'read_downloaded_file',
diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.js
index 7ba817553..c62700e8d 100644
--- a/src/firefox/src/content/content.js
+++ b/src/firefox/src/content/content.js
@@ -2412,6 +2412,11 @@
}
// Adopt (or refresh) the copy of `root`'s :hover rules.
+ // A sheet's copies are read again only when its rule count changes (a
+ // page adding or removing rules), so a large site is read once.
+ const _visionHoverSheetCopies = new WeakMap();
+ const _visionHoverSheetTexts = new WeakMap();
+
function _visionHoverAdopt(root) {
const view = (root.ownerDocument || root).defaultView;
if (!view || typeof view.CSSStyleSheet !== 'function' || !root.adoptedStyleSheets) return;
@@ -2421,11 +2426,20 @@
if (sheet === ours) continue;
let rules = null;
try { rules = sheet.cssRules; } catch { rules = null; }
- copies.push(..._visionHoverCopies(rules));
+ if (!rules) continue;
+ const cached = _visionHoverSheetCopies.get(sheet);
+ if (cached && cached.length === rules.length) { copies.push(...cached.copies); continue; }
+ const own = _visionHoverCopies(rules);
+ _visionHoverSheetCopies.set(sheet, { length: rules.length, copies: own });
+ copies.push(...own);
}
if (!copies.length) { _visionHoverDrop(root); return; }
const sheet = ours || new view.CSSStyleSheet();
- try { sheet.replaceSync(copies.join('\n')); } catch { return; }
+ const text = copies.join('\n');
+ if (_visionHoverSheetTexts.get(sheet) !== text) {
+ try { sheet.replaceSync(text); } catch { return; }
+ _visionHoverSheetTexts.set(sheet, text);
+ }
if (!Array.from(root.adoptedStyleSheets).includes(sheet)) root.adoptedStyleSheets = [...Array.from(root.adoptedStyleSheets), sheet];
_visionHoverSheets.set(root, sheet);
}
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index 4c94118dd..35def1ea3 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -795,6 +795,44 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
}
});
+ test(`${build}: a requested frame verifies an action only once it was captured`, async () => {
+ let failing = false;
+ const ctx = runAgent(step => {
+ if (step === 1) { failing = true; return { toolCalls: [call('click_at', { x: 300, y: 200 })] }; }
+ // The screen update after the click could not be captured. Asking for a
+ // frame and finishing in the same batch is not a verified finish.
+ if (step === 2) { failing = false; return { toolCalls: [call('take_screenshot', {}), call('done', { summary: 'Saved.', outcome: 'success' })] }; }
+ // The frame asked for was captured after that batch: now it is.
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Saved.', outcome: 'success' })] };
+ return { content: 'done' };
+ }, { maxSteps: 8 });
+ // The completion check advises here (lean mode skips it).
+ ctx.agent.leanMode = 'off';
+ const capture = ctx.agent._visionPlatformCapture;
+ ctx.agent._visionPlatformCapture = async (...args) => {
+ if (failing) throw new Error('The browser returned no screenshot.');
+ return capture(...args);
+ };
+ const result = await runLoop(ctx.agent, nextTab++, false);
+ assert.equal(ctx.requests.length, 3, JSON.stringify(result).slice(0, 300));
+ assert.deepEqual(ctx.dispatched.map(entry => entry.name), ['click', 'take_screenshot', 'done']);
+ assert.ok(toolTexts(ctx.requests[2]).some(text => /not been verified by a successful page observation/.test(text)), toolTexts(ctx.requests[2]).join('\n').slice(0, 2000));
+ assert.doesNotMatch(String(result?.content ?? result), /could not verify/i, 'the finish after the captured frame carries no caveat');
+ });
+
+ test(`${build}: a drag finished in the same batch is not a verified finish`, async () => {
+ const ctx = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('drag_at', { start_x: 300, start_y: 300, end_x: 600, end_y: 300 }), call('done', { summary: 'Moved the slider.', outcome: 'success' })] };
+ if (step === 2) return { toolCalls: [call('done', { summary: 'Moved the slider.', outcome: 'success' })] };
+ return { content: 'done' };
+ }, { maxSteps: 8 });
+ ctx.agent.leanMode = 'off';
+ const result = await runLoop(ctx.agent, nextTab++, false);
+ assert.equal(ctx.requests.length, 2, JSON.stringify(result).slice(0, 300));
+ assert.deepEqual(ctx.dispatched.map(entry => entry.name), ['drag_at', 'done']);
+ assert.ok(toolTexts(ctx.requests[1]).some(text => /verification|verified|result was seen|same step/i.test(text) && /blockedDone/.test(text)), toolTexts(ctx.requests[1]).join('\n').slice(0, 2000));
+ });
+
test(`${build}: a window resized while the model decides gets a fresh full frame before any point is used`, async () => {
let tab;
const ctx = runAgent(step => {
diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjs
index 8f7cc97d4..56b31d16b 100644
--- a/test/vision-mode-guards.mjs
+++ b/test/vision-mode-guards.mjs
@@ -351,12 +351,21 @@ for (const build of ['chrome', 'firefox']) {
assert.ok(names.includes('click_at') && names.includes('gmail_count_results'), names.join(','));
});
- test(`${build}: after a vision action a zoom or a fresh frame is the observation, and the refusal names what the run has`, () => {
+ test(`${build}: after a vision action a zoom or a captured frame is the observation, and the refusal names what the run has`, () => {
const acted = recordCompletionToolResult(createCompletionInvariantState('run-1'), 'click',
{ x: 40, y: 60, coordinate_space: 'css' }, { success: true, dispatched: true, method: 'cdp-vision-pointer' });
assert.equal(acted.verificationDebt, true);
assert.equal(recordCompletionToolResult(acted, 'zoom', { x: 0, y: 0, width: 50, height: 40 }, { success: true, image: 'z1' }).verificationDebt, false);
- assert.equal(recordCompletionToolResult(acted, 'take_screenshot', {}, { success: true }).verificationDebt, false);
+ // take_screenshot only asks for a frame: the capture after the batch,
+ // recorded as auto_screenshot when it succeeds, is the observation.
+ const asked = recordCompletionToolResult(acted, 'take_screenshot', {}, { success: true, note: 'A fresh full frame follows this result.' });
+ assert.equal(asked.verificationDebt, true, 'a requested frame that was never captured verifies nothing');
+ assert.equal(recordCompletionToolResult(asked, 'auto_screenshot', {}, { success: true, method: 'image_attach', _attachImage: true }).verificationDebt, false);
+ // A drag is a consequential action like the drag_drop it replaces.
+ const dragged = recordCompletionToolResult(createCompletionInvariantState('run-2'), 'drag_at',
+ { start_x: 10, start_y: 10, end_x: 200, end_y: 10 }, { success: true, dispatched: true, method: 'synthetic-vision-drag' });
+ assert.equal(dragged.verificationDebt, true);
+ assert.equal(dragged.lastAction?.name, 'drag_at');
const agent = agentFor();
const tab = 33;
agent._effectiveRunMode = () => 'act';
diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjs
index fde794fac..2acb10e17 100644
--- a/test/vision-pipeline-browser.mjs
+++ b/test/vision-pipeline-browser.mjs
@@ -553,6 +553,10 @@ test('content-script vision hover (Firefox path) reveals what CSS :hover reveals
// A shadow root's :host(:hover) rule applies inside it.
await run({ type: 'hover', x: 750, y: 120 });
assert.equal((await state()).tip, 'block');
+ // A rule the page adds later is read on the next input.
+ await page.evaluate(() => document.styleSheets[0].insertRule('#away:hover { color: rgb(0, 128, 0) }', document.styleSheets[0].cssRules.length));
+ await run({ type: 'hover', x: 150, y: 420 });
+ assert.equal(await page.evaluate(() => getComputedStyle(document.getElementById('away')).color), 'rgb(0, 128, 0)');
// A click leaves the pointer, and :hover, where it clicked.
await run({ type: 'click', x: 150, y: 120 });
assert.equal((await state()).submenu, 'block');