Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: a zoom after an earlier action of its batch stays usable

Emre Sokulluf9ddc1b89bba
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 800531726..4b089118f 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -45,7 +45,7 @@ Gemini, Qwen3-VL, GLM-4.xV and InternVL point in relative 0..1000 coordinates; f
A point is refused before any dispatch when:
-- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`);
+- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`); a zoom taken after an earlier action of its batch already shows what that action changed, so it counts from the update the batch produced and only a change after the zoom's capture makes its points stale (a frame-scale capture beside the zoom tells, `_visionRefreshZooms`);
- it lies outside the image;
- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate; a `drag_at` is checked at both ends against one capture);
- the run has no frame of its own yet (`no_frame`).
@@ -150,7 +150,7 @@ npm run test:vision-mode
```
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs, text selection, disabled controls, the file-chooser guard); the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index cf022a8c9..2348bc73e 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -543,6 +543,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const zooms = this._visionTakePendingZooms(tabId);
if (plan.kind === 'none') {
session.registerNoChange();
+ this._visionRefreshZooms(session, zooms, frame);
const note = visionNoChangeNote(session.keyframe);
if (zooms.length) {
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), zooms,
@@ -562,6 +563,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
encodedPatches.push({ region, encoded });
}
const patches = session.registerPatches(plan.regions.map(regionLabel), frame.raster.pixels);
+ this._visionRefreshZooms(session, zooms, frame);
const note = visionPatchNote(patches, session.keyframe, { coordinates: session.coordinates });
const images = patches.map((patch, index) => ({ id: patch.id, kind: 'patch', dataUrl: encodedPatches[index].encoded.dataUrl }));
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), [...images, ...zooms],
@@ -581,6 +583,20 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return { kind: 'patch', patches };
};
+ /**
+ * Zooms of this batch sent with this update: one taken after the batch's
+ * earlier actions (with a frame-scale capture beside it) already shows
+ * what this update reports, so it is made as current as the update and
+ * stays stale only where the screen changed after the zoom.
+ */
+ proto._visionRefreshZooms = function _visionRefreshZooms(session, zooms, frame) {
+ for (const zoom of zooms || []) {
+ const base = zoom?.base;
+ if (!base || base.width !== frame.size.width || base.height !== frame.size.height || base.url !== frame.probe.url) continue;
+ session.markZoomCurrent(zoom.id, diffFrames(base.pixels, frame.raster.pixels, frame.size.width, frame.size.height).regions);
+ }
+ };
+
/**
* Rewrite click_at into the canonical coordinate click and mark which
* coordinate calls follow an earlier action in the same batch (those are
@@ -753,6 +769,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (!session.keyframe) return { success: false, noDispatch: true, error: 'Take a screenshot before zooming.' };
const region = session.resolveRegion({ x: args.x, y: args.y, width: args.width, height: args.height, image: args.image });
if (!region.ok) return { success: false, noDispatch: true, visionCoordinateError: region.code, error: region.error };
+ // After an earlier action of the batch the zoom shows the screen that
+ // action left, which the update after the batch reports as changed. A
+ // frame-scale capture now lets that update keep the zoom current where
+ // nothing changed after it (_visionRefreshZooms).
+ let base = null;
+ if (vision?.needsStabilityCheck) {
+ try {
+ const frame = await this._visionCaptureFrame(tabId, state);
+ base = { pixels: frame.raster.pixels, width: frame.size.width, height: frame.size.height, url: frame.probe.url };
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ }
+ }
const profile = VISION_PROFILES[state.profileId] || VISION_PROFILES.default;
const probe = await this._visionPlatformProbe(tabId);
const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });
@@ -779,6 +808,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
session.pendingZooms.push({
id: zoom.id, kind: 'zoom', dataUrl: encoded.dataUrl,
note: visionZoomNote(zoom, session.keyframe, region.keyframeRegion, { coordinates: session.coordinates }),
+ ...(base ? { base } : {}),
});
this._visionTraceNote(tabId, 'vision_frame', { kind: 'zoom', id: zoom.id, region: region.keyframeRegion, width: plan.width, height: plan.height, bytes: encoded.bytes });
return { success: true, image: zoom.id, width: plan.width, height: plan.height, note: `Zoom ${zoom.id} follows this result.` };
diff --git a/src/chrome/src/agent/vision-session.js b/src/chrome/src/agent/vision-session.js
index e9c5c4c9b..dd9f1ee17 100644
--- a/src/chrome/src/agent/vision-session.js
+++ b/src/chrome/src/agent/vision-session.js
@@ -187,6 +187,21 @@ export class VisionSession {
return image;
}
+ /**
+ * A zoom taken after earlier actions of its batch shows the screen those
+ * actions left, which the update after the batch then reports as changed
+ * (patches newer than the zoom). That update found `changedSince` (frame
+ * px regions) changed after the zoom's capture: the zoom becomes as
+ * current as the update and stays stale only there.
+ */
+ markZoomCurrent(id, changedSince = []) {
+ const image = this.images.get(id);
+ if (image?.kind !== 'zoom' || image.epoch !== this.epoch) return false;
+ image.seq = this.updateSeq;
+ image.staleRegions = (changedSince || []).map(region => ({ x: region.x, y: region.y, width: region.width, height: region.height }));
+ return true;
+ }
+
/** Should the next observation be a full frame regardless of the diff? */
needsKeyframe(meta = {}) {
if (!this.keyframe || this.forceKeyframe || !this.seenPixels) return true;
@@ -247,7 +262,7 @@ export class VisionSession {
const kx = Math.min(this.keyframe.width - 0.5, target.offsetX + Math.min(ix, target.width - 0.5) * target.scaleX);
const ky = Math.min(this.keyframe.height - 0.5, target.offsetY + Math.min(iy, target.height - 0.5) * target.scaleY);
if (!lookup.composite) {
- const later = this.changes.filter(change => change.seq > target.seq).flatMap(change => change.regions);
+ const later = [...this.changes.filter(change => change.seq > target.seq).flatMap(change => change.regions), ...(target.staleRegions || [])];
if (pointTouchesRegions({ x: kx, y: ky }, later, STALE_MARGIN_PX)) {
const newest = [...this.images.values()].filter(img => img.kind === 'patch' && img.seq > target.seq)
.find(img => pointTouchesRegions({ x: kx, y: ky }, [{ x: img.offsetX, y: img.offsetY, width: img.width, height: img.height }], STALE_MARGIN_PX));
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index cf022a8c9..2348bc73e 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -543,6 +543,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
const zooms = this._visionTakePendingZooms(tabId);
if (plan.kind === 'none') {
session.registerNoChange();
+ this._visionRefreshZooms(session, zooms, frame);
const note = visionNoChangeNote(session.keyframe);
if (zooms.length) {
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), zooms,
@@ -562,6 +563,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
encodedPatches.push({ region, encoded });
}
const patches = session.registerPatches(plan.regions.map(regionLabel), frame.raster.pixels);
+ this._visionRefreshZooms(session, zooms, frame);
const note = visionPatchNote(patches, session.keyframe, { coordinates: session.coordinates });
const images = patches.map((patch, index) => ({ id: patch.id, kind: 'patch', dataUrl: encodedPatches[index].encoded.dataUrl }));
messages.push(this._visionImageMessage([note, ...zooms.map(zoom => zoom.note)].join('\n'), [...images, ...zooms],
@@ -581,6 +583,20 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return { kind: 'patch', patches };
};
+ /**
+ * Zooms of this batch sent with this update: one taken after the batch's
+ * earlier actions (with a frame-scale capture beside it) already shows
+ * what this update reports, so it is made as current as the update and
+ * stays stale only where the screen changed after the zoom.
+ */
+ proto._visionRefreshZooms = function _visionRefreshZooms(session, zooms, frame) {
+ for (const zoom of zooms || []) {
+ const base = zoom?.base;
+ if (!base || base.width !== frame.size.width || base.height !== frame.size.height || base.url !== frame.probe.url) continue;
+ session.markZoomCurrent(zoom.id, diffFrames(base.pixels, frame.raster.pixels, frame.size.width, frame.size.height).regions);
+ }
+ };
+
/**
* Rewrite click_at into the canonical coordinate click and mark which
* coordinate calls follow an earlier action in the same batch (those are
@@ -753,6 +769,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (!session.keyframe) return { success: false, noDispatch: true, error: 'Take a screenshot before zooming.' };
const region = session.resolveRegion({ x: args.x, y: args.y, width: args.width, height: args.height, image: args.image });
if (!region.ok) return { success: false, noDispatch: true, visionCoordinateError: region.code, error: region.error };
+ // After an earlier action of the batch the zoom shows the screen that
+ // action left, which the update after the batch reports as changed. A
+ // frame-scale capture now lets that update keep the zoom current where
+ // nothing changed after it (_visionRefreshZooms).
+ let base = null;
+ if (vision?.needsStabilityCheck) {
+ try {
+ const frame = await this._visionCaptureFrame(tabId, state);
+ base = { pixels: frame.raster.pixels, width: frame.size.width, height: frame.size.height, url: frame.probe.url };
+ } catch (error) {
+ if (isControlFlowError(error)) throw error;
+ }
+ }
const profile = VISION_PROFILES[state.profileId] || VISION_PROFILES.default;
const probe = await this._visionPlatformProbe(tabId);
const plan = planZoomImageSize({ regionWidth: region.cssRegion.width, regionHeight: region.cssRegion.height, devicePixelRatio: probe?.devicePixelRatio || 1, profile });
@@ -779,6 +808,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
session.pendingZooms.push({
id: zoom.id, kind: 'zoom', dataUrl: encoded.dataUrl,
note: visionZoomNote(zoom, session.keyframe, region.keyframeRegion, { coordinates: session.coordinates }),
+ ...(base ? { base } : {}),
});
this._visionTraceNote(tabId, 'vision_frame', { kind: 'zoom', id: zoom.id, region: region.keyframeRegion, width: plan.width, height: plan.height, bytes: encoded.bytes });
return { success: true, image: zoom.id, width: plan.width, height: plan.height, note: `Zoom ${zoom.id} follows this result.` };
diff --git a/src/firefox/src/agent/vision-session.js b/src/firefox/src/agent/vision-session.js
index e9c5c4c9b..dd9f1ee17 100644
--- a/src/firefox/src/agent/vision-session.js
+++ b/src/firefox/src/agent/vision-session.js
@@ -187,6 +187,21 @@ export class VisionSession {
return image;
}
+ /**
+ * A zoom taken after earlier actions of its batch shows the screen those
+ * actions left, which the update after the batch then reports as changed
+ * (patches newer than the zoom). That update found `changedSince` (frame
+ * px regions) changed after the zoom's capture: the zoom becomes as
+ * current as the update and stays stale only there.
+ */
+ markZoomCurrent(id, changedSince = []) {
+ const image = this.images.get(id);
+ if (image?.kind !== 'zoom' || image.epoch !== this.epoch) return false;
+ image.seq = this.updateSeq;
+ image.staleRegions = (changedSince || []).map(region => ({ x: region.x, y: region.y, width: region.width, height: region.height }));
+ return true;
+ }
+
/** Should the next observation be a full frame regardless of the diff? */
needsKeyframe(meta = {}) {
if (!this.keyframe || this.forceKeyframe || !this.seenPixels) return true;
@@ -247,7 +262,7 @@ export class VisionSession {
const kx = Math.min(this.keyframe.width - 0.5, target.offsetX + Math.min(ix, target.width - 0.5) * target.scaleX);
const ky = Math.min(this.keyframe.height - 0.5, target.offsetY + Math.min(iy, target.height - 0.5) * target.scaleY);
if (!lookup.composite) {
- const later = this.changes.filter(change => change.seq > target.seq).flatMap(change => change.regions);
+ const later = [...this.changes.filter(change => change.seq > target.seq).flatMap(change => change.regions), ...(target.staleRegions || [])];
if (pointTouchesRegions({ x: kx, y: ky }, later, STALE_MARGIN_PX)) {
const newest = [...this.images.values()].filter(img => img.kind === 'patch' && img.seq > target.seq)
.find(img => pointTouchesRegions({ x: kx, y: ky }, [{ x: img.offsetX, y: img.offsetY, width: img.width, height: img.height }], STALE_MARGIN_PX));
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index 709678f63..8d031c6d6 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -247,6 +247,46 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.ok(results.some(text => /screen changed near this point/.test(text)), results.join('\n'));
});
+ test(`${label}: a zoom after an earlier action of its batch stays usable beside that batch's patch, except where the screen changed after it`, async () => {
+ // The click draws a block at CSS 600–800 x 420–500; the zoom (f1 px
+ // 540,380 200x100 = CSS 600,422 222x111) is taken after it, and its
+ // point (100,50) is CSS 650,447, inside the block.
+ for (const changedAfterZoom of [false, true]) {
+ const { agent, requests, dispatched, scene } = runAgent((step, messages) => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 200, y: 150 }), call('zoom', { x: 540, y: 380, width: 200, height: 100 })] };
+ if (step === 2) {
+ const zoomId = /UNTRUSTED ZOOM (z\d+)/.exec(messages.map(textOf).findLast(text => /UNTRUSTED ZOOM z\d+/.test(text)) || '')?.[1];
+ assert.ok(zoomId, 'the zoom reached the model with the update');
+ return { toolCalls: [call('click_at', { x: 100, y: 50, image: zoomId })] };
+ }
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Done.', outcome: 'partial' })] };
+ return { content: 'done' };
+ }, { onDispatch: (name, _args, scene) => {
+ if (name === 'click' && scene.rects.length === 1) scene.rects.push({ x: 600, y: 420, w: 200, h: 80, color: [0, 160, 0] });
+ } });
+ if (changedAfterZoom) {
+ // The page draws again inside the zoomed area right after the zoom's capture.
+ const capture = agent._visionPlatformCapture;
+ agent._visionPlatformCapture = async function (tab, options) {
+ const result = await capture.call(this, tab, options);
+ if (options?.cssRegion && scene.rects.length === 2) scene.rects.push({ x: 620, y: 430, w: 80, h: 40, color: [20, 20, 200] });
+ return result;
+ };
+ }
+ const tab = nextTab++;
+ await runLoop(agent, tab, streaming);
+ const clicks = dispatched.filter(entry => entry.name === 'click');
+ const results = requests[2].messages.filter(message => message.role === 'tool').map(message => String(message.content));
+ if (changedAfterZoom) {
+ assert.equal(clicks.length, 1, JSON.stringify(dispatched.map(entry => entry.name)));
+ assert.ok(results.some(text => /changed after z\d+ was taken/.test(text)), results.join('\n'));
+ } else {
+ assert.equal(clicks.length, 2, results.join('\n'));
+ assert.ok(Math.abs(clicks[1].args.x - 650) <= 2 && Math.abs(clicks[1].args.y - 447) <= 2, JSON.stringify(clicks[1].args));
+ }
+ }
+ });
+
test(`${label}: a drag after an earlier action of the batch is re-checked at both ends`, async () => {
// The click draws a block at CSS 650–810 x 420–500. The drag starts far
// from it; only its end (model 650,420 = CSS 722,467) sits on the block.