Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: check the screen again after a failed screen update

Emre Sokullu35a43b398785
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index fb68b7998..b0ee08f96 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -53,12 +53,14 @@ A point is refused before any dispatch when:
- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`); a zoom taken after an earlier action of its batch already shows what that action changed, so it counts from the update the batch produced and only a change after the zoom's capture makes its points stale (a frame-scale capture beside the zoom tells, `_visionRefreshZooms`);
- it lies outside the image;
-- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate; a `drag_at` is checked at both ends against one capture);
+- it is the second or later coordinate action of one model turn, or any coordinate action while the last screen update failed (`session.observationFailed`), and the live screen changed near it since the screen the model last saw, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate; a `drag_at` is checked at both ends against one capture);
- the run has no frame of its own yet (`no_frame`);
- the browser window was resized or zoomed (CSS viewport size or device pixel ratio) after its frame was taken (`viewport_changed`, `_visionStaleLayoutRefusal`): checked once before each batch that uses points, so a resize page feedback has not reported yet is caught too; a full frame follows. A resize page feedback did report while the model decided sets the decision aside for a fresh full frame before that (`visionScreenFeedback`).
A refused point also stops the rest of the model's batch: the calls after it (a `type_text`, say) were planned on that point.
+Typing goes to whatever has focus, which only the screen shows. While the run has no frame of this page, or the last screen update failed (a capture or encoding error, or a re-check capture that failed; any later frame, patch or "no visible change" clears it), `type_text` and `press_keys` are refused before dispatch (`visionScreenUnverified`) unless a `click_at` earlier in the same response passed its check, in which case they type into the field it focused. The refusal stops the rest of the batch, and the model is told to take a screenshot or click the field first.
+
Every run starts on a frame it captured itself: `_beginVisionRun` retires the previous run's screen, so if the first capture fails the model is told so, earlier screenshots leave the request, and coordinates are refused until `take_screenshot` succeeds. Image ids and epochs continue across sessions of one conversation (another model, a restarted worker), and the current screen is the last full frame in the history by position.
### Capture, retina and zoom
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index a9c0537a7..f51394376 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -95,6 +95,21 @@ function visionUserMessage(content, tag) {
return { role: 'user', content, webbrainAppOwned: true, webbrainAppOwnedKind: 'vision_frame', webbrainVisionFrame: tag };
}
+// Typing goes to whatever has focus, which only the screen shows.
+const VISION_TYPING_TOOLS = new Set(['type_text', 'press_keys']);
+
+/** Typing refused while the model has not seen the current screen. */
+function visionScreenUnverifiedResult(name, { noFrame = false } = {}) {
+ const verb = name === 'press_keys' ? 'pressed' : 'typed';
+ const why = noFrame
+ ? 'no screenshot of this page has been taken yet'
+ : 'the screen could not be updated after the last action';
+ return {
+ success: false, dispatched: false, noDispatch: true, visionScreenUnverified: true,
+ error: `Not ${verb}: ${why}, so what has focus now is unknown. Call take_screenshot and look first, or click_at the field in the same response before typing.`,
+ };
+}
+
// The page an upload handoff belongs to: its address without the fragment.
function pageKey(url) {
return String(url || '').split('#')[0];
@@ -718,8 +733,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (!observe) return null;
const failed = (stage, error) => {
// The next update is a full frame; until then the model knows the
- // screen it last saw may be out of date.
+ // screen it last saw may be out of date, and its points and typing are
+ // checked against the live screen (_visionTranslateToolCalls).
session.forceKeyframe = true;
+ session.observationFailed = true;
messages.push(visionUserMessage(`[SCREEN: the screen could not be ${stage === 'observe' ? 'captured' : 'prepared'} after the action (${errorText(error).slice(0, 160)}). Call take_screenshot to look again.]`,
{ epoch: session.epoch, kind: 'error' }));
this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });
@@ -844,8 +861,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
*/
proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls, { staleLayout = null } = {}) {
const state = this._visionRunState(tabId);
- if (!state.active || !Array.isArray(toolCalls) || !toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name))) return toolCalls;
+ if (!state.active || !Array.isArray(toolCalls)) return toolCalls;
+ const coordinates = toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name));
+ if (!coordinates && !toolCalls.some(call => VISION_TYPING_TOOLS.has(call?.function?.name))) return toolCalls;
const session = this._visionSessionFor(tabId, state);
+ // No screen of this page yet, or the last update failed: the model does
+ // not know what the screen shows now. Its points are re-checked against
+ // the live screen, and typing waits for a fresh look or a checked click.
+ const unverified = !session.keyframe || session.observationFailed === true;
+ if (!coordinates && !unverified) return toolCalls;
let actedBefore = false;
// Where the last click_at of this batch landed: type_text / press_keys
// after it type into the field that click focused.
@@ -856,7 +880,8 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// A point is re-checked against the live screen when an earlier action
// of the batch ran, or when the page changed while the model decided
// (page feedback notes a response it kept despite that).
- const recheck = actedBefore || (!!call && typeof call === 'object' && this._visionRecheckCalls?.has(call) === true);
+ const recheck = actedBefore || session.observationFailed === true
+ || (!!call && typeof call === 'object' && this._visionRecheckCalls?.has(call) === true);
if (VISION_TRANSLATED_TOOLS.has(name) || VISION_NATIVE_TOOLS.has(name)) {
let args = {};
try { args = typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {}); }
@@ -884,8 +909,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck,
...(point?.ok ? { monitorPoint: { x: point.cssX, y: point.cssY } } : {}) } };
}
- } else if ((name === 'type_text' || name === 'press_keys') && clickedAt) {
+ } else if (VISION_TYPING_TOOLS.has(name) && clickedAt) {
translated = { ...call, _vision: { tool: name, focusFollowUp: { ...clickedAt } } };
+ } else if (VISION_TYPING_TOOLS.has(name) && unverified) {
+ translated = { ...call, _vision: { tool: name, error: visionScreenUnverifiedResult(name, { noFrame: !session.keyframe }) } };
}
if (!VISION_NO_OBSERVE_TOOLS.has(name) && name !== 'take_screenshot') {
actedBefore = true;
@@ -975,6 +1002,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// The point was chosen before the earlier action; without a fresh
// capture nothing shows it is still right.
session.forceKeyframe = true;
+ session.observationFailed = true;
return refuse(`WebBrain could not re-check the screen around ${which}`);
}
if (frame.probe.url !== session.keyframe.url) return refuse(`the page navigated since you read ${which}`);
diff --git a/src/chrome/src/agent/vision-session.js b/src/chrome/src/agent/vision-session.js
index b8dab9f32..919b9e682 100644
--- a/src/chrome/src/agent/vision-session.js
+++ b/src/chrome/src/agent/vision-session.js
@@ -53,6 +53,10 @@ export class VisionSession {
this.patchChainLength = 0;
this.seenPixels = null;
this.forceKeyframe = false;
+ // The last screen update (or a re-check capture) failed: what the model
+ // last saw may no longer be the screen, until the next frame, patch or
+ // "no visible change" (vision-runtime.js gates points and typing on it).
+ this.observationFailed = false;
this.pendingZooms = [];
// Bumped each time the model is shown a screen that differs from the one
// it saw (a patch, a new page or layout, a frame with other pixels), never
@@ -95,6 +99,7 @@ export class VisionSession {
this.changes = [];
this.patchChainLength = 0;
this.forceKeyframe = false;
+ this.observationFailed = false;
const id = this._nextId('f');
for (const retired of this.retired.values()) if (!retired.replacedBy) retired.replacedBy = id;
this.keyframe = {
@@ -143,6 +148,7 @@ export class VisionSession {
registerPatches(regions = [], pixels = null) {
if (!this.keyframe) throw new Error('A keyframe is required before patches.');
this.updateSeq += 1;
+ this.observationFailed = false;
if (regions.length) this.screenVersion += 1;
const seq = this.updateSeq;
const images = regions.map(region => {
@@ -172,7 +178,9 @@ export class VisionSession {
/** The model was told nothing visible changed; the seen frame stays. */
registerNoChange() {
- if (this.keyframe) this.updateSeq += 1;
+ if (!this.keyframe) return;
+ this.updateSeq += 1;
+ this.observationFailed = false;
}
/**
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index a9c0537a7..f51394376 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -95,6 +95,21 @@ function visionUserMessage(content, tag) {
return { role: 'user', content, webbrainAppOwned: true, webbrainAppOwnedKind: 'vision_frame', webbrainVisionFrame: tag };
}
+// Typing goes to whatever has focus, which only the screen shows.
+const VISION_TYPING_TOOLS = new Set(['type_text', 'press_keys']);
+
+/** Typing refused while the model has not seen the current screen. */
+function visionScreenUnverifiedResult(name, { noFrame = false } = {}) {
+ const verb = name === 'press_keys' ? 'pressed' : 'typed';
+ const why = noFrame
+ ? 'no screenshot of this page has been taken yet'
+ : 'the screen could not be updated after the last action';
+ return {
+ success: false, dispatched: false, noDispatch: true, visionScreenUnverified: true,
+ error: `Not ${verb}: ${why}, so what has focus now is unknown. Call take_screenshot and look first, or click_at the field in the same response before typing.`,
+ };
+}
+
// The page an upload handoff belongs to: its address without the fragment.
function pageKey(url) {
return String(url || '').split('#')[0];
@@ -718,8 +733,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
if (!observe) return null;
const failed = (stage, error) => {
// The next update is a full frame; until then the model knows the
- // screen it last saw may be out of date.
+ // screen it last saw may be out of date, and its points and typing are
+ // checked against the live screen (_visionTranslateToolCalls).
session.forceKeyframe = true;
+ session.observationFailed = true;
messages.push(visionUserMessage(`[SCREEN: the screen could not be ${stage === 'observe' ? 'captured' : 'prepared'} after the action (${errorText(error).slice(0, 160)}). Call take_screenshot to look again.]`,
{ epoch: session.epoch, kind: 'error' }));
this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });
@@ -844,8 +861,15 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
*/
proto._visionTranslateToolCalls = function _visionTranslateToolCalls(tabId, toolCalls, { staleLayout = null } = {}) {
const state = this._visionRunState(tabId);
- if (!state.active || !Array.isArray(toolCalls) || !toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name))) return toolCalls;
+ if (!state.active || !Array.isArray(toolCalls)) return toolCalls;
+ const coordinates = toolCalls.some(call => VISION_TOOL_NAMES.has(call?.function?.name));
+ if (!coordinates && !toolCalls.some(call => VISION_TYPING_TOOLS.has(call?.function?.name))) return toolCalls;
const session = this._visionSessionFor(tabId, state);
+ // No screen of this page yet, or the last update failed: the model does
+ // not know what the screen shows now. Its points are re-checked against
+ // the live screen, and typing waits for a fresh look or a checked click.
+ const unverified = !session.keyframe || session.observationFailed === true;
+ if (!coordinates && !unverified) return toolCalls;
let actedBefore = false;
// Where the last click_at of this batch landed: type_text / press_keys
// after it type into the field that click focused.
@@ -856,7 +880,8 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// A point is re-checked against the live screen when an earlier action
// of the batch ran, or when the page changed while the model decided
// (page feedback notes a response it kept despite that).
- const recheck = actedBefore || (!!call && typeof call === 'object' && this._visionRecheckCalls?.has(call) === true);
+ const recheck = actedBefore || session.observationFailed === true
+ || (!!call && typeof call === 'object' && this._visionRecheckCalls?.has(call) === true);
if (VISION_TRANSLATED_TOOLS.has(name) || VISION_NATIVE_TOOLS.has(name)) {
let args = {};
try { args = typeof call.function.arguments === 'string' ? JSON.parse(call.function.arguments || '{}') : (call.function.arguments || {}); }
@@ -884,8 +909,10 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
translated = { ...call, _vision: { tool: name, needsStabilityCheck: recheck,
...(point?.ok ? { monitorPoint: { x: point.cssX, y: point.cssY } } : {}) } };
}
- } else if ((name === 'type_text' || name === 'press_keys') && clickedAt) {
+ } else if (VISION_TYPING_TOOLS.has(name) && clickedAt) {
translated = { ...call, _vision: { tool: name, focusFollowUp: { ...clickedAt } } };
+ } else if (VISION_TYPING_TOOLS.has(name) && unverified) {
+ translated = { ...call, _vision: { tool: name, error: visionScreenUnverifiedResult(name, { noFrame: !session.keyframe }) } };
}
if (!VISION_NO_OBSERVE_TOOLS.has(name) && name !== 'take_screenshot') {
actedBefore = true;
@@ -975,6 +1002,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// The point was chosen before the earlier action; without a fresh
// capture nothing shows it is still right.
session.forceKeyframe = true;
+ session.observationFailed = true;
return refuse(`WebBrain could not re-check the screen around ${which}`);
}
if (frame.probe.url !== session.keyframe.url) return refuse(`the page navigated since you read ${which}`);
diff --git a/src/firefox/src/agent/vision-session.js b/src/firefox/src/agent/vision-session.js
index b8dab9f32..919b9e682 100644
--- a/src/firefox/src/agent/vision-session.js
+++ b/src/firefox/src/agent/vision-session.js
@@ -53,6 +53,10 @@ export class VisionSession {
this.patchChainLength = 0;
this.seenPixels = null;
this.forceKeyframe = false;
+ // The last screen update (or a re-check capture) failed: what the model
+ // last saw may no longer be the screen, until the next frame, patch or
+ // "no visible change" (vision-runtime.js gates points and typing on it).
+ this.observationFailed = false;
this.pendingZooms = [];
// Bumped each time the model is shown a screen that differs from the one
// it saw (a patch, a new page or layout, a frame with other pixels), never
@@ -95,6 +99,7 @@ export class VisionSession {
this.changes = [];
this.patchChainLength = 0;
this.forceKeyframe = false;
+ this.observationFailed = false;
const id = this._nextId('f');
for (const retired of this.retired.values()) if (!retired.replacedBy) retired.replacedBy = id;
this.keyframe = {
@@ -143,6 +148,7 @@ export class VisionSession {
registerPatches(regions = [], pixels = null) {
if (!this.keyframe) throw new Error('A keyframe is required before patches.');
this.updateSeq += 1;
+ this.observationFailed = false;
if (regions.length) this.screenVersion += 1;
const seq = this.updateSeq;
const images = regions.map(region => {
@@ -172,7 +178,9 @@ export class VisionSession {
/** The model was told nothing visible changed; the seen frame stays. */
registerNoChange() {
- if (this.keyframe) this.updateSeq += 1;
+ if (!this.keyframe) return;
+ this.updateSeq += 1;
+ this.observationFailed = false;
}
/**
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index f72f3af64..da27faf04 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -460,6 +460,71 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
const latest = visionMessages(ctx.requests[2].messages).at(-1);
assert.equal(latest.webbrainVisionFrame.kind, 'key', textOf(latest));
assert.match(textOf(latest), /full viewport \(requested\)/);
+ // The next batch's first point was read from a screen that may be out of
+ // date: it was re-checked against the live screen before it went out.
+ const clicks = ctx.dispatched.filter(entry => entry.name === 'click');
+ assert.equal(clicks.length, 2);
+ assert.equal(clicks[0].vision.needsStabilityCheck, false);
+ assert.equal(clicks[1].vision.needsStabilityCheck, true);
+ });
+
+ test(`${build}: after a failed screen update, a point where the screen changed is not clicked`, async () => {
+ let failPatch = true;
+ const ctx = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };
+ // Read from f1, which does not show what the first click opened.
+ if (step === 2) return { toolCalls: [call('click_at', { x: 600, y: 410 })] };
+ if (step === 3) return { toolCalls: [call('done', { summary: 'Stopped.', outcome: 'partial' })] };
+ return { content: 'done' };
+ }, { onDispatch: (name, _args, scene) => { if (name === 'click') scene.rects.push({ x: 560, y: 380, w: 200, h: 120, color: [200, 40, 40] }); } });
+ const encode = ctx.agent._visionEncodeRegion;
+ ctx.agent._visionEncodeRegion = async (canvas, region, options) => {
+ if (region && failPatch) { failPatch = false; throw new Error('convertToBlob failed'); }
+ return encode(canvas, region, options);
+ };
+ await runLoop(ctx.agent, nextTab++, false);
+ assert.equal(ctx.dispatched.filter(entry => entry.name === 'click').length, 1, 'the second point was refused');
+ const refusal = toolTexts(ctx.requests[2]).find(text => /Not dispatched/.test(text)) || '';
+ assert.match(refusal, /the screen changed near this point/, toolTexts(ctx.requests[2]).join('\n'));
+ });
+
+ test(`${build}: typing waits for a look at the screen when there is none or the last update failed`, async () => {
+ let failPatch = false;
+ const ctx = runAgent(step => {
+ // The first capture failed, and so did the update after this step:
+ // nothing shows what has focus.
+ if (step === 1) return { toolCalls: [call('type_text', { text: 'blind' })] };
+ if (step === 2) return { toolCalls: [call('press_keys', { key: 'Enter' })] };
+ if (step === 3) { failPatch = true; return { toolCalls: [call('take_screenshot', {})] }; }
+ if (step === 4) return { toolCalls: [call('click_at', { x: 200, y: 150 })] };
+ // That click's screen update failed.
+ if (step === 5) return { toolCalls: [call('type_text', { text: 'stale' })] };
+ // A click checked against the live screen focuses the field first.
+ if (step === 6) return { toolCalls: [call('click_at', { x: 200, y: 150 }), call('type_text', { text: 'checked' })] };
+ // The update after it succeeded: typing on its own is fine again.
+ if (step === 7) return { toolCalls: [call('type_text', { text: 'seen' })] };
+ if (step === 8) return { toolCalls: [call('done', { summary: 'Typed.', outcome: 'success' })] };
+ return { content: 'done' };
+ }, { maxSteps: 12, onDispatch: (name, _args, scene) => { if (name === 'click') scene.rects.push({ x: 900, y: 700 - scene.rects.length * 30, w: 60, h: 20, color: [0, 0, 200] }); } });
+ const capture = ctx.agent._visionPlatformCapture;
+ let failedCaptures = 0;
+ ctx.agent._visionPlatformCapture = async (tab, options) => {
+ if (failedCaptures < 2) { failedCaptures++; throw new Error('capture failed'); }
+ return capture(tab, options);
+ };
+ const encode = ctx.agent._visionEncodeRegion;
+ ctx.agent._visionEncodeRegion = async (canvas, region, options) => {
+ if (region && failPatch) { failPatch = false; throw new Error('convertToBlob failed'); }
+ return encode(canvas, region, options);
+ };
+ await runLoop(ctx.agent, nextTab++, false);
+ const typed = ctx.dispatched.filter(entry => ['type_text', 'press_keys'].includes(entry.name)).map(entry => entry.args.text || entry.args.key);
+ assert.deepEqual(typed, ['checked', 'seen']);
+ assert.match(toolTexts(ctx.requests[1]).at(-1), /Not typed: no screenshot of this page has been taken yet/);
+ assert.match(toolTexts(ctx.requests[2]).at(-1), /Not pressed: no screenshot/);
+ assert.match(toolTexts(ctx.requests[5]).at(-1), /Not typed: the screen could not be updated after the last action/);
+ const checked = ctx.dispatched.find(entry => entry.name === 'type_text' && entry.args.text === 'checked');
+ assert.deepEqual(Object.keys(checked.vision.focusFollowUp).sort(), ['x', 'y']);
});
test(`${build}: a new run never shows or uses the previous run's frame when its first capture fails`, async () => {
diff --git a/test/vision-mode.mjs b/test/vision-mode.mjs
index 63e60f6ac..fc7bf8cf5 100644
--- a/test/vision-mode.mjs
+++ b/test/vision-mode.mjs
@@ -508,6 +508,23 @@ test('session: numbering continues across sessions and a retired screen refuses
assert.equal(session.resolvePoint({ x: 10, y: 10, image: 'f8' }).code, 'stale_image');
});
+test('session: a failed update stays recorded until the model is shown the screen again', () => {
+ const session = new VisionSession();
+ assert.equal(session.observationFailed, false);
+ session.registerKeyframe({ imageWidth: 100, imageHeight: 50, cssWidth: 100, cssHeight: 50 }, new Uint8ClampedArray(100 * 50 * 4));
+ for (const shown of [
+ () => session.registerNoChange(),
+ () => session.registerPatches([{ x: 0, y: 0, width: 10, height: 10 }], new Uint8ClampedArray(100 * 50 * 4)),
+ () => session.registerKeyframe({ imageWidth: 100, imageHeight: 50, cssWidth: 100, cssHeight: 50 }, new Uint8ClampedArray(100 * 50 * 4)),
+ ]) {
+ session.observationFailed = true;
+ session.registerZoom({ x: 0, y: 0, width: 10, height: 10 }, 20, 20);
+ assert.equal(session.observationFailed, true, 'a zoom is not the screen');
+ shown();
+ assert.equal(session.observationFailed, false);
+ }
+});
+
test('session: an image value that is not an id is refused, not read as the current screen', () => {
const session = new VisionSession();
session.registerKeyframe({ imageWidth: 1000, imageHeight: 500, cssWidth: 2000, cssHeight: 1000 });