Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Vision mode: re-check a drag's end, keep an empty late-tab watch armed

Emre Sokullu19b1e5260730
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index f65b105d2..1f00cf31a 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -47,7 +47,7 @@ A point is refused before any dispatch when:
- its image belongs to an earlier frame (`stale_image`), or the area around it changed in a later patch (`stale_area`);
- it lies outside the image;
-- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate);
+- it is the second or later coordinate action of one model turn and the live screen changed near it after the earlier action, or the screen could not be re-captured to check (`_visionStabilityBlock`, checked right before dispatch, after every gate; a `drag_at` is checked at both ends against one capture);
- the run has no frame of its own yet (`no_frame`).
A refused point also stops the rest of the model's batch: the calls after it (a `type_text`, say) were planned on that point.
@@ -110,7 +110,7 @@ With an instant model the MV3 fixture's four steps (click and type, send, scroll
- the settle wait ends on micro changes instead of running its full ~1 s on every step with a caret or a clock on screen;
- the forced re-observation after page churn and the tree read in feedback are gone (one screen update per step);
-- in Chrome the file-chooser guard's 500 ms settle doubles as the quiet window for a new tab (`watchOpenedTabs` sees tabs as they are created) instead of a further 900 ms poll; a tab the click opens later is still rerouted into the run's tab before the next screen update (`_visionPlatformLateTab`);
+- in Chrome the file-chooser guard's 500 ms settle doubles as the quiet window for a new tab (`watchOpenedTabs` sees tabs as they are created) instead of a further 900 ms poll; a tab the click opens later is still rerouted into the run's tab (`_visionPlatformLateTab`): the watch stays armed for 4 s, a tab that opens while the update's screen settles is rerouted and the screen read again, and one that opens while the model decides is rerouted before the next update;
- the first frame is taken after a cookie-banner decline in progress has finished, so it does not show a banner about to disappear.
## Input dispatch and safety
@@ -150,7 +150,7 @@ npm run test:vision-mode
```
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; in Chrome, a click that opened no tab costing no poll and tabs opened at once or later being rerouted.
+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools, fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.
- `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames.
- `test/quality/vision-mode-mv3.mjs` (`--dpr=2`, `--zoom=1.25`, `--scroll=100`, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form with `click_at` and `type_text` (sent together in one response), scrolls with `scroll_at` and follows a `target=_blank` link; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 17bb8618a..5af5e7655 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -375,7 +375,8 @@ function urlWithoutFragment(url) {
}
// How long after a vision click a tab it opens is still rerouted into the
-// source tab (checked before the next screen update; see watchOpenedTabs).
+// source tab (checked at each screen update while the watch lasts; see
+// watchOpenedTabs and _visionPlatformLateTab).
const VISION_LATE_TAB_WATCH_MS = 4000;
// Tabs that `openerTabId` opens from now on (a target=_blank link or
@@ -20701,11 +20702,15 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
/**
* A tab the last vision click opened after its quiet window (watchOpenedTabs)
* is rerouted into this tab before the next screen update, as one opened at
- * once would have been. `cancel` drops the watch (a new click or run).
+ * once would have been. A watch that has seen no tab yet stays armed until
+ * its deadline, for a tab the page opens while the screen settles or the
+ * model decides (a later update reroutes it); `cancel` drops the watch (a
+ * new click or run).
*/
async _visionPlatformLateTab(tabId, { cancel = false } = {}) {
const entry = this._visionLateTabWatches?.get(tabId);
if (!entry) return null;
+ if (!cancel && !entry.watch.opened.length && entry.watch.listening) return null;
this._visionLateTabWatches.delete(tabId);
clearTimeout(entry.timer);
entry.watch.stop();
diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js
index ac93a391e..cf022a8c9 100644
--- a/src/chrome/src/agent/vision-runtime.js
+++ b/src/chrome/src/agent/vision-runtime.js
@@ -490,12 +490,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });
return { kind: 'error' };
};
- // A tab a click of this batch opened late is rerouted into this tab
- // first, so the update shows what the click opened (Chrome only).
- try { await this._visionPlatformLateTab?.(tabId); } catch (error) { if (isControlFlowError(error)) throw error; }
+ // A tab a click opened late is rerouted into this tab first, so the
+ // update shows what the click opened (Chrome only); one that opens while
+ // the screen settles is rerouted and the screen read again.
+ const rerouteLateTab = async () => {
+ try { return await this._visionPlatformLateTab?.(tabId); }
+ catch (error) { if (isControlFlowError(error)) throw error; return null; }
+ };
+ await rerouteLateTab();
let frame;
try {
frame = await this._visionCaptureSettled(tabId, state);
+ if ((await rerouteLateTab())?.redirected) frame = await this._visionCaptureSettled(tabId, state);
} catch (error) {
if (isControlFlowError(error)) throw error;
return failed('observe', error);
@@ -640,15 +646,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
/**
* Before a coordinate action that follows another action in the same
- * batch: refuse it when the screen changed near its point.
+ * batch: refuse it when the screen changed near its point. `cssPoint` may
+ * be a list (a drag's start and end), checked against one fresh capture;
+ * a point's `label` names it in the refusal.
*/
proto._visionStabilityBlock = async function _visionStabilityBlock(tabId, cssPoint) {
const state = this._visionRunState(tabId);
const session = this._visionSessions?.get(tabId);
if (!state.active || !session?.keyframe || !session.seenPixels) return null;
+ const points = (Array.isArray(cssPoint) ? cssPoint : [cssPoint]).filter(Boolean);
+ const which = points.length > 1 ? 'these points' : 'this point';
const refuse = reason => ({
success: false, dispatched: false, noDispatch: true, visionScreenChanged: true,
- error: `Not dispatched: ${reason}. Look at the updated screen WebBrain shows next and choose the point again.`,
+ error: `Not dispatched: ${reason}. Look at the updated screen WebBrain shows next and choose the ${points.length > 1 ? 'points' : 'point'} again.`,
});
let frame;
try { frame = await this._visionCaptureFrame(tabId, state); }
@@ -657,32 +667,51 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// The point was chosen before the earlier action; without a fresh
// capture nothing shows it is still right.
session.forceKeyframe = true;
- return refuse('WebBrain could not re-check the screen around this point');
+ return refuse(`WebBrain could not re-check the screen around ${which}`);
}
- if (frame.probe.url !== session.keyframe.url) return refuse('the page navigated since you read this point');
- if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse('the viewport changed since you read this point');
+ if (frame.probe.url !== session.keyframe.url) return refuse(`the page navigated since you read ${which}`);
+ if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse(`the viewport changed since you read ${which}`);
const d = diffFrames(session.seenPixels, frame.raster.pixels, frame.size.width, frame.size.height);
- const kfPoint = session.cssToKeyframe(cssPoint.x, cssPoint.y);
- if (d.regionAreaRatio > 0.6 || pointTouchesRegions(kfPoint, d.regions, STABILITY_MARGIN_PX)) {
- return refuse('the screen changed near this point since you read it');
+ if (d.regionAreaRatio > 0.6) return refuse(`most of the screen changed since you read ${which}`);
+ for (const point of points) {
+ const kfPoint = session.cssToKeyframe(point.x, point.y);
+ if (pointTouchesRegions(kfPoint, d.regions, STABILITY_MARGIN_PX)) {
+ return refuse(`the screen changed near ${point.label || 'this point'} since you read it`);
+ }
}
return null;
};
- /** Resolve a model point to CSS for a native tool, with batch stability. */
- proto._visionResolveForTool = async function _visionResolveForTool(tabId, point, vision) {
+ /**
+ * Resolve model points to CSS for a native tool, with batch stability:
+ * every point (a drag's end as much as its start) is re-checked against
+ * the live screen when an earlier action of the batch ran.
+ */
+ proto._visionResolvePointsForTool = async function _visionResolvePointsForTool(tabId, points, vision) {
const state = this._visionRunState(tabId);
if (!state.active) return { error: { success: false, noDispatch: true, error: 'Vision mode is not active for this run.' } };
const session = this._visionSessionFor(tabId, state);
- const resolved = session.resolvePoint(point);
- if (!resolved.ok) return { error: { success: false, dispatched: false, noDispatch: true, visionCoordinateError: resolved.code, error: resolved.error } };
+ const resolved = [];
+ for (const point of points) {
+ const result = session.resolvePoint(point);
+ if (!result.ok) return { error: { success: false, dispatched: false, noDispatch: true, visionCoordinateError: result.code, error: result.error } };
+ resolved.push(result);
+ }
if (vision?.needsStabilityCheck) {
- const blocked = await this._visionStabilityBlock(tabId, { x: resolved.cssX, y: resolved.cssY });
+ const blocked = await this._visionStabilityBlock(tabId, resolved.map((result, index) => ({
+ x: result.cssX, y: result.cssY, ...(points[index].label ? { label: points[index].label } : {}),
+ })));
if (blocked) return { error: blocked };
}
return { resolved, session, state };
};
+ /** Resolve one model point to CSS for a native tool, with batch stability. */
+ proto._visionResolveForTool = async function _visionResolveForTool(tabId, point, vision) {
+ const result = await this._visionResolvePointsForTool(tabId, [point], vision);
+ return result.error ? result : { resolved: result.resolved[0], session: result.session, state: result.state };
+ };
+
/**
* Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. A
* failure becomes a tool result (no dispatch, or an unknown outcome once
@@ -787,11 +816,16 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return { success: result?.success !== false, at: point, ...(result || {}) };
}
if (name === 'drag_at') {
- const start = await this._visionResolveForTool(tabId, { x: args.start_x, y: args.start_y, image: args.image }, vision);
- if (start.error) return start.error;
- const end = session.resolvePoint({ x: args.end_x, y: args.end_y, image: args.image });
- if (!end.ok) return { success: false, dispatched: false, noDispatch: true, visionCoordinateError: end.code, error: end.error };
- const from = { x: start.resolved.cssX, y: start.resolved.cssY };
+ // Both ends are where the model read them: an earlier action of the
+ // batch that changed the drop target refuses the drag as surely as one
+ // that changed its start.
+ const ends = await this._visionResolvePointsForTool(tabId, [
+ { x: args.start_x, y: args.start_y, image: args.image, label: "the drag's start point" },
+ { x: args.end_x, y: args.end_y, image: args.image, label: "the drag's end point" },
+ ], vision);
+ if (ends.error) return ends.error;
+ const [start, end] = ends.resolved;
+ const from = { x: start.cssX, y: start.cssY };
const to = { x: end.cssX, y: end.cssY };
// A press and release on the same control is a click that no click
// guard has seen; a frame WebBrain cannot inspect could hide one.
diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js
index ac93a391e..cf022a8c9 100644
--- a/src/firefox/src/agent/vision-runtime.js
+++ b/src/firefox/src/agent/vision-runtime.js
@@ -490,12 +490,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
this._visionTraceNote(tabId, 'vision_frame_error', { stage, error: errorText(error) });
return { kind: 'error' };
};
- // A tab a click of this batch opened late is rerouted into this tab
- // first, so the update shows what the click opened (Chrome only).
- try { await this._visionPlatformLateTab?.(tabId); } catch (error) { if (isControlFlowError(error)) throw error; }
+ // A tab a click opened late is rerouted into this tab first, so the
+ // update shows what the click opened (Chrome only); one that opens while
+ // the screen settles is rerouted and the screen read again.
+ const rerouteLateTab = async () => {
+ try { return await this._visionPlatformLateTab?.(tabId); }
+ catch (error) { if (isControlFlowError(error)) throw error; return null; }
+ };
+ await rerouteLateTab();
let frame;
try {
frame = await this._visionCaptureSettled(tabId, state);
+ if ((await rerouteLateTab())?.redirected) frame = await this._visionCaptureSettled(tabId, state);
} catch (error) {
if (isControlFlowError(error)) throw error;
return failed('observe', error);
@@ -640,15 +646,19 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
/**
* Before a coordinate action that follows another action in the same
- * batch: refuse it when the screen changed near its point.
+ * batch: refuse it when the screen changed near its point. `cssPoint` may
+ * be a list (a drag's start and end), checked against one fresh capture;
+ * a point's `label` names it in the refusal.
*/
proto._visionStabilityBlock = async function _visionStabilityBlock(tabId, cssPoint) {
const state = this._visionRunState(tabId);
const session = this._visionSessions?.get(tabId);
if (!state.active || !session?.keyframe || !session.seenPixels) return null;
+ const points = (Array.isArray(cssPoint) ? cssPoint : [cssPoint]).filter(Boolean);
+ const which = points.length > 1 ? 'these points' : 'this point';
const refuse = reason => ({
success: false, dispatched: false, noDispatch: true, visionScreenChanged: true,
- error: `Not dispatched: ${reason}. Look at the updated screen WebBrain shows next and choose the point again.`,
+ error: `Not dispatched: ${reason}. Look at the updated screen WebBrain shows next and choose the ${points.length > 1 ? 'points' : 'point'} again.`,
});
let frame;
try { frame = await this._visionCaptureFrame(tabId, state); }
@@ -657,32 +667,51 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
// The point was chosen before the earlier action; without a fresh
// capture nothing shows it is still right.
session.forceKeyframe = true;
- return refuse('WebBrain could not re-check the screen around this point');
+ return refuse(`WebBrain could not re-check the screen around ${which}`);
}
- if (frame.probe.url !== session.keyframe.url) return refuse('the page navigated since you read this point');
- if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse('the viewport changed since you read this point');
+ if (frame.probe.url !== session.keyframe.url) return refuse(`the page navigated since you read ${which}`);
+ if (frame.size.width !== session.keyframe.width || frame.size.height !== session.keyframe.height) return refuse(`the viewport changed since you read ${which}`);
const d = diffFrames(session.seenPixels, frame.raster.pixels, frame.size.width, frame.size.height);
- const kfPoint = session.cssToKeyframe(cssPoint.x, cssPoint.y);
- if (d.regionAreaRatio > 0.6 || pointTouchesRegions(kfPoint, d.regions, STABILITY_MARGIN_PX)) {
- return refuse('the screen changed near this point since you read it');
+ if (d.regionAreaRatio > 0.6) return refuse(`most of the screen changed since you read ${which}`);
+ for (const point of points) {
+ const kfPoint = session.cssToKeyframe(point.x, point.y);
+ if (pointTouchesRegions(kfPoint, d.regions, STABILITY_MARGIN_PX)) {
+ return refuse(`the screen changed near ${point.label || 'this point'} since you read it`);
+ }
}
return null;
};
- /** Resolve a model point to CSS for a native tool, with batch stability. */
- proto._visionResolveForTool = async function _visionResolveForTool(tabId, point, vision) {
+ /**
+ * Resolve model points to CSS for a native tool, with batch stability:
+ * every point (a drag's end as much as its start) is re-checked against
+ * the live screen when an earlier action of the batch ran.
+ */
+ proto._visionResolvePointsForTool = async function _visionResolvePointsForTool(tabId, points, vision) {
const state = this._visionRunState(tabId);
if (!state.active) return { error: { success: false, noDispatch: true, error: 'Vision mode is not active for this run.' } };
const session = this._visionSessionFor(tabId, state);
- const resolved = session.resolvePoint(point);
- if (!resolved.ok) return { error: { success: false, dispatched: false, noDispatch: true, visionCoordinateError: resolved.code, error: resolved.error } };
+ const resolved = [];
+ for (const point of points) {
+ const result = session.resolvePoint(point);
+ if (!result.ok) return { error: { success: false, dispatched: false, noDispatch: true, visionCoordinateError: result.code, error: result.error } };
+ resolved.push(result);
+ }
if (vision?.needsStabilityCheck) {
- const blocked = await this._visionStabilityBlock(tabId, { x: resolved.cssX, y: resolved.cssY });
+ const blocked = await this._visionStabilityBlock(tabId, resolved.map((result, index) => ({
+ x: result.cssX, y: result.cssY, ...(points[index].label ? { label: points[index].label } : {}),
+ })));
if (blocked) return { error: blocked };
}
return { resolved, session, state };
};
+ /** Resolve one model point to CSS for a native tool, with batch stability. */
+ proto._visionResolveForTool = async function _visionResolveForTool(tabId, point, vision) {
+ const result = await this._visionResolvePointsForTool(tabId, [point], vision);
+ return result.error ? result : { resolved: result.resolved[0], session: result.session, state: result.state };
+ };
+
/**
* Execute take_screenshot, zoom, scroll_at, hover_at, drag_at and wait. A
* failure becomes a tool result (no dispatch, or an unknown outcome once
@@ -787,11 +816,16 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {
return { success: result?.success !== false, at: point, ...(result || {}) };
}
if (name === 'drag_at') {
- const start = await this._visionResolveForTool(tabId, { x: args.start_x, y: args.start_y, image: args.image }, vision);
- if (start.error) return start.error;
- const end = session.resolvePoint({ x: args.end_x, y: args.end_y, image: args.image });
- if (!end.ok) return { success: false, dispatched: false, noDispatch: true, visionCoordinateError: end.code, error: end.error };
- const from = { x: start.resolved.cssX, y: start.resolved.cssY };
+ // Both ends are where the model read them: an earlier action of the
+ // batch that changed the drop target refuses the drag as surely as one
+ // that changed its start.
+ const ends = await this._visionResolvePointsForTool(tabId, [
+ { x: args.start_x, y: args.start_y, image: args.image, label: "the drag's start point" },
+ { x: args.end_x, y: args.end_y, image: args.image, label: "the drag's end point" },
+ ], vision);
+ if (ends.error) return ends.error;
+ const [start, end] = ends.resolved;
+ const from = { x: start.cssX, y: start.cssY };
const to = { x: end.cssX, y: end.cssY };
// A press and release on the same control is a click that no click
// guard has seen; a frame WebBrain cannot inspect could hide one.
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index fb2b35381..709678f63 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -247,6 +247,29 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.ok(results.some(text => /screen changed near this point/.test(text)), results.join('\n'));
});
+ test(`${label}: a drag after an earlier action of the batch is re-checked at both ends`, async () => {
+ // The click draws a block at CSS 650–810 x 420–500. The drag starts far
+ // from it; only its end (model 650,420 = CSS 722,467) sits on the block.
+ for (const [end, refused] of [[{ end_x: 650, end_y: 420 }, true], [{ end_x: 1100, end_y: 150 }, false]]) {
+ const { agent, requests, platform } = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 200, y: 150 }), call('drag_at', { start_x: 100, start_y: 700, ...end })] };
+ if (step === 2) return { toolCalls: [call('done', { summary: 'Done.', outcome: 'partial' })] };
+ return { content: 'done' };
+ }, { onDispatch: (_name, _args, scene) => { scene.rects.push({ x: 650, y: 420, w: 160, h: 80, color: [0, 160, 0] }); } });
+ const tab = nextTab++;
+ await runLoop(agent, tab, streaming);
+ const drags = platform.filter(entry => entry.kind === 'drag');
+ const results = requests[1].messages.filter(message => message.role === 'tool').map(message => String(message.content));
+ if (refused) {
+ assert.equal(drags.length, 0, JSON.stringify(platform));
+ assert.ok(results.some(text => /screen changed near the drag's end point since you read it.*choose the points again/.test(text)), results.join('\n'));
+ } else {
+ assert.equal(drags.length, 1, results.join('\n'));
+ assert.ok(!results.some(text => /Not dispatched/.test(text)), results.join('\n'));
+ }
+ }
+ });
+
test(`${label}: scroll_at, hover_at, drag_at and zoom map points through the session`, async () => {
const { agent, requests, platform } = runAgent(step => {
if (step === 1) return { toolCalls: [call('scroll_at', { direction: 'down', amount: 3, x: 648, y: 405 })] };
@@ -642,6 +665,49 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
Object.assign(api.tabs, saved);
}
});
+
+ test('chrome: a late-tab watch that has seen nothing stays armed, and a tab opened while the screen settles is rerouted before the update', async () => {
+ const { agent, scene } = runAgent(() => ({ content: 'done' }));
+ agent._effectiveRunMode = () => 'act';
+ const tab = nextTab++;
+ const messages = [];
+ await agent._visionEnsureKeyframe(tab, messages, () => {});
+ assert.equal(visionMessages(messages).length, 1, 'a first frame');
+ const redirects = [];
+ agent._redirectTargetBlankClick = async (tabId, beforeTabIds) => {
+ redirects.push({ tabId, before: [...beforeTabIds] });
+ scene.url = 'https://news.example.test/late';
+ scene.rects = [{ x: 0, y: 0, w: 1440, h: 900, color: [10, 10, 10] }];
+ return { redirected: true, url: scene.url };
+ };
+ const watch = { opened: [], listening: true, stop() { this.listening = false; } };
+ agent._visionLateTabWatches = new Map([[tab, { watch, beforeTabIds: new Set([tab]), timer: null }]]);
+ // Nothing opened yet: the update goes ahead and the watch stays armed.
+ assert.equal(await agent._visionPlatformLateTab(tab), null);
+ assert.equal(agent._visionLateTabWatches.get(tab)?.watch, watch);
+ assert.equal(watch.listening, true);
+ // The page opens the tab while the update's screen settles.
+ const settle = agent._visionCaptureSettled;
+ let captures = 0;
+ agent._visionCaptureSettled = async function (...args) {
+ const frame = await settle.apply(this, args);
+ if (++captures === 1) watch.opened.push(4242);
+ return frame;
+ };
+ const update = await agent._visionObserveAfterBatch(tab, messages, () => {}, { toolCalls: [call('click_at', { x: 1, y: 1 })] });
+ assert.deepEqual(redirects, [{ tabId: tab, before: [tab] }], 'rerouted before the update');
+ assert.equal(captures, 2, 'and the screen read again');
+ assert.equal(update?.kind, 'keyframe', JSON.stringify(update));
+ assert.match(textOf(visionMessages(messages).at(-1)), /full viewport \(page changed\)/);
+ assert.equal(agent._visionLateTabWatches.has(tab), false, 'a rerouted watch is done');
+ assert.equal(watch.listening, false);
+ // A watch past its deadline that saw nothing is dropped, rerouting nothing.
+ const expired = { opened: [], listening: false, stop() {} };
+ agent._visionLateTabWatches.set(tab, { watch: expired, beforeTabIds: new Set([tab]), timer: null });
+ assert.equal(await agent._visionPlatformLateTab(tab), null);
+ assert.equal(agent._visionLateTabWatches.has(tab), false);
+ assert.equal(redirects.length, 1);
+ });
}
// A scene with something that never stops moving: `tick` repaints a block