Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Vision mode: stop the Chrome flicker on every screenshot
Barack Sokullu0fe752e3df94
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 957be5873..894f859a2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,6 +12,10 @@ This changelog was generated from the repository Git history and release tags. V### Changed- The agent cursor no longer disappears three seconds after an action. It now keeps one, smaller size: it glides from target to target, presses once with a ripple when WebBrain clicks, and between actions stays where it last acted, slowly fading in and out until the task ends. The ring around the target still fades after a moment. Chrome and Firefox.+### Fixed+- Vision mode no longer makes the page flicker in Chrome. Each screenshot made Chrome redraw the tab shrunk into its top-left corner for a moment, several times after every action; WebBrain now captures the tab as it is on screen and crops and scales the image itself. Only a `zoom` that needs more detail than the screen has, as most do on a display at 100% scaling, still re-renders the page for its capture. Right-to-left pages wider than the window are now captured as they appear rather than as a blank frame.+- While the screen settles after an action, WebBrain's border, cursor and Stop button now disappear once and come back after the last screenshot, instead of blinking around each one, and the Stop button keeps its layout afterwards. Chrome and Firefox.+## [44.0.0] - 2026-10-27### Addeddiff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 01ff6eed0..3ea72fd33 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -84,7 +84,9 @@ Every run starts on a frame it captured itself: `_beginVisionRun` retires the prAn image never has more pixels than the CSS viewport: a retina capture is downscaled locally, and `zoom` is the path to fine detail. WebP is ~30% smaller than JPEG for UI screenshots, so uploads are faster; providers bill images by pixels, so the token cost is the same.-Chrome captures with CDP `Page.captureScreenshot`. Two properties of that call drive the implementation (verified in `test/vision-pipeline-browser.mjs`):+Chrome captures with CDP `Page.captureScreenshot` **without a clip**: the tab's surface as it is on screen, `devicePixelRatio` image pixels per CSS pixel from the visual viewport's top left (Chromium keeps the page scrollbar on the right, in right-to-left pages too). The visual viewport, or a zoom's region, is cut out and scaled to the plan locally (`cropImage` in `vision-encoder.js`), which also leaves classic scrollbars out. A clip would have the GPU do the scaling, but Chromium serves one by resizing the tab's view to clip Γ scale and rendering the page into it until the capture is done: a visible tab shows a shrunken page in its top-left corner, white around it, on every capture of every screen update. `test/vision-pipeline-browser.mjs` watches the screencast for that resize.++A clip is still used where only a re-render can serve: a `zoom` that asks for more detail than the screen has (more image pixels per CSS pixel than the DPR, as most zooms at DPR 1 do), and a pinch-zoomed page, whose surface is not the visual viewport. Two properties of a clip drive that path (verified in the same test):- **Clip coordinates are document coordinates.** A clip at `0,0` on a scrolled page captures the unrendered top of the document, a blank frame. The clip starts at the visual viewport's page offset (`Page.getLayoutMetrics`).- **The clip is in device-independent pixels**, CSS pixels times the browser zoom (`Page.getLayoutMetrics().cssVisualViewport.zoom`), and **the output is clip Γ scale Γ device scale factor.** With the DPR (device scale factor Γ zoom) divided out of the scale, the image comes back at the planned size; whatever size actually comes back is resampled to the plan (`rasterizeFrame`), so emulated DPRs and compositor quirks cannot change the coordinate map. The legacy auto-screenshot clip (`_viewportCaptureClip`) applies the same zoom factor.@@ -95,7 +97,7 @@ Screenshot redaction maps element boxes into the captured visual-viewport box (p### Diffs-After a batch WebBrain waits until two consecutive captures match (bounded, ~1 s; a micro change between them, a few 16 px cells such as a blinking caret, a clock or a spinner, counts as a match, `isMicroScreenChange`, since it never stops), then compares the new frame with what the model has seen (`vision-diff.js`): 16 px cells, a per-pixel threshold that ignores encoder noise and caret blinks, nearby changes merged, at most 4 padded regions. `planFrameUpdate` sends:+After a batch WebBrain waits until two consecutive captures match (bounded, ~1 s; a micro change between them, a few 16 px cells such as a blinking caret, a clock or a spinner, counts as a match, `isMicroScreenChange`, since it never stops). Its own page UI (the activity effect, cursor and Stop button) is hidden once for all of those captures and shown again after the last, rather than around each one, so it does not blink on every step; each capture still asks, which re-hides the UI of a page that navigated in between (`_withIndicatorsHidden`). It then compares the new frame with what the model has seen (`vision-diff.js`): 16 px cells, a per-pixel threshold that ignores encoder noise and caret blinks, nearby changes merged, at most 4 padded regions. `planFrameUpdate` sends:- **nothing but a one-line note** when nothing changed: "no visible change" is also the clearest signal that a click missed;- **patches** for local changes (typing, a menu, a toast). For a provider with an image grid (28 px for Qwen2/2.5-VL, UI-TARS, GLM-4.xV, InternVL and local servers; 32 px for Qwen3-VL) each patch is grown to multiples of the grid, like frames and zooms, so the server never resizes it and its pixels map 1:1 to the frame;diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 85c7f34c7..b27be85c8 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -19538,23 +19538,37 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d* chrome-extension:// / file:// pages), the sendMessage rejects and* we capture as-is. Same on tabs the user opened before the extension* had a chance to inject.+ *+ * Calls for a tab nest: a vision screen update holds one around all of+ * its captures (vision-runtime.js _visionCaptureSettled). Every call+ * still hides, which a hidden indicator ignores but a page that+ * navigated in between needs, and the indicator comes back once, when+ * the last call ends, instead of blinking around every capture.*/async _withIndicatorsHidden(tabId, fn) {- let needsRestore = false;- try {- await chrome.tabs.sendMessage(tabId, { type: 'WB_HIDE_FOR_TOOL_USE' });- needsRestore = true;- // One paint frame for Chrome to apply the display:none the- // content script just set, before CDP grabs the surface.- await new Promise((r) => setTimeout(r, 16));- } catch { /* content script absent β that's fine, no UI to hide */ }+ const holds = (this._indicatorHolds ??= new Map());+ const hold = holds.get(tabId) || { count: 0, hidden: false };+ hold.count += 1;+ holds.set(tabId, hold);try {+ try {+ const reply = await chrome.tabs.sendMessage(tabId, { type: 'WB_HIDE_FOR_TOOL_USE' });+ hold.hidden = true;+ // One paint frame for Chrome to apply the display:none the+ // content script just set, before CDP grabs the surface (none+ // when it was hidden already).+ if (reply?.changed !== false) await new Promise((r) => setTimeout(r, 16));+ } catch { /* content script absent β that's fine, no UI to hide */ }return await fn();} finally {- if (needsRestore) {- try {- chrome.tabs.sendMessage(tabId, { type: 'WB_SHOW_AFTER_TOOL_USE' }).catch(() => {});- } catch { /* ignore */ }+ hold.count -= 1;+ if (hold.count === 0) {+ holds.delete(tabId);+ if (hold.hidden) {+ try {+ chrome.tabs.sendMessage(tabId, { type: 'WB_SHOW_AFTER_TOOL_USE' }).catch(() => {});+ } catch { /* ignore */ }+ }}}}@@ -21160,17 +21174,33 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout ddevicePixelRatio: Number(value.dpr) || 1,// Browser zoom (CSS px -> device-independent px); part of the DPR.zoom: Number(visual?.zoom) > 0 ? Number(visual.zoom) : 1,+ // Pinch zoom of the visual viewport (_visionPlatformCapture).+ pageScale: Number(visual?.scale) > 0 ? Number(visual.scale) : 1,url,};}/*** Capture the viewport (or `cssRegion` of it) close to outWidth x- * outHeight. The clip is in device-independent pixels (CSS px x browser- * zoom) and the compositor renders clip x scale x device scale factor, so- * with the DPR (= device scale factor x zoom) divided out of the scale the- * output is outWidth wide. Whatever size actually comes back is resampled- * to the plan by the runtime.+ * outHeight.+ *+ * The tab's surface is captured as it is on screen, and the region is+ * cut out and scaled locally (_visionCropImage). Given a `clip`,+ * Chromium instead resizes the tab's view to clip x scale and re-renders+ * the page into it for the capture (page_handler.cc CaptureScreenshot):+ * a visible tab flashes a shrunken page in its top-left corner, on each+ * of the captures of a screen update. The surface starts at the visual+ * viewport's top left (Chromium keeps the page scrollbar on the right,+ * in RTL pages too) and has devicePixelRatio pixels per CSS pixel.+ *+ * A clip is still used where the surface cannot serve: a zoom that asks+ * for more detail than the screen has (more image pixels per CSS pixel+ * than the DPR), which only a re-render provides, and a pinch-zoomed page.+ * The clip is in device-independent pixels (CSS px x browser zoom) and the+ * compositor renders clip x scale x device scale factor, so with the DPR+ * (= device scale factor x zoom) divided out of the scale the output is+ * outWidth wide. Whatever size actually comes back is resampled to the+ * plan by the runtime.*/async _visionPlatformCapture(tabId, { probe, cssRegion = null, outWidth, outHeight }) {await cdpClient.attach(tabId);@@ -21180,9 +21210,24 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// (vision-runtime.js _visionBounded no longer waits for it).this._throwIfAborted(this._runAbortSignal(tabId));const region = cssRegion || { x: 0, y: 0, width: probe.cssWidth, height: probe.cssHeight };+ const width = Math.max(1, region.width);+ const height = Math.max(1, region.height);const dpr = Math.max(0.25, Number(probe?.devicePixelRatio) || 1);+ const pinched = Math.abs((Number(probe?.pageScale) || 1) - 1) > 0.01;+ // 2% slack: a zoom planned at the DPR itself is a crop.+ const needsDetail = !!cssRegion && Number(outWidth) > width * dpr * 1.02;+ if (!pinched && !needsDetail) {+ const shot = await this._withIndicatorsHidden(tabId, () => cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {+ format: 'jpeg',+ quality: 90,+ fromSurface: true,+ }));+ if (!shot?.data) return null;+ return this._visionCropImage(`data:image/jpeg;base64,${shot.data}`,+ { x: region.x * dpr, y: region.y * dpr, width: width * dpr, height: height * dpr }, outWidth, outHeight);+ }const zoom = Math.max(0.25, Number(probe?.zoom) || 1);- const scale = Math.max(0.05, Math.min(4, Number(outWidth) / (Math.max(1, region.width) * dpr)));+ const scale = Math.max(0.05, Math.min(4, Number(outWidth) / (width * dpr)));const shot = await this._withIndicatorsHidden(tabId, () => cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {format: 'jpeg',quality: 90,@@ -21190,8 +21235,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dclip: {x: ((Number(probe?.scrollX) || 0) + region.x) * zoom,y: ((Number(probe?.scrollY) || 0) + region.y) * zoom,- width: Math.max(1, region.width) * zoom,- height: Math.max(1, region.height) * zoom,+ width: width * zoom,+ height: height * zoom,scale,},}));diff --git a/src/chrome/src/agent/vision-encoder.js b/src/chrome/src/agent/vision-encoder.jsindex dfe1407d9..21aae4505 100644--- a/src/chrome/src/agent/vision-encoder.js+++ b/src/chrome/src/agent/vision-encoder.js@@ -19,8 +19,8 @@ export function createCanvas(width, height) {throw new Error('No canvas implementation is available for vision screenshots.');}-function context2d(canvas) {- const ctx = canvas.getContext('2d', { willReadFrequently: true });+function context2d(canvas, { willReadFrequently = true } = {}) {+ const ctx = canvas.getContext('2d', { willReadFrequently });if (!ctx) throw new Error('The browser could not create a 2D canvas context.');return ctx;}@@ -104,6 +104,34 @@ async function canvasToBlob(canvas, type, quality) {});}+/**+ * Cut `rect` (source pixels, fractions allowed) out of an encoded image and+ * scale it to `width`x`height`, as a data URL. Chrome's vision capture takes+ * the tab's surface as it is on screen and crops it here (agent+ * _visionPlatformCapture): a CDP clip instead resizes the tab's view while+ * the capture renders, which a visible tab shows as a flash.+ */+export async function cropImage(source, rect, width, height, { mimeType = 'image/jpeg', quality = 0.92 } = {}) {+ const image = await decodeImage(source);+ try {+ const decoded = sourceSize(image);+ const x = Math.min(Math.max(0, Number(rect?.x) || 0), Math.max(0, decoded.width - 1));+ const y = Math.min(Math.max(0, Number(rect?.y) || 0), Math.max(0, decoded.height - 1));+ const w = Math.max(1, Math.min(decoded.width - x, Number(rect?.width) || decoded.width));+ const h = Math.max(1, Math.min(decoded.height - y, Number(rect?.height) || decoded.height));+ const canvas = createCanvas(width || w, height || h);+ // Drawn once and encoded, never read back: no CPU-backed canvas needed.+ const ctx = context2d(canvas, { willReadFrequently: false });+ ctx.imageSmoothingEnabled = true;+ if ('imageSmoothingQuality' in ctx) ctx.imageSmoothingQuality = 'high';+ ctx.drawImage(image, x, y, w, h, 0, 0, canvas.width, canvas.height);+ const blob = await canvasToBlob(canvas, mimeType, quality);+ return `data:${blob.type || mimeType};base64,${arrayBufferToBase64(await blob.arrayBuffer())}`;+ } finally {+ try { image?.close?.(); } catch {}+ }+}+let webpSupport = null;/** Whether this browser's canvas can encode WebP (Safari cannot). */export async function supportsWebpEncoding() {diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.jsindex 87f8404ed..cf9cd6d1a 100644--- a/src/chrome/src/agent/vision-runtime.js+++ b/src/chrome/src/agent/vision-runtime.js@@ -21,7 +21,7 @@ import {VISION_PROFILES, planVisionImageSize, planZoomImageSize, estimateVisionTokens, visionEncodingFor,normalizeVisionQuality,} from './vision-budget.js';-import { rasterizeFrame, encodeCanvasRegion, supportsWebpEncoding } from './vision-encoder.js';+import { rasterizeFrame, encodeCanvasRegion, cropImage, supportsWebpEncoding } from './vision-encoder.js';import {VISION_NATIVE_TOOLS, VISION_TRANSLATED_TOOLS, VISION_NO_OBSERVE_TOOLS, VISION_TOOL_NAMES,VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,@@ -647,6 +647,11 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return encodeCanvasRegion(canvas, region, options);};+ /** Chrome's capture cuts the viewport out of the tab surface here (agent _visionPlatformCapture). */+ proto._visionCropImage = function _visionCropImage(dataUrl, rect, width, height) {+ return cropImage(dataUrl, rect, width, height);+ };+/*** Runs one platform read (`what` names it in the error) against the run's* Stop and VISION_CAPTURE_DEADLINE_MS (`_visionCaptureDeadlineMs` overrides@@ -737,21 +742,26 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {/*** Capture until two consecutive frames match (bounded). Micro changes (a* caret, a clock, a spinner) count as a match: they never stop, and waiting- * for them only cost the full settle time on every step.+ * for them only cost the full settle time on every step. WebBrain's own+ * page UI (border, cursor, Stop button) is hidden once for all of the+ * captures, not around each (agent _withIndicatorsHidden), so it does not+ * blink on every step.*/proto._visionCaptureSettled = async function _visionCaptureSettled(tabId, state) {await sleep(SETTLE_INITIAL_DELAY_MS);- let frame = await this._visionCaptureFrame(tabId, state);- for (let attempt = 0; attempt < SETTLE_MAX_EXTRA_CAPTURES; attempt++) {- await sleep(SETTLE_POLL_MS);- let next;- try { next = await this._visionCaptureFrame(tabId, state); } catch { break; }- const sameShape = next.size.width === frame.size.width && next.size.height === frame.size.height;- const moving = !sameShape || !isMicroScreenChange(diffFrames(frame.raster.pixels, next.raster.pixels, next.size.width, next.size.height));- frame = next;- if (!moving) break;- }- return frame;+ return this._withIndicatorsHidden(tabId, async () => {+ let frame = await this._visionCaptureFrame(tabId, state);+ for (let attempt = 0; attempt < SETTLE_MAX_EXTRA_CAPTURES; attempt++) {+ await sleep(SETTLE_POLL_MS);+ let next;+ try { next = await this._visionCaptureFrame(tabId, state); } catch { break; }+ const sameShape = next.size.width === frame.size.width && next.size.height === frame.size.height;+ const moving = !sameShape || !isMicroScreenChange(diffFrames(frame.raster.pixels, next.raster.pixels, next.size.width, next.size.height));+ frame = next;+ if (!moving) break;+ }+ return frame;+ });};/**diff --git a/src/chrome/src/content/agent-visual-indicator.js b/src/chrome/src/content/agent-visual-indicator.jsindex 413d43c22..f2c09eb09 100644--- a/src/chrome/src/content/agent-visual-indicator.js+++ b/src/chrome/src/content/agent-visual-indicator.js@@ -1791,7 +1791,8 @@if (styleLoaded) startEffect();if (stopContainerEl) {- stopContainerEl.style.display = '';+ // The container lays its button out as flex (createStopButton).+ stopContainerEl.style.display = 'flex';} else {stopContainerEl = createStopButton();root.appendChild(stopContainerEl);@@ -1852,9 +1853,12 @@/*** Hide everything without tearing it down β for the brief window around a* screenshot capture, so the agent doesn't see its own effect or cursor in- * the screenshots it sends back to the model.+ * the screenshots it sends back to the model. The captures of one screen+ * update each ask while it is already hidden: a repeat keeps what the+ * first call saved. Returns whether anything was hidden.*/function hideForToolUse() {+ if (suspendedForToolUse) return false;suspendedForToolUse = true;savedStopVisible = !!(stopContainerEl && stopContainerEl.style.display !== 'none');savedTargetOutlineVisible = !!(targetOutlineEl && targetOutlineEl.style.display !== 'none');@@ -1862,13 +1866,14 @@cursor?.suspend();if (stopContainerEl) stopContainerEl.style.display = 'none';if (targetOutlineEl) targetOutlineEl.style.display = 'none';+ return true;}function showAfterToolUse() {suspendedForToolUse = false;effect?.resume();cursor?.resume();- if (savedStopVisible && stopContainerEl) stopContainerEl.style.display = '';+ if (savedStopVisible && stopContainerEl) stopContainerEl.style.display = 'flex';if (savedTargetOutlineVisible && targetOutlineEl) targetOutlineEl.style.display = '';savedStopVisible = false;savedTargetOutlineVisible = false;@@ -1894,8 +1899,7 @@sendResponse({ ok: true, active: indicatorsActive });break;case 'WB_HIDE_FOR_TOOL_USE':- hideForToolUse();- sendResponse({ ok: true });+ sendResponse({ ok: true, changed: hideForToolUse() });break;case 'WB_SHOW_AFTER_TOOL_USE':showAfterToolUse();diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 62ffc336d..72ba562fe 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -18617,23 +18617,37 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d* Best-effort: if the content script isn't loaded (about:* / file://* pages, or pre-existing tabs that loaded before the extension), the* sendMessage rejects and we capture as-is.+ *+ * Calls for a tab nest: a vision screen update holds one around all of+ * its captures (vision-runtime.js _visionCaptureSettled). Every call+ * still hides, which a hidden indicator ignores but a page that+ * navigated in between needs, and the indicator comes back once, when+ * the last call ends, instead of blinking around every capture.*/async _withIndicatorsHidden(tabId, fn) {- let needsRestore = false;- try {- await browser.tabs.sendMessage(tabId, { type: 'WB_HIDE_FOR_TOOL_USE' });- needsRestore = true;- // Give the renderer one paint frame to apply the display:none the- // content script just set, before we capture.- await new Promise((r) => setTimeout(r, 16));- } catch { /* content script absent β capture without hiding */ }+ const holds = (this._indicatorHolds ??= new Map());+ const hold = holds.get(tabId) || { count: 0, hidden: false };+ hold.count += 1;+ holds.set(tabId, hold);try {+ try {+ const reply = await browser.tabs.sendMessage(tabId, { type: 'WB_HIDE_FOR_TOOL_USE' });+ hold.hidden = true;+ // Give the renderer one paint frame to apply the display:none the+ // content script just set, before we capture (none when it was+ // hidden already).+ if (reply?.changed !== false) await new Promise((r) => setTimeout(r, 16));+ } catch { /* content script absent β capture without hiding */ }return await fn();} finally {- if (needsRestore) {- try {- browser.tabs.sendMessage(tabId, { type: 'WB_SHOW_AFTER_TOOL_USE' }).catch(() => {});- } catch { /* ignore */ }+ hold.count -= 1;+ if (hold.count === 0) {+ holds.delete(tabId);+ if (hold.hidden) {+ try {+ browser.tabs.sendMessage(tabId, { type: 'WB_SHOW_AFTER_TOOL_USE' }).catch(() => {});+ } catch { /* ignore */ }+ }}}}diff --git a/src/firefox/src/agent/vision-encoder.js b/src/firefox/src/agent/vision-encoder.jsindex dfe1407d9..21aae4505 100644--- a/src/firefox/src/agent/vision-encoder.js+++ b/src/firefox/src/agent/vision-encoder.js@@ -19,8 +19,8 @@ export function createCanvas(width, height) {throw new Error('No canvas implementation is available for vision screenshots.');}-function context2d(canvas) {- const ctx = canvas.getContext('2d', { willReadFrequently: true });+function context2d(canvas, { willReadFrequently = true } = {}) {+ const ctx = canvas.getContext('2d', { willReadFrequently });if (!ctx) throw new Error('The browser could not create a 2D canvas context.');return ctx;}@@ -104,6 +104,34 @@ async function canvasToBlob(canvas, type, quality) {});}+/**+ * Cut `rect` (source pixels, fractions allowed) out of an encoded image and+ * scale it to `width`x`height`, as a data URL. Chrome's vision capture takes+ * the tab's surface as it is on screen and crops it here (agent+ * _visionPlatformCapture): a CDP clip instead resizes the tab's view while+ * the capture renders, which a visible tab shows as a flash.+ */+export async function cropImage(source, rect, width, height, { mimeType = 'image/jpeg', quality = 0.92 } = {}) {+ const image = await decodeImage(source);+ try {+ const decoded = sourceSize(image);+ const x = Math.min(Math.max(0, Number(rect?.x) || 0), Math.max(0, decoded.width - 1));+ const y = Math.min(Math.max(0, Number(rect?.y) || 0), Math.max(0, decoded.height - 1));+ const w = Math.max(1, Math.min(decoded.width - x, Number(rect?.width) || decoded.width));+ const h = Math.max(1, Math.min(decoded.height - y, Number(rect?.height) || decoded.height));+ const canvas = createCanvas(width || w, height || h);+ // Drawn once and encoded, never read back: no CPU-backed canvas needed.+ const ctx = context2d(canvas, { willReadFrequently: false });+ ctx.imageSmoothingEnabled = true;+ if ('imageSmoothingQuality' in ctx) ctx.imageSmoothingQuality = 'high';+ ctx.drawImage(image, x, y, w, h, 0, 0, canvas.width, canvas.height);+ const blob = await canvasToBlob(canvas, mimeType, quality);+ return `data:${blob.type || mimeType};base64,${arrayBufferToBase64(await blob.arrayBuffer())}`;+ } finally {+ try { image?.close?.(); } catch {}+ }+}+let webpSupport = null;/** Whether this browser's canvas can encode WebP (Safari cannot). */export async function supportsWebpEncoding() {diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.jsindex 87f8404ed..cf9cd6d1a 100644--- a/src/firefox/src/agent/vision-runtime.js+++ b/src/firefox/src/agent/vision-runtime.js@@ -21,7 +21,7 @@ import {VISION_PROFILES, planVisionImageSize, planZoomImageSize, estimateVisionTokens, visionEncodingFor,normalizeVisionQuality,} from './vision-budget.js';-import { rasterizeFrame, encodeCanvasRegion, supportsWebpEncoding } from './vision-encoder.js';+import { rasterizeFrame, encodeCanvasRegion, cropImage, supportsWebpEncoding } from './vision-encoder.js';import {VISION_NATIVE_TOOLS, VISION_TRANSLATED_TOOLS, VISION_NO_OBSERVE_TOOLS, VISION_TOOL_NAMES,VISION_SCROLL_STEP_CSS_PX, VISION_DEFAULT_SCROLL_STEPS, VISION_MAX_WAIT_SECONDS,@@ -647,6 +647,11 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {return encodeCanvasRegion(canvas, region, options);};+ /** Chrome's capture cuts the viewport out of the tab surface here (agent _visionPlatformCapture). */+ proto._visionCropImage = function _visionCropImage(dataUrl, rect, width, height) {+ return cropImage(dataUrl, rect, width, height);+ };+/*** Runs one platform read (`what` names it in the error) against the run's* Stop and VISION_CAPTURE_DEADLINE_MS (`_visionCaptureDeadlineMs` overrides@@ -737,21 +742,26 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) {/*** Capture until two consecutive frames match (bounded). Micro changes (a* caret, a clock, a spinner) count as a match: they never stop, and waiting- * for them only cost the full settle time on every step.+ * for them only cost the full settle time on every step. WebBrain's own+ * page UI (border, cursor, Stop button) is hidden once for all of the+ * captures, not around each (agent _withIndicatorsHidden), so it does not+ * blink on every step.*/proto._visionCaptureSettled = async function _visionCaptureSettled(tabId, state) {await sleep(SETTLE_INITIAL_DELAY_MS);- let frame = await this._visionCaptureFrame(tabId, state);- for (let attempt = 0; attempt < SETTLE_MAX_EXTRA_CAPTURES; attempt++) {- await sleep(SETTLE_POLL_MS);- let next;- try { next = await this._visionCaptureFrame(tabId, state); } catch { break; }- const sameShape = next.size.width === frame.size.width && next.size.height === frame.size.height;- const moving = !sameShape || !isMicroScreenChange(diffFrames(frame.raster.pixels, next.raster.pixels, next.size.width, next.size.height));- frame = next;- if (!moving) break;- }- return frame;+ return this._withIndicatorsHidden(tabId, async () => {+ let frame = await this._visionCaptureFrame(tabId, state);+ for (let attempt = 0; attempt < SETTLE_MAX_EXTRA_CAPTURES; attempt++) {+ await sleep(SETTLE_POLL_MS);+ let next;+ try { next = await this._visionCaptureFrame(tabId, state); } catch { break; }+ const sameShape = next.size.width === frame.size.width && next.size.height === frame.size.height;+ const moving = !sameShape || !isMicroScreenChange(diffFrames(frame.raster.pixels, next.raster.pixels, next.size.width, next.size.height));+ frame = next;+ if (!moving) break;+ }+ return frame;+ });};/**diff --git a/src/firefox/src/content/agent-visual-indicator.js b/src/firefox/src/content/agent-visual-indicator.jsindex a4bd6a65a..00ead662d 100644--- a/src/firefox/src/content/agent-visual-indicator.js+++ b/src/firefox/src/content/agent-visual-indicator.js@@ -1791,7 +1791,8 @@if (styleLoaded) startEffect();if (stopContainerEl) {- stopContainerEl.style.display = '';+ // The container lays its button out as flex (createStopButton).+ stopContainerEl.style.display = 'flex';} else {stopContainerEl = createStopButton();root.appendChild(stopContainerEl);@@ -1852,9 +1853,12 @@/*** Hide everything without tearing it down β for the brief window around a* screenshot capture, so the agent doesn't see its own effect or cursor in- * the screenshots it sends back to the model.+ * the screenshots it sends back to the model. The captures of one screen+ * update each ask while it is already hidden: a repeat keeps what the+ * first call saved. Returns whether anything was hidden.*/function hideForToolUse() {+ if (suspendedForToolUse) return false;suspendedForToolUse = true;savedStopVisible = !!(stopContainerEl && stopContainerEl.style.display !== 'none');savedTargetOutlineVisible = !!(targetOutlineEl && targetOutlineEl.style.display !== 'none');@@ -1862,13 +1866,14 @@cursor?.suspend();if (stopContainerEl) stopContainerEl.style.display = 'none';if (targetOutlineEl) targetOutlineEl.style.display = 'none';+ return true;}function showAfterToolUse() {suspendedForToolUse = false;effect?.resume();cursor?.resume();- if (savedStopVisible && stopContainerEl) stopContainerEl.style.display = '';+ if (savedStopVisible && stopContainerEl) stopContainerEl.style.display = 'flex';if (savedTargetOutlineVisible && targetOutlineEl) targetOutlineEl.style.display = '';savedStopVisible = false;savedTargetOutlineVisible = false;@@ -1894,8 +1899,7 @@sendResponse({ ok: true, active: indicatorsActive });break;case 'WB_HIDE_FOR_TOOL_USE':- hideForToolUse();- sendResponse({ ok: true });+ sendResponse({ ok: true, changed: hideForToolUse() });break;case 'WB_SHOW_AFTER_TOOL_USE':showAfterToolUse();diff --git a/test/page-monitor-ui.mjs b/test/page-monitor-ui.mjsindex 185d94634..35b00dfe0 100644--- a/test/page-monitor-ui.mjs+++ b/test/page-monitor-ui.mjs@@ -2299,10 +2299,21 @@ for (const [build, engine] of [['chrome', chromium], ['firefox', firefox]]) {assert.equal(await page.evaluate(() => document.querySelectorAll('#webbrain-agent-effect').length), 1, 'the previous effect is removed after a switch');assert.equal((await page.evaluate(() => feedback)).length, 0, JSON.stringify(await page.evaluate(() => feedback)));// Screenshot capture hides every part, and nothing intercepts real input.- await page.evaluate(() => messageListeners.forEach(listener => listener({ type: 'WB_HIDE_FOR_TOOL_USE' }, {}, () => {})));- assert.deepEqual(await page.evaluate(() => ['webbrain-agent-effect', 'webbrain-agent-target-cursor', 'webbrain-agent-stop-container']- .map(id => getComputedStyle(document.getElementById(id)).display)), ['none', 'none', 'none']);+ // The captures of one screen update each ask: a repeat changes nothing,+ // and the one show after the last capture brings every part back.+ const parts = () => page.evaluate(() => ['webbrain-agent-effect', 'webbrain-agent-target-cursor', 'webbrain-agent-stop-container']+ .map(id => getComputedStyle(document.getElementById(id)).display));+ const shown = await parts();+ const hideReplies = await page.evaluate(() => [1, 2].map(() => {+ const replies = [];+ messageListeners.forEach(listener => listener({ type: 'WB_HIDE_FOR_TOOL_USE' }, {}, reply => replies.push(reply)));+ return replies.find(reply => reply && 'changed' in reply) || null;+ }));+ assert.deepEqual(hideReplies, [{ ok: true, changed: true }, { ok: true, changed: false }]);+ assert.deepEqual(await parts(), ['none', 'none', 'none']);await page.evaluate(() => messageListeners.forEach(listener => listener({ type: 'WB_SHOW_AFTER_TOOL_USE' }, {}, () => {})));+ assert.deepEqual(await parts(), shown);+ assert.ok(shown.every(display => display !== 'none'), shown.join());await page.locator('#human').click();assert.ok((await page.evaluate(() => feedback)).some(event => event.kind === 'click' && event.source === 'user'));// Reduced motion: the canvas keeps one still frame, and the glow and the ring stop pulsing.diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjsindex a78d005cc..cc52e3abc 100644--- a/test/vision-mode-agent.mjs+++ b/test/vision-mode-agent.mjs@@ -1367,6 +1367,99 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))}});+ test(`${build}: a screen update hides WebBrain's page UI once for all of its captures`, async () => {+ const scene = tickingScene({ x: 300, y: 200, w: 400, h: 300 });+ const { agent } = runAgent(() => ({ content: 'done' }), { scene });+ const tab = nextTab++;+ const state = agent._visionRunState(tab, 'act');+ const tabs = (build === 'chrome' ? globalThis.chrome : globalThis.browser).tabs;+ const send = tabs.sendMessage;+ const log = [];+ let hidden = false;+ // The indicator content script's answers: a hide while hidden changes nothing.+ tabs.sendMessage = async (id, message) => {+ if (id !== tab) return send(id, message);+ log.push(message.type);+ if (message.type === 'WB_HIDE_FOR_TOOL_USE') {+ const changed = !hidden;+ hidden = true;+ return { ok: true, changed };+ }+ if (message.type === 'WB_SHOW_AFTER_TOOL_USE') hidden = false;+ return { ok: true };+ };+ // Each capture hides around itself, as both platform layers do; the page+ // navigates after the second, and the new page shows its UI again.+ const capture = agent._visionPlatformCapture;+ agent._visionPlatformCapture = function (tabId, options) {+ return this._withIndicatorsHidden(tabId, async () => {+ log.push(`capture, hidden: ${hidden}`);+ if (log.filter(entry => entry.startsWith('capture')).length === 2) hidden = false;+ return capture.call(this, tabId, options);+ });+ };+ try {+ await agent._visionCaptureSettled(tab, state);+ } finally {+ tabs.sendMessage = send;+ }+ const captures = log.filter(entry => entry.startsWith('capture'));+ assert.deepEqual(captures, Array(4).fill('capture, hidden: true'), log.join('\n'));+ assert.equal(log.filter(entry => entry === 'WB_SHOW_AFTER_TOOL_USE').length, 1, log.join('\n'));+ assert.equal(log.at(-1), 'WB_SHOW_AFTER_TOOL_USE', 'shown again after the last capture');+ assert.equal(agent._indicatorHolds?.has(tab), false);+ });++ if (build === 'chrome') {+ test('chrome: a vision capture reads the tab surface as it is and crops it; only a zoom needing more detail than the screen has, or a pinch-zoomed page, uses a clip', async () => {+ const { agent } = runAgent(() => ({ content: 'done' }));+ // The real CDP platform capture, not the fake scene's.+ delete agent._visionPlatformCapture;+ const tab = nextTab++;+ const shots = [];+ const crops = [];+ const saved = { attach: cdpClient.attach, sendCommand: cdpClient.sendCommand };+ Object.assign(cdpClient, {+ attach: async () => {},+ sendCommand: async (_tab, method, params) => {+ if (method !== 'Page.captureScreenshot') return {};+ shots.push(params);+ return { data: 'c2hvdA==' };+ },+ });+ agent._visionCropImage = async (dataUrl, rect, width, height) => {+ crops.push({ dataUrl, rect, width, height });+ return 'data:image/jpeg;base64,Y3JvcA==';+ };+ const probe = { cssWidth: 1263, cssHeight: 760, scrollX: 0, scrollY: 600, devicePixelRatio: 2, zoom: 1, pageScale: 1 };+ const region = { x: 100, y: 50, width: 300, height: 200 };+ try {+ // A frame: the surface without a clip (a clip resizes the tab's view+ // for the capture), the visual viewport cut out of it.+ assert.equal(await agent._visionPlatformCapture(tab, { probe, outWidth: 1024, outHeight: 616 }), 'data:image/jpeg;base64,Y3JvcA==');+ assert.deepEqual(shots.at(-1), { format: 'jpeg', quality: 90, fromSurface: true });+ assert.deepEqual(crops.at(-1), { dataUrl: 'data:image/jpeg;base64,c2hvdA==', rect: { x: 0, y: 0, width: 2526, height: 1520 }, width: 1024, height: 616 });+ // A zoom the screen's pixels cover (2 per CSS px at DPR 2) is a crop too.+ await agent._visionPlatformCapture(tab, { probe, cssRegion: region, outWidth: 600, outHeight: 400 });+ assert.equal(shots.at(-1).clip, undefined);+ assert.deepEqual(crops.at(-1).rect, { x: 200, y: 100, width: 600, height: 400 });+ // At DPR 1 the same zoom asks for 1.5 image px per CSS px, which only a re-render has.+ const flat = { ...probe, devicePixelRatio: 1 };+ await agent._visionPlatformCapture(tab, { probe: flat, cssRegion: region, outWidth: 450, outHeight: 300 });+ assert.deepEqual(shots.at(-1).clip, { x: 100, y: 650, width: 300, height: 200, scale: 1.5 });+ // A frame never re-renders, even zoomed out below one device pixel per CSS pixel.+ await agent._visionPlatformCapture(tab, { probe: { ...flat, devicePixelRatio: 0.8, zoom: 0.8 }, outWidth: 1200, outHeight: 722 });+ assert.equal(shots.at(-1).clip, undefined);+ // A pinch-zoomed page: its surface is not the visual viewport.+ await agent._visionPlatformCapture(tab, { probe: { ...probe, pageScale: 2 }, outWidth: 1024, outHeight: 616 });+ assert.ok(shots.at(-1).clip, JSON.stringify(shots.at(-1)));+ assert.equal(crops.length, 3);+ } finally {+ Object.assign(cdpClient, saved);+ }+ });+ }+test(`${build}: a requested frame verifies an action only once it was captured`, async () => {let failing = false;const ctx = runAgent(step => {diff --git a/test/vision-pipeline-browser.mjs b/test/vision-pipeline-browser.mjsindex e439aaeb3..7e964b285 100644--- a/test/vision-pipeline-browser.mjs+++ b/test/vision-pipeline-browser.mjs@@ -35,6 +35,15 @@ const FIXTURE = `<!doctype html><meta charset="utf-8"><title>Vision fixture</tit});</script>`;+// A right-to-left page wider than the window opens scrolled to its right end.+const RTL_FIXTURE = `<!doctype html><html dir="rtl"><meta charset="utf-8"><title>RTL fixture</title>+<style>+ body { margin: 0; width: 3000px; height: 3000px; background: #fff; }+ #start { position: fixed; left: 0; top: 0; width: 60px; height: 60px; background: rgb(0, 200, 0); }+ #end { position: fixed; right: 0; top: 0; width: 60px; height: 60px; background: rgb(200, 0, 200); }+</style>+<div id="start"></div><div id="end"></div></html>`;+function serve() {const server = http.createServer((request, response) => {if (request.url.startsWith('/agent/')) {@@ -45,7 +54,7 @@ function serve() {return;}response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });- response.end(FIXTURE);+ response.end(request.url === '/rtl' ? RTL_FIXTURE : FIXTURE);});return new Promise(resolve => server.listen(0, '127.0.0.1', () => resolve(server)));}@@ -86,14 +95,16 @@ for (const dpr of [1, 2, 3]) {}, { probe });assert.ok(plan.width < probe.cssWidth, `economy downscales: ${JSON.stringify(plan)} for ${probe.cssWidth}`);- // Capture exactly like _visionPlatformCapture.+ // Capture exactly like _visionPlatformCapture: the surface as it is on+ // screen, the visual viewport cut out of it and scaled to the plan.const capture = async () => {- const scale = plan.width / (probe.cssWidth * probe.devicePixelRatio);- const shot = await cdp.send('Page.captureScreenshot', {- format: 'jpeg', quality: 90, fromSurface: true,- clip: { x: probe.scrollX, y: probe.scrollY, width: probe.cssWidth, height: probe.cssHeight, scale },- });- return `data:image/jpeg;base64,${shot.data}`;+ const shot = await cdp.send('Page.captureScreenshot', { format: 'jpeg', quality: 90, fromSurface: true });+ return page.evaluate(async ({ data, plan, probe }) => {+ const encoder = await import('/agent/vision-encoder.js');+ const dpr = probe.devicePixelRatio;+ return encoder.cropImage(`data:image/jpeg;base64,${data}`,+ { x: 0, y: 0, width: probe.cssWidth * dpr, height: probe.cssHeight * dpr }, plan.width, plan.height);+ }, { data: shot.data, plan, probe });};const first = await capture();const analysis = await page.evaluate(async ({ dataUrl, plan, probe }) => {@@ -116,11 +127,11 @@ for (const dpr of [1, 2, 3]) {jpeg: { bytes: jpeg.bytes, mime: jpeg.mimeType },};}, { dataUrl: first, plan, probe });- // The compositor honoured the DPR-divided scale (or the runtime resampled).+ // The crop came back at the planned size.assert.deepEqual(analysis.size, { width: plan.width, height: plan.height });- assert.ok(Math.abs(analysis.decoded.width - plan.width) <= 2, `capture came back at the planned size: ${JSON.stringify(analysis.decoded)} vs ${plan.width}`);+ assert.deepEqual(analysis.decoded, { width: plan.width, height: plan.height });// The pixel at the mapped point is the blue target, not white page: the- // clip used document coordinates of the scrolled viewport.+ // surface shows the scrolled viewport, from its top left.const [r, g, b] = analysis.rgb;assert.ok(b > 180 && r < 80 && g > 80 && g < 170, `mapped pixel is the target button: ${analysis.rgb}`);assert.equal(analysis.webp.mime, 'image/webp');@@ -165,6 +176,88 @@ for (const dpr of [1, 2, 3]) {});}+test('a vision capture leaves the tab on screen as it is, where a clipped capture shrinks it', async () => {+ // DPR 2, where the clip's scale is smallest. The screencast reports every+ // frame the tab shows; an animation keeps them coming during captures.+ const browser = await chromium.launch({ channel: 'chromium', args: ['--force-device-scale-factor=2', '--window-size=1280,860'] });+ try {+ const context = await browser.newContext({ viewport: null });+ const page = await context.newPage();+ await page.setContent(`<!doctype html><style>body { margin: 0; height: 3000px; }+ #spin { position: fixed; left: 300px; top: 200px; width: 40px; height: 40px; background: #c0c; animation: spin 1s linear infinite; }+ @keyframes spin { to { transform: rotate(360deg); } }</style><div id="spin"></div>`);+ await page.evaluate(() => window.scrollTo(0, 400));+ const cdp = await context.newCDPSession(page);+ const { cssVisualViewport: vv } = await cdp.send('Page.getLayoutMetrics');+ const frames = [];+ cdp.on('Page.screencastFrame', frame => {+ const png = Buffer.from(frame.data, 'base64');+ frames.push(`${png.readUInt32BE(16)}x${png.readUInt32BE(20)} image, ${frame.metadata.deviceWidth}x${frame.metadata.deviceHeight} view`);+ cdp.send('Page.screencastFrameAck', { sessionId: frame.sessionId }).catch(() => {});+ });+ await cdp.send('Page.startScreencast', { format: 'png', everyNthFrame: 1 });+ // Frames arrive slower on a loaded machine: wait for a few, bounded.+ const awaitFrames = async (count, timeoutMs = 10000) => {+ const deadline = Date.now() + timeoutMs;+ while (frames.length < count && Date.now() < deadline) await page.waitForTimeout(50);+ assert.ok(frames.length >= count, `the screencast sent ${frames.length} frames`);+ };+ await awaitFrames(3);+ const onScreen = [...new Set(frames)];+ assert.equal(onScreen.length, 1, onScreen.join('; '));+ // Like _visionPlatformCapture: no clip. Every frame shown during and+ // after the captures keeps the tab's size and scale.+ frames.length = 0;+ for (let i = 0; i < 3; i++) await cdp.send('Page.captureScreenshot', { format: 'jpeg', quality: 90, fromSurface: true });+ await awaitFrames(3);+ assert.deepEqual([...new Set(frames)], onScreen, 'the tab keeps its size and scale through the captures');+ // The capture before: Chromium resizes the view to the clip's scaled size+ // and renders the page into it, which the screen shows.+ const clip = { x: vv.pageX, y: vv.pageY, width: vv.clientWidth, height: vv.clientHeight, scale: 1024 / (vv.clientWidth * 2) };+ const seen = [];+ for (let i = 0; i < 6 && !seen.some(size => size !== onScreen[0]); i++) {+ frames.length = 0;+ await cdp.send('Page.captureScreenshot', { format: 'jpeg', quality: 90, fromSurface: true, clip });+ await awaitFrames(2);+ seen.push(...frames);+ }+ assert.ok(seen.some(size => size !== onScreen[0]), `a clipped capture is seen on screen: ${[...new Set(seen)].join('; ')}`);+ } finally {+ await browser.close();+ }+});++test('a vision capture of a right-to-left page wider than the window shows the screen, scrollbar left out', async () => {+ // Classic scrollbars, as on Windows and Linux. Chromium keeps the page's+ // own on the right in RTL too, so the surface's left edge is CSS x 0.+ const browser = await chromium.launch({ channel: 'chromium', ignoreDefaultArgs: ['--hide-scrollbars'], args: ['--window-size=1000,700'] });+ try {+ const context = await browser.newContext({ viewport: null });+ const page = await context.newPage();+ await page.goto(`${origin}/rtl`);+ const cdp = await context.newCDPSession(page);+ const { cssVisualViewport: vv } = await cdp.send('Page.getLayoutMetrics');+ const scrollbar = await page.evaluate(() => innerWidth - document.documentElement.clientWidth);+ assert.ok(scrollbar > 0, 'the page has a classic scrollbar');+ assert.equal(vv.clientWidth, await page.evaluate(() => document.documentElement.clientWidth));+ // Like _visionPlatformCapture, at one image pixel per CSS pixel.+ const shot = await cdp.send('Page.captureScreenshot', { format: 'jpeg', quality: 90, fromSurface: true });+ const result = await page.evaluate(async ({ data, vv }) => {+ const encoder = await import('/agent/vision-encoder.js');+ const dataUrl = await encoder.cropImage(`data:image/jpeg;base64,${data}`, { x: 0, y: 0, width: vv.clientWidth, height: vv.clientHeight }, vv.clientWidth, vv.clientHeight);+ const raster = await encoder.rasterizeFrame(dataUrl, vv.clientWidth, vv.clientHeight);+ const at = x => Array.from(raster.pixels.slice((30 * raster.width + x) * 4, (30 * raster.width + x) * 4 + 3));+ return { width: raster.width, start: at(30), end: at(raster.width - 30), right: at(raster.width - 1) };+ }, { data: shot.data, vv });+ const near = (rgb, expected) => rgb.every((value, i) => Math.abs(value - expected[i]) < 40);+ assert.ok(near(result.start, [0, 200, 0]), `the page's left edge: ${result.start}`);+ assert.ok(near(result.end, [200, 0, 200]), `its right edge: ${result.end}`);+ assert.ok(near(result.right, [200, 0, 200]), `the image ends where the page does, before the scrollbar: ${result.right}`);+ } finally {+ await browser.close();+ }+});+test('screenshot redaction pixelates every region, or returns the image untouched when one region cannot be', async () => {const browser = await chromium.launch({ channel: 'chromium' });try {