Open-source AI browser agent for Chrome and Firefox (monorepo) đź§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Tree mode: stop the Chrome flicker on automatic and tool screenshots
Barack Sokullu29280068731b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 894f859a2..3c0aad78d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -15,6 +15,7 @@ This changelog was generated from the repository Git history and release tags. V### Fixed- Vision mode no longer makes the page flicker in Chrome. Each screenshot made Chrome redraw the tab shrunk into its top-left corner for a moment, several times after every action; WebBrain now captures the tab as it is on screen and crops and scales the image itself. Only a `zoom` that needs more detail than the screen has, as most do on a display at 100% scaling, still re-renders the page for its capture. Right-to-left pages wider than the window are now captured as they appear rather than as a blank frame.- While the screen settles after an action, WebBrain's border, cursor and Stop button now disappear once and come back after the last screenshot, instead of blinking around each one, and the Stop button keeps its layout afterwards. Chrome and Firefox.+- Screenshots outside vision mode no longer make the page flicker in Chrome either. Automatic screenshots (after each navigation by default), the **screenshot** and **inspect_viewport** tools and the screenshot crop of **download_social_media** redrew the tab shrunk into its top-left corner for a moment: on retina displays, and on any display when the image budget scales the screenshot down. They now capture the tab as it is on screen and crop and scale the image in WebBrain; image sizes, coordinates and redaction are unchanged. A right-to-left page wider than the window, which came out blank, is now captured as shown.## [44.0.0] - 2026-10-27diff --git a/docs/architecture.md b/docs/architecture.mdindex e549b2dec..b5d9e3b1d 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -1399,7 +1399,7 @@ The primary page-interaction path. Produces a flat, indented text tree of the paWraps `chrome.debugger` API for:- **Trusted events** — `Input.dispatchMouseEvent`, `Input.dispatchKeyEvent` (event.isTrusted === true)-- **Screenshots** — `Page.captureScreenshot` with clip/scale control+- **Screenshots** — `Page.captureScreenshot` of the tab's surface as it is on screen, cropped and scaled locally (a clip would resize the visible tab while it captures); full-page tiles keep their clip- **DOM queries** — `Runtime.evaluate` for shadow DOM piercing, `DOM.getDocument` for closed roots- **WebMCP** — `WebMCP.enable` maintains a bounded live catalog and`WebMCP.invokeTool` executes a page-registered structured capability. WebBraindiff --git a/docs/fr/architecture.md b/docs/fr/architecture.mdindex a126ad334..4459fe0eb 100644--- a/docs/fr/architecture.md+++ b/docs/fr/architecture.md@@ -337,7 +337,7 @@ Le principal chemin d'interaction avec la page. Produit un arbre textuel indentEnveloppe l'API `chrome.debugger` pour :- **Événements de confiance** — `Input.dispatchMouseEvent`, `Input.dispatchKeyEvent` (event.isTrusted === true)-- **Captures d'écran** — `Page.captureScreenshot` avec contrôle de recadrage/échelle+- **Captures d'écran** — `Page.captureScreenshot` de la surface de l'onglet telle qu'elle est à l'écran, recadrée et mise à l'échelle localement (un recadrage CDP redimensionnerait l'onglet visible pendant la capture) ; les tuiles pleine page gardent leur recadrage- **Requêtes DOM** — `Runtime.evaluate` pour la percée du shadow DOM, `DOM.getDocument` pour les racines ferméesSans CDP (Firefox), tous les événements sont synthétiques (`el.click()`, `new KeyboardEvent()`).diff --git a/package.json b/package.jsonindex 954f24234..bd19288ca 100644--- a/package.json+++ b/package.json@@ -110,7 +110,7 @@"test:shard:browser-a": "node --test test/captcha-draft-content.mjs test/captcha-challenge-surfaces.mjs && node test/bot-memory-dom.mjs && node test/bot-memory-click-dom.mjs && node test/ui-content-lifecycle.mjs && node --test test/native-telegram-chat.mjs test/native-telegram-recipient.mjs test/telegram-native-baseline.mjs test/telegram-native-latest.mjs test/native-chat-pre-enter-draft.mjs test/chat-draft-dispatch-ownership.mjs test/native-telegram-alias-preparation.mjs test/telegram-read-preparation-content.mjs test/queued-composer-ownership.mjs test/onboarding-provider-ready.mjs test/sidepanel-permission-gate-ui.mjs test/permission-skip-dialog-ui.mjs test/permission-skip-session.mjs test/model-picker-status-ui.mjs test/workflow-native-targets.mjs test/native-whatsapp-chat.mjs test/native-chat-context-identity.mjs test/native-whatsapp-contact.mjs test/whatsapp-chat-selection.mjs test/whatsapp-native-bootstrap.mjs test/whatsapp-drawer-identity.mjs test/whatsapp-read-latest.mjs test/whatsapp-read-preparation.mjs test/whatsapp-chat-completion.mjs test/whatsapp-chat-restart.mjs test/chat-goal-lifecycle.mjs test/chat-completion.mjs test/chat-generic-dispatch.mjs test/completion-live-status-browser.mjs test/native-confirmation-visibility.mjs test/native-button-preparation.mjs test/read-continuation-browser.mjs test/iframe-preparation-focus.mjs test/contenteditable-literal.mjs test/native-chat-empty-editor.mjs test/native-chat-trusted-clear.mjs test/chat-steering-ui.mjs test/activity-brain-ui.mjs test/recipient-confirmation-ui.mjs test/assistant-message-visibility-ui.mjs test/slash-command-timeline-ui.mjs test/page-monitor-ui.mjs test/x-compose-trusted-typing.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-native-preparation.mjs test/native-preparation-cold.mjs test/page-monitor-navigation.mjs test/page-monitor-main-owner.mjs test/page-monitor-collateral-controls.mjs test/accessibility-tree-scoped-budget.mjs test/ax-reference-origin.mjs test/page-monitor-decorative-hit.mjs test/social-ui-navigation.mjs test/native-resource-deletion-binding.mjs test/native-mastodon-confirmation.mjs test/dropdown-menu-inventory.mjs test/page-monitor-passive-progress.mjs test/page-monitor-feed-items.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/social-media-downloader-reinjection.mjs test/upload-file-local-state-guidance.mjs test/upload-file-source-approval.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-scroll-churn.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/lifecycle-run-ui.mjs test/lifecycle-menu-prompts-held.mjs test/delivery-cdp-input.mjs test/delivery-click-ax-response-loss.mjs test/delivery-native-button-geometry.mjs test/delivery-hover-transition.mjs test/delivery-native-button-route.mjs test/delivery-dispatch-boundary.mjs test/delivery-stop-compound.mjs test/guards-control-classification.mjs test/guards-unbound-deletion.mjs test/submit-probe-readings.mjs test/declined-submission-identity.mjs test/improver-content-account-chrome.mjs test/improver-content-recipient.mjs test/improver-content-truncated-replacement.mjs test/improver-forms-probe.mjs test/improver-misc-trace-browser.mjs test/page-feedback-held-time.mjs test/page-monitor-held-observation.mjs","test:shard:browser-b1": "node test/message-recipient-upload.mjs && node test/message-recipient-clarify-binding.mjs && node --test test/message-recipient-whatsapp.mjs test/message-control-readings.mjs && node test/quality/integration-mv3.mjs --full-form && node test/quality/integration-mv3.mjs --batch && node test/quality/literal-calculate-midrun-mv3.mjs && node test/quality/ask-completion-churn-mv3.mjs && node test/quality/ask-person-activity-mv3.mjs && node test/quality/telegram-cold-preparation-mv3.mjs && node test/quality/native-chat-mv3.mjs && node test/quality/native-chat-presubmit-mv3.mjs && node test/quality/cloud-bot-memory-mv3.mjs","test:shard:browser-b2": "node test/quality/whatsapp-contact-matrix.mjs && node test/quality/pdf-detection-mv3.mjs && node test/quality/download-upload-mv3.mjs && node test/quality/upload-release-binding-mv3.mjs && node test/quality/upload-release-binding-mv3.mjs --selector-recovery && node test/quality/upload-release-binding-mv3.mjs --sequential-assets && node test/quality/release-review-world-mv3.mjs && node test/quality/downloaded-file-saved-bytes-mv3.mjs && npm run test:quality:resolved-address && node test/quality/file-upload-accept-mv3.mjs && node test/quality/bulk-resume-mv3.mjs && node test/quality/shared-action-guards-mv3.mjs && node test/quality/video-upload-attachment-mv3.mjs && node --test test/iframe-probe-match-index.mjs",- "test:shard:browser-c": "node test/quality/literal-frame-mv3.mjs && node test/quality/mastodon-publication-mv3.mjs && node test/quality/mastodon-reply-route-mv3.mjs && node test/quality/mastodon-deletion-mv3.mjs && node test/quality/mastodon-delete-confirmation-mv3.mjs && node test/quality/mastodon-delete-confirmation-mv3.mjs --menu-read && node test/quality/mastodon-delete-confirmation-mv3.mjs --vague && node test/quality/mastodon-delete-after-correction-mv3.mjs && node test/quality/unbound-deletion-mv3.mjs && node test/quality/recipient-clarify-mv3.mjs && node test/quality/open-chat-recipients-mv3.mjs && node test/quality/correction-cross-site-mv3.mjs && node test/quality/anchored-action-mv3.mjs && node test/quality/inline-social-composer-mv3.mjs && node test/quality/x-compose-dialog-mv3.mjs && node test/quality/bluesky-composer-mv3.mjs && node test/quality/dropdown-menu-matrix.mjs && node test/quality/mastodon-deletion-navigation-mv3.mjs && node test/quality/semantic-navigation-matrix.mjs && node test/quality/workflow-replay-mv3.mjs && node test/quality/generative-media-mv3.mjs && node test/quality/cdp-worker-restart-mv3.mjs && node test/quality/steering-worker-restart-mv3.mjs && npm run test:vision-mode",+ "test:shard:browser-c": "node test/quality/literal-frame-mv3.mjs && node test/quality/mastodon-publication-mv3.mjs && node test/quality/mastodon-reply-route-mv3.mjs && node test/quality/mastodon-deletion-mv3.mjs && node test/quality/mastodon-delete-confirmation-mv3.mjs && node test/quality/mastodon-delete-confirmation-mv3.mjs --menu-read && node test/quality/mastodon-delete-confirmation-mv3.mjs --vague && node test/quality/mastodon-delete-after-correction-mv3.mjs && node test/quality/unbound-deletion-mv3.mjs && node test/quality/recipient-clarify-mv3.mjs && node test/quality/open-chat-recipients-mv3.mjs && node test/quality/correction-cross-site-mv3.mjs && node test/quality/anchored-action-mv3.mjs && node test/quality/inline-social-composer-mv3.mjs && node test/quality/x-compose-dialog-mv3.mjs && node test/quality/bluesky-composer-mv3.mjs && node test/quality/dropdown-menu-matrix.mjs && node test/quality/mastodon-deletion-navigation-mv3.mjs && node test/quality/semantic-navigation-matrix.mjs && node test/quality/workflow-replay-mv3.mjs && node test/quality/generative-media-mv3.mjs && node test/quality/cdp-worker-restart-mv3.mjs && node test/quality/steering-worker-restart-mv3.mjs && node --test test/viewport-capture-browser.mjs && npm run test:vision-mode","test:shard:tail": "node test/run.js && node test/selection-scope-restoration.mjs && node scripts/benchmark-offline-relevance.mjs && npm run test:security && node --test test/placeholder-submit-probe.mjs","test:composer-context": "node --test test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs && node test/quality/inline-social-composer-mv3.mjs && node test/quality/x-compose-dialog-mv3.mjs && node test/quality/bluesky-composer-mv3.mjs","test:bot-memory": "node --test test/bot-memory.mjs test/bot-memory-registries.mjs test/bot-memory-cloud-controller.mjs && node test/bot-memory-dom.mjs && node test/quality/cloud-bot-memory-mv3.mjs",diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 14e16b535..67345178c 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -45,6 +45,7 @@ import { LoopDetector } from './loop-detector.js';import { recordCollectionReadProgress } from './collection-read-progress.js';import { parseToolCallsFromText } from './tool-call-parser.js';import { IMAGE_BUDGET, estimateImageTokens, fitImageDimensions } from './image-budget.js';+import { cropImage } from './vision-encoder.js';import { BROWSER_MUTATION_TOOLS, STATE_CHANGE_TOOLS as SHARED_STATE_CHANGE_TOOLS } from './mutation-tools.js';import { guardRecentSubmitClick } from './submit-click-guard.js';import { isExplicitGuardOverrideRequest } from './guard-override.js';@@ -19863,7 +19864,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d* document coordinates (a clip at 0,0 on a scrolled page captures the* unrendered top of the document: a blank frame), and the output is* clip x scale x devicePixelRatio (scale 1 on a retina screen is twice the- * CSS size), so the scale divides the DPR out.+ * CSS size), so the scale divides the DPR out. Only pinch-zoomed pages and+ * failed local crops still capture with a clip (_captureViewportImage).*/_viewportCaptureClip(probe, cssW, cssH, scale = 1) {const dpr = Math.max(0.25, Number(probe?.dpr) || 1);@@ -19879,15 +19881,64 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d};}+ /**+ * The visible viewport, cssW x cssH CSS px from its top left, as a+ * `width` x `height` image, or at native device pixels without `width`.+ * Returns what Page.captureScreenshot does ({ data }).+ *+ * The tab's surface is captured as it is on screen and the viewport is+ * cut out and scaled locally. Given a clip, Chromium instead resizes the+ * tab's view to clip x scale and re-renders the page into it until the+ * capture ends (page_handler.cc CaptureScreenshot): a visible tab flashes+ * a shrunken page in its top-left corner whenever that size differs from+ * the view, as at scale 1 / DPR on a retina screen or in a budget-scaled+ * capture. The surface starts at the viewport's top left (Chromium keeps+ * the page scrollbar on the right, in RTL pages too) and has+ * devicePixelRatio pixels per CSS pixel, as in _visionPlatformCapture.+ *+ * A pinch-zoomed page keeps the clip: its surface shows the magnified+ * visual viewport, not the layout viewport these CSS sizes describe. So+ * does a capture whose local crop fails.+ */+ async _captureViewportImage(tabId, probe, cssW, cssH, { format = 'png', quality, width, height } = {}) {+ const dpr = Math.max(0.25, Number(probe?.dpr) || 1);+ const jpeg = format === 'jpeg';+ const capture = params => this._withIndicatorsHidden(tabId, () =>+ cdpClient.sendCommand(tabId, 'Page.captureScreenshot', { format, fromSurface: true, ...params }));+ const clipped = () => capture({+ ...(jpeg ? { quality } : {}),+ clip: this._viewportCaptureClip(probe, cssW, cssH, width ? width / cssW : dpr),+ });+ if (Math.abs((Number(probe?.pageScale) || 1) - 1) > 0.01) return clipped();+ // Native device pixels are the surface itself. A scaled image is cut+ // from a finer JPEG and encoded once more at the caller's quality.+ const shot = await capture(jpeg ? { quality: width ? 90 : quality } : {});+ if (!shot?.data || !width) return shot;+ const mimeType = jpeg ? 'image/jpeg' : 'image/png';+ try {+ const dataUrl = await this._cropAndScaleImage(`data:${mimeType};base64,${shot.data}`,+ { x: 0, y: 0, width: cssW * dpr, height: cssH * dpr }, width, height,+ { mimeType, ...(jpeg ? { quality: quality / 100 } : {}) });+ return { data: dataUrl.slice(dataUrl.indexOf(',') + 1) };+ } catch {+ return clipped();+ }+ }++ /** cropImage (vision-encoder.js); tests without a canvas stand in for it. */+ _cropAndScaleImage(dataUrl, rect, width, height, options) {+ return cropImage(dataUrl, rect, width, height, options);+ }+async _captureAutoScreenshot(tabId, { coordAligned = false } = {}) {try {await cdpClient.attach(tabId);await cdpClient.sendCommand(tabId, 'Page.enable');await this._preparePageForCapture(tabId);- // Probe the CSS viewport first so we can either (a) clip exactly- // to it for pixel-accurate captures, or (b) compute a budget-aware- // CDP-side scale that downsizes during capture rather than after.+ // Probe the CSS viewport first so we can either (a) capture it at+ // exactly its CSS size for pixel-accurate captures, or (b) compute+ // budget-aware target dims to capture it at.const probe = await this._captureViewportProbe(tabId);const cssW = Math.max(1, Math.round(probe?.innerWidth || 1024));const cssH = Math.max(1, Math.round(probe?.innerHeight || 768));@@ -19899,13 +19950,12 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// #311 review). When a side exceeds the cap, resize and report scale.const budget = this._budgetForCoordAlignedCapture();const captureOnce = async () => {- const shot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'jpeg',- quality: 60,- clip: this._viewportCaptureClip(probe, cssW, cssH, 1),- })- );+ const shot = await this._captureViewportImage(tabId, probe, cssW, cssH, {+ format: 'jpeg',+ quality: 60,+ width: cssW,+ height: cssH,+ });if (!shot?.data) return null;const rawDataUrl = `data:image/jpeg;base64,${shot.data}`;const needsResize = cssW > budget.maxTargetPx || cssH > budget.maxTargetPx;@@ -19936,27 +19986,24 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}// Non-coord-aligned mode: pre-compute target dims via the budget- // binary-search, then ask CDP to capture + scale in one pass. This- // avoids ever materializing a multi-MB native-DPR JPEG that we'd- // then have to decode and resize in the service worker.+ // binary-search, then capture the viewport at them. The tab's surface+ // is scaled to them in the service worker (_captureViewportImage):+ // a CDP-side scale would resize the visible tab while it captures.const budget = this._budgetForCapture();const [targetW, targetH] = Agent._fitImageDimensions(cssW, cssH, budget);- const scale = targetW < cssW ? targetW / cssW : 1;const captureOnce = async () => {- const shot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'jpeg',- quality: Math.round(budget.initialJpegQuality * 100),- fromSurface: true,- clip: this._viewportCaptureClip(probe, cssW, cssH, scale),- })- );+ const shot = await this._captureViewportImage(tabId, probe, cssW, cssH, {+ format: 'jpeg',+ quality: Math.round(budget.initialJpegQuality * 100),+ width: targetW,+ height: targetH,+ });if (!shot?.data) return null;const rawDataUrl = `data:image/jpeg;base64,${shot.data}`;- // CDP-side resize + JPEG q=75 usually fits. Iterative quality- // downgrade is the safety net for high-DPR screens where the- // captured image can still exceed the base64 ceiling.+ // Resize + JPEG q=75 usually fits. Iterative quality downgrade is+ // the safety net for detailed pages whose image can still exceed+ // the base64 ceiling.const shrunk = await this._compressJpegToByteCeiling(rawDataUrl, budget);const redacted = await this._redactScreenshotDataUrl(tabId, shrunk, { coordinateSpace: 'viewport' });return {@@ -20400,13 +20447,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst cssW = Math.max(1, Math.round(probe?.innerWidth || 1024));const cssH = Math.max(1, Math.round(probe?.innerHeight || 768));const captureOnce = async () => {- const shot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'png',- fromSurface: true,- clip: this._viewportCaptureClip(probe, cssW, cssH, 1),- })- );+ const shot = await this._captureViewportImage(tabId, probe, cssW, cssH, {+ format: 'png',+ width: cssW,+ height: cssH,+ });if (!shot?.data) return null;const cropDataUrl = `data:image/png;base64,${shot.data}`;// Model-facing path honors maxImageDimension (issue #311). Keep the raw@@ -20694,9 +20739,13 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst value = probe?.result?.value || null;if (value) {// Browser zoom: capture clips are in CSS px x zoom (_viewportCaptureClip).+ // Pinch zoom: the surface is not the layout viewport (_captureViewportImage).try {- const zoom = Number((await cdpClient.sendCommand(tabId, 'Page.getLayoutMetrics'))?.cssVisualViewport?.zoom);+ const visual = (await cdpClient.sendCommand(tabId, 'Page.getLayoutMetrics'))?.cssVisualViewport;+ const zoom = Number(visual?.zoom);if (zoom > 0) value.zoom = zoom;+ const pageScale = Number(visual?.scale);+ if (pageScale > 0) value.pageScale = pageScale;} catch {}}return value;@@ -20771,6 +20820,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// (2) capturing at that reduced size via CDP's clip.scale — and if// the resulting JPEG is still over a byte ceiling, iteratively// dropping JPEG quality (0.75 → 0.1 in 0.05 steps) until it fits.+ // WebBrain scales the tab's surface to that size itself instead+ // (_captureViewportImage): a clip resizes the visible tab while it captures.//// Defaults below match Anthropic's exactly (pxPerToken=28, 1568/1568,// initial 0.75, step 0.05, min 0.1, ~1.4 MB base64 ceiling) — those@@ -49028,13 +49079,11 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// maxImageDimension (sides), not the token budget (issue #311).// PNG capture (lossless) so glyph edges stay sharp before re-encode.const budget = this._budgetForCoordAlignedCapture();- const screenshot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'png',- fromSurface: true,- clip: this._viewportCaptureClip(probe, cssW, cssH, 1),- })- );+ const screenshot = await this._captureViewportImage(tabId, probe, cssW, cssH, {+ format: 'png',+ width: cssW,+ height: cssH,+ });if (!screenshot?.data) return null;const rawUrl = `data:image/png;base64,${screenshot.data}`;const needsResize = cssW > budget.maxTargetPx || cssH > budget.maxTargetPx;@@ -49065,14 +49114,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// shrink a model-facing copy (mirrors full_page_screenshot).const budget = this._budgetForCapture();if (wantSave) {- const screenshot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'png',- fromSurface: true,- // Downloads keep native device pixels (scale 1 is CSS px x DPR).- clip: { ...this._viewportCaptureClip(probe, cssW, cssH), scale: 1 },- })- );+ // Downloads keep native device pixels: the tab's surface as it is.+ const screenshot = await this._captureViewportImage(tabId, probe, cssW, cssH, { format: 'png' });if (!screenshot?.data) return null;const rawUrl = `data:image/png;base64,${screenshot.data}`;const shrunk = await this._shrinkImageForBudget(rawUrl, cssW, cssH, budget);@@ -49086,21 +49129,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d};}- // No save: pick target dims via binary search, ask CDP to capture- // + scale in one pass, then iterative-quality fallback if needed.+ // No save: pick target dims via binary search, capture the+ // viewport at them, then iterative-quality fallback if needed.const [targetW, targetH] = Agent._fitImageDimensions(cssW, cssH, budget);- const scale = targetW < cssW ? targetW / cssW : 1;- const screenshot = await this._withIndicatorsHidden(tabId, () =>- cdpClient.sendCommand(tabId, 'Page.captureScreenshot', {- format: 'jpeg',- quality: Math.round(budget.initialJpegQuality * 100),- fromSurface: true,- clip: this._viewportCaptureClip(probe, cssW, cssH, scale),- })- );+ const screenshot = await this._captureViewportImage(tabId, probe, cssW, cssH, {+ format: 'jpeg',+ quality: Math.round(budget.initialJpegQuality * 100),+ width: targetW,+ height: targetH,+ });if (!screenshot?.data) return null;const rawUrl = `data:image/jpeg;base64,${screenshot.data}`;- const resized = scale < 1 ? ` (resized ${cssW}×${cssH} → ${targetW}×${targetH} for vision-token budget)` : '';+ const resized = targetW < cssW ? ` (resized ${cssW}×${cssH} → ${targetW}×${targetH} for vision-token budget)` : '';return {dataUrl: await this._compressJpegToByteCeiling(rawUrl, budget),description: `Screenshot captured via CDP (${screenshot.data.length} bytes, JPEG)${resized}`,diff --git a/test/run.js b/test/run.jsindex 1d1e238a4..6ef3c9ced 100644--- a/test/run.js+++ b/test/run.js@@ -21852,11 +21852,134 @@ test('chrome screenshot tool saves pre-budget data URL when save:true', () => {);assert.match(body,- /wantSave[\s\S]*captureScreenshot[\s\S]*scale: 1[\s\S]*_shrinkImageForBudget/,- 'save:true non-coord path should full-capture then shrink for the model',+ /wantSave[\s\S]*_captureViewportImage\(tabId, probe, cssW, cssH, \{ format: 'png' \}\)[\s\S]*_shrinkImageForBudget/,+ 'save:true non-coord path should capture native device pixels, then shrink for the model',);});+test('chrome tree-mode viewport captures read the tab surface without a clip and scale it locally', async () => {+ // A CDP clip makes Chromium resize the tab's view to clip x scale while it+ // captures, which a visible tab shows as a shrunken page: at scale 1/DPR on+ // a retina screen, and in every budget-scaled capture. Only a pinch-zoomed+ // page, or a capture whose local crop fails, still uses one.+ const originalAttach = cdpClientCh.attach;+ const originalSendCommand = cdpClientCh.sendCommand;+ const shots = [];+ const crops = [];+ let viewport = { innerWidth: 1263, innerHeight: 760, scrollX: 0, scrollY: 600, dpr: 2 };+ let visual = { zoom: 1, scale: 1 };+ let failCrop = false;+ try {+ cdpClientCh.attach = async () => ({ attached: true });+ cdpClientCh.sendCommand = async (_tabId, method, params) => {+ if (method === 'Page.captureScreenshot') {+ shots.push(params);+ return { data: 'c3VyZmFjZQ==' };+ }+ if (method === 'Runtime.evaluate') return { result: { value: { readyState: 'complete', ...viewport } } };+ if (method === 'Page.getLayoutMetrics') return { cssVisualViewport: visual };+ return {};+ };+ const agent = new AgentCh({+ getActive: () => ({ name: 'vision-test', supportsVision: true }),+ getVisionProvider: async () => null,+ });+ agent._withIndicatorsHidden = async (_tabId, capture) => capture();+ agent._cropAndScaleImage = async (dataUrl, rect, width, height, options) => {+ crops.push({ dataUrl, rect, width, height, options });+ if (failCrop) throw new Error('no canvas');+ return `data:${options.mimeType};base64,Y3JvcA==`;+ };+ const surfaceJpeg = { format: 'jpeg', quality: 90, fromSurface: true };+ const next = () => { shots.length = 0; crops.length = 0; };++ // Coordinate-aligned: the CSS-sized JPEG at q60, cut from the DPR 2 surface.+ const aligned = await agent._captureAutoScreenshot(51, { coordAligned: true });+ assert.deepEqual(shots, [surfaceJpeg]);+ assert.deepEqual(crops, [{+ dataUrl: 'data:image/jpeg;base64,c3VyZmFjZQ==',+ rect: { x: 0, y: 0, width: 2526, height: 1520 },+ width: 1263,+ height: 760,+ options: { mimeType: 'image/jpeg', quality: 0.6 },+ }]);+ assert.deepEqual(aligned, { dataUrl: 'data:image/jpeg;base64,Y3JvcA==', width: 1263, height: 760, coordAligned: true });++ // Budget: the planned size and JPEG quality, from the same surface.+ viewport = { ...viewport, innerWidth: 1728, innerHeight: 1080 };+ const [targetW, targetH] = AgentCh._fitImageDimensions(1728, 1080, agent._budgetForCapture());+ assert.ok(targetW < 1728, `the budget scales this viewport down (${targetW}x${targetH})`);+ next();+ const budgeted = await agent._captureAutoScreenshot(51);+ assert.deepEqual(shots, [surfaceJpeg]);+ assert.deepEqual(crops.map(({ rect, width, height, options }) => ({ rect, width, height, options })), [{+ rect: { x: 0, y: 0, width: 3456, height: 2160 },+ width: targetW,+ height: targetH,+ options: { mimeType: 'image/jpeg', quality: 0.75 },+ }]);+ assert.deepEqual(budgeted, { dataUrl: 'data:image/jpeg;base64,Y3JvcA==', width: targetW, height: targetH, coordAligned: false });++ // The screenshot tool: lossless at CSS size for coord_aligned, the budget otherwise.+ viewport = { ...viewport, innerWidth: 1263, innerHeight: 760 };+ next();+ const tool = await agent.executeTool(51, 'screenshot', { coord_aligned: true });+ assert.equal(tool.success, true, JSON.stringify(tool));+ assert.deepEqual(shots, [{ format: 'png', fromSurface: true }]);+ assert.deepEqual(crops.map(({ rect, width, height, options }) => ({ rect, width, height, options })), [{+ rect: { x: 0, y: 0, width: 2526, height: 1520 }, width: 1263, height: 760, options: { mimeType: 'image/png' },+ }]);+ next();+ assert.equal((await agent.executeTool(51, 'screenshot', {})).success, true);+ assert.deepEqual(shots, [surfaceJpeg]);+ assert.deepEqual(crops.map(({ width, height, options }) => ({ width, height, options })), [{+ width: 1263, height: 760, options: { mimeType: 'image/jpeg', quality: 0.75 },+ }]);++ // Visible-media localization keeps a CSS-sized PNG.+ next();+ const media = await agent._captureVisibleMediaScreenshot(51);+ assert.deepEqual(shots, [{ format: 'png', fromSurface: true }]);+ assert.equal(media.cropDataUrl, 'data:image/png;base64,Y3JvcA==');+ assert.deepEqual([media.width, media.height], [1263, 760]);++ // Native device pixels (screenshot save:true) are the surface itself.+ const probe = await agent._captureViewportProbe(51);+ next();+ assert.deepEqual(await agent._captureViewportImage(51, probe, 1263, 760, { format: 'png' }), { data: 'c3VyZmFjZQ==' });+ assert.deepEqual(shots, [{ format: 'png', fromSurface: true }]);+ assert.equal(crops.length, 0);++ // A crop that fails falls back to the clipped capture.+ failCrop = true;+ next();+ const fallback = await agent._captureAutoScreenshot(51, { coordAligned: true });+ assert.deepEqual(shots, [surfaceJpeg, {+ format: 'jpeg', quality: 60, fromSurface: true,+ clip: { x: 0, y: 600, width: 1263, height: 760, scale: 0.5 },+ }]);+ assert.equal(fallback.dataUrl, 'data:image/jpeg;base64,c3VyZmFjZQ==');+ failCrop = false;++ // A pinch-zoomed page: its surface shows the magnified visual viewport,+ // so the layout viewport is still captured with a clip.+ visual = { zoom: 1, scale: 2 };+ next();+ await agent._captureAutoScreenshot(51, { coordAligned: true });+ const pinched = await agent._captureViewportProbe(51);+ assert.equal(pinched.pageScale, 2);+ await agent._captureViewportImage(51, pinched, 1263, 760, { format: 'png' });+ assert.deepEqual(shots, [+ { format: 'jpeg', quality: 60, fromSurface: true, clip: { x: 0, y: 600, width: 1263, height: 760, scale: 0.5 } },+ { format: 'png', fromSurface: true, clip: { x: 0, y: 600, width: 1263, height: 760, scale: 1 } },+ ]);+ assert.equal(crops.length, 0);+ } finally {+ cdpClientCh.attach = originalAttach;+ cdpClientCh.sendCommand = originalSendCommand;+ }+});+test('existing dims under caps stay within the monotonic bound', () => {// Fuzz a range of common viewport sizes; invariant: output ≤ input on// every dimension, and output token count ≤ maxTargetTokens.diff --git a/test/viewport-capture-browser.mjs b/test/viewport-capture-browser.mjsnew file mode 100644index 000000000..e68edf0e7--- /dev/null+++ b/test/viewport-capture-browser.mjs@@ -0,0 +1,209 @@+// Chrome's tree-mode viewport captures (automatic screenshots, the screenshot+// tool, visible-media capture: Agent._captureViewportImage) in real Chromium,+// driven by the agent's own code over CDP. A screencast reports every frame+// the tab shows: a clipped capture resizes the tab's view for its duration,+// which the screen shows as a shrunken page, and these captures must leave it+// as it is while still returning the planned image, mapped 1:1 to CSS pixels.+// Vision mode's captures: test/vision-pipeline-browser.mjs.+import assert from 'node:assert/strict';+import http from 'node:http';+import fs from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { test } from 'node:test';+import { chromium } from 'playwright';+import { cdpClient } from '../src/chrome/src/cdp/cdp-client.js';++const ROOT = fileURLToPath(new URL('../', import.meta.url));+const AGENT_DIR = path.join(ROOT, 'src/chrome/src/agent');++const area = { get: async () => ({}), set: async () => {}, remove: async () => {} };+globalThis.chrome ??= {+ storage: { local: area, session: area, onChanged: { addListener() {} } },+ runtime: { getURL: value => `extension://test/${value}`, sendMessage: async () => ({}), getManifest: () => ({ version: '0.0.0' }) },+ tabs: { get: async id => ({ id, active: true }), query: async () => [], sendMessage: async () => ({ ok: true }) },+ webNavigation: { getAllFrames: async () => [] },+ scripting: { executeScript: async () => [{ result: null }] },+};+const { Agent } = await import('../src/chrome/src/agent/agent.js');++// A scrolled page with fixed marks at known CSS positions, and a spinner that+// keeps the screencast sending frames while the captures run.+const FIXTURE = `<!doctype html><meta charset="utf-8"><title>Capture fixture</title>+<style>+ body { margin: 0; height: 3000px; background: linear-gradient(#fff, #eef); }+ #mark { position: fixed; left: 600px; top: 300px; width: 60px; height: 60px; background: rgb(20, 120, 220); }+ #corner { position: fixed; right: 0; bottom: 0; width: 40px; height: 40px; background: rgb(220, 60, 40); }+ #spin { position: fixed; left: 200px; top: 200px; width: 40px; height: 40px; background: #c0c; animation: spin 1s linear infinite; }+ @keyframes spin { to { transform: rotate(360deg); } }+</style><div id="mark"></div><div id="corner"></div><div id="spin"></div>`;++// A right-to-left page wider than the window opens scrolled to its right end.+const RTL_FIXTURE = `<!doctype html><html dir="rtl"><meta charset="utf-8"><title>RTL fixture</title>+<style>+ body { margin: 0; width: 3000px; height: 3000px; background: #fff; }+ #start { position: fixed; left: 0; top: 0; width: 60px; height: 60px; background: rgb(0, 200, 0); }+ #end { position: fixed; right: 0; top: 0; width: 60px; height: 60px; background: rgb(200, 0, 200); }+</style>+<div id="start"></div><div id="end"></div></html>`;++function serve() {+ const server = http.createServer((request, response) => {+ if (request.url.startsWith('/agent/')) {+ const file = path.join(AGENT_DIR, path.basename(request.url));+ if (!file.startsWith(AGENT_DIR) || !fs.existsSync(file)) { response.writeHead(404); response.end(); return; }+ response.writeHead(200, { 'Content-Type': 'text/javascript' });+ response.end(fs.readFileSync(file));+ return;+ }+ response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });+ response.end(request.url === '/rtl' ? RTL_FIXTURE : FIXTURE);+ });+ return new Promise(resolve => server.listen(0, '127.0.0.1', () => resolve(server)));+}++const server = await serve();+const origin = `http://127.0.0.1:${server.address().port}`;+test.after(() => new Promise(resolve => server.close(resolve)));++const TAB = 7001;+const savedCdp = { attach: cdpClient.attach, sendCommand: cdpClient.sendCommand };+test.afterEach(() => Object.assign(cdpClient, savedCdp));++/** A real page, with the agent's CDP traffic going to it. */+async function open({ dpr = 1, windowSize = '1280,860', url = '/fixture', scrollbars = false } = {}) {+ const browser = await chromium.launch({+ channel: 'chromium',+ args: [`--force-device-scale-factor=${dpr}`, `--window-size=${windowSize}`],+ ...(scrollbars ? { ignoreDefaultArgs: ['--hide-scrollbars'] } : {}),+ });+ const context = await browser.newContext({ viewport: null });+ const page = await context.newPage();+ await page.goto(`${origin}${url}`);+ const session = await context.newCDPSession(page);+ Object.assign(cdpClient, {+ attach: async () => ({ attached: true }),+ sendCommand: (_tabId, method, params) => session.send(method, params),+ });+ const agent = new Agent({ getActive: () => null, getProvider: () => null, getVisionProvider: async () => null });+ // The local crop runs where a canvas is: in the page, with the shared encoder.+ agent._cropAndScaleImage = (dataUrl, rect, width, height, options) => page.evaluate(async args => {+ const { cropImage } = await import('/agent/vision-encoder.js');+ return cropImage(args.dataUrl, args.rect, args.width, args.height, args.options);+ }, { dataUrl, rect, width, height, options });+ return { browser, page, session, agent };+}++/** Every frame the tab shows: its image size and the view it was drawn for. */+function watchScreen(page, session) {+ const frames = [];+ session.on('Page.screencastFrame', frame => {+ const png = Buffer.from(frame.data, 'base64');+ frames.push(`${png.readUInt32BE(16)}x${png.readUInt32BE(20)} image, ${frame.metadata.deviceWidth}x${frame.metadata.deviceHeight} view`);+ session.send('Page.screencastFrameAck', { sessionId: frame.sessionId }).catch(() => {});+ });+ // Frames arrive slower on a loaded machine: wait for a few, bounded.+ const awaitFrames = async (count, timeoutMs = 10000) => {+ const deadline = Date.now() + timeoutMs;+ while (frames.length < count && Date.now() < deadline) await page.waitForTimeout(50);+ assert.ok(frames.length >= count, `the screencast sent ${frames.length} frames`);+ };+ return { frames, awaitFrames, start: () => session.send('Page.startScreencast', { format: 'png', everyNthFrame: 1 }) };+}++/** The decoded size of a data URL and its RGB at image points. */+function readPixels(page, dataUrl, points) {+ return page.evaluate(async ({ dataUrl, points }) => {+ const { decodeImage } = await import('/agent/vision-encoder.js');+ const image = await decodeImage(dataUrl);+ const canvas = new OffscreenCanvas(image.width, image.height);+ const ctx = canvas.getContext('2d');+ ctx.drawImage(image, 0, 0);+ const rgb = points.map(([x, y]) => Array.from(ctx.getImageData(Math.round(x), Math.round(y), 1, 1).data.slice(0, 3)));+ return { width: image.width, height: image.height, rgb };+ }, { dataUrl, points });+}++const near = (rgb, expected) => rgb.every((value, i) => Math.abs(value - expected[i]) < 40);+const BLUE = [20, 120, 220];+const RED = [220, 60, 40];++// DPR 2 shrinks a clipped capture by 1/DPR; at DPR 1 a window wider than the+// image budget shrinks the budget-scaled one.+for (const { dpr, windowSize } of [{ dpr: 2, windowSize: '1280,860' }, { dpr: 1, windowSize: '1920,1080' }]) {+ test(`DPR ${dpr}: tree-mode captures leave the tab on screen as it is and map 1:1 to CSS pixels, where a clipped capture shrinks it`, async () => {+ const { browser, page, session, agent } = await open({ dpr, windowSize });+ try {+ await page.evaluate(() => window.scrollTo(0, 400));+ const { cssW, cssH } = await page.evaluate(() => ({ cssW: innerWidth, cssH: innerHeight }));+ const screen = watchScreen(page, session);+ await screen.start();+ await screen.awaitFrames(3);+ const onScreen = [...new Set(screen.frames)];+ assert.equal(onScreen.length, 1, onScreen.join('; '));++ screen.frames.length = 0;+ const aligned = await agent._captureAutoScreenshot(TAB, { coordAligned: true });+ const budgeted = await agent._captureAutoScreenshot(TAB);+ const media = await agent._captureVisibleMediaScreenshot(TAB);+ const native = await agent._captureViewportImage(TAB, await agent._captureViewportProbe(TAB), cssW, cssH, { format: 'png' });+ await screen.awaitFrames(3);+ assert.deepEqual([...new Set(screen.frames)], onScreen, 'the tab keeps its size and scale through the captures');++ // Each image is the scrolled viewport from its top left, at the size it+ // reports: the mark and the bottom-right corner sit where CSS has them.+ const [targetW, targetH] = Agent._fitImageDimensions(cssW, cssH, agent._budgetForCapture());+ if (dpr === 1) assert.ok(targetW < cssW, `the budget scales this window down: ${targetW} of ${cssW}`);+ for (const [label, dataUrl, width, height] of [+ ['coordinate-aligned', aligned?.dataUrl, aligned?.width, aligned?.height],+ ['budget', budgeted?.dataUrl, targetW, targetH],+ ['visible media', media?.cropDataUrl, cssW, cssH],+ ['native', native?.data && `data:image/png;base64,${native.data}`, Math.round(cssW * dpr), Math.round(cssH * dpr)],+ ]) {+ assert.ok(dataUrl, `${label}: captured`);+ const sx = width / cssW;+ const sy = height / cssH;+ const shot = await readPixels(page, dataUrl, [[630 * sx, 330 * sy], [(cssW - 20) * sx, (cssH - 20) * sy]]);+ assert.deepEqual([shot.width, shot.height], [width, height], `${label}: the reported size`);+ assert.ok(near(shot.rgb[0], BLUE), `${label}: the mark at CSS 630,330: ${shot.rgb[0]}`);+ assert.ok(near(shot.rgb[1], RED), `${label}: the corner: ${shot.rgb[1]}`);+ }+ assert.equal(budgeted.width, targetW);+ assert.equal(budgeted.height, targetH);+ if (dpr === 2) assert.deepEqual([aligned.width, aligned.height, aligned.coordAligned], [cssW, cssH, true]);++ // The capture before this change, which a failed local crop still falls+ // back to: Chromium resizes the view to the clip's scaled size and+ // renders the page into it, which the screen shows.+ agent._cropAndScaleImage = async () => { throw new Error('no canvas'); };+ const seen = [];+ for (let i = 0; i < 6 && !seen.some(size => size !== onScreen[0]); i++) {+ screen.frames.length = 0;+ assert.ok(await agent._captureAutoScreenshot(TAB), 'the clipped fallback still captures');+ await screen.awaitFrames(2);+ seen.push(...screen.frames);+ }+ assert.ok(seen.some(size => size !== onScreen[0]), `a clipped capture is seen on screen: ${[...new Set(seen)].join('; ')}`);+ } finally {+ await browser.close();+ }+ });+}++test('a tree-mode capture of a right-to-left page wider than the window shows the screen', async () => {+ // Classic scrollbars, as on Windows and Linux; Chromium keeps the page's+ // own on the right in RTL too. The clip captured this page's left end,+ // off screen, as a blank frame.+ const { browser, page, agent } = await open({ url: '/rtl', windowSize: '1000,700', scrollbars: true });+ try {+ const { cssW, clientW } = await page.evaluate(() => ({ cssW: innerWidth, clientW: document.documentElement.clientWidth }));+ assert.ok(cssW > clientW, 'the page has a classic scrollbar');+ const shot = await agent._captureAutoScreenshot(TAB, { coordAligned: true });+ assert.deepEqual([shot?.width, shot?.coordAligned], [cssW, true]);+ const { rgb: [start, end] } = await readPixels(page, shot.dataUrl, [[30, 30], [clientW - 30, 30]]);+ assert.ok(near(start, [0, 200, 0]), `the page's left edge: ${start}`);+ assert.ok(near(end, [200, 0, 200]), `its right edge, before the scrollbar: ${end}`);+ } finally {+ await browser.close();+ }+});